Parking support device and parking support method

Abstract

To provide a parking support device, parking support method, and parking support program capable of selecting a feature point so as to improve accuracy in automatic parking.SOLUTION: A parking support device includes: an image acquisition unit acquiring camera images from cameras capturing respective different directions around a vehicle; a feature point detection unit extracting feature points from the camera images; a feature point selection unit evaluating the feature points and selecting feature points to be registered when performing learning travel where the vehicle is manually parked and a parking route and a parking position of the vehicle are registered in a map; and a vehicle control unit causing the vehicle to park on the basis of the map when the vehicle is automatically parked. The feature point selection unit varies priority for registration of the feature points or the number of the feature points for registration according to a position on the parking route, or selects, at the position on the parking route, the feature point to be registered on the map on the basis of a position of the camera or a relative position of the feature point with respect to an optical axis direction of the camera.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a parking assistance device, a parking assistance method, and a parking assistance program. [Background technology]

[0002] Patent Document 1 discloses a method for registering only characteristic points around the parking position on a map, and not registering characteristic points near the parking start position on a map. Patent Document 2 discloses a method for registering more characteristic points on a map the closer they are to the parking position.

[0003] The technologies in Patent Documents 1 and 2 focus on the fact that the closer to the parking position, the more accurate the position estimation is required, and aim to reduce the map capacity and the amount of calculations required for automatic parking by eliminating or reducing the number of feature points registered on the map for positions farther from the parking position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-114526 A [Patent Document 2] JP 2018-75866 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technologies of Patent Documents 1 and 2 do not take into consideration that the obtained accuracy differs depending on the angle of the camera with respect to the optical axis direction, that the difference in height between the feature point and the camera affects the position accuracy, etc., so the position accuracy may decrease even at the parking position. In addition, they do not take into consideration that position accuracy is required even at a position away from the parking position, such as when turning around at a position to avoid an obstacle when backing into a parking space, or when passing by an obstacle on the way to the turning around position.

[0006] Furthermore, this technique does not take into consideration the fact that the required level of accuracy differs between the longitudinal positional accuracy of the vehicle and the lateral positional accuracy, and that the direction in which a feature point contributes to positional accuracy differs depending on the orientation of the feature point relative to the vehicle, and so it may not be possible to properly register on a map a feature point that is in a position requiring accuracy and that is advantageous for obtaining positional accuracy in the direction requiring accuracy.

[0007] An object of the present disclosure is to provide a parking assistance device, a parking assistance method, and a parking assistance program that are capable of selecting feature points to improve the accuracy of automatic parking. [Means for solving the problem]

[0008] In order to solve the above problems, one aspect of a parking assistance device according to the present disclosure includes an image acquisition unit that acquires camera images from cameras that each capture a different direction around the vehicle, a feature point detection unit that extracts feature points from the camera images, a feature point selection unit that evaluates the feature points and selects the feature points to be registered on the map during a learning drive in which the vehicle is manually parked and the parking path and parking position are registered on a map, and a vehicle control unit that parks the vehicle based on the map during automatic parking, wherein the feature point selection unit varies a priority for registering feature points or the number of feature points to be registered depending on a position on the parking path, or selects the feature points to be registered on the map based on the position of the camera or the relative position of the feature points with respect to the optical axis direction of the camera at the position on the parking path.

[0009] Moreover, one aspect of the parking assistance method according to the present disclosure includes the steps of acquiring camera images from cameras each capturing a different direction around the vehicle, extracting feature points from the camera images, evaluating the feature points and selecting feature points to be registered on the map during a learning drive in which the vehicle is manually parked and the parking path and parking position are registered on a map, and parking the vehicle based on the map during automatic parking, wherein the step of selecting the feature points varies a priority for registering feature points or the number of feature points to be registered depending on a position on the parking path, or selects feature points to be registered on the map based on a relative position of the feature points with respect to the position of the camera or the optical axis direction of the camera at a position on the parking path.

[0010] Moreover, one aspect of a parking assistance program according to the present disclosure is a parking assistance program that causes a computer to execute the steps of acquiring camera images from cameras each capturing a different direction around a vehicle, extracting feature points from the camera images, evaluating the feature points and selecting feature points to be registered on the map during a learning drive in which the vehicle is manually parked and the parking path and parking position are registered on a map, and parking the vehicle based on the map during automatic parking, wherein the step of selecting the feature points varies a priority for registering feature points or the number of feature points to be registered depending on a position on the parking path, or selects feature points to be registered on the map based on a relative position of the feature points with respect to the position of the camera or the optical axis direction of the camera at a position on the parking path. Effect of the Invention

[0011] According to the present disclosure, feature points can be selected to improve the accuracy of automatic parking. [Brief description of the drawings]

[0012] [Figure 1] 5A and 5B are diagrams illustrating an example of feature point detection performed by the parking assistance device according to the present embodiment. [Diagram 2]FIG. 2 is a diagram showing automatic parking using self-position estimation in the present embodiment. [Diagram 3] 1 is a diagram showing a vehicle to which the parking assistance device according to the present embodiment can be applied. [Figure 4] 1 is a block diagram showing a parking assistance device according to an embodiment of the present invention; [Diagram 5] 1 is a diagram showing a hardware configuration of a parking assistance device according to an embodiment of the present invention; [Figure 6] FIG. 2 is a diagram showing a positional relationship between a vehicle and an object. [Figure 7] FIG. 13 is a diagram illustrating the effect of the angle relative to the optical axis direction of the camera on accuracy. [Figure 8] FIG. 13 is a diagram showing the effect of the height difference between the camera and feature points on accuracy. [Figure 9] 1 is a diagram showing the positions of feature points and sensitivities in the front-rear and left-right directions of a vehicle; [Figure 10] FIG. 13 is a diagram showing the relationship between the distance from the vehicle path to a feature point and the sensitivity. [Figure 11] FIG. 13 is a diagram illustrating changes in sensitivity of feature points. [Figure 12] FIG. 13 is a diagram showing the positions of feature points and sensitivity in the left-right direction. [Figure 13] FIG. 13 is a diagram showing a case where a feature point is present on the path of a vehicle. [Figure 14] FIG. 13 is a diagram showing changes in sensitivity of feature points when a vehicle moves in the X direction. [Figure 15] FIG. 13 is a diagram illustrating the positions of feature points and the sensitivity of the vehicle body posture. [Figure 16] 1A and 1B are diagrams illustrating a state in which the orientation or position of a camera changes. [Figure 17] FIG. 2 is a diagram showing the position of a vehicle requiring high accuracy in self-location estimation; [Figure 18] FIG. 13 is a diagram showing the arrangement of feature points around an end point. [Figure 19] FIG. 13 is a diagram showing the arrangement of feature points around an end point. [Figure 20] FIG. 13 is a diagram showing characteristic points around a straight line section. [Figure 21]FIG. 13 is a diagram showing an attitude convergence section in which an occupant aligns the vehicle in the left-right direction by steering. [Figure 22] FIG. 13 is a diagram showing an attitude convergence section in which an occupant aligns the vehicle in the left-right direction by steering. [Diagram 23] FIG. 1 is a diagram showing a turning section requiring precision in the left-right direction; [Figure 24] FIG. 11 is a diagram showing sensitivity in the left and right directions when the vehicle is turning. [Diagram 25] FIG. 11 is a diagram showing sensitivity in the left and right directions when the vehicle is turning. [Figure 26] FIG. 1 is a diagram showing a situation in which a vehicle requires accuracy in the lateral direction. [Figure 27] 4 is a flowchart of map generation according to the present embodiment. [Figure 28] FIG. 13 is a diagram illustrating the adjustment of end points. [Figure 29] FIG. 13 is a diagram showing an example of endpoints and allocation of points to sections. [Diagram 30] FIG. 13 is a diagram illustrating feature point registration at end points. [Diagram 31] FIG. 13 is a diagram showing a range of feature points to be registered in a curved line section. [Diagram 32] FIG. 13 is a diagram showing a range of feature points to be registered in a straight line section. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each embodiment described below shows a specific example of the present disclosure. Therefore, each component, the arrangement position and connection form of each component, each step and the order of each step shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0014] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations are omitted or simplified.

[0015] Fig. 1 is a diagram showing an example of feature point detection performed by the parking assistance device in this embodiment. In the learning-type automatic parking, if you manually park the vehicle and store the starting position and parking route, you can automatically drive and park the vehicle along the same parking route simply by activating the automatic parking function at the same starting position.

[0016] More specifically, a vehicle 1 equipped with a parking assistance device generates a map including feature points during manual parking (called learning driving), and during automatic parking, the vehicle 1 uses the map to estimate the position and attitude of the vehicle 1 while driving. The feature points registered in the map are representative points of the subject captured in the camera image, and are points that can identify the position of the subject (image). Figure 1 shows how the image of the subject is captured by a right camera 2a (described later) provided on the right side mirror of the vehicle 1. The image captured by the camera is called a camera image.

[0017] A feature point is a point in an image captured by a camera image whose position can be specified. For example, as shown in FIG. 1, if there is a linear image 3, its end point may be the feature point, if there is a circular image 4, its center point may be the feature point, and if there is a polygonal image 5, its corner may be the feature point. Since the camera image may be a color image or a black and white image, the parking assistance device may use a point of change in luminance or a point of change in hue as the feature point. Since the position at which the feature point is captured in the camera image depends on the orientation of the feature point relative to the vehicle 1, the parking assistance device can calculate the orientation of the feature point from the camera image.

[0018] When the vehicle 1 travels and the position of the right camera 2a moves from point A to point B, the orientation of the feature points changes. At this time, the position of each feature point can be identified by applying the principle of triangulation to each feature point with the line segment AB as the baseline. Many pairs of the characteristics (color and shape) and position (coordinates) of the feature points are registered on the map.

[0019] FIG. 2 is a diagram showing automatic parking by self-location estimation in this embodiment. In the learning type automatic parking, a map including data of a parking route is generated during learning driving, and during automatic parking, the vehicle drives according to the parking route registered in the map. The parking assistance device may register the parking route as a set of points such as a start position 11, a parking position 14, and a start point 12 and an end point 13 of a curved section, or may register the parking route as a combination of a straight section and a curved section. The curved section may be accompanied by information on a steering angle and a turning radius. In addition, the positions of characteristic points captured by a camera on the parking route are also registered in the map.

[0020] In other words, feature points and parking routes are registered on the map. The feature points registered on the map are called markers to distinguish them from feature points detected from camera images. In other words, the map has markers and parking routes registered.

[0021] If there are a large number of markers and there are markers in all directions around the vehicle, the vehicle's position can be accurately estimated while following a parking path and parking in an accurate location. However, if the number of markers is limited due to memory capacity considerations, the error in the estimated vehicle position may become large, causing the parking path or parking position 14 to deviate or automatic parking to fail.

[0022] This embodiment focuses on the fact that the accuracy of the automatic parking route and the parking position 14 changes depending on the selection of feature points to be registered on a map, and discloses a method of selecting feature points to be registered on a map so as to improve the accuracy of automatic parking. Note that, since the selection of feature points is premised on the registration on a map, the selection of feature points to be registered on a map is sometimes simply referred to as "selecting feature points." For example, the method disclosed in this embodiment may be rephrased as a method of selecting feature points so as to improve the accuracy of position estimation.

[0023] The map generation method in this embodiment preferentially registers feature points located within a predetermined range based on the camera equipped on the vehicle 1 on the map, and the predetermined range is determined based on at least one of the optical axis direction of the camera or the position of the camera.

[0024] A range where the angle with respect to the optical axis of the camera is small is a direction where the angle of the feature point can be accurately specified, so if the feature point in that direction is registered on a map, it is advantageous for controlling the steering angle. A feature point with a small difference in height from the camera is advantageous for controlling the steering angle because the angle of the detected feature point is less affected even if there is a change in the posture of the vehicle body. A feature point with a small distance from the camera is advantageous for accurately estimating the position because the amount of change in the angle of the detected feature point is large when the position of the camera changes.

[0025] For example, by using the position and optical axis of the camera when the vehicle 1 is parked during the learning run as a reference, a feature point that is close to the camera, close to the optical axis of the camera, and has a small difference in elevation from the camera is selected and registered on the map, so that the vehicle can be accurately parked in the same position during automatic parking. Also, a position and direction requiring positional accuracy may be specified, and the position of a feature point that is advantageous for obtaining accuracy in that direction may be specified at that position, and the feature point in the more advantageous position may be preferentially selected, so that the required accuracy can be obtained with a small number of feature points. The method of specifying the position and direction requiring positional accuracy will be described after the configuration example shown below.

[0026] 3 is a diagram showing a vehicle to which the parking assistance device of this embodiment can be applied. Cameras 2 are provided at four locations on the front, rear, left and right sides of the body of vehicle 1. Each camera 2 is equipped with a fisheye lens and has a horizontal viewing range of 180 degrees or more (see dashed lines).

[0027] Since each camera 2 is mounted at a depression angle in order to capture the road surface, when the range of the road surface captured is converted into a horizontal field of view, a range of about 240 degrees of the road surface is captured by each camera 2. For example, the front and rear wheels and the side of the vehicle are captured in the image captured by the side cameras 2a installed on the left and right of the vehicle body.

[0028] FIG. 4 is a block diagram showing the parking assistance device 100 in this embodiment. The camera 2 outputs a captured camera image, which is received by an image processing unit 130 of the parking assistance device 100 to generate a display image, etc. The image processing unit 130 may be called an image acquisition unit. The display image is output from the notification unit 180 to the HMI device 20, and the notification unit 180 superimposes a message on the display image or outputs a voice message in response to an instruction from the state management unit 110. The state management unit 110 accepts a user's operation via the operation device 10, and controls the functions of the parking assistance device 100 in response to the user's operation. Here, the touch panel of the navigation device 40 is included in the operation device 10.

[0029] The state management unit 110 receives position information from the main body (not shown) of the navigation device 40. When performing a learning drive, the map generation unit 120 generates a map. The map generation unit 120 adds position information acquired by the navigation device 40 using GPS to the map and records the map in the storage unit 170. In this way, when performing automatic parking, the position information of the navigation device 40 can be compared with the position information added to the map to select the map to use for automatic parking.

[0030] The image processing unit 130 generates a display image and an image for detection. The image for detection may be, for example, an image in which changes in brightness or color are emphasized, or an image to which a process for extracting a contour or edge has been added. The feature point detection unit 140 extracts feature points from the image for detection. A feature point is not a point inside the surface of an image or in the middle of a side, but a point at a corner or edge whose position can be specified, so it is sufficient to extract the contour of the image and specify the corner or edge. The information on the feature points detected by the feature point detection unit 140 includes information on the color and shape of the image, and information on the position of the feature point on the camera image. The map generation unit 120 registers information on the feature points when generating a map, but after evaluating the feature points, registers only the selected feature points on the map. The map generation unit 120 selects the feature points to be registered on the map, so it may be called a feature point selection unit.

[0031] The color information of a feature point may include the color of the acute angle side and the color of the obtuse angle side if the feature point is a corner. Furthermore, the color information of a feature point may include the color of the line and the color of the background if the feature point is the end of a line. Since feature points are set by identifying the position of an image on an image, sharp images, that is, images with high contrast, are preferable. Feature points detected from images with low contrast may change position or may become undetectable depending on conditions such as the light source, and are therefore undesirable as feature points to be registered on a map. Therefore, they may be excluded at the feature point detection stage.

[0032] For example, in the image of an object with a gently curved surface, the position of the boundary between the illuminated part and the shadow changes depending on the direction of the light ray, so registering feature points on the boundary in the map may cause position estimation errors. Therefore, it is recommended that the map generating unit 120 recognizes that the boundary between light and shadow has a gradation-like appearance (i.e., low contrast) and does not register such feature points.

[0033] The map generating unit 120 may also include the favorability of a feature point as a basic score of the feature point in the feature point information, and add it to an evaluation value based on the position of the feature point when selecting the feature point to be registered on the map. Alternatively, when there are multiple feature points with the same evaluation based on the feature point position, the map generating unit 120 may select the feature point with the higher basic score.

[0034] If only feature points in the same direction as vehicle 1 are selected, there may be no difference in the orientation of the feature points as seen by vehicle 1. Since the principle of triangulation is to identify a position based on the difference in the orientation of feature points, if the distribution of feature points is biased, it may become impossible to identify the position of vehicle 1 or the accuracy of position estimation may decrease.

[0035] Therefore, the positions and directions of the selected feature points are prevented from concentrating in a specific position or direction. For example, the map generating unit 120 sets an interval threshold that limits the interval of the selected feature points, and when a certain feature point is selected, the map generating unit 120 prevents subsequent selection of feature points that are less than the interval threshold from the selected feature point.

[0036] Alternatively, when the map generating unit 120 selects a certain feature point, it performs lateral inhibition, which reduces the evaluation value as the distance from the feature point decreases, making feature points less likely to be selected near the selected feature point. By setting such distance thresholds and lateral inhibition characteristics for each position or section on the parking route, it is possible to control the number and density of selected feature points.

[0037] In addition, the map generating unit 120 may add an evaluation value to a feature point at a position necessary for obtaining positional accuracy according to a situation occurring on the parking path, so that the feature point at the necessary position is likely to be selected. Alternatively, the map generating unit 120 may increase the number and density of feature points registered on the map at the necessary positions by changing the interval threshold or the characteristics of lateral inhibition between positions where positional accuracy is required and positions where it is not required. The setting for controlling the number and density of the feature points to be selected may be called a score allocation. For example, the map generating unit 120 allocates scores according to the position on the parking path, and allocates high scores near parking positions and turning positions, and reduces the score allocation in other positions, thereby efficiently ensuring the necessary positional accuracy.

[0038] When the learning drive starts, the map generation unit 120 temporarily stores the feature points detected at the start position 11 of the learning drive in the memory unit 170. This storage of the feature points detected at the start position 11 is not a final registration to the map, so it may be called a provisional registration. The feature points to be registered to the map are determined after the learning drive ends. Alternatively, unnecessary feature points among the provisionally registered feature points may be deleted after the learning drive ends. Information on the feature points detected at the start position 11 is necessary to identify the position and attitude of the vehicle 1 when starting automatic parking.

[0039] When the vehicle 1 starts learning travel, the map generating unit 120 tracks the positions of the feature points on the camera image. For example, as shown in FIG. 1, the map generating unit 120 compares the feature points captured in the camera image at time A with the feature points captured in the camera image at a later time B, and detects a pair of feature points whose color and shape information and the amount of movement of the feature points match. Matching of the amount of movement means that, assuming that the feature point at time A moves to the position of the feature point at time B due to the movement of the vehicle, the movement direction and movement distance of the feature point roughly correspond to the movement direction and movement distance of the vehicle. This process of matching feature points captured at different times is called tracking. The change in the positions of the pair of feature points on the camera image is a motion parallax caused by the movement of the vehicle 1 between time A and time B, so the three-dimensional coordinates of the feature points can be identified by applying the principle of triangulation to the motion parallax.

[0040] The map generating unit 120 further evaluates the feature points whose three-dimensional coordinates can be identified, selects those that meet predetermined conditions, and registers them on the map.

[0041] The storage unit 170 has a volatile area and a non-volatile area, and the map is stored in the non-volatile area. Storing in a non-volatile area is also called recording or registering.

[0042] When automatic parking is started, the feature point detection unit 140 compares the information of the feature points detected by the feature point detection unit 140 from the camera image (such as information on the color and shape of the image, and information on the position of the feature points on the camera image) with the information of the feature points read from the map to detect feature points that match the map. If the starting position 11 of automatic parking is close to the position where the learning drive started and there is no significant difference in the orientation of the vehicle body, the positions of the feature points on the camera image when automatic parking starts should not be significantly different from the positions of the feature points on the camera image when the learning drive started. Since this difference in the positions of the feature points corresponds to the difference in the position and orientation of the vehicle, it is possible to identify how the position and orientation of the vehicle differs between when the learning drive started and when automatic parking started, from the difference in the positions of the feature points.

[0043] Specifically, the position of a feature point on a camera image corresponds to the angle of the feature point (the angle of the line connecting the feature point and the camera with respect to the optical axis direction), and indicates the angle at which the feature point is visible from the vehicle 1. When the position estimation unit 150 compares the angles of the feature points on all four sides with the three-dimensional coordinates of the feature points on the map, the position of the vehicle that appears at the angle that matches the feature points on all four sides is limited to a specific range. Therefore, the position estimation unit 150 specifies the position and attitude of the vehicle 1 by finding an optimal solution of the position and attitude of the vehicle 1 that matches the angles of multiple feature points. This process is called self-position estimation. The degree to which the angle of the feature points matches the conditions may be referred to as likelihood, and the process of finding the optimal solution may be referred to as maximum likelihood estimation.

[0044] Self-location estimation can be performed using an existing method, and detailed description will be omitted. However, regardless of the method used, if the coordinates of a feature point registered on a map are offset from the coordinates of the actual feature point, or if the angle of a feature point estimated from the position where the feature point is captured on a camera image is offset from the actual angle, the estimated position will be offset from the actual position. In self-location estimation by the position estimation unit 150, if a large number of feature points are used, even if there is an offset in the information of some of the feature points, the offset in the position estimation can be kept small, but if there are few feature points, the offset in the position estimation may be significant.

[0045] The driving control unit 160 communicates with the vehicle control device during learning driving, acquires information on the number of rotations of the wheels and the steering angle, and calculates the amount of movement and direction of the vehicle for each unit time. For example, a small section is defined as the amount of movement and direction of the vehicle for each unit time expressed as the length and direction of a line segment. In other words, a small section has data on length and direction. A small section may also be expressed as the number of rotations of the wheels and the steering angle. The driving control unit 160 calculates the position, attitude, and route of the vehicle every moment by integrating the data for the small sections.

[0046] The map generating unit 120 may receive route information from the driving control unit 160 in units of data for small sections, or may receive a parking route (a collection of data for small sections) at the time of parking. This parking route will be a broken line made up of small sections, so both ends of each small section will be called break points. The parking route during learning driving may be a collection of data for break points.

[0047] When performing automatic parking, the driving control unit 160 estimates the position, attitude, and route of the vehicle based on the information on the wheel rotation speed and steering angle, and outputs instruction values ​​to the vehicle control device 30 every moment so that the route of the vehicle reproduces the route when the vehicle was learned. In other words, the parking route when performing automatic parking is a reproduction of the parking route when the vehicle was learned.

[0048] When the position and attitude of the vehicle deviate from the position and attitude of the vehicle when the vehicle was learned, the driving control unit 160 first controls the steering angle, changes the steering angle so that the path intersects with the path when the vehicle was learned, and controls the steering angle so that the attitude of the vehicle coincides with the path when the vehicle was learned. In other words, the driving control unit 160 estimates the position, attitude, and path of the vehicle, and feedback controls the steering angle so that the path of the vehicle follows the parking path of the learned path. Therefore, the driving control unit 160 may be called a vehicle control unit. Alternatively, focusing on parking the vehicle based on a map, the position estimation unit 150 and the memory unit 170 may be included in the vehicle control unit.

[0049] When automatically parking, the driving control unit 160 may reproduce the vehicle speed at which the vehicle was driven during learning, but may also limit the vehicle speed. This is because if the vehicle speed is high, the wheels may slip and the vehicle may deviate from the route. The driving control unit 160 may keep the vehicle speed at, for example, 5 km per hour.

[0050] Since the position estimation unit 150 estimates the position and attitude of the vehicle separately from the driving control unit 160, for example, when it is estimated that slip has occurred from the number of rotations of the wheels, the vehicle position and attitude data of the driving control unit 160 may be overwritten with the position and attitude data estimated by the position estimation unit 150, thereby correcting the subsequent steering angle control.

[0051] Fig. 5 is a diagram showing a hardware configuration of parking assistance device 100 in this embodiment. The functions of parking assistance device 100 may be implemented in the hardware shown in Fig. 5. Parking assistance device 100 may be a computer including CPU 101, ROM 102, RAM 103, I / O (input / output interface) 104, and IMP (Image Processor) 105, with each element being connected via a bus.

[0052] Parking assistance device 100 may accommodate multiple elements on one chip, or one element may be composed of multiple chips. A bus need not be a single one, but may be a combination of multiple types of buses. For example, CPU 101, ROM 102, RAM 103, and IMP 105 may be accommodated on one chip and connected by a parallel bus, and I / O 104 may be composed of multiple chips and connected to the chip accommodating CPU 101 via a serial bus.

[0053] The CPU 101 controls the entire parking assistance device 100. The functions of each part of the parking assistance device 100 may be implemented in the form of a program executed by the CPU 101. The ROM 102 and the RAM 103 correspond to storage parts, and the ROM 102 corresponds to a non-volatile area. The RAM 103 is used for temporary storage as a working area of ​​the CPU 101. For example, the RAM 103 temporarily stores camera images such as display images and detection images, and information on detected feature points. The IMP 105 is a processor with improved processing performance specialized for image processing and parallel processing, and the processing of the image processing unit 130, the feature point detection unit 140, the position estimation unit 150, and the like may be executed by the IMP 105. In other words, the parking assistance device shown in FIG. 4 may be realized by a combination of hardware such as that shown in FIG. 5 and a program executed on the hardware.

[0054] From here, the relationship between the direction of the feature point and the accuracy will be explained. FIG. 6 is a diagram showing the positional relationship between the vehicle 1 and an object. The optical axis of the side camera 2a housed in the left side mirror is oriented in a direction perpendicular to the longitudinal axis of the vehicle body, and an object C close to the optical axis direction and an object D located approximately 90 degrees from the optical axis direction are captured in one side camera image. However, the accuracy of the position of the image captured in the camera image is not uniform. For example, the position of the image of object D captured in the periphery (near the left edge) of the side camera image is less accurate than the position of the image of object C captured in the center. The reason for this inaccuracy will be explained below.

[0055] Fig. 7 is a diagram showing the effect on accuracy of the angle of the camera with respect to the optical axis direction. A lens that can capture images in directions perpendicular to the optical axis direction is called a fisheye lens, and is configured so that even light rays that enter from directions perpendicular to the optical axis direction hit the image sensor by repeatedly bending light rays with multiple lenses. However, the way images are captured in a fisheye image 201 is very different between the center, where objects close to the optical axis direction are captured on the image sensor, and the peripheral area, where objects outside the center and far from the optical axis direction are captured.

[0056] In the center of the sensor, the light coming from the optical axis direction is weakly converged and strikes the sensor surface at an angle close to a right angle, whereas in the periphery of the sensor, the light rays from the oblique to the right angle direction are strongly converged and strike the sensor surface obliquely in a narrow range. For example, the distance that the light spot moves on the sensor surface while the incident angle of a thin light beam changes from 90 degrees to 45 degrees is shorter than the distance that the light spot moves while the incident angle changes from 45 degrees to 0 degrees. In other words, the light outside the 45 degree angle is converged and strikes a narrow range on the sensor surface, and since the light is more strongly converged toward the outside, the image of the subject appears to be squashed toward the center in the periphery of the fisheye image 201.

[0057] In addition, light contains wavelengths from red to blue, and the angle at which it is bent by the lens varies depending on the wavelength, so red and blue light rays hit different positions on the image sensor. This is called chromatic aberration, and the effect of chromatic aberration is greater in the peripheral areas where the light rays are bent more. In other words, color shift occurs in the peripheral areas, where the position at which the image is projected changes depending on the color. In addition, the focal plane where the image is focused at a single point is not flat but curved, and since the focal plane is adjusted to match the center of the image sensor, the peripheral areas become out of focus and out of focus. In addition, since the image sensor has a three-dimensional structure, light hitting the image sensor at an angle in the peripheral areas also causes blurring. In addition, the image projected in the peripheral areas becomes blurred due to being stretched in the next distortion correction. In other words, due to various factors, the image projected in the peripheral areas becomes blurred more than the image projected in the center.

[0058] Since a lens bends light rays more strongly toward the outside, the relationship between the orientation of the subject and the position of the subject's image is a nonlinear function. Therefore, image processing unit 130 corrects the distortion of fisheye image 201 by stretching it outward according to the inverse function of this nonlinear function, and generates distortion-corrected image 203. This is called lens distortion correction 202. Specifically, image processing unit 130 stretches fisheye image 201 in a direction away from the image center according to the inverse function of the nonlinear image height characteristic that indicates the relationship between the distance from the center and the size of the image, and creates distortion-corrected image 203.

[0059] The image height characteristic is expressed by a graph with the image height on the vertical axis and θ degrees on the horizontal axis, where the image height is the distance between the image of light incident on the optical axis at an angle of θ degrees to the optical axis and the image center, when the position of the image of light incident on the optical axis is the image center. The detection image may be one that has been subjected to image processing on the distortion-corrected image 203, or the distortion-corrected image 203 may be used as is. Since the distance from the image center of the image on the detection image (image height) is directly proportional to the angle θ with respect to the optical axis, in self-location estimation, the position of the image on the detection image is converted into the orientation of the subject.

[0060] However, in reality, distortion remains in the distortion-corrected image 203 after distortion correction to a greater or lesser extent, and the image reflected in the peripheral portion is more distorted than the image reflected in the center. One of the causes of the remaining distortion is the problem of lens precision (variation). The lens of the vehicle-mounted camera is constructed by combining lenses manufactured by a die-cutting method, and the precision of lenses manufactured by the die-cutting method is lower than the precision of lenses manufactured by a polishing method. Therefore, the image height characteristics of the lens of the vehicle-mounted camera vary within the range of shipping standards, and the range of image height variation tends to be wider in lower-cost cameras.

[0061] In addition, due to the extremely small size of the lenses used in vehicle-mounted cameras and cost constraints, when multiple lenses are stacked together, the position and angle of the lenses are not adjusted so that the optical axes of the lenses are aligned. If the optical axes of the lenses are not aligned, unequal image heights occur, where the image height changes depending on the direction relative to the center of the image. For example, when comparing the distance from the center of the image of an object located 90 degrees to the right of the optical axis with that of an object located 90 degrees to the left, the distance from the center of the image will be different for the left and right images.

[0062] In principle, if the image height characteristics are measured in multiple directions for each camera and recorded for each camera, and the image height characteristics applied are changed for each camera and direction, distortion can be corrected fairly accurately even if unequal image heights or image height variations occur. However, in reality, due to cost constraints, individual image height characteristics are not measured, and distortion correction is performed using the same standard image height characteristics for each camera and each direction. In other words, since distortion correction does not deal with unequal image heights or image height variations, the image position varies in the peripheral areas. Also, although image height characteristics change depending on the wavelength, in vehicle-mounted cameras, distortion correction is performed using the same standard image height characteristics for all color components due to cost constraints. In other words, since distortion correction does not correct chromatic aberration, the image position changes depending on the color of the subject in the peripheral areas.

[0063] In other words, the position of the image shifts by the amount that the image height characteristics of the lens deviate from the standard value, and the position of the image also shifts depending on the color. Furthermore, blurring also makes the identification of the position unstable. For example, if the image is blurred and the contour is in a gradation shape, the position of the quantified image may change in relation to the detection threshold. Since the blurring and position shift of the image are large in the peripheral area, it can be said that the position of the image reflected in the peripheral area is inaccurate. In other words, since self-location estimation converts the position of an object on the image, specifically, the lateral position of the image of the object on the corrected image, into the orientation of the object, the orientation of the object reflected in the peripheral area is less accurate than that of the object reflected in the center.

[0064] From the above, it can be said that it is more advantageous in terms of accuracy to preferentially register feature points in the direction of the optical axis of the camera on the map. Therefore, when selecting feature points to be registered on the map, the feature point selection unit evaluates the feature points based on the angle of the feature points with respect to the optical axis of a specified camera, and preferentially registers feature points with a smaller angle with respect to the optical axis of the camera on the map over feature points with a larger angle with respect to the optical axis of the camera.

[0065] Next, the effect of the height difference between the camera and the feature points on the accuracy will be described. FIG. 8 is a diagram showing the effect of the height difference between the camera and the feature points on the accuracy. The upper and lower figures are views of the vehicle body seen from the front, and the upper figure is a diagram during learning driving, with the vehicle body standing upright. The lower figure is a diagram during automatic parking, with one tire of the vehicle body running over a protrusion 213 on the road surface and tilting in the roll direction. The protrusion 213 may be, for example, a mound of snow or soil, or a situation in which the other wheel has fallen into a rut may be assumed.

[0066] Point E represents the position of the front camera that captures the image ahead of the vehicle 1. There is a pole 211 directly in front of the vehicle, point F represents the position of the characteristic point at the base of the pole 211, and point G represents the position of the characteristic point at the tip of the pole 211. In addition, there is a utility pole 212 on the right side of the vehicle body as seen by the driver.

[0067] The position estimation unit 150 converts the position of the feature point on the camera image, specifically, the lateral position of the feature point on the image, into the orientation of the object. In the upper diagram, feature points F and G are both captured in the center of the image, so the identified orientation is the same. In the lower diagram, feature points F and G are also in front of the camera, but point H on the road surface is captured in the center of the camera image, and feature points F and G are both captured to the right of the center of the image. The displacements of feature points F and G are both caused by the inclination of the vehicle body, but feature point F, which has a large difference in height from the camera, is displaced more by the inclination of the vehicle body than feature point G, which has a small difference in height from the camera.

[0068] If characteristic point F is registered on the map in the state shown in the upper diagram during learning driving, and self-location estimation is performed in the state shown in the lower diagram during automatic parking, characteristic point F will appear to the right of the center of the image, so even though there is a pole 211 in front of the camera, it is assumed that the vehicle body has shifted to the left (right in the diagram) in the direction of travel, and the steering angle is corrected to the right so that characteristic point F is displayed in the center of the camera image. This causes the course of vehicle 1 to change, and vehicle 1 approaches utility pole 212 on the right side. In this way, the roll of the vehicle body can cause errors in self-location estimation, resulting in erroneous steering angle correction.

[0069] Here, consider a case where, instead of feature point F with a large difference in elevation from the camera, feature point G with a small difference in elevation is registered on the map. In this case, the amount of displacement of feature point G due to the inclination of the vehicle body is smaller than the amount of displacement of feature point F, so it is expected that the amount of change in steering angle due to an error in self-position estimation will also be small. In other words, registering feature point G with a small difference in elevation from the camera is advantageous in terms of accuracy. Therefore, when selecting feature points, a feature point selection unit that selects feature points to be registered on the map evaluates the feature points based on the difference in elevation between a predetermined camera and the feature points, and gives priority to registering feature points with a small difference in elevation from the camera on the map over feature points with a large difference in elevation from the camera.

[0070] The conditions for feature points that are advantageous in terms of accuracy of position estimation have been described above based on the height difference with the camera and the positional relationship with the optical axis direction, but since the position of the camera and the optical axis direction change with the movement of the vehicle 1, the positions of feature points that contribute to accuracy cannot be specified unless the positions on the route are specified. In other words, it is necessary to specify the positions on the route and select feature points that are advantageous in terms of accuracy at those positions.

[0071] In addition, it is better to vary the priority of registration and the number of feature points to be registered depending on the position on the route. Therefore, the feature point selection unit may repeat the steps of identifying a specific point on the parking route or a specific section of the parking route, setting the priority of registering feature points or the number of feature points to be registered according to the position on the specific parking route, evaluating the feature points based on the position of the camera or the relative position of the camera to the optical axis direction at the specific parking route, and preferentially registering the feature points that are advantageous in terms of accuracy at that position, for each position on the parking route.

[0072] Alternatively, the parking route may be evaluated from a bird's-eye view, and a process for setting the registration priority and the number of feature points to be registered for each position on the route may be performed first, and then a process for selecting feature points to be registered on the map may be performed based on the position of the camera or the relative position of the camera to the optical axis direction at the position on the parking route. Alternatively, the effect may be obtained by either a process for varying the registration priority or the number of feature points to be registered depending on the position on the parking route, or a process for selecting feature points to be registered on the map based on the position of the camera or the relative position of the feature points to the optical axis direction at the position on the parking route.

[0073] In addition, depending on the position on the route, there are directions that require precision and directions that do not, so the cameras that should capture the feature points that contribute to precision are different. Therefore, when identifying a position on the route and evaluating feature points based on the camera position or the relative position of the camera to the optical axis direction, it is preferable to identify a camera that should be used as the evaluation standard at that position and select feature points that contribute to precision based on the position and optical axis direction of that specific camera. For example, at a position where precision in the left-right direction is required but precision in the front-back direction is not required, feature points captured in the optical axis direction of the front and back cameras are registered first, and the number of feature points registered near the front and back cameras is also increased. In that case, the registration of feature points captured by the left and right cameras may be delayed, or the number of feature points to be registered may be reduced.

[0074] Alternatively, a higher score may be assigned to positions where accuracy is highly necessary, and many feature points may be registered, and a lower score may be assigned to positions where accuracy is low. Alternatively, the stage for evaluating feature points and the stage for selecting feature points may be clearly separated, and selection may be performed after all evaluations have been completed. For example, an evaluation step in which feature points are evaluated at a certain position and a high evaluation score is given to feature points that are in advantageous positions in terms of obtaining accuracy may be performed at all positions, and then feature points may be selected in descending order of evaluation score. In this case as well, a similar effect can be expected by identifying a camera that contributes to accuracy at a position where accuracy is highly necessary, and giving a high evaluation score to feature points that contribute to accuracy.

[0075] The information to be specified and matters to be considered in self-location estimation for automatic parking are summarized below. The information to be specified in self-location estimation for automatic parking is the vehicle's forward / rearward position, the vehicle's left / right position, and the vehicle's orientation (posture). When applying this to an automatic parking route, the forward / rearward direction of vehicle 1 corresponds to the tangent direction of the parking route, the left / right direction of vehicle 1 corresponds to the normal direction of the parking route, and the orientation (posture) of vehicle 1 corresponds to the inclination of the tangent or normal line of the parking route. In automatic parking, the vehicle is steered according to the left / right position, and the vehicle speed is controlled and gears are changed according to the vehicle's forward / rearward position.

[0076] Since the feature points contribute to the position estimation by being captured by the camera, the direction of the optical axis of the camera is the criterion for evaluating the contribution of the feature points to the position estimation. The tangent direction of the parking path corresponds to the optical axis direction of the front and rear cameras 2 of the vehicle 1, and the normal direction of the parking path corresponds to the optical axis direction of the left and right cameras 2a of the vehicle 1.

[0077] A parking path can be divided into sections, such as straight sections and curved sections, and the direction and degree of estimation accuracy required vary depending on the nature of the section. The estimated position is obtained by triangulating the orientation of the feature points, but the accuracy may or may not be improved depending on the positional relationship between the section and the feature points. Here, feature points that contribute greatly to accuracy are called "sensitive" feature points, and feature points that contribute little to accuracy are called "low sensitive" feature points.

[0078] The relationship between the position of the feature point and the accuracy obtained thereby will be summarized below. FIG. 9 is a diagram showing the position of the feature point and the sensitivity in the front, back, left and right directions of the vehicle 1. Consider a case where the feature points IJKL are located on the front, back, left and right sides of the vehicle 1, and when the vehicle 1 moves forward, its movement is detected based on the change in the orientation of the feature points (motion parallax). The orientation of the feature points is a vehicle-based orientation with the forward direction of the vehicle 1 in front being 0 degrees. The cameras 2 on the front, back, left and right sides of the vehicle 1 capture images of the feature points, and the orientation of the feature points is detected as a lateral position on the camera image. For example, the feature point I is at 0 degrees, and the orientation does not change even if the vehicle 1 moves forward. In other words, the motion parallax of the feature point I is zero. Since the self-location estimation estimates the amount of movement based on the motion parallax, the amount of movement cannot be estimated with the feature point I. This can be said to be insensitive to the movement in the front and back directions.

[0079] Feature point J at the rear of vehicle 1 has sensitivity to longitudinal movement because its orientation changes with longitudinal movement of vehicle 1. Feature points KL on the left and right sides of vehicle 1 have greater changes in orientation with longitudinal movement of vehicle 1, and therefore can be said to have greater sensitivity than feature points I and J in the longitudinal direction of vehicle 1.

[0080] Comparing the sensitivity of feature point K and feature point L, the closer feature point K has a larger amount of change in orientation and therefore a higher sensitivity than the farther feature point L. In general, closer feature points have a larger motion parallax than farther feature points, so they can be said to have a higher sensitivity.

[0081] Therefore, in situations where high accuracy in detecting the position in the forward and backward directions is required, the map generating unit 120 should select feature points located in the left and right directions, and even better, among the feature points located in the left and right directions, select nearby feature points with priority over distant feature points.

[0082] FIG. 10 is a diagram showing the relationship between the distance from the path of the vehicle 1 to the feature point and the sensitivity. FIG. 11 is a diagram showing the change in sensitivity of the feature point when the vehicle 1 moves. The sensitivity on the vertical axis may be interpreted as motion parallax. FIG. 11 corresponds to FIG. 10, and is a graph showing the change in sensitivity of each feature point when the vehicle 1 passes beside the feature points shown in FIG. 10. For example, when the vehicle 1 passes directly beside the feature points K and L, the sensitivity (motion parallax) between the feature points K and L is maximized. At this time, the sensitivity of the feature point K is greater than the sensitivity of the feature point L, but as the vehicle 1 moves away, the sensitivity of the feature point K decreases and becomes less sensitive than the feature point L. In other words, the closer the feature point is to the path of the vehicle 1, the narrower the range of high sensitivity is.

[0083] If the vehicle 1 requires positional accuracy in the longitudinal direction not only directly beside the feature point K but also before and after it, the feature points K' and K'' before and after the feature point K may be registered. In other words, a narrow range of high sensitivity may be compensated for by increasing the number of feature points. For example, when setting an interval threshold limiting the interval of selected feature points in order to avoid bias in the feature points registered on a map, the interval threshold may be reduced for sections requiring accuracy in the longitudinal direction, thereby reducing the interval of the selected feature points. In other words, by identifying sections requiring accuracy during learning driving and densely registering a large number of feature points close to the route of the sections, the required accuracy can be obtained in sections requiring accuracy during automatic parking.

[0084] Alternatively, the map generating unit 120 may register both feature points K close to the route and feature points L far from the route on the map, so that when the vehicle 1 moves away from feature point K, the required accuracy can be maintained using feature point L. Alternatively, in sections where low accuracy in the forward / rearward direction is acceptable, the required accuracy can be maintained with a small number of feature points by selecting only feature points L far from the route of the vehicle 1. Note that the sensitivity (motion parallax) of a feature point decreases as the distance increases, so that even when selecting feature points that are far away, those that are, for example, 10 meters or more away should be excluded from selection.

[0085] A position where accuracy is maintained with a small number of feature points may be said to be a position with a low priority and a small number of registrations. Since it is better to register a nearby feature point at a position with a high priority and a large number of registrations, the distance of the prioritized feature point may be changed depending on the position on the parking route. In other words, the feature point selection unit evaluates the feature point based on the distance between the feature point and the position on the parking route, and when the position on the parking route is a position with a low priority for registering feature points or a position with a small number of registrations, the feature point with a longer distance is preferentially registered on the map compared to a position with a high priority for registering feature points or a position with a large number of registrations. For example, since the number of registrations is reduced at a position where accuracy in the forward and backward directions is not required, it is possible to preferentially register feature points that are about 5 meters away from the route among feature points captured by the left and right cameras, and not register feature points within 2 meters of the route.

[0086] 12 is a diagram showing the position of feature points and their sensitivity in the left-right direction. Considering a case where the position of the vehicle 1 changes in the left-right direction, the orientation of the left and right feature points KL does not change, whereas the orientation of the front and rear feature points I and J changes. In other words, the front and rear feature points I and J have a larger motion parallax in terms of the left-right position than the left and right feature points K and L, and therefore have a higher sensitivity. Therefore, in a situation where high accuracy in detecting the left-right position is required, the map generating unit 120 should preferentially select feature points that have a small angle with respect to the front-rear direction of the vehicle.

[0087] FIG. 13 is a diagram showing a case where a feature point is on the path of the vehicle. FIG. 14 is a diagram showing changes in the sensitivity of the feature point when the vehicle moves in the X direction. FIG. 14 corresponds to FIG. 13, and shows the time changes in the sensitivity in the left-right direction of feature point I and feature point J when vehicle 1 passes from left to right as shown in FIG. 13. Here, it is assumed that feature point I and feature point J are on the road surface, and vehicle 1 passes over the feature points. The fact that a closer feature point has a higher sensitivity than a more distant feature point is the same in the case of left-right sensitivity as in the case of front-rear position change, but as shown in FIG. 13, when a feature point is below vehicle 1, the feature point is in the blind spot of camera 2 and cannot be detected, and the sensitivity becomes zero.

[0088] In other words, the sensitivity of nearby feature points is good, but the sensitivity is lost when the feature point is in a blind spot. Therefore, among the feature points that can be detected at a position that requires accuracy in the left and right direction (for example, parking position P), the feature point that is closest to the front camera may be selected.

[0089] Fig. 15 is a diagram showing the positions of feature points and the sensitivity of the vehicle body posture. When the orientation of the vehicle 1 changes, the orientations of the front, rear, left and right feature points I, J, K, and L all change, so there is no difference in sensitivity due to the orientation. The fact that nearby feature points have higher sensitivity than distant feature points is the same as with the sensitivity in the front-rear and left-right directions. Therefore, in situations where high detection accuracy of the vehicle orientation is required, the map generating unit 120 may select nearby feature points with priority over distant feature points.

[0090] The difference in sensitivity of the feature points due to the distance when the direction of the vehicle 1 changes is a secondary effect, which is caused by the motion parallax that occurs when the position of the camera 2 changes when the direction of the vehicle 1 changes. Figure 16 is a diagram showing a state in which the direction or position of the camera 2 changes.

[0091] For example, as shown in the figure above, when the position of camera 2 remains the same and only the orientation of camera 2 changes, the image of a nearby object and the image of a distant object move the same amount on the screen. In other words, when only the orientation of camera 2 changes, there is no motion parallax, so there is no difference in sensitivity of feature points due to distance. As shown in the figure below, when the orientation of camera 2 changes as the orientation of vehicle 1 changes and the position of camera 2 changes at the same time, the image of a nearby object moves more than the image of a distant object due to motion parallax. In other words, Figure 16 shows that the image of a distant object moves the same amount as the image of the nearby object, or less. Patent Document 2 states that "the farther an object is from the vehicle, the more it moves in response to a change in attitude," which is the exact opposite of the phenomenon shown in the figure, but this is thought to be a misunderstanding of the secondary effects of motion parallax.

[0092] Next, the positions of the vehicle 1 that require high accuracy in self-location estimation will be described. Fig. 17 is a diagram showing the positions of the vehicle 1 that require high accuracy in self-location estimation. M is the parking start position, N is the turning start position, Q is the steering position, R is the turning end position, and S is the parking position. If the parking path is divided into a straight section and a curved section, M to S are all the end points (start point or end point) of the section. Of the end points, points N and R do not change the traveling direction of the vehicle (gear position), so they are called passing points, and points M, Q, and S are called stopping points, because the vehicle stops at these points.

[0093] At the stopping point, the steering angle is changed, the direction of travel is changed, and the vehicle is stopped, so if the positional accuracy in the forward / backward direction at the end point is poor, the vehicle may deviate from the parking path or the parking position may shift. Therefore, positional accuracy in the forward / backward direction is required at the end point. Also, if the position in the left / right direction is shifted at the parking position S or the turning position Q, the vehicle may approach an obstacle, so positional accuracy in the left / right direction is required at the stopping point. In other words, positional accuracy in both the forward / backward direction and the left / right direction is required at the stopping point.

[0094] Of the passing points N and R, N does not require high positional accuracy in the left and right direction because there are no obstacles on either side, but R is the point where the vehicle starts moving straight toward the parking position, and since left and right alignment is performed here, positional accuracy in the left and right direction is also required. At positions where positional accuracy is required, a feature point close to the camera and the optical axis direction can be selected based on the position of camera 2 and the optical axis direction at that point. Also, left and right alignment begins just before R, so positional accuracy in the left and right direction is also required at the position just before R.

[0095] Figures 18 and 19 are diagrams showing the arrangement of feature points around the end points. If the left and right positions need to be accurate at the position of R in Figure 17, for example, the alignment can be started from point R' just before R, as shown in Figure 18. In that case, it is sufficient to arrange the feature points so that positional accuracy can be obtained between R' and R.

[0096] 18, the map generating unit 120 may place feature points in ranges 221 located in the directions of the optical axes of the front, rear, left and right cameras 2 of the vehicle 1 when the vehicle 1 is at position R. In this way, it can be expected that feature points in the same range 221 will be captured in directions close to the optical axes of the front, rear, left and right cameras 2 even at the position of point R' just before R. In other words, by placing feature points around the end points, sufficient positional accuracy can be obtained even just before the end points.

[0097] Furthermore, the range of registered feature points may be adjusted so that they are easier to capture even in front of the end points, as shown in Fig. 19. Range 222 in Fig. 19 is adjusted by shifting the position of range 221 in response to the fact that the head of vehicle 1 is positioned to the left at the position of point R' in front of R, and is adjusted so that feature points are positioned approximately directly in front of camera 2 of vehicle 1 at both R and R'. Feature points positioned around these end points, including those in front of the end points, are called end point feature points.

[0098] In the route of an automatic parking, the part divided by a pair of end points is called a section. Based on its shape, a section can be divided into a straight section where the vehicle goes straight and a curved section where the vehicle turns at a fixed steering angle. For example, MN and RS in Figure 17 are straight sections, and NQ and QR are curved sections.

[0099] Geometrically, a section includes end points, but in processing feature points, end point feature points (feature points placed around end points) and section feature points are treated separately, and end point feature points are not included in section feature points. This is because end point feature points need to be treated preferentially. Specifically, when focusing on one section, feature points are placed around the end points first, and then, if necessary, feature points are placed around the section. When a vehicle is near an end point, position accuracy can be obtained with the feature points placed around the end points, so if position accuracy needs to be obtained in the middle between the end points, they are used as section feature points. In other words, section feature points can be treated as auxiliary points, and do not need to be registered if not necessary.

[0100] The position in the forward / backward direction within a section (the direction along the section if it is a straight section, or the tangent direction if it is a curved section) only needs to be known when approaching the end points, so positional accuracy in the forward / backward direction is not necessary in the intermediate portions between the end points. In other words, positional accuracy in the left / right direction (the direction perpendicular to the section if it is a straight section, or the normal direction if it is a curved section) is sufficient within the section. Within a section, positional accuracy is not obtained only from feature points placed as feature points of the section, but is obtained using all detected feature points. In other words, to obtain positional accuracy in the left / right direction within a section, feature points at the end points can be used, so feature points can be used to supplement feature points at positions in the middle of the section where accuracy is insufficient.

[0101] FIG. 20 is a diagram showing the ranges of feature points of the end points, arranged around the end points R and S of a straight section RS where a vehicle retreats from R to S, by using ellipses. For example, in the straight section from R to S, there is a range 231 of feature points arranged on the left and right of the vehicle at the position R, a range 233 of feature points arranged in the front and rear, and a range 232 of feature points arranged on the left and right of the vehicle at the position S, and a range 234 of feature points arranged in the front and rear. When the vehicle is located near the midpoint between the end points R and S, feature points in the blind spot below the vehicle cannot be detected, but all other detection points can be detected, and the feature points in the ranges 233 and 234 in the front and rear of the vehicle 1 are in the optical axis direction of the camera in the front-rear direction, so sufficient accuracy can be obtained in estimating the position in the left-right direction. Therefore, for example, as shown in FIG. 20, when the section is a short straight section, there may be no feature points to be registered as feature points of the section.

[0102] In this way, if the feature points of the end points are registered with priority at the end points, the number of feature points to be registered as feature points of the section will be reduced, and the majority of the feature points to be registered will be feature points of the end points. Alternatively, the feature points of the end points may be registered with priority by distributing (allocating) the feature points so that there are more of them than the feature points of the section. This can also be rephrased as the feature point selection unit registering feature points according to the position on the parking route, and registering the feature points to be registered at the end points with priority over the feature points to be registered in the section, or registering more feature points to be registered at the end points than the feature points to be registered in the section.

[0103] In addition, each section may be divided into a section before the end and the rest, the section before the end being called the end and the rest being called the middle section, and the end may be treated as having the same priority as the end when registering feature points. The end is the end point of the section that has the shortest route length to the parking position, and the route length to the parking position is the distance from that point along the parking route to the parking position.

[0104] This is to deal with the fact that even if it is found that the vehicle is off the route near the end point, it is difficult to correct the vehicle back onto the route. For example, even if it is found that the vehicle is off the route 20 cm to the side 1 m before the parking position, it is difficult to correct the deviation, but if it is found 2 m before the parking position, it is possible to correct the deviation. In other words, even if it is found that the vehicle is off the route at the end point, it is too late, so the system makes it possible to accurately estimate the vehicle's position from a position before the end point where the deviation can be corrected.

[0105] Therefore, the feature point selection unit evaluates the path length between the position on the parking path and the parking position, and either prioritizes feature points to be registered at a position with a shorter path length over feature points to be registered at a position with a longer path length, or registers more feature points at a position with a shorter path length than at a position with a longer path length. For example, even within one section, feature points to be registered at an end portion may be prioritized over feature points to be registered at an intermediate portion, and more feature points to be registered at an end portion may be registered than feature points to be registered at an intermediate portion.

[0106] The difference from the former, which prioritizes only the end points, is that the range in which feature points are preferentially allocated to obtain accuracy is expanded to the portion just before the end (the end portion), but as described above, sufficient accuracy can often be obtained even at the end portion if many feature points are allocated around the end points, so there need not be any substantial difference. In this embodiment, a distinction is made between the end points and the end portion in order to explain the background of the accuracy required at the end points and the end portion. However, if feature points are allocated with priority given to the end points, and when arranging feature points of a section, feature points are allocated with priority given to the portion close to the end point, it can be expected that the result will be the same as when the end portion is prioritized, so in practice, either is acceptable.

[0107] 21 and 22 are diagrams showing the posture convergence section in which the occupant aligns the vehicle in the left-right direction by steering the vehicle 1 in front of the parking position. Generally, the front wheels of the vehicle 1 are steerable, and the rear wheel shafts are fixed and do not rotate in the left-right direction. Therefore, while the vehicle 1 can move freely in the front-rear direction, it is difficult to move the vehicle in the left-right direction, and the rear part of the vehicle is particularly difficult to move in the left-right direction. Because of these characteristics of the vehicle 1, when parking the vehicle, the driver first steers the vehicle in front of the parking position to align the vehicle in the left-right direction, and then aligns the vehicle in the front-rear direction to the parking position before parking the vehicle.

[0108] That is, the last section of the parking path is a straight section that ends at the parking position, and before that there is a section where the vehicle is aligned left and right by steering. The vehicle's attitude (direction) changes while the vehicle is aligned left and right, but when the alignment is completed, the steering angle is returned to neutral and the change in attitude converges, so this section is called the attitude convergence section. The position of this attitude convergence section depends on the state of the parking space 241. For example, if the parking space 241 is a parking frame drawn on the road surface and there are no three-dimensional objects around it, the vehicle can be aligned even within the parking space 241.

[0109] Therefore, as shown in Fig. 21, the posture convergence section continues from the outside of the parking space to the inside of the parking space, and the length of the final straight section may be less than one vehicle length. However, as shown in Fig. 22, when parking on pallet 242 in a mechanical parking lot, the vehicle can only go straight on pallet 242, so left and right alignment must be completed just before pallet 242. Therefore, the posture convergence section ends just before pallet 242, and the length of the final straight section is more than one vehicle length.

[0110] Since the parking space 241 is often set at a substantially right angle to the road or passageway facing the parking space 241, it often has a curved section where the vehicle body is turned before the straight section. Therefore, when returning the steering angle to neutral at the end of the curved section, the left and right positioning is often performed by adjusting the speed at which the steering angle is returned. In that case, as shown in FIG. 22, the end part of the last curved section, in other words, the end part of the last curved section, becomes the attitude convergence section.

[0111] Therefore, a section where the attitude angle of the vehicle 1 has changed and the difference between the attitude angle and the attitude angle of the vehicle at the parking position is equal to or less than a predetermined angle threshold is set as the attitude convergence section, and the position accuracy in the left-right direction is improved in the attitude convergence section. For example, if the parking path ends with a straight section and there is a curved section before that, the attitude convergence section where the left-right alignment is performed is the end part of the curved section. By definition, the attitude convergence section does not include the last straight section. This is because the straight section is a section where the attitude angle of the vehicle is maintained and the left-right alignment is completed. In other words, the position accuracy in the attitude convergence section may be given more importance than the position accuracy at the parking position, and the feature point registration in the attitude convergence section may be given priority over the feature point registration at the parking position.

[0112] As described above, the section where the driver has performed left-right positioning may be set as the posture convergence section, or the posture convergence section may be determined geometrically. In the posture convergence section, it is preferable to arrange the feature points so that left-right positioning accuracy can be obtained. For example, as shown in FIG. 22, when parking on a pallet 242, if the feature point 244 at the entrance of the pallet 242 can be registered in the map, it is preferable for left-right positioning in the posture convergence section. When the vehicle 1 enters onto the pallet 242, the feature point 244 at the entrance of the pallet 242 is hidden under the vehicle body and loses sensitivity, but since left-right positioning is completed just before the pallet 242, there is no problem even if the feature point 244 at the entrance has no sensitivity while the vehicle 1 is moving straight on the pallet 242.

[0113] Feature points that are highly sensitive to positional changes in the left-right direction are feature points in the optical axis direction of the front and rear cameras 2 of the vehicle 1, and the optical axis direction of the front and rear cameras 2 is the tangential direction of the parking path when viewed from above. Furthermore, feature points closer to the front and rear cameras 2 of the vehicle 1 have higher sensitivity. Therefore, in an attitude convergence section where the attitude angle of the vehicle 1 has changed and the difference between the attitude angle and the attitude angle of the vehicle 1 at the parking position is equal to or less than a predetermined angle threshold, the map generating unit 120 may preferentially register in the map feature points that are in the tangential direction of the attitude convergence section and close to the attitude convergence section.

[0114] Fig. 23 is a diagram showing a turning section that requires precision in the left-right direction. Vehicle 1 has the tendency to go straight when the driver releases the steering wheel, which is called straight-line stability. The vehicle can turn by turning the steering wheel to change the direction of the steered wheels, but if the road surface is wet and the front wheels slip or the grip on the road surface weakens, the straight-line stability may cause the vehicle to deviate from the normal course to the outside. As shown in Fig. 23, if the vehicle deviates from the left-right position when turning at the entrance of parking space 251, it may not be possible to park the vehicle.

[0115] To prevent such a situation, it is necessary to detect left-right position deviations in curved sections and control the steering angle so as not to deviate from the route. Left-right position deviations are likely to occur when the steering angle is large and the turning radius is small, so curved sections where the absolute value of the steering angle is equal to or greater than a predetermined steering angle threshold value should be classified as turning sections in particular, and the left-right position accuracy should be improved in the turning sections.

[0116] As described above, since the posture convergence section may be the end of a turning section, the section corresponding to either the posture convergence section or the turning section may be set as the section for improving the positional accuracy in the left-right direction. Also, the entire curved section with a large steering angle may be set as the section for improving the positional accuracy in the left-right direction.

[0117] FIG. 24 is a diagram showing sensitivity in the left-right direction when the vehicle 1 turns. The feature points captured in the optical axis direction of the camera 2 in front of and behind the vehicle 1 are sensitive to positional deviation in the left-right direction, but since the orientation (travel direction) of the vehicle 1 changes in curved sections, the optical axis direction of the camera 2 is not constant. Therefore, the definition of the direction in curved sections and the conditions for feature points to be prioritized are summarized below. In curved sections, the vehicle turns with a turning radius according to the steering angle, so the model is organized assuming that the turning radius is constant and the curved section forms an arc that is part of a circumference. In this case, the left-right direction may be rephrased as the radial direction, and the front-rear direction as the circumferential direction.

[0118] Figure 24 shows a case where the curved section is an arc acb that runs from point a to point b via point c. If vehicle 1 is at point c and camera 2 observes the directions of points a and b relative to vehicle 1, the direction difference between points a and b, represented by ∠acb, is constant according to the circular angle theorem no matter where point c is on the arc acb.

[0119] However, if vehicle 1 deviates from the curved section (arc acb) to the outside and is at point d, the azimuth difference between points a and b, represented by ∠adb, becomes smaller than when vehicle 1 is on the parking path, such as ∠adb<∠acb. This shows that points a and b, or characteristic points on arc acb, have a certain degree of sensitivity in detecting the left-right position.

[0120] FIG. 25 is a diagram showing sensitivity in the left-right direction when the vehicle 1 turns. In FIG. 25, points on the arc ef are sampled at equal intervals, and an arrow 261 representing the front-rear direction of the vehicle 1 is plotted assuming that the vehicle 1 is located at the sampled point. The front-rear direction of the vehicle 1 is also the tangent direction of the arc ef. As shown in FIG. 25, in the section where the vehicle 1 turns, the front-rear direction of the vehicle 1 faces the outside of the arc, so it can be said that a feature point with good sensitivity in the left-right direction is outside the arc. Therefore, the map generating unit 120 may preferentially select a feature point that is outside the arc and close to the arc as a feature point with high sensitivity in the left-right direction.

[0121] In contrast, the area near the center of the arc is always located directly to the side regardless of the left-right or front-back position of the vehicle 1, and the angle seen from the vehicle 1 does not change much, so it can be said that the motion parallax is small and the sensitivity is low (the contribution to position accuracy is small). Therefore, when the route forms an arc, the map generating unit 120 evaluates the feature points based on the positional relationship between the feature points and the route, and it is preferable that feature points in the front-back direction of the vehicle or on the outside of the arc are easier to register on the map than feature points on the inside of the arc.

[0122] In addition, since the front-rear direction of the vehicle 1 is also a tangential direction, the map generating unit 120 may preferentially select characteristic points that are in the tangential direction and close to the curved section, as in the attitude convergence section, so that, as a result, characteristic points that are on the outside of the arc and close to the arc are preferentially selected.

[0123] 26 is a diagram showing a situation where the vehicle 1 requires accuracy in the left-right direction. When passing beside an obstacle (e.g., a utility pole 271) or turning around in front of a fence, position accuracy is required to avoid contact. In manual parking, the driver often decelerates the vehicle 1 in such cases, so the position where the vehicle 1 decelerates or travels at a low speed may be estimated to be close to the obstacle.

[0124] The map generating unit 120 may also determine that an obstacle is approaching based on the distance information from the obstacle obtained from the obstacle detection device. That is, when the vehicle speed is low or decelerating, or when the distance to the obstacle is close, it may determine that position accuracy is necessary. Also, when the distance to the obstacle is close and the vehicle speed is low or decelerating, it may determine that the need for position accuracy is particularly high.

[0125] Specifically, the feature point selection unit (map generation unit 120) registers feature points based on vehicle speed information indicating the vehicle speed or distance information indicating the distance from an obstacle, and registers feature points on the map in preference to locations where the vehicle speed is low or deceleration is occurring, rather than locations where the vehicle speed is high and there is no deceleration, or in preference to locations where the distance between the vehicle 1 and an obstacle is short, rather than locations where the distance between the vehicle 1 and an obstacle is long.

[0126] The direction requiring accuracy near an obstacle is not limited to the left-right direction. For example, when turning around in front of an obstacle, position accuracy in the front-rear direction is also required to avoid going too far and colliding. Therefore, the range in which feature points are preferentially selected may be different between a passing point where the vehicle passes without stopping and a stopping point where the vehicle stops, and the position of the front and rear cameras or the feature points close to the optical axis direction of the front and rear cameras may be preferentially selected at the passing point, and the position of the front, rear, left, right cameras or the feature points close to the optical axis direction of the front, rear, left, right cameras may be preferentially selected at the stopping point.

[0127] Alternatively, the priority of registering feature points may be changed according to the detection information of the obstacle. For example, if no obstacle is detected ahead at the turning position, there is no risk of collision even if the positional accuracy in the forward and backward directions is low, so the priority of the feature points captured by the left and right cameras may be lowered.

[0128] Furthermore, position accuracy should not be evaluated in two stages, "necessary" or "not necessary," but should be evaluated steplessly according to distance, vehicle speed, etc., and should be evaluated by combining information on the distance to an obstacle and vehicle speed when the driver passes by or approaches an obstacle during manual parking. For example, the map generating unit 120 may evaluate feature points based on the vehicle speed and changes in the vehicle speed, and may preferentially register feature points near the camera in front of the garage, capturing that the vehicle speed is low when entering the garage and that the vehicle speed decreases in front of the garage.

[0129] Alternatively, the map generating unit 120 may use the vehicle speed as the only condition and make it easier to register feature points located in the front-rear direction of the vehicle 1 on the map at locations where the vehicle speed is low after deceleration than at locations where the vehicle speed is high and there is no deceleration, or may evaluate whether the driver is reducing the vehicle speed as an indicator of the level of positional accuracy required, and may register more feature points closer to the vehicle 1 as the vehicle speed is lower. In other words, evaluation may be performed using a combination of multiple conditions, but a combination is not essential.

[0130] 27 is a flowchart of map generation in this embodiment. The process of generating a map by learning driving may be executed by the map generating unit 120 of the parking assistance device 100 in the following steps.

[0131] In the start point process, the map generating unit 120 registers the GPS coordinates of the start position and collects information on detected feature points (step S1). In the tracking process, the map generating unit 120 tracks the feature points and identifies their coordinates, and collects route information such as steering angle and movement amount (step S2). In the end point process, the map generating unit 120 determines that the learning drive is complete (step S3).

[0132] In the analysis process, the map generation unit 120 identifies the end points and sections of the parking route and organizes the feature points (step S4). In the point allocation process, the map generation unit 120 determines the number of feature points to be registered for each end point and each section (step S5). In the registration process, the map generation unit 120 determines the feature points to be registered for each end point and each section (step S6).

[0133] 27 corresponds to a case where the parking route is evaluated from a bird's-eye view, a process of setting the registration priority and the number of feature points to be registered for each position on the route is performed, and then a process of selecting the feature points to be registered on the map is performed based on the position of the camera on the parking route or the relative position of the camera with respect to the optical axis direction. In other words, the process of setting the registration priority and the number of feature points to be registered corresponds to step S5, and the process of selecting the feature points to be registered on the map based on the position of the camera on the parking route or the relative position of the camera with respect to the optical axis direction corresponds to step S6. Each step will be described in detail below.

[0134] For the explanation of the parking route, refer to FIG. 2. The start point of the start point process of S1 is the start position 11 of the learning drive. At the start point, the map to be used is specified from the GPS coordinates during automatic parking, and information is acquired to enable the initial self-location estimation. First, the map generating unit 120 acquires the GPS coordinates at the start position 11 from the navigation device 40 and stores them in the volatile area of ​​the storage unit 170. In addition, in the start point process, the map generating unit 120 collects information of characteristic points detected at the start position in order to use the information for the initial self-location estimation for automatic parking. Since the collected information is temporarily stored in the volatile area of ​​the storage unit 170, hereinafter, the map generating unit 120 collecting information may be rephrased as the map generating unit 120 writing information in the volatile area of ​​the storage unit 170.

[0135] The tracking process of S2 is performed while the vehicle 1 travels from the start position 11 to the parking position 14. During this time, the map generation unit 120 tracks feature points, calculates the coordinates of the feature points that have been successfully tracked, and collects feature point information including the coordinates. In addition, in the tracking process, the map generation unit 120 collects route information such as the steering angle, vehicle speed, gear position, moving direction, and moving distance, as well as obstacle detection information.

[0136] The end point of the end point process in S3 is the parking position 14 where the vehicle was parked during the learning drive, and when the gear position becomes parking (P), that position is determined as the end point. If the map generating unit 120 is executing tracking processing at the time when the end point is determined, the tracking processing ends, and the process transitions to analysis processing on the condition that the gear position is P. In other words, the end point process is a step for determining the start of analysis processing.

[0137] In the analysis process of S4, the map generation unit 120 analyzes the parking route and identifies the end points and sections. In the next point allocation process, the map generation unit 120 allocates points to each end point and section, so the analysis process can be said to be a pre-processing for the point allocation process.

[0138] S1 to S3 are processes during learning driving, and S4 and onwards are processes after learning driving. During learning driving, the map generating unit 120 prioritizes data collection, and performs other processes after learning driving. For example, during learning driving, the map generating unit 120 records the data of the small section or the time series of data of the break points received from the driving control unit 160 as is in the storage unit 170, and after learning driving, it analyzes the parking route to identify the sections and end points.

[0139] For example, the map generating unit 120 receives a parking route in the shape of a broken line composed of many small sections from the driving control unit 160. Therefore, in the analysis process of S4, the map generating unit 120 merges a plurality of small sections to reconstruct the parking route into a small number of sections divided by a small number of end points. For example, the map generating unit 120 may analyze the time series of data of the small sections, merge a series of small sections in which the steering angle does not change before and after to form a section, and set the break points (change points of the steering angle) where the steering angle changes before and after as the end points. Alternatively, the parking route may be analyzed from a bird's-eye view, and the parking route may be approximated by a small number of straight sections and a small number of curved sections divided by fewer end points than the break points.

[0140] The map generating unit 120 registers the parking route simplified by the reconstruction and approximation as the route for automatic parking. In other words, the map generating unit 120 reduces the number of end points of the route for automatic parking compared to the number of turning points of the parking route during learning driving. In automatic parking, the occupants feel unsettled every time the vehicle passes an end point (a point where the steering angle changes), so by simplifying the parking route during learning driving from a bird's-eye view and reducing the number of end points of the route for automatic parking, the number of times the occupants feel unsettled is reduced, resulting in a better usability.

[0141] It is also advantageous in terms of accuracy for the map generating unit 120 to have fewer end points than break points. Since steering angle change points are points where control is performed to change the steering angle during automatic parking, feature points should be preferentially placed around them in order to obtain positional accuracy. However, if all break points are made into end points, the number of feature points per end point will be reduced. Therefore, it is preferable for the map generating unit 120 to simplify the route to reduce the number of sections and reduce the number of end points to which feature points are assigned. In this way, positions that do not require positional accuracy are excluded from the end points, so that more feature points can be placed at positions (end points) that require positional accuracy.

[0142] Furthermore, the map generating unit 120 may treat the break points (change points of steering angle) of the route for automatic parking and the end points to which feature points are assigned as separate entities, reduce the number of end points to which feature points are assigned, but leave the break points (change points of steering angle) of the parking route unchanged. In other words, the map generating unit 120 may reduce the number of end points to which feature points are assigned without changing the parking route to be registered, by making most of the change points of steering angle break points to which feature points are not assigned. The break points (change points of steering angle) to which feature points are not assigned may be called recessive end points, and making a break point to which feature points are not assigned may be rephrased as recessive end points.

[0143] Fig. 28 is a diagram showing the adjustment of end points. For example, when sections gh and ij of a parking route passing through points g, h, i, and j are straight sections and section hi is a short curved section, the map generating unit 120 evaluates the relationship between the parking route and the parking position or obstacles. Then, depending on the evaluation result, the map generating unit 120 may determine one or both of points h and i as inferior end points and exclude them from the end points to which feature points are assigned.

[0144] For example, when point j is the parking position 14 and the section between points h and i, that is, section hi, is evaluated as a posture convergence section for adjusting the posture and lateral position of the vehicle body, the map generating unit 120 may leave both points h and i as end points to which feature points are assigned. For example, when an obstacle is detected near point h, the map generating unit 120 may evaluate that the lateral position accuracy of the vehicle body is required near point h, and leave point h as an end point to which feature points are assigned, but may make point i a recessive end point (break point) and exclude it from the end points to which feature points are assigned.

[0145] For example, if point j is not near the parking position 14 and no obstacles are detected near points h and i, the map generation unit 120 may evaluate that positional accuracy is not required at points h and i, and may treat points h and i as inferior end points (break points), excluding both of them from the end points to which feature points are assigned.

[0146] In this case, the map generating unit 120 may treat the points h and i as non-existent, approximate the section gj with a large arc, and register the section gj on the map as a single curved section gj with no break points along the way, or may leave the points h and i as recessive end points (break points) and register the section gj on the map as three sections gh, hi, ij with a change in steering angle at a break point along the way. In either case, the number of feature points decreases near the points h and i, and the number of feature points assigned to one end point can be increased. In summary, the analysis process performed by the map generating unit 120 can be said to be a process of analyzing the parking route during learning driving and reducing the number of end points to which feature points are assigned to fewer than the break points that are the change points in the steering angle during learning driving.

[0147] 29 is a diagram showing an example of point allocation to end points and sections. In the point allocation process of S5, the map generation unit 120 determines the upper limit (point allocation) of the number of feature points to be registered at the end points and sections. At that time, the map generation unit 120 evaluates the parking positions of the end points and sections, their relationships with obstacles, and the amount of change in steering angle and attitude, and changes the point allocation accordingly.

[0148] For example, the map generating unit 120 assigns a higher score to an end point or section closer to the parking position S. Since the user evaluates the quality of parking at the parking position S, the shorter the route length to the parking position S, the higher the accuracy. The score may be adjusted according to risk. For example, since risk is high near an obstacle, it is good to assign a higher score to an end point or section closer to the obstacle. Furthermore, the map generating unit 120 may estimate that a section where the driver traveled at a low speed is riskier than a section where the driver traveled at a faster speed, and assign a higher score to the section.

[0149] The map generating unit 120 evaluates the amount of change in the steering angle and attitude angle of the vehicle and reflects it in the allocation of points. This is because the greater the amount of change in the steering angle and attitude angle, the higher the risk of positional deviation in the left-right direction. The amount of change in attitude angle is the amount of change in the attitude angle of the vehicle (the direction in which the front of the vehicle body points) between the start point and the end point of the section. Positional accuracy in the left-right direction is also necessary in the attitude convergence section, which is a section in which the difference between the attitude angle of the vehicle and the attitude angle of the vehicle at the parking position is equal to or less than a predetermined angle threshold. Therefore, the feature point selecting unit evaluates the amount of change in the steering angle and attitude angle of the vehicle, and in either a turning section in which the steering angle is equal to or more than a predetermined threshold, a turning section in which the amount of change in the attitude angle in the section is equal to or more than a predetermined threshold, or a posture convergence section in which the difference between the attitude angle of the vehicle and the attitude angle of the vehicle at the parking position is equal to or less than a predetermined angle threshold, the feature point in the tangential direction of the section may be preferentially registered on the map, or the number of feature points to be registered may be increased by increasing the allocation of points.

[0150] Between endpoints and sections, the points allocated to endpoints are given priority over the points allocated to sections, and the points allocated to endpoints are greater than the points allocated to sections. In the example of Fig. 29, 75% of the points are allocated to endpoints M, N, Q, R, and S, but since feature points around endpoints can often be detected in the sections before and after them, there is no problem if the points allocated to sections are smaller. Points are allocated to sections so that position estimation can be performed even between endpoints, so the map generating unit 120 may allocate more points to particularly long sections if there are any.

[0151] End point M is the starting point, and since automatic parking cannot begin unless self-location estimation can be performed at the starting point, a high score is allocated to it. The end point with the next highest score is R. Vehicle 1 has finished adjusting its lateral position and attitude at R, and the end part of section QR is also an attitude convergence section, so it is given priority in score allocation over parking position S. In other words, the priority is not determined solely by the path length to parking position S, but the priority and score allocation are determined according to the need for positional accuracy. For example, in section RS, the vehicle is close to obstacles on the left and right, and it is difficult to correct its attitude after end point R, so end point R is given a high score.

[0152] Q is the turning position, and S is the parking position. Both are stopping points and close to obstacles, so a high score is assigned to them to obtain positional accuracy in the forward and backward directions. N is the starting point of a turn, but it is a passing point, and the route to the parking position S is long and it is not close to obstacles, so it is assigned a lower score than the other end points.

[0153] Among the sections, the QR section, which includes the posture convergence section, is assigned the highest score. The NQ section, which has the same turning radius as the QR section, is assigned a lower score because it is farther from the parking position, but it is assigned a higher score than the straight section MN and straight section RS. The straight section MN is assigned a lower score because the route length to the parking position S is longer than the straight section RS.

[0154] In the registration process of S6, the map generating unit 120 first determines the order of determining the feature points, and then determines the feature points to be registered for each end point and each section. The order of determining the feature points is such that the determination of the feature points of the end points is prioritized over the determination of the feature points of the sections, but the order may be such that the feature points have a shorter route length to the parking position S, or the feature points with a higher score are prioritized. For example, the order may be point S, section RS, point R, section QR, point Q, section NQ, point N, section MN, and point M, or the order may be point R, point S, point Q, point M, section QR, section NQ, section RS, point N, and section MN. The map generating unit 120 may also process the sections after completing the processing of all the end points, such as in the order of point R, point S, point Q, point M, point N, section QR, section NQ, section RS, and section MN.

[0155] In the registration process, the map generating unit 120 imposes density restrictions to prevent feature points from concentrating in a small range. For example, in the vicinity of a previously registered feature point (marker), the registration of feature points may be prohibited or the evaluation value of the feature point may be deducted (lateral inhibition). In other words, as a side effect of the density restrictions, a previously registered feature point prevents the later registration of surrounding feature points. The reason for registering feature points of end points or sections with high importance first when determining the order in which feature points are determined is to prevent feature points with high sensitivity from being unable to be registered due to density restrictions.

[0156] In addition, in order to allow a sufficient number of feature points to be registered at locations requiring positional accuracy, the map generating unit 120 adjusts the density restriction parameters according to the points allocated, and adjusts the density restriction threshold and the strength of lateral inhibition so that the density is high at locations with high points allocated. In other words, the feature point selecting unit includes a density restriction unit that restricts the density of feature points to be registered, and the density restriction unit varies the density according to the location on the parking route, and the density is higher at locations with a high priority for registering feature points or locations with a large number of registered feature points than at locations with a low priority and a small number of registered feature points. In other words, at locations requiring positional accuracy, the restriction by the density restriction unit is relaxed, and feature points are concentrated in a narrow range.

[0157] 30 is a diagram showing the registration of feature points at endpoint R, which is the end point of the curved section QR that turns toward the parking position. When registering the feature point of endpoint R, the map generating unit 120 selects a camera to be used as the evaluation standard depending on the direction in which accuracy is required, evaluates the angle of the feature point with respect to the optical axis of the camera, the elevation difference between the feature point and the camera, and the distance, using the selected camera as the standard, and preferentially registers feature points that are in advantageous positions for obtaining accuracy.

[0158] For example, since the end point R requires positional accuracy in the left-right direction, the map generating unit 120 selects a feature point that is captured in the center of the front and rear cameras. When feature points k, l, and m are captured in the rear camera, k and l are in the direction of the optical axis of the camera, but k is closer to the camera, so the map generating unit 120 gives a higher priority to k. Although l and m are at the same distance from the camera, m has a larger angle from the optical axis and is captured in the peripheral area, so the map generating unit 120 gives a lower priority to m. If there is a feature point l' that is at the same position on the plane as the feature point l and has a smaller elevation difference with the camera than l, the map generating unit 120 gives priority to l' over l. If only one of l or l' can be registered due to density restrictions, the map generating unit 120 may register the feature point l' that has a smaller elevation difference with the camera and exclude the feature point l that has a larger elevation difference.

[0159] In this way, the map generating unit 120 determines the priority, for example, k>l'>m, and determines the range of feature points to be registered according to the allocation of points. For example, when the number of feature points (allocation of points) to be registered in the field of view of the rear camera is 3, the map generating unit 120 registers all of k, l', and m, but when the allocation of points is low, it registers only k and l', or only k. In reality, the number of feature points to be registered in the field of view of the rear camera at the end point of the posture convergence section is 4 or more, so that even feature points l, m, and m' that are in unfavorable positions in terms of obtaining accuracy may be registered. Also, feature points within the field of view (not shown) may be registered until the number determined by the allocation of points is reached, and in that case, feature points that are advantageous in terms of obtaining accuracy may be registered preferentially.

[0160] For example, the evaluation value of each feature point is calculated based on the angle relative to the optical axis direction of the rear camera and the elevation difference from the camera, and the feature point with the highest evaluation value is registered first. After one feature point is registered, the evaluation values ​​of feature points near it are deducted due to density restrictions. Next, the feature points are sorted again by the updated evaluation values, and the feature point with the next highest evaluation value is registered. This process of density restriction, sorting, and registration as one cycle can be repeated the number of times determined by the point allocation.

[0161] When selecting a feature point of an end point, the map generating unit 120 may select the feature point by considering the sections before and after the end point. For example, when feature points n and p are in the field of view of the front camera and only one of them can be registered due to density restriction, the map generating unit 120 may evaluate that n is closer to the optical axis direction of the front camera while traveling in the section QR and may preferentially register feature point n. Alternatively, the positions of the feature points in the sections before and after the end point may be evaluated, and the feature points with higher evaluation values ​​may be registered first. Both feature point n and feature point p are targets for adding points because they are located outside the arc of the curve section QR, but since feature point n is closer to the arc, it is better to add more points. In addition, since the portion before R corresponds to the posture convergence section, the map generating unit 120 may adjust the threshold of the density restriction and register both feature point n and feature point p.

[0162] When registering feature points of the end points, the map generating unit 120 allocates the number of feature points to be registered to each camera according to the direction requiring accuracy and the traveling direction. For example, the map generating unit 120 allocates points to all cameras while assigning points in a slanted manner according to the direction requiring accuracy and the traveling direction, such as 40% to the rear camera facing the traveling direction, 30% to the front camera facing the opposite direction, 20% to the right camera facing the outside of the route QR, and 10% to the left camera facing the inside.

[0163] In this way, if not only feature points in the direction requiring accuracy but also feature points in other directions are registered, it becomes easier to maintain the function even if the environment changes. In other words, robustness is improved. For example, if only rear feature points k, l' and front feature point n are registered because position accuracy in the left-right direction is required, and left-right feature points m and m' are not registered, automatic parking may not be possible due to changes in the environment. For example, when rear feature points k, l' are in a garage and the rear camera image is completely black due to the direction of sunlight, and the feature points cannot be detected, the position of the vehicle cannot be determined by only the front feature point n, and self-location estimation cannot be performed, so automatic parking cannot be continued. In this regard, if feature points captured by the left and right cameras such as m and m' are also registered, self-location estimation can be performed based on the detection points in the three directions of the front, left and right, even if the rear camera image is completely black.

[0164] The purpose of density restriction, which avoids the concentration of feature points in a small area, is the same. For example, if only feature points located in a small area nearby are registered and those feature points belong to a vehicle parked nearby, when the vehicle disappears or moves, automatic parking may not be possible or the parking position may shift.

[0165] For example, if you select a feature point that is higher than the camera, the probability that it is another vehicle will be low, and if you select a feature point that is far from the camera, the probability that it is not on the road will be high. In other words, it is better to distribute the placement of feature points in terms of the height difference and distance from the camera.

[0166] 31 is a diagram showing the range of feature points to be registered in a curved section. At a feature point, the camera position and optical axis direction are fixed, whereas the camera position and optical axis direction change in a curved section. Therefore, the map generating unit 120 may evaluate feature points based on the trajectory of the camera position movement within the section and the range in which the optical axis direction changes.

[0167] For example, if the arc qr is the trajectory of the camera in the traveling direction, and the optical axis direction changes from the direction qs to the direction rt, then the range 281 enclosed by the arcs qr and st, and the line segments qs and rt, corresponds to the optical axis direction and can be said to be a range close to the camera. Also, the range 282 enclosed by the arcs st and uv, and the line segments su and tv in Fig. 31 has the same conditions in the optical axis direction and can be said to be a range farther from the camera. Therefore, the feature points in the range 281 may be given a higher evaluation value than the feature points in the range 282, and feature points with higher evaluation values ​​may be registered preferentially.

[0168] In this way, the map generating unit 120 may specify a range where the conditions in the optical axis direction are the same in the curved section, and within that range, evaluate the feature points according to the distance to the camera position. Alternatively, as a simpler method, the map generating unit 120 may evaluate the feature points according to the distance to the camera trajectory.

[0169] FIG. 32 is a diagram showing the range of feature points to be registered in a straight section. In a straight section, the optical axis direction of the front and rear cameras does not change, and the optical axis direction of the left and right cameras moves in parallel, so the relationship between the camera and the feature point is determined by the distance between the camera trajectory and the feature point. Therefore, the map generating unit 120 may, for example, use the trajectories of the left and right cameras as a reference to divide the space into tree-ring-shaped (or Baumkuchen-shaped) regions 291 that are the same distance from the camera trajectory, and evaluate the feature points. For example, in a range on one tree ring where the distance from the camera trajectory is the same, a feature point with a small angle difference with the optical axis of the camera may be prioritized, and for feature points with the same angle difference with the optical axis of the camera, a feature point with a short distance from the camera trajectory may be prioritized. Furthermore, the map generating unit 120 may change the evaluation criteria and prioritize a feature point with a small height difference with the camera in a range on one tree ring where the distance from the camera is the same. This evaluation method can also be implemented in a curved section.

[0170] In addition, when registering feature points, the distance of feature points to be registered with priority may be adjusted according to the allocation of points. This is because nearby feature points have a high maximum sensitivity, but the range of high sensitivity is narrow, so the sensitivity drops rapidly when moving away from the feature points. Also, nearby feature points may be in the blind spot of the vehicle, and when the feature point is in the blind spot of the vehicle, the sensitivity becomes zero. Therefore, the feature point selection unit evaluates the feature points based on the distance between the feature points and the position on the parking route, and when the position on the parking route is a position with a low priority for registering feature points or a position with a small number of registrations, the feature points with a longer distance are registered with priority compared to positions with a high priority for registering feature points or a position with a large number of registrations. A position with a high priority and many registrations of feature points is a position with a high allocation of points, so in other words, if the allocation of points is high, the priority is given to nearby feature points, and if the allocation of points is low, the priority is given to distant feature points. In this way, in positions with a small allocation of points, the distant feature points are registered with priority, so that the position accuracy within the section can be ensured with a small number of feature points.

[0171] Furthermore, the map generating unit 120 may give priority to the feature points with the shorter route length to the parking position, and may register feature points in order from the closest to the end point, even within one section. This is to prevent feature points that are in advantageous positions for obtaining accuracy near an end point, which requires higher accuracy, from not being registered due to density restrictions. For this reason, for example, a gradient addition may be applied to the evaluation values ​​of feature points that are candidates for registration in one section, with more points being added the closer they are to the end point, so that feature points closer to the end point are registered first.

[0172] When feature point registration at the end points is performed first, feature points are already registered around the start and end points of the section. In particular, in straight sections, positional accuracy in the left-right direction can often be ensured by feature points registered at the front and rear end points. Therefore, when registering feature points at the end points first, the map generating unit 120 may register left and right feature points based on the left and right cameras as feature point registration for the section to ensure positional accuracy in the front-back direction. Furthermore, when registering feature points in curved sections, the map generating unit 120 may preferentially register feature points on the outside of the arc over feature points in the direction of the center of the arc.

[0173] In addition, the present disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present disclosure. [Industrial Applicability]

[0174] The present disclosure can be used in a parking assistance device and a parking assistance method. [Explanation of symbols]

[0175] 1 vehicle 2 Camera 10 Operating device 11 Starting position 12 end points 14 Parking Location 20 HMI device 30 Vehicle control device 40 Navigation devices 100 Parking assistance device 101 CPU 102 ROM 103 RAM 104 I / O 105 IMP 110 Status Management Unit 120 Map Generation Unit 130 Image processing section 140 Feature point detection unit 150 Position estimation part 160 Driving control unit 170 Storage section 180 Notification Department 201 Fisheye Images 202 Distortion correction 203 Image after distortion correction 211 Paul 213 Convex 241 Parking Spaces 242 Palettes 261 Arrow 271 Electric pole

Claims

1. an image acquisition unit that acquires camera images from cameras each capturing an image in a different direction around the vehicle; A feature point detection unit that extracts feature points from the camera image; a feature point selection unit that evaluates the feature points and selects the feature points to be registered on the map during a learning drive in which the vehicle is manually parked and the parking route and parking position are registered on a map; A vehicle control unit that parks the vehicle based on the map during automatic parking, a parking assistance device in which the feature point selection unit varies a priority for registering feature points or a number of feature points to be registered depending on a position on the parking path, or selects feature points to be registered on the map based on a relative position of the feature point with respect to a position of the camera or a direction of an optical axis of the camera at a position on the parking path.

2. 2. The parking assistance device according to claim 1, wherein the feature point selection unit evaluates the feature points based on an elevation difference between the camera and the feature points, and registers on the map feature points with a smaller elevation difference with a higher priority than feature points with a larger elevation difference.

3. 2. The parking assistance device according to claim 1, wherein the feature point selection unit evaluates the feature points based on angles of the feature points with respect to an optical axis direction of the camera, and registers the feature points with smaller angles on the map with priority over feature points with larger angles.

4. 2. The parking assistance device according to claim 1, wherein the feature point selection unit evaluates the feature points based on a distance between the position on the parking path and the feature point, and when the position on the parking path is a position with a low priority for registering the feature point or a position with a small number of registrations, the feature point selection unit preferentially registers the feature point with a longer distance on the map compared to when the position on the parking path is a position with a high priority for registering the feature point or a position with a large number of registrations.

5. 2. The parking assistance device according to claim 1, wherein the feature point selection unit registers feature points according to positions on the parking path, and registers feature points at end points in priority to feature points registered in sections, or registers more feature points at the end points than feature points registered in the sections.

6. 2. The parking assistance device according to claim 1, wherein the feature point selection unit evaluates a steering angle or an attitude angle of the vehicle, and in a turning section where the steering angle is equal to or greater than a predetermined threshold value, or a change in the attitude angle within the section is equal to or greater than a predetermined threshold value, or in an attitude convergence section where a difference between the attitude angle of the vehicle and the attitude angle of the vehicle at a parking position is equal to or less than a predetermined angle threshold value, the feature point selection unit preferentially registers on the map feature points located in a tangential direction of the section, or increases the number of feature points to be registered.

7. 2. The parking assistance device according to claim 1, wherein the feature point selection unit registers feature points based on vehicle speed information indicating a speed of the vehicle or distance information indicating a distance between the vehicle and an obstacle, and registers feature points at locations where the vehicle speed is low or deceleration is occurring with priority over locations where the vehicle speed is high and no deceleration is occurring, or registers feature points at locations where the distance is short with priority over locations where the distance is long.

8. 2. The parking assistance device according to claim 1, wherein the feature point selection unit evaluates a path length between a position on the parking path and the parking position, and either prioritizes registering feature points at positions where the path length is short over feature points at positions where the path length is long, or registers more feature points at positions where the path length is short than feature points at positions where the path length is long.

9. the feature point selection unit includes a density restriction unit that restricts a density of the feature points to be registered; 2. The parking assistance device according to claim 1, wherein the density limiting unit varies the density depending on a position on the parking path, and sets the density higher at a position where a priority of registering the feature points is high or where a large number of feature points are registered than at a position where the priority is low and where a small number of feature points are registered.

10. acquiring camera images from cameras each of which views a different direction around the vehicle; Extracting feature points from the camera image; During a learning drive in which the vehicle is manually parked and the parking route and parking position are registered on a map, the feature points are evaluated and selected to be registered on the map; and during automated parking, parking the vehicle based on the map. In the step of selecting the feature points, a priority for registering the feature points or the number of feature points to be registered is varied depending on the position on the parking path, or the feature points to be registered on the map are selected based on the position of the camera or a relative position of the feature points with respect to the optical axis direction of the camera at the position on the parking path.

11. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature points includes evaluating the feature points based on an elevation difference between the camera and the feature points, and registering the feature points with a smaller elevation difference on the map with a higher priority than the feature points with a larger elevation difference.

12. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature points includes evaluating the feature points based on angles of the feature points with respect to an optical axis of the camera, and registering the feature points with smaller angles on the map with higher priority than feature points with larger angles.

13. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature point includes evaluating the feature point based on a distance between the position on the parking route and the feature point, and when the position on the parking route is a position having a low priority for registering the feature point or a position having a small number of registrations, a feature point having a longer distance is preferentially registered on the map compared to a position having a high priority for registering the feature point or a position having a large number of registrations.

14. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature points includes registering feature points according to positions on the parking route, and registering feature points at end points in preference to feature points registered in a section, or registering more feature points at end points than feature points registered in the section.

15. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature points comprises evaluating a steering angle or an attitude angle of the vehicle, and preferentially registering on the map feature points located in a tangential direction of a section in either a turning section in which the steering angle is equal to or greater than a predetermined threshold value, or in which an amount of change in the attitude angle within the section is equal to or greater than a predetermined threshold value, or in an attitude convergence section in which a difference between the attitude angle of the vehicle and the attitude angle of the vehicle at a parking position is equal to or less than a predetermined angle threshold value, or increasing the number of feature points to be registered.

16. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature points registers feature points based on vehicle speed information indicating a vehicle speed of the vehicle or distance information indicating a distance between the vehicle and an obstacle, and registers feature points at positions where the vehicle speed is low or deceleration is occurring with priority over positions where the vehicle speed is high and deceleration is not occurring, or registers feature points at positions where the distance is short with priority over positions where the distance is long.

17. 11. The parking assistance method according to claim 10, wherein the step of selecting the feature points comprises evaluating a path length between a position on the parking path and the parking position, and registering feature points at positions where the path length is shorter in priority over feature points at positions where the path length is longer, or registering more feature points at positions where the path length is shorter than feature points at positions where the path length is longer.

18. The step of selecting the feature points includes a step of limiting a density of the feature points to be registered; 11. The parking assistance method according to claim 10, wherein the step of limiting the density varies the density depending on a position on the parking path, and the density is set higher at a position where a priority of registering the feature points is high or where a large number of feature points are registered than at a position where the priority is low and where a small number of feature points are registered.

19. acquiring camera images from cameras each of which views a different direction around the vehicle; Extracting feature points from the camera image; During a learning drive in which the vehicle is manually parked and the parking route and parking position are registered on a map, the feature points are evaluated and selected to be registered on the map; During automated parking, parking the vehicle based on the map; A parking assistance program for causing a computer to execute the following: a step of selecting the feature points by varying a priority for registering the feature points or the number of feature points to be registered depending on the position on the parking path, or by selecting the feature points to be registered on the map based on a relative position of the feature points with respect to the position of the camera or the optical axis direction of the camera at the position on the parking path.

Images