Parking assistance method and parking assistance device

Abstract

In the parking assistance method of the present invention, the position of the landmark object around the vehicle, i.e., the landmark object position, is detected from the image obtained by photographing the surroundings of the vehicle (S2), a first shooting condition of the shooting condition when shooting the image is detected (S3), and from a specified storage device storing a combination of one or more relative position relationships between the landmark object detected from images obtained from past shots of the surroundings of the target parking position, i.e., past images, and the target parking position and the shooting condition when shooting the past images, i.e., a second shooting condition, one or more relative position relationships and a second shooting condition combination are read out (S4), and from the read relative position relationships, a relative position relationship stored in the combination of the second shooting condition whose difference from the first shooting condition is less than a specified difference is selected (S5), and based on the selected relative position relationship and the landmark object position of the landmark object stored around the vehicle, the relative position between the current position of the vehicle and the target parking position is calculated, and parking assistance for the vehicle to the target parking position is implemented (S7 to S9).

Description

Technical Field

[0001] The invention relates to a parking assistance method and a parking assistance device. Background Art

[0002] As a technology related to parking assistance for assisting a vehicle to park at a target parking position, the following patent document 1 is known. In patent document 1, a marker object is detected and stored from an image of the surroundings of the target parking position taken in the past, and the relative position of the target parking position with respect to the vehicle is calculated based on the stored position of the marker object and the position of the marker object detected from an image of the surroundings of the vehicle taken during automatic parking, and the vehicle is automatically moved to the target parking position based on the calculated relative position.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-138664

[0006] Problems to be Solved by the Invention

[0007] However, the landmark object detected from the image may change depending on the shooting conditions of the image. Therefore, if the past shooting conditions of the stored landmark object are detected and different from the shooting conditions during parking assistance, the accuracy of parking assistance may be reduced. Summary of the invention

[0008] An object of the present invention is to suppress a decrease in parking assistance accuracy due to a change in a shooting environment of a captured image, in parking assistance for assisting a host vehicle to park at a target parking position based on a position of a landmark object detected from a captured image.

[0009] In a parking assistance method of one embodiment of the present invention, the position of a marker object around the vehicle, i.e., a first marker object position, is detected from an image obtained by photographing the surroundings of the vehicle, and the photographing conditions when photographing the image, i.e., the first photographing conditions, are detected. Regarding the marker object position of the marker object detected from an image obtained by photographing the surroundings of the target parking position in the past, i.e., a past image, i.e., a second marker object position, from a specified storage device storing a combination of a relative position relationship between one or more second marker object positions and the target parking position and the photographing conditions when photographing the past image, i.e., the second photographing conditions, one or more sets of relative position relationships and a combination of the second photographing conditions are read out. Among the read relative position relationships, a relative position relationship stored in a combination of the second photographing conditions whose difference from the first photographing condition is less than a specified difference is selected. Based on the selected relative position relationship and the first marker object position, the relative position between the current position of the vehicle and the target parking position is calculated, and parking assistance for the vehicle to the target parking position is implemented.

[0010] Effects of the Invention

[0011] According to the present invention, in parking assistance for assisting a host vehicle to park at a target parking position based on the position of a landmark object detected from a captured image, it is possible to suppress a decrease in parking assistance accuracy due to a change in the capturing environment of the captured image.

[0012] The objects and advantages of the present invention are realized by using the elements and combinations thereof shown in the scope of the claims.The above general description and the following detailed description are only exemplary and illustrative and should not be construed as limiting the present invention in the manner of the scope of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a diagram showing an example of a schematic configuration of a parking assist device according to an embodiment.

[0014] Figure 2 It is a schematic diagram of the positions of landmark objects around the target parking position.

[0015] Figure 3 This is an explanatory diagram of an example of a parking assistance method.

[0016] Figure 4 1 is an explanatory diagram of a first example of parking assistance data.

[0017] Figure 5 2 is an explanatory diagram of a second example of parking assist data.

[0018] Figure 6 yes Figure 1 A block diagram showing an example of the functional structure of a controller in FIG.

[0019] Figure 7 This is a flowchart of an example of a parking assistance method according to the embodiment. DETAILED DESCRIPTION

[0020] (structure)

[0021] Figure 1 1 is a diagram showing an example of a schematic configuration of a parking assistance device according to an embodiment. A vehicle 1 is provided with a parking assistance device 10 for assisting the vehicle 1 in parking at a target parking position. In parking assistance by the parking assistance device 10, a target driving trajectory from the current position of the vehicle 1 to the target parking position is calculated. Then, the vehicle 1 is assisted in driving along the target driving trajectory.

[0022] The parking assistance of the parking assistance device 10 includes various methods of assisting the vehicle 1 to travel along the target driving trajectory. For example, the parking of the vehicle 1 may be assisted by controlling the automatic driving of the vehicle 1 so as to travel to the target parking position along the target driving trajectory of the vehicle 1. In addition, the automatic driving of the vehicle 1 so as to travel to the target parking position along the target driving trajectory of the vehicle 1 refers to controlling all or part of the steering angle, driving force, and braking force of the vehicle 1, automatically implementing all or part of the driving of the vehicle 1 along the target driving trajectory, and assisting the control of the parking operation of the occupant.

[0023] In addition, for example, the target driving trajectory and the current position of the vehicle 1 can be displayed on a display device that can be visually recognized by the occupants of the vehicle 1, making it easier for the occupants of the vehicle 1 to perform driving operations so that the vehicle 1 travels along the target driving trajectory, thereby assisting the parking of the vehicle 1.

[0024] The parking assist device 10 includes a positioning device 11, an object sensor 12, a vehicle sensor 13, a communication device 14, a human-machine interface 15, an actuator 16, and a controller 17. In the drawings, the human-machine interface is labeled as "HMI".

[0025] The positioning device 11 measures the current position of the host vehicle 1. The positioning device 11 may include, for example, a global positioning system (GNSS) receiver. The GNSS receiver is, for example, a global positioning system (GPS) receiver, and measures the current position of the host vehicle 1 by receiving radio waves from a plurality of navigation satellites.

[0026] The object sensor 12 detects objects within a specified distance range from the host vehicle 1 (e.g., the detection area of ​​the object sensor 12). The object sensor 12 detects the surrounding environment of the host vehicle 1, such as the relative position of the objects existing around the host vehicle 1 and the host vehicle 1, the distance between the host vehicle 1 and the objects, and the direction in which the objects exist. The object sensor 12 may include, for example, a camera that captures the surrounding environment of the vehicle 1. The camera may also be, for example, a surround monitor camera that captures the surroundings of the host vehicle 1 and generates a captured image that is converted into a bird's-eye view (surround monitor image). The object sensor 12 may also include a distance measuring device such as a laser rangefinder (LRF), a radar, or a LiDAR (Light Detection and Ranging) laser radar.

[0027] The vehicle sensor 13 detects various information (vehicle information) obtained from the host vehicle 1. The vehicle sensor 13 may include, for example, a vehicle speed sensor for detecting the travel speed (vehicle speed) of the host vehicle 1, a wheel speed sensor for detecting the rotation speed of each tire of the host vehicle 1, a three-axis acceleration sensor (G sensor) for detecting the acceleration (including deceleration) of the three axes of the host vehicle 1, a steering angle sensor for detecting the steering angle of the steering wheel, a steering angle sensor for detecting the steering angle of the steering wheel, a gyro sensor for detecting the angular velocity generated by the host vehicle 1, and a yaw rate sensor for detecting the yaw rate.

[0028] The communication device 14 performs wireless communication with a communication device outside the host vehicle 1. The communication method of the communication device 14 may be, for example, wireless communication of a public mobile communication network, vehicle-to-vehicle communication, road-to-vehicle communication, or satellite communication.

[0029] The human-machine interface 15 is an interface device for transmitting and receiving information between the parking assist device 10 and the occupant. The human-machine interface 15 includes a display device (for example, a display screen of a navigation system or a display device provided near the instrument in front of the driver's seat) that can be visually recognized by the occupant of the vehicle 1, a speaker and a buzzer that output alarm sounds, notification sounds, and voice information. In addition, the human-machine interface 15 includes an operating element that receives an operation input from the occupant to the parking assist device 10. For example, the operating element may be a button, a switch, a joystick, a dial, a keyboard, a touch panel, etc.

[0030] The actuator 16 includes a steering actuator, an accelerator actuator, and a brake control actuator. The steering actuator controls the steering angle of the steering mechanism of the vehicle 1 according to the control signal from the controller 17. The accelerator actuator controls the accelerator opening of the drive device as the engine or the drive motor according to the control signal from the controller 17. The brake actuator operates the brake device according to the control signal from the controller 17.

[0031] The controller 17 is an electronic control unit (ECU) that performs parking assist control of the vehicle 1. The controller 14 includes peripheral components such as a processor 18 and a storage device 19. The processor 18 may be a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit).

[0032] The storage device 19 may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. The storage device 19 may include a register, a cache memory, a ROM (Read Only Memory) used as a main storage device, and a RAM (Random Access Memory) and other memories. The functions of the controller 17 described below are realized by, for example, the processor 18 executing a computer program stored in the storage device 19.

[0033] In addition, the controller 17 may be formed by dedicated hardware for executing each information processing described below. For example, the controller 17 may also include a functional logic circuit set in a general-purpose semiconductor integrated circuit. For example, the controller 17 may include a programmable logic device (PLD: Programmable Logic Device) such as a field programmable gate array (FPGA: Field-Programmable Gate Array) and the like.

[0034] Next, an example of parking assist control by the parking assist device 10 will be described. Figure 2 . When starting to use the parking assistance device 10 for parking assistance, first, the relative position relationship between the marker object position of the marker object existing near the target parking position 2 and the target parking position 2 is stored in the storage device 19. Here, the target parking position 2 refers to the target position for parking the vehicle 1. The marker object is a ground object that serves as a mark for determining the current position of the vehicle 1. The marker object can be, for example, a road surface marking (lane boundary line 3a, stop line 3b, road sign, etc.), a road boundary (curbstones 3c~3e, guardrails, etc.), and an obstacle (house 3f, fence 3g, object 3h, etc.).

[0035] When the relative positional relationship between the landmark position and the target parking position 2 is stored in the storage device 19 for the first time, the operation mode of the parking assist device 10 is set to the "landmark object learning mode". Then, the host vehicle 1 is parked at the target parking position 2 by manual driving. When the host vehicle 1 is parked at the target parking position 2 by manual driving, the parking assist device 10 may automatically set the operation mode to the "landmark object learning mode".

[0036] While the vehicle 1 is parked at the target parking position 2 by manual driving, the object sensor 12 detects the position of the landmark object around the vehicle 1. The object sensor 12 detects the position of the landmark object existing in the detection area within the detection distance range specified by the object sensor 12.

[0037] For example, the parking assist device 10 detects a road marking (in Figure 2 In the example of A, the lane boundary line 3a, the stop line 3b), the road boundary (in Figure 2 In the example of A, the grounding part of the curbstones 3c to 3e), obstacles (in Figure 2 In the example of A, the edge or corner part of the landmark object (such as the house 3f, the wall 3g, the grounding part of the object 3h) is used as the feature point, and the position of the feature point can also be used as the position of the landmark object.

[0038] Furthermore, the parking assist device 10 calculates the feature value of the detected feature point (e.g., the density and attribute of the feature point). Here, the detection of the feature point and the calculation of the feature value may also be performed using methods such as SIFT, SURF, ORB, BRIAK, KAZE, and AKAZE. In addition, the feature value of the feature point does not necessarily need to be detected, but at least the position of the feature point can be detected.

[0039] Next, the parking assist device 10 obtains the relative positional relationship between the position of the marker object (the position of the feature point) detected by the object sensor 12 and the target parking position 2. For example, the parking assist device 10 calculates the relative position of the feature point with respect to the host vehicle 1 based on the position of the feature point on the image and the camera information regarding the installation state of the camera on the host vehicle 1 (installation position, optical axis angle, and viewing angle). Next, the current position of the host vehicle 1 in the fixed coordinate system at the time when the position of the marker object is detected by the object sensor 12 is estimated, and the position of the marker object in the fixed coordinate system is calculated based on the estimated current position and the relative position of the marker object with respect to the host vehicle 1. Here, the fixed coordinate system refers to a coordinate system (e.g., a map coordinate system) with a specific location as the coordinate origin.

[0040] The current position of the host vehicle 1 in the fixed coordinate system can be estimated, for example, by the positioning device 11, the odometry method, or the dead reckoning method. The current position of the host vehicle 1 in the fixed coordinate system can also be estimated based on the landmark object detected by the object sensor 12, the map mapping with the known landmark object position and the high-precision map information, etc.

[0041] Next, the parking assist device 10 determines the target parking position 2 in the fixed coordinate system. For example, the position of the host vehicle 1 when the host vehicle 1 is located at the target parking position 2 can be detected as the target parking position 2. By determining the position of the marker object and the position of the target parking position 2 in the fixed coordinate system, the relative positional relationship between the marker object position and the target parking position 2 is determined.

[0042] The parking assist device 10 stores the relative positional relationship between the marker object position and the target parking position 2 in the storage device 19. For example, the target position in the fixed coordinate system and the position of the target parking position 2 may be stored in the storage device 19, respectively. Alternatively, the marker object position in the relative coordinate system with the target parking position 2 as the coordinate origin may be calculated and stored in the storage device 19. In the following description, an example of a case where the marker object position in the fixed coordinate system and the position of the target parking position 2 are stored in the storage device 19 is described.

[0043] In the following description, the landmark objects stored in the storage device 19 may be referred to as “learning completed landmark objects”. Figure 2 The circular icon indicates the marker object position of the learning completed marker object stored in the storage device 19.

[0044] Next, refer to Figure 3 A parking assistance method using the parking assistance device 10 will be described. In order to utilize parking assistance, the operation mode of the parking assistance device 10 is set to a "parking assistance mode."

[0045] In the parking assist mode, the parking assist device 10 reads out the target position and the position of the target parking position 2 stored in the storage device 19 . Figure 3 The circle icon indicates the marker object position of the learning completed marker object read from the storage device 19 in the parking assist mode.

[0046] The parking assist device 10 detects the relative positions of the landmark objects around the host vehicle 1 with respect to the host vehicle 1 as landmark object positions of the landmark objects around the host vehicle 1 by the object sensor 12. The landmark object position detection method is the same as that in the landmark object learning mode.

[0047] In the parking assist mode, the position of the landmark object detected by the object sensor 12 is indicated by a triangular icon. Figure 3 In the example, the lane boundary line 3a, the curbstones 3c and 3e, and the corner portions of the fence 3g are detected as the marker object positions.

[0048] The parking assist device 10 compares each marker object position (triangle icon) detected by the object sensor 12 in the parking assist mode with the marker object position (circle icon) of the learning completed marker object read from the storage device 19 to make the same marker object positions correspond to each other. For example, the parking assist device 10 can determine the marker object position with the same or similar feature quantity as the same marker object position. Or, regardless of the feature quantity, by comparing the relative positional relationship between the marker object positions (triangle icons) detected by the object sensor 12 and the relative positional relationship between the marker object positions (circle icons) of the learning completed marker object read from the storage device 19, the marker object position (triangle icon) detected by the object sensor 12 and the marker object position (circle icon) of the learning completed marker object read from the storage device 19 can also be made to correspond to each other. Alternatively, both the feature quantity and relative positional relationship of the above-mentioned feature points can be used to make the marker object position (triangle icon) detected by the object sensor 12 and the marker object position (circle icon) of the learning completed marker object read from the storage device 19 correspond to each other. Figure 3 In the example, the lane boundary line 3a, the curbstones 3c and 3e, and the corner portions of the fence 3g correspond to each other.

[0049] The parking assistance device 10 determines the current position of the vehicle 1 based on the relative position relationship between the marker object position (triangle icon) detected in the parking assistance mode and the vehicle 1, and the relative position relationship between the marker object position (circle icon) of the learned marker object corresponding to these feature points (triangle icon) and the target parking position 2.

[0050] For example, the parking assist device 10 may calculate the position of the target parking position 2 on a relative coordinate system (hereinafter referred to as the "vehicle coordinate system") based on the current position of the host vehicle 1. Alternatively, the current position of the host vehicle 1 on the fixed coordinate system may be calculated based on the relative positional relationship between the positions of the marker objects (triangular icons) detected in the parking assist mode and the host vehicle 1, and the marker object position (circle icon) of the learning completion marker object on the fixed coordinate system. By determining the position of the host vehicle 1 and the position of the target parking position 2 in the fixed coordinate system, the relative position of the target parking position 2 relative to the current position of the host vehicle 1 is determined.

[0051] The parking assist device 10 calculates a target driving trajectory from the current position of the host vehicle 1 to the target parking position 2 based on the relative position of the target parking position 2 to the current position of the host vehicle 1. The parking assist device 10 performs automatic driving to control the host vehicle 1 to drive to the target parking position along the calculated target driving trajectory.

[0052] However, when detecting feature points of a landmark object from an image, the feature points detected from the image change depending on the conditions under which the image was taken. For example, the feature points detected from the image change depending on the illumination (amount of light) and the direction of the light when the image was taken. In addition, due to the difference in vegetation in different seasons, for example, the feature points detected from the image (e.g., feature points detected from a fence or lawn) change.

[0053] If the shooting conditions at the time when the mark object position of the learning completion mark object was detected in the past are significantly different from the shooting conditions at the time when parking assistance is used, the number of mark object positions of the learning completion mark object (circle icons) that can correspond to the mark object positions (triangle icons) detected at the time of using parking assistance will decrease, the detection accuracy of the relative position of the target parking position 2 relative to the current position of the vehicle 1 will decrease, and the accuracy of parking assistance may be reduced.

[0054] Therefore, when assisting the vehicle 1 in parking to a target parking position, the parking assistance device 10 of the embodiment detects the positions of landmark objects around the vehicle 1 from an image obtained by photographing the surroundings of the vehicle 1, and detects the shooting conditions when the image was shot (sometimes referred to as "first shooting conditions" in the following description).

[0055] In addition, a shooting condition (sometimes referred to as a "second shooting condition" in the following description) when shooting an image in which a learning completion mark object was detected in the past (sometimes referred to as a "past image" below) and one or more combinations of the relative position relationship between the learning completion mark object detected from the past image and the target parking position 2 are stored in the storage device 19 as parking assistance data 20.

[0056] Figure 4 1 is an explanatory diagram of a first example of the parking assistance data 20 . Figure 4 The parking assistance data 20 in includes j sets of parking assistance data 20a1 to 20aj. The parking assistance data 20a1 to 20aj are combinations of a plurality of different second imaging conditions and marker object positions of learning-completed marker objects detected from images captured in the second imaging conditions, respectively.

[0057] For example, the parking assistance data 20a1 to 20aj include information on the date, time, and weather when the shooting is performed as the second shooting condition. In addition, as the marker object position of the learning completed marker object, the image feature information (the position and feature amount of the feature point in the fixed coordinate system) of the learning completed marker object is included.

[0058] In addition, the parking assistance data 20 also includes position information of the target parking position 2 on the fixed coordinate system.

[0059] The parking assistance device 10 selects the mark object position of the learned mark object after combining with the second shooting condition whose difference with the first shooting condition is less than the specified difference from these parking assistance data 20a1~20aj, and calculates the relative position of the target parking position 2 relative to the current position of the vehicle 1 based on the relative position relationship between the selected learned mark object and the target parking position 2, and the mark object position of the mark object detected around the vehicle 1.

[0060] In this way, by selecting a learning completion mark object detected from an image captured under a second shooting condition whose difference from the first shooting condition is less than a specified difference, it is possible to select and utilize a learning completion mark object detected from an image captured under a shooting condition close to (similar to) the first shooting condition when parking assistance is performed for the vehicle 1 to the target parking position.

[0061] As a result, it is possible to suppress a decrease in the marker object positions (circle icons) of the learning-completed marker objects corresponding to the marker object positions (triangle icons) detected when performing parking assistance, thereby suppressing a decrease in the accuracy of parking assistance.

[0062] In addition, the parking assist data 20 may be stored in the storage device 19 for a plurality of different target parking positions.

[0063] Figure 5 2 is an explanatory diagram of a second example of the parking assist data 20 . Figure 5 The parking assistance data 20 includes j sets of parking assistance data 20a1 to 20aj, k sets of parking assistance data 20b1 to 20bk, m sets of parking assistance data 20c1 to 20cm, and n sets of parking assistance data 20d1 to 20dn for different target parking positions, respectively.

[0064] The parking assistance data 20a1 to 20aj are a combination of a plurality of different second shooting conditions and the marker object positions of the learning completion marker objects detected from the images shot around the target parking position at the location A under these second shooting conditions. The parking assistance data 20b1 to 20bk are a combination of a plurality of different second shooting conditions and the marker object positions of the learning completion marker objects detected from the images shot around the target parking position at the location B under these second shooting conditions. The parking assistance data 20c1 to 20cm are a combination of a plurality of different second shooting conditions and the marker object positions of the learning completion marker objects detected from the images shot around the target parking position at the location C under these second shooting conditions. The parking assistance data 20d1 to 20dn are a combination of a plurality of different second shooting conditions and the marker object positions of the learning completion marker objects detected from the images shot around the target parking position at the location D under these second shooting conditions.

[0065] In the parking assist mode, the parking assist device 10 selects and reads out the stored parking assist data 20 regarding the target parking position where the host vehicle 1 is intended to park among the plurality of spots A to D.

[0066] For example, the parking assist device 10 can read out the parking assist data 20 corresponding to the current position of the host vehicle 1 measured by the positioning device 11. For example, the parking assist data stored about the target parking position closest to the current position of the host vehicle 1 can be read out. In addition, attribute data of the parking assist data 20 (for example, "parking lot at home" or "parking lot at the workplace") can be stored in the storage device 19, and the occupant (for example, the driver) can select the parking assist data based on the attribute data.

[0067] Instead of storing the parking assist data 20 in the storage device 19 , the parking assist data 20 may be stored in an external server device and the parking assist data 20 may be transmitted and received via the communication device 14 .

[0068] The functional structure of the controller 17 is described in more detail below. Figure 6 The controller 17 functions as an image conversion unit 40, a self-position calculation unit 41, a shooting condition detection unit 42, a feature point detection unit 43, an auxiliary data generation unit 44, an auxiliary data selection unit 45, a relative position estimation unit 46, a target trajectory generation unit 47, a steering control unit 48, a vehicle speed control unit 49, and an auxiliary image generation unit 50.

[0069] The image conversion unit 40 converts the image captured by the camera of the object sensor 12 into Figure 2 A bird's-eye view image (surround view monitor image) as shown in 2A and 2B is viewed from a virtual viewpoint directly above the vehicle 1. Hereinafter, the bird's-eye view image converted by the image conversion unit 40 may be referred to as a "surrounding image".

[0070] The own position calculation unit 41 calculates the current position of the own vehicle 1 on the fixed coordinate system by a dead reckoning method or the like based on the vehicle information output from the vehicle sensor 13. The own position calculation unit 41 can correct the calculated current position by mapping the position of the landmark object detected by the object sensor 12 with the known landmark object position and high-precision map information.

[0071] The feature point detection unit 43 detects feature points of landmark objects around the vehicle 1 based on the surrounding image output from the image conversion unit 40, and calculates feature values ​​of the feature points. The feature point detection unit 43 outputs the detected feature points and their feature values ​​together with the current position of the vehicle 1 received from the own position calculation unit 41 to the auxiliary data generation unit 44 and the relative position estimation unit 46.

[0072] The shooting condition detection unit 42 detects the shooting condition of the camera of the object sensor 12 at the shooting time of the image in which the feature point is detected by the feature point detection unit 43. The shooting condition detection unit 42 may include a clock for detecting the time, time and date of the shooting condition. The shooting condition detection unit 42 may also obtain weather information at the current position of the vehicle 1 as the shooting condition. The shooting condition detection unit 42 may obtain weather information from an external device via the communication device 14.

[0073] In addition, the shooting condition detection unit 42 can detect the brightness (illuminance) of the current position of the vehicle 1 as the shooting condition through the camera of the object sensor 12 or other sensors. In addition, the shooting condition detection unit 42 can also calculate the position of the sun at the shooting position based on the date and time of the shooting time, and use it as the shooting condition.

[0074] The shooting condition detection unit 42 outputs information on the detected shooting condition to the auxiliary data selection unit 45 and the auxiliary data generation unit 44. The shooting condition output to the auxiliary data selection unit 45 is used as the first shooting condition. The shooting condition output to the auxiliary data generation unit 44 is used as the second shooting condition.

[0075] The auxiliary data generating unit 44 calculates the position of the feature point on the fixed coordinate system based on the feature point output from the feature point detecting unit 43 and the current position of the host vehicle 1. The auxiliary data generating unit 44 combines the image feature information including the calculated position and feature amount information of the feature point with the second shooting condition output from the shooting condition detecting unit 42, and stores it in the storage device 19 as the parking assist data 20.

[0076] When the current position of the vehicle 1 reaches the target parking position 2 , the auxiliary data generating unit 44 receives the current position of the vehicle 1 on the fixed coordinate system from the positioning device 11 or the own position calculating unit 41 as the target parking position 2 , and stores it in the storage device 19 as parking auxiliary data 20 .

[0077] In addition to generating the parking assist data 20 based on the feature points detected while the host vehicle 1 is manually parked at the target parking position 2 in the landmark object learning mode, the assist data generating unit 44 also generates the parking assist data 20 based on the feature points detected while the host vehicle 1 is subsequently parked at the target parking position 2 in the parking assist mode. In this way, the parking assist data 20 can be easily generated under different shooting conditions.

[0078] Next, when parking assistance is performed for the host vehicle 1 to the target parking position in the parking assistance mode, the assistance data selection unit 45 obtains the current position of the host vehicle 1 from the own position calculation unit 41. Then, the data stored about the plurality of locations (e.g. Figure 5 The parking assistance data 20 corresponding to the current position among the parking assistance data 20 of the target parking position of the location A to the location d) is read. Here, the case where the parking assistance data 20a1 to 20aj of the location A are read out will be described.

[0079] The assist data selection unit 45 compares the second imaging condition included in the parking assist data 20a1 to 20aj of the point A with the first imaging condition output from the imaging condition detection unit 42, and obtains a difference between the first imaging condition and the second imaging condition.

[0080] For example, the auxiliary data selection unit 45 may calculate the difference between the variables (e.g., shooting time, brightness, and sun position) representing the first shooting condition and the second shooting condition as the difference between the first shooting condition and the second shooting condition. In addition, when using weather (e.g., sunny, cloudy, rainy, and snowy) as the variables representing the first shooting condition and the second shooting condition, corresponding values ​​corresponding to the respective weather conditions are assigned (e.g., a value of "0" for sunny, a value of "0.4" for cloudy, a value of "1" for rainy, and a value of "1.2" for snowy), and the difference between the corresponding values ​​may be calculated as the difference between the first shooting condition and the second shooting condition.

[0081] Furthermore, corresponding values ​​corresponding to variables representing the first shooting condition and the second shooting condition may be derived using a function, a map, or a lookup table, respectively, and a difference between the corresponding values ​​may be calculated as a difference between the first shooting condition and the second shooting condition.

[0082] For example, a corresponding value corresponding to the shooting date and time of the first shooting condition and a corresponding value corresponding to the shooting date and time of the second shooting condition may be derived respectively, and the difference between the corresponding values ​​may be calculated as the difference between the first shooting condition and the second shooting condition. As the corresponding value corresponding to the shooting date and time, for example, binary value information distinguishing whether the shooting time is in the daytime time period or the nighttime time period for each shooting day may be derived, or an estimated value of brightness in the shooting date and time may be derived.

[0083] It is also possible to use a function, mapping or lookup table to derive corresponding values ​​of the shooting date and time and weather combination, and calculate the difference between the corresponding values ​​as the difference between the first shooting condition and the second shooting condition. In addition, for example, it is also possible to separately obtain the corresponding value corresponding to the shooting date and time and the corresponding value corresponding to the weather, and calculate the difference between these weighted sums as the difference between the first shooting condition and the second shooting condition.

[0084] The auxiliary data selection unit 45 selects, from the parking auxiliary data 20a1 to 20aj of the point A, image feature information of feature points of the learned marker object combined with the second imaging condition having a difference from the first imaging condition of not more than a predetermined difference.

[0085] The difference between the first shooting condition and the second shooting condition being equal to or less than a predetermined difference may be, for example, that the difference between the first shooting condition and the second shooting condition is equal to or less than a predetermined value, or that the difference between the first shooting condition and the second shooting condition is the smallest.

[0086] Furthermore, when the differences between the first imaging condition and the second imaging condition are both greater than a predetermined value, the auxiliary data selection unit 45 may select the image feature information combined with the second imaging condition having the smallest difference from the first imaging condition.

[0087] Furthermore, when there are a plurality of parking assistance data having a difference between the first shooting condition and the second shooting condition that is less than a predetermined difference among the parking assistance data 20a1 to 20aj at the point A, the assistance data selection unit 45 may select the parking assistance data including the image feature information of the most feature points from among the parking assistance data.

[0088] The auxiliary data selection unit 45 outputs the selected image feature information and the information of the target parking position 2 corresponding to the current position of the host vehicle 1 to the relative position estimation unit 46 .

[0089] The relative position estimation unit 46 calculates the marker object position ( Figure 3 The circle icon ) and the landmark object position ( Figure 3 The triangular icon in the figure is matched and the feature points detected from the same landmark object are made to correspond to each other.

[0090] The relative position estimation unit 46 estimates the relative position of the target parking position 2 relative to the current position of the vehicle 1 based on the relative position relationship between the marker object position (triangle icon) detected in the parking assistance mode and the vehicle 1, and the relative position relationship between the feature point information (circle icon) of the learned marker object corresponding to these feature points (triangle icon) and the target parking position 2.

[0091] For example, the feature points detected in the parking assist mode are recorded as (x i ,y i ), will be combined with the feature point (x i ,y i ) respectively correspond to the feature points of the learning completed landmark objects and are recorded as (x mi ,y mi)(i=1 to N). For example, the relative position estimation unit 45 may calculate the affine transformation matrix M by the following equation based on the least squares method: affine .

[0092] (Equation 1)

[0093]

[0094] in,

[0095]

[0096] You can also use the weighted least squares method to calculate the column vector (a1, a2, a3, a4) as follows: T .

[0097] (Formula 2)

[0098] W=diag(w1…wy)

[0099]

[0100] The relative position estimating unit 46 calculates the position (target x) of the target parking position 2 on the fixed coordinate system included in the parking assistance data 20 according to the following equation: m , targety m ) is transformed into the position (targetx, targety) of the vehicle coordinate system.

[0101] (Equation 3)

[0102]

[0103] The target trajectory generation unit 47 generates a target travel trajectory from the current position of the host vehicle 1 on the vehicle coordinate system (ie, the coordinate origin) to the position (targetx, targety) of the target parking position 2 on the vehicle coordinate system.

[0104] The target driving trajectory from the current position of the host vehicle 1 to the target parking position 2 can be calculated by a known method that has been used in a generally known automatic parking device. As an example, the target driving trajectory can be calculated by connecting the current position of the host vehicle 1 to the target parking position 2 with a clothoid curve. In the case where the target driving trajectory includes a turning point, the target parking position 2 can be calculated by connecting the current position of the host vehicle through the turning point with a clothoid curve.

[0105] In addition, the target trajectory generation unit 47 calculates a target vehicle speed curve for setting the moving speed at each position on the target driving trajectory from the current position of the host vehicle to the target parking position 2. For example, the target vehicle speed curve may be calculated based on a predetermined set speed, in which the vehicle 1 accelerates from the current position to the set speed and then stops at the target parking position 2. In the case where the target driving trajectory includes a turning point, a vehicle speed curve may be calculated in which the vehicle decelerates before the turning point and stops at the turning point, accelerates from the turning point to the set speed, decelerates before the target parking position 2, and stops at the target parking position 2. The set speed when calculating the speed curve may be set to a lower speed as the curvature increases, according to the calculated curvature of the target driving trajectory.

[0106] The steering control unit 48 controls the steering actuator of the actuator 16 so that the host vehicle 1 travels along the target travel trajectory.

[0107] The vehicle speed control unit 49 controls the accelerator actuator and the brake actuator of the actuator 16 so as to change the vehicle speed of the vehicle 1 according to the moving speed plan calculated by the target trajectory generation unit 47. Thus, the vehicle 1 is controlled to travel along the target travel trajectory.

[0108] The auxiliary image generation unit 50 generates a parking assistance image showing the target driving trajectory calculated by the target trajectory generation unit 47 and the current position of the host vehicle 1. For example, the parking assistance image may be an image in which the target driving trajectory and the current position of the host vehicle 1 are superimposed on a bird's-eye view of the surroundings of the host vehicle 1 from above and a bird's-eye view. The auxiliary image generation unit 50 displays the parking assistance image on the display device of the human-machine interface 15.

[0109] (action)

[0110] Figure 7 This is a flowchart of an example of the operation of the parking assist device 10 in the parking assist mode.

[0111] In step S1 , the positioning device 11 measures the current position of the host vehicle 1 on the fixed coordinate system.

[0112] In step S2 , the feature point detector 43 of the controller 17 detects the landmark object positions (feature points) of landmark objects around the own vehicle 1 .

[0113] In step S3 , the imaging situation detection unit 42 detects the imaging situation around the host vehicle 1 captured by the camera of the object sensor 12 as the first imaging situation.

[0114] In step S4 , the assist data selection unit 45 reads the parking assist data 20 corresponding to the current position of the host vehicle 1 from the storage device 19 .

[0115] In step S5 , the auxiliary data selection unit 45 selects the parking auxiliary data 20 for which the difference between the first imaging situation and the second imaging situation detected in step S3 is less than a predetermined difference from the parking auxiliary data 20 read out in step S4 .

[0116] In step S6 , the auxiliary data selection unit 45 selects data including the most characteristic points in the parking auxiliary data 20 selected in step S5 .

[0117] In step S7 , the relative position estimating unit 46 estimates the relative position of the target parking position 2 with respect to the host vehicle 1 based on the position of the landmark object detected in step S2 and the parking assistance data 20 selected in step S6 .

[0118] In step S8 , the target trajectory generation unit 47 generates a target travel trajectory and a target vehicle speed curve for causing the host vehicle 1 to travel from the current position of the host vehicle 1 to the target parking position 2 based on the relative position of the target parking position 2 with respect to the host vehicle 1 .

[0119] In step S9, the steering control unit 48 controls the steering angle so that the host vehicle 1 travels along the target travel trajectory. The vehicle speed control unit 49 controls the moving speed of the host vehicle 1 according to the target vehicle speed curve. When the host vehicle 1 reaches the target parking position 2, the process ends.

[0120] (Effects of Embodiments)

[0121] (1) The controller 17 detects the position of a marker object around the vehicle 1, i.e., a first marker object position, from an image obtained by photographing the surroundings of the vehicle 1, and detects the photographing conditions when photographing the image, i.e., the first photographing conditions. With respect to the marker object position of a marker object detected from an image obtained by photographing the surroundings of the target parking position in the past, i.e., a past image, from a specified storage device 19 storing a combination of a relative position relationship between one or more second marker object positions and the target parking position, and a photographing condition when photographing the past image, i.e., the second photographing condition, the controller 17 reads out a combination of one or more relative position relationships and the second photographing conditions, and selects a relative position relationship stored in a combination of the second photographing conditions whose difference from the first photographing condition is less than a specified difference from the relative position relationship read out from the read relative position relationships. Based on the selected relative position relationship and the marker object position of the marker object stored around the vehicle 1, the controller 17 calculates the relative position between the current position of the vehicle 1 and the target parking position, and implements parking assistance for the vehicle 1 to the target parking position.

[0122] Thus, the marker object is detected from the image of the surroundings of the target parking position captured in advance, and the relative positional relationship between the target parking position and the marker object position is stored in advance. At the subsequent parking assistance time, the marker object position is detected from the image captured around the host vehicle 1, and the detected marker object position is made to correspond to the pre-stored marker object position. When calculating the relative position between the current position of the host vehicle 1 and the target parking position, the marker object position detected from the image captured in the second shooting condition close to (similar to) the shooting condition at the parking assistance time, that is, the first shooting condition, can be calculated to correspond to the marker object position detected at the parking assistance time. As a result, it is possible to suppress the reduction of the marker object position that can be matched due to the change of the shooting environment, and thus it is possible to suppress the reduction of the parking assistance accuracy.

[0123] (2) The controller 17 may select, from among the relative positional relationships stored in the storage device, a relative positional relationship stored in combination with the second imaging condition that has the smallest difference from the first imaging condition as the selected relative positional relationship.

[0124] Thus, the relative position between the current position of the vehicle 1 and the target parking position can be calculated by correspondingly linking the position of the marker object detected from the image captured in the second capturing condition closest to the first capturing condition (most similar) with the position of the marker object detected during parking assistance.

[0125] (3) The first shooting condition and the second shooting condition may include at least one of the brightness or weather of the shooting location, the shooting time, the shooting date and time, or the position of the sun at the time of shooting.

[0126] In this way, the difference between the first shooting condition and the second shooting condition can be obtained based on the factors affecting the detection of the marker object position.

[0127] (4) The controller 17 may also calculate a target driving trajectory from the current position of the vehicle 1 to the target parking position based on the relative position relationship between the current position of the vehicle 1 and the target parking position, and control the vehicle 1 so that the vehicle 1 travels from the current position of the vehicle 1 to the target parking position along the target driving trajectory.

[0128] Thereby, parking assistance can be realized in which the host vehicle 1 is controlled so that the host vehicle 1 travels to the target parking position 2 .

[0129] (5) The controller 17 may also calculate a target driving trajectory from the current position of the vehicle 1 to the target parking position based on the relative position relationship between the current position of the vehicle 1 and the target parking position, and display the target driving trajectory and the position of the vehicle 1 on a display device that can be visually recognized by the occupants.

[0130] This allows the occupant to visually recognize the target travel trajectory for driving the host vehicle 1 to the target parking position 2 .

[0131] All examples and conditional terms described herein are intended for the purpose of teaching in order to help readers understand the present invention and the concepts provided by the inventors for technological advancement, and should be interpreted as not being limited to the above examples and conditions specifically described, as well as the structure of the examples of this specification related to the superiority and deterioration of the present invention. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be applied thereto without departing from the spirit and scope of the present invention.

[0132] Explanation of symbols

[0133] 1: own vehicle, 10: parking assistance device, 11: positioning device, 12: object sensor, 13: vehicle sensor, 14: communication device, 15: human-machine interface, 16: actuator, 17: controller, 18: processor, 19: storage device, 40: image conversion unit, 41: own position calculation unit, 42: shooting status detection unit, 43: feature point detection unit, 44: auxiliary data generation unit, 45: auxiliary data selection unit, 46: relative position estimation unit, 47: target trajectory generation unit, 48: steering control unit, 49: vehicle speed control unit, 50: auxiliary image generation unit.

Claims

1. A parking assistance method, characterized in that: The position of a landmark object around the vehicle, that is, the first landmark object position, is detected from an image obtained by photographing the surroundings of the vehicle. detecting a shooting condition when the image is shot, namely a first shooting condition, Regarding the marker object position, i.e., the second marker object position, of the marker object detected from the image obtained from the past photographing the surroundings of the target parking position, i.e., the past image, one or more sets of relative positional relationships between the second marker object position and the target parking position and the shooting conditions when the past image was shot, i.e., the second shooting conditions, are read out from a prescribed storage device storing a combination of the relative positional relationships between the second marker object position and the target parking position and the shooting conditions when the past image was shot, i.e., the second shooting conditions, The relative position relationship read out is selected to be stored in combination with the second shooting condition whose difference from the first shooting condition is less than a predetermined difference, Based on the selected relative position relationship and the first marker object position, the relative position between the current position of the host vehicle and the target parking position is calculated, and parking assistance for the host vehicle to the target parking position is implemented.

2. The parking assistance method according to claim 1, characterized in that: Among the read relative positional relationships, the relative positional relationship that is stored in combination with the second shooting condition that has the smallest difference from the first shooting condition is selected as the selected relative positional relationship.

3. The parking assistance method according to claim 1 or 2, characterized in that: The first shooting condition and the second shooting condition include at least one of brightness or weather at a shooting location, shooting time, shooting date and time, or the position of the sun at the time of shooting.

4. The parking assistance method according to any one of claims 1 to 3, characterized in that: Based on the relative position relationship between the current position of the host vehicle and the target parking position, a target driving trajectory from the current position of the host vehicle to the target parking position is calculated, The host vehicle is controlled so that the host vehicle travels from a current position of the host vehicle along the target travel trajectory to the target parking position.

5. The parking assistance method according to any one of claims 1 to 4, characterized in that: Based on the relative position relationship between the current position of the host vehicle and the target parking position, a target driving trajectory from the current position of the host vehicle to the target parking position is calculated, The target driving trajectory and the position of the host vehicle are displayed on a display device that can be visually recognized by an occupant.

6. A parking assist device, characterized in that: have: A camera device that captures the surroundings of the vehicle; A controller that performs the following control: detecting the position of a marker object around the host vehicle, i.e., a first marker object position, from an image generated by the camera device; detecting a shooting condition when shooting the image, i.e., a first shooting condition; regarding the marker object position of a marker object detected from an image obtained from a past shooting of the surroundings of a target parking position, i.e., a past image, i.e., a second marker object position; reading out one or more sets of relative position relationships between the second marker object position and the target parking position, and a combination of shooting conditions when shooting the past image, i.e., a second shooting condition, from a specified storage device storing a combination of the relative position relationship between the second marker object position and the target parking position, and the shooting condition when shooting the past image, i.e., a second shooting condition; selecting, from the read relative position relationships, the relative position relationship stored in the combination of the second shooting conditions whose difference from the first shooting condition is less than a specified difference; calculating the relative position between the current position of the host vehicle and the target parking position based on the selected relative position relationship and the first marker object position, and implementing parking assistance for the host vehicle to the target parking position.

Images