Parking assistance device, parking assistance method, and parking assistance program

Abstract

The invention relates to a parking assist device, a parking assist method, and a parking assist program. A parking assist device (10) is provided with: an odometer information acquisition unit (30) that acquires odometer information indicating a position change amount and a posture change amount of a vehicle between a parking space and a first position; a road surface region specifying unit (32) that specifies, on the basis of the odometer information, a road surface region of a parking space in a first image captured from the first position; and a position / orientation estimation unit (34) that calculates relative position / orientation information indicating the amount of change in position and the amount of change in orientation of the vehicle between the first position and the second position by comparing a road surface region in a second image captured from the second position with a road surface region in the first image. On the basis of the odometer information and the relative position / orientation information, position / orientation estimation information indicating the position and orientation of the vehicle with respect to the parking space when the vehicle is located at the second position is output.

Description

TECHNICAL FIELD

[0001] The present application relates to a parking assistance device, a parking assistance method, and a parking assistance program. BACKGROUND

[0002] As a parking assistance device that performs assistance when parking a vehicle, for example, the following technology is known. That is, a parking assistance device described in Japanese Patent Application Publication No. 2021-062718 has a camera installed on a vehicle in a manner to capture the surroundings of the vehicle, and a control mechanism that acquires information about a parking area based on an image of the parking area captured by the camera and registers the information as parking area information, and that causes the vehicle to automatically park in the parking area using the parking area information.

[0003] The control mechanism is configured to acquire and register the parking area information when the vehicle stops near an entrance of the parking area, and to acquire and register the parking area information when parking of the vehicle into the parking area ends. In addition, the control mechanism is configured to register information about a feature point in the image of the parking area captured by the camera as the parking area information. According to this parking assistance device, when the vehicle is caused to automatically travel in order to park the vehicle in the parking area, the positional relationship of the vehicle with respect to the parking area can be accurately grasped.

[0004] In the above-described parking assistance device, in an environment where the lighting condition is good, by detecting a feature point (for example, a corner of an object or a pattern) from an image and correlating the feature points between images using a brightness pattern around the feature points, it is possible to infer the position and posture of the vehicle with respect to the parking space. However, in an environment where the lighting condition is poor, such as an outdoor parking area at night, there are many false correspondences of feature points, so it is difficult to accurately infer the position and posture of the vehicle.

[0005] Here, if a region where a plane appears in an image can be determined, it is possible to infer the position and posture of the vehicle. In addition, as a method of determining a region where a plane appears, there are methods such as semantic segmentation, but in the above-described methods, the amount of calculation is large. SUMMARY

[0006] The present application has been achieved in view of the above circumstances, and aims to provide a parking assistance device, a parking assistance method, and a parking assistance program that can infer the position and posture of a vehicle with respect to a parking space with less amount of calculation than methods such as semantic segmentation in an environment where the lighting condition is poor.

[0007] A first embodiment related to the technology of the present application is a parking assistance device including: an odometer information acquisition unit that acquires odometer information indicating a positional change amount and a posture change amount of a vehicle between a parking space and a first position other than the parking space; a road surface region determination unit that determines a road surface region of the parking space within a first image captured by a vehicle-mounted camera mounted on the vehicle from the first position based on the odometer information; and a position-posture estimation unit that calculates relative position-posture information indicating a positional change amount and a posture change amount of the vehicle between the first position and a second position different from the first position by comparing the road surface region within a second image captured by the vehicle-mounted camera from the second position with the road surface region within the first image, and outputs position-posture estimation information indicating a position and a posture of the vehicle with respect to the parking space when the vehicle is positioned at the second position based on the odometer information and the relative position-posture information.

[0008] A second embodiment related to the technology of the present application is a parking assistance device in which, in the parking assistance device of the first embodiment, the position-posture estimation unit calculates the relative position-posture information by calculating a homography matrix between the first image and the second image based on luminance values of pixels included in the road surface region within the first image and luminance values of pixels included in the road surface region within the second image, and decomposing the homography matrix.

[0009] A third embodiment related to the technology of the present application is a parking assistance device in which, in the parking assistance device of the first embodiment or the second embodiment, a parking control unit performs control to cause the vehicle to park from the second position to the parking space based on the position-posture estimation information.

[0010] A fourth embodiment related to the technology of the present application is a parking assistance method including: an odometer information acquisition step of acquiring odometer information indicating a position change amount and a posture change amount of a vehicle between a parking space and a first position other than the parking space; a road surface region determination step of determining a road surface region of the parking space within a first image captured by a vehicle-mounted camera mounted on the vehicle from the first position, based on the odometer information; and a position / posture estimation step of calculating relative position / posture information indicating a position change amount and a posture change amount of the vehicle between the first position and a second position different from the first position other than the parking space, by comparing the road surface region within a second image captured by the vehicle-mounted camera from the second position, with the road surface region within the first image, and outputting position / posture estimation information of a position and a posture of the vehicle with respect to the parking space when the vehicle is at the second position, based on the odometer information and the relative position / posture information.

[0011] A fifth embodiment related to the technology of the present application is a parking assistance program for causing a computer to execute: an odometer information acquisition step of acquiring odometer information indicating a position change amount and a posture change amount of a vehicle between a parking space and a first position other than the parking space; a road surface region determination step of determining a road surface region of the parking space within a first image captured by a vehicle-mounted camera mounted on the vehicle from the first position, based on the odometer information; and a position / posture estimation step of calculating relative position / posture information indicating a position change amount and a posture change amount of the vehicle between the first position and a second position different from the first position other than the parking space, by comparing the road surface region within a second image captured by the vehicle-mounted camera from the second position, with the road surface region within the first image, and outputting position / posture estimation information of a position and a posture of the vehicle with respect to the parking space when the vehicle is at the second position, based on the odometer information and the relative position / posture information.

[0012] According to the technology of the present application, in an environment where the lighting condition is poor, it is possible to estimate a position and a posture of a vehicle with respect to a parking space with less computational load than a method such as semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram showing one example of a configuration of a parking assistance system of an embodiment.

[0014] Figure 2 is a block diagram showing one example of a functional configuration of a parking assistance device of an embodiment.

[0015] Figure 3is a plan view showing an example of a positional relationship of a parking space and a first position outside the parking space.

[0016] Figure 4 is a view showing an example of a first image captured from the first position by the on-vehicle camera.

[0017] Figure 5 is a plan view showing a positional relationship of a camera coordinate system of each on-vehicle camera and a vehicle coordinate system.

[0018] Figure 6 is a view showing an example of a positional relationship of an image coordinate system within the first image and each end point of a road surface region.

[0019] Figure 7 is a plan view showing an example of a positional relationship of a parking space, a first position, and a second position.

[0020] Figure 8 is a view showing an example of a second image captured from the second position by the on-vehicle camera.

[0021] Figure 9 is a flowchart showing an example of a flow of a parking assistance process of the embodiment.

[0022] Figure 10 is a flowchart showing an example of a flow of a homography matrix optimum value calculation process of the embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, one example of a mode for implementing the technology of the present application will be described in detail with reference to the drawings. Further, the same reference numerals are assigned to the same action structure members and processing having the same action, effect, and function in all the drawings, and redundant description will be appropriately omitted. Each drawing is shown only schematically to the extent that the technology of the present application can be sufficiently understood. Therefore, the technology of the present application is not limited to the illustrated examples. In addition, in the present embodiment, the description of a structure not directly related to the technology of the present application or a well-known structure will be omitted at times.

[0024] Figure 1 is a block diagram showing an example of a structure of a parking assistance system 100 of the present embodiment. The parking assistance system 100 of the present embodiment is mounted on a vehicle 40. The vehicle 40 can be, for example, a vehicle such as a passenger car. The parking assistance system 100 is a system that performs assistance when the vehicle 40 is parked, and includes a parking assistance device 10, a wheel speed sensor 20, a steering angle sensor 21, and an on-vehicle camera 22.

[0025] The in-vehicle camera 22 is provided in the vehicle 40 and captures the surroundings of the vehicle 40. The in-vehicle camera 22 is provided in the vehicle 40 in a state in which the in-vehicle camera 22 can capture the road surface, and the position of the in-vehicle camera 22 in the vehicle 40 is not particularly limited. The in-vehicle camera 22 is, for example, a monocular camera, but is not limited thereto and can be a stereo camera or the like.

[0026] The in-vehicle camera 22 is, for example, configured so that the optical axis of the in-vehicle camera 22 is slightly directed downward from the horizontal direction. The in-vehicle camera 22 is communicably connected to the parking assistance device 10 and transmits the captured image to the parking assistance device 10. In addition, the vehicle 40 can be equipped with one in-vehicle camera 22 or a plurality of in-vehicle cameras 22. Hereinafter, an example in which the vehicle 40 is equipped with a plurality of in-vehicle cameras 22 (for example, four in-vehicle cameras 22 provided at the front, rear, left, and right of the vehicle body) will be described.

[0027] The wheel speed sensor 20 measures the wheel speed of the wheels provided in the vehicle 40. The wheel speed sensor 20 transmits data of the measured wheel speed to the parking assistance device 10. The wheel speed sensor 20 can also use an encoder provided with respect to the wheels. In addition, in the case where the vehicle 40 is a hybrid vehicle or the like provided with a drive motor, an encoder provided in the drive motor can also be used as the wheel speed sensor 20.

[0028] The steering angle sensor 21 measures the steering angle of the vehicle 40. The steering angle sensor 21 transmits data of the measured steering angle to the parking assistance device 10.

[0029] The parking assistance device 10 can be realized by a part of an ECU (Electronic Control Unit) that is a computer for vehicle control, or can be realized by an in-vehicle computer different from the ECU.

[0030] The parking assistance device 10 is provided with a CPU (Central Processing Unit) 11, a ROM (ReadOnly Memory) 12, a RAM (Random Access Memory) 13, an input / output interface (I / O) 14, a storage portion 15, and an external interface (external I / F) 16.

[0031] The CPU 11, the ROM 12, the RAM 13, and the I / O 14 are respectively connected via a bus. The respective functional portions including the storage portion 15 and the external I / F 16 are connected to the I / O 14. The respective functional portions described above can communicate with each other via the I / O 14.

[0032] The control section is configured by the CPU 11, the ROM 12, the RAM 13, and the I / O 14. The control section can be configured as a sub-control section that controls the operation of a part of the parking assistance device 10, or as a part of a main control section that controls the operation of the entire parking assistance device 10.

[0033] A part or all of each module of the control section can also be configured using an integrated circuit such as an LSI (Large Scale Integration) or an IC (Integrated Circuit) chip set. Alternatively, a separate circuit can be used for each module, or a circuit in which a part or all of the modules are integrated can be used. Alternatively, the modules can be provided integrally with each other, or a part of the modules can be provided separately. Alternatively, a part of each module can be provided separately. The integration of the control section is not limited to an LSI, and a dedicated circuit or a general-purpose processor can be used.

[0034] The storage section 15 is configured using, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The storage section 15 stores a parking assistance program 15A of the present embodiment. Alternatively, the parking assistance program 15A can be stored in the ROM 12.

[0035] The parking assistance program 15A can also be installed in the parking assistance device 10 in advance. Alternatively, the parking assistance program 15A can be distributed by being stored in a non-volatile storage medium or via a network, and installed in the parking assistance device 10 as appropriate. Examples of the non-volatile storage medium include a CD-ROM (Compact Disc-Read Only Memory), a magneto-optical disk, an HDD, a DVD-ROM (Digital Versatile Disc-Read Only Memory), a flash memory, a memory card, and the like.

[0036] The external I / F 16 is an interface for connecting to each of the wheel speed sensor 20, the steering angle sensor 21, and the in-vehicle camera 22 in a communicable manner.

[0037] The CPU 11 of the parking assistance device 10 of the present embodiment functions as a Figure 2 each of the components illustrated in FIG. 1 by writing the parking assistance program 15A stored in the storage section 15 in the RAM 13 and executing the program.

[0038] Figure 2This is a block diagram showing an example of the functional configuration of the parking assistance device 10 according to this embodiment. The CPU 11 of the parking assistance device 10 according to this embodiment functions as an odometer information acquisition unit 30 , a road surface area identification unit 32 , a position and posture estimation unit 34 , and a parking control unit 36 ​​.

[0039] Figure 3 This is a top view illustrating an example of the positional relationship between parking space P0 and first position P1. First position P1 is an arbitrary position outside of parking space P0. As an example, first position P1 is near the exit of parking space P0. First position P1 can be the position of vehicle 40 when entering parking space P0 or when exiting parking space P0. Furthermore, vehicle 40 can be driven manually by a driver or automatically by an autonomous driving system.

[0040] When the vehicle 40 moves between the parking space P0 and the first position P1 , the odometer information acquisition unit 30 acquires the measured values ​​from the wheel speed sensor 20 and the steering angle sensor 21 , and calculates the odometer information based on the acquired measured values ​​to acquire the odometer information.

[0041] The values ​​measured by wheel speed sensor 20 and steering angle sensor 21 are examples of driving history information of vehicle 40 obtained when vehicle 40 moves between parking space P0 and first position P1. Odometer information is information indicating the amount of change in position and posture of vehicle 40 between parking space P0 and first position P1.

[0042] In the parking lot coordinate system (X p -Y p When the origin of the coordinate system is set to the center of the rear wheel axle of the vehicle 40 parked in the parking space P0, the odometer information is obtained by the position of the rear wheel axle center (x ν0 、y ν0 ) and yaw angle θ ν0 Indicates. Among them, X p The y axis is a coordinate axis extending along the front-rear direction of the vehicle 40 parked in the parking space P0. p The x-axis is a coordinate axis extending along the width direction of the vehicle 40 parked in the parking space P0. Hereinafter, the odometer information may be referred to as odometer information (x ν0 、y ν0 ,θ ν0 ).

[0043] The following describes the odometer information (x ν0 、y ν0 ,θ ν0 ) is calculated. In addition, the rear wheel axle center is relative to the parking lot coordinate system (Xp Y p The position of the origin of the coordinate system is called the vehicle position (x ν , y ν ).

[0044] Here, T[s] denotes a sampling period, and ·(k) denotes various state quantities at the k-th sampling. Hereinafter, as an example, the case of departure from the warehouse will be described. In k = 0, the vehicle 40 is parked in the parking space P0.

[0045] From the wheel speed ω t (k) [rad / s] measured by the wheel speed sensor 20, using the radius R t [m] of the wheel (tire), the vehicle speed V(k) [m / s] is calculated by the following equation.

[0046] [Equation 1]

[0047] V(k) = R t ω t (k)

[0048] From the steering angle θ h (k) [rad] measured by the steering angle sensor 21, using the steering gear ratio G s , the front wheel steering angle θ f (k) [rad] is calculated by the following equation.

[0049] [Equation 2]

[0050] θ f (k) = θ h (k) / G s

[0051] From the vehicle speed V(k) and the front wheel steering angle θ f (k), using the wheel base L wb [m], the yaw rate ω ν (k) [rad / s] is calculated by the following equation.

[0052] [Equation 3]

[0053] ω v (k) = V(k) θ f (k) / L wb

[0054] From the yaw rate ω ν (k), the yaw angle θ ν (k) [rad] is calculated by the following equation. In addition, θ ν (0) = 0.

[0055] [Equation 4]

[0056] θ v (k) = θ v (k - 1) + Tω v (k)

[0057] The vehicle position (x ν (k), y ν (k)) is calculated from the vehicle speed V(k) and the yaw angle θ ν (k) using the following equations. In addition, x ν (0) = 0, y ν (0) = 0.

[0058] [Equation 5]

[0059] x v (k) = x v (k - 1) + TV(k)cosθ v (k)

[0060] y v (k) = y v (k - 1) + TV(k)sinθ v (k)

[0061] The odometry information (x ν (k), y ν (k), θ ν (k)) when the vehicle 40 is at the first position PI is represented by (x ν0 , y ν0 , θ ν0 ).

[0062] Figure 4 is a diagram illustrating one example of the first image 51 captured by the on-vehicle camera 22 from the first position PI. The road surface region determination section 32 acquires the first image 51 captured by the on-vehicle camera 22 from the first position PI, and determines the road surface region A of the parking space PO within the acquired first image 51 based on the odometry information (x ν0 , y ν0 , θ ν0 ).

[0063] Here, the road surface region A of the parking space PO in the actual space is regarded as a rectangle determined by the overall length and the overall width of the vehicle 40. The first image 51 can also be an image captured by any one of the plurality of on-vehicle cameras 22 (i.e., the four on-vehicle cameras 22 respectively arranged at the front, rear, left, and right of the vehicle body) mounted on the vehicle 40.

[0064] Figure 5 is a diagram illustrating the camera coordinate system (X c -Z ccoordinate system) and vehicle coordinate system (X ν -Y ν The top view of the positional relationship of the vehicle coordinate system (X ν -Y ν In the following description, it is assumed that the camera coordinate system (X c -Z c coordinate system) relative to the vehicle coordinate system (X v -Y v Therefore, the camera coordinate system (X c -Z c coordinate system) relative to the vehicle coordinate system (X v -Y v coordinate system), and the relative position and relative posture of the odometer information (x v0 、y v0 ,θ v0 ) to determine the road surface area A of the parking space P0 in the first image 51. In this way, by determining the road surface area A in the first image 51 based on the odometer information, computational resources can be reduced and costs can be lowered.

[0065] Next, the calculation formula for the road surface area A in the first image 51 will be described. Here, regarding the vehicle 40, the total length is L v [m], set the total width to W v [m], the distance from the rear axle center to the rear end of the vehicle body is set to L r [m]. Figure 3 The parking lot coordinate system (X p -Y p When the four endpoints of the road surface area A in the coordinate system are set as endpoints P1, P2, P3, and P4, the positions of the endpoints P1, P2, P3, and P4 are represented by the vector p 1_p 、p 2_p 、p 3_p 、p 4_p In addition, T represents transpose.

[0066] [Formula 6]

[0067] p 1_p =[-L p , -0.5W v ,0] T

[0068] p 2_p =[-L r , 0.5W v ,0] T

[0069] p 3_p = [L v -L r , 0.5W v , 0] T

[0070] p 4_p = [L v -L r , -0.5W v , 0] T

[0071] If the positions of the end points Pi (i = 1, 2, 3, 4) in the vehicle coordinate system (X v -Y v coordinate system) are set as vectors p i_v , then the vectors p i_v are calculated by the following equations.

[0072] [Equation 7]

[0073] p i_v = R v0 T (p i_p -t v0 )

[0074]

[0075] Figure 6 is a diagram showing one example of the positional relationship of the image coordinate system (u-v coordinate system) within the first image 51 and the end points P1, P2, P3, P4 of the road surface region A. The origin of the image coordinate system (u-v coordinate system) is set at one of the end points (as one example, the upper left end point) of the first image 51. The u coordinate axis is a coordinate axis extending in the lateral direction of the first image 51, and the v coordinate axis is a coordinate axis extending in the longitudinal direction of the first image 51. The positions of the end points P1, P2, P3, P4 of the road surface region A within the first image 51 are represented by (U p1 , v p1 ), (U p2 , v p2 ), (U p3 , v p3 ), and (U p4 , v p4 ).

[0076] According to the image coordinate system (u-v coordinate system) shown in Equation 2, the end points P i_ν = [x i_ν , y i_ν , 0] T , Figure 6 i ​(u pi , v pi ) are calculated using the internal parameter matrix K and the external parameter matrix T ext of the on-vehicle camera 22, and by the following equation. Here, f is the focal length of the on-vehicle camera 22, (c x , c y ) are the center coordinates of the first image 51, T 11 ~ T 34 are the external parameter matrix, and all are known.

[0077] [Equation 8]

[0078]

[0079] Figure 7 Fig. 8 is a plan view showing an example of the positional relationship of the parking space P0, the first position PI, and the second position P2. The second position P2 is, for example, a parking start position at the time of automatic parking of the vehicle 40 in the parking space P0. The second position P2 is a position different from the first position PI outside the parking space P0, and is an arbitrary position. As an example, the second position P2 is a position on the side opposite to the parking space P0 with respect to the first position PI. In other words, the first position PI corresponds to a midway position between the second position P2 and the parking space P0.

[0080] Figure 8 Fig. 9 is a view showing an example of the second image 52 captured by the on-vehicle camera 22 from the second position P2. The position and posture inference section 34 acquires the second image 52 captured by the on-vehicle camera 22 from the second position P2 in a case where the vehicle 40 is automatically parked from the second position P2 to the parking space P0. The second image 52 can also be an image captured by any one of the plurality of on-vehicle cameras 22 (i.e., four on-vehicle cameras 22 respectively provided on the front, rear, left, and right of the vehicle body) mounted on the vehicle 40. In addition, the second image 52 can also be an image captured by an on-vehicle camera 22 different from the on-vehicle camera 22 that captured the first image 51.

[0081] Next, the position and posture inference section 34 calculates the relative position and posture information indicating the amount of change in position and the amount of change in posture of the vehicle 40 between the first position PI and the second position P2 by comparing the road surface region A in the second image 52 and the road surface region A in the first image 51. The origin of the vehicle coordinate system (X ν -Y ν coordinates) when the vehicle 40 is positioned at the first position PI is taken as a reference, and the relative position and posture information is calculated by (x νd , y νd , θ νd) indicates relative position posture information. Hereinafter, the relative position posture information is sometimes referred to as relative position posture information (x νd , y νd , θ νd ).

[0082] Specifically, the position posture inferring section 34 calculates the relative position posture information (x νd , y νd , θ νd ) as follows. First, the position posture inferring section 34 calculates a homography matrix between the first image 51 and the second image 52 based on the luminance values of the pixels included in the road surface region A in the first image 51 and the luminance values of the pixels included in the road surface region A in the second image 52. For example, the homography matrix between the first image 51 and the second image 52 is found by searching for a region in which the luminance values of the pixels included in the road surface region A in the first image 51 and the luminance values of the pixels included in the road surface region A in the second image 52 are minimized. Further, the homography refers to a projection of a certain plane to another plane using a projective transformation.

[0083] Next, the position posture inferring section 34 calculates the relative position posture information (x νd , y νd , θ νd ) by decomposing the homography matrix. In this case, the position posture inferring section 34 finds the relative position and the relative posture of the on-vehicle camera 22 between the first position PI and the second position P2 by decomposing the homography matrix. Next, the position posture inferring section 34 finds the relative position posture information (x νd , y νd , θ νd ) by converting the relative position and the relative posture of the on-vehicle camera 22 between the first position PI and the second position P2 into the relative position and the relative posture of the vehicle 40 using the relative position and the relative posture of the camera coordinate system with respect to the vehicle coordinate system.

[0084] Then, the position posture inferring section 34 infers and outputs position posture inferring information (x ν1 , y ν1 , θ ν1 ) of the position and the posture of the vehicle 40 with respect to the parking stall PO when the vehicle 40 is at the second position P2 based on the odometry information (x ν0 , y ν0 , θ ν0 ) and the relative position posture information (x νd , y νd , θ νd ). In this way, the relative position posture information (x νd , y νd , θνd ) in the case where the brightness values of the pixels included in the road surface region A in the first image 51 and the second image 52 are used. Thus, even if the lighting condition of the parking stall PO is a poor environment, the position and the posture of the vehicle 40 with respect to the parking stall PO can be inferred.

[0085] The calculation of the position / posture inference information (x ν1 , y ν1 , θ ν1 ) will be described below. First, the initial value G0 of the homography matrix between the first image 51 and the second image 52 is set to a three-by-three unit matrix, and the optimal value G opt of the homography matrix is calculated. The calculation steps for obtaining the optimal value G opt of the homography matrix will be described later with reference to Figure 10 . Next, the optimal value G opt of the homography matrix is decomposed as shown in the following equation.

[0086] [Equation 9]

[0087]

[0088] Here, K is the internal parameter matrix of the on-vehicle camera 22 described above, R est is a rotation matrix (inferred value) indicating the amount of change in the posture of the on-vehicle camera 22 with respect to the camera coordinate system (X c -Z c coordinate system), t est is a translation vector (inferred value) indicating the amount of change in the position of the on-vehicle camera 22 with respect to the camera coordinate system, n est is a road surface normal vector (inferred value) with respect to the camera coordinate system, and h is the installation height of the on-vehicle camera 22 from the road surface (previously measured value).

[0089] Further, a method of decomposing the optimal value G opt of the homography matrix is disclosed in the following document.

[0090] E. Malis, et al., “Deeper understanding of the homography decomposition for vision-based control,” Research Report, RR-6303, INRIA, 2007.

[0091] Next, the position and the posture of the vehicle 40 with respect to the parking stall PO are calculated by the following equation.Figure 7 The vehicle coordinate system (X ν -Y ν The rotation matrix R of the posture change amount of the coordinate system) νd and represent the vehicle 40 relative to Figure 7 The vehicle coordinate system (X ν -Y ν The translation vector t of the position change of the coordinate system) νd .

[0092] [Formula 10]

[0093]

[0094] If (x νd 、y νd ,θ νd ) indicates that Figure 7 The vehicle coordinate system (X ν -Y ν The relative position and posture information of the second position P2 represented by the coordinate system) is the translation vector t νd The first row and first column of the component is equivalent to x νd , translation vector t νd The second row and first column of is equivalent to yνd. In addition, θ νd Calculate by the following formula. Here, R 12 It is R νd The first row and second column component, R 22 It is R νd The second row and second column component.

[0095] [Mathematical formula 11]

[0096]

[0097] Moreover, according to the odometer information (x ν0 、y ν0 ,θ ν0 ) and relative position and posture information (x νd 、y νd ,θ νd ), and calculated by the following formula Figure 7 The parking lot coordinate system (X p -Y p The position and posture estimation information (x ν1 、y ν1 ,θ ν1 ).

[0098] [Mathematical formula 12]

[0099]

[0100] θ v1 = θ v0 + θ vd

[0101] The parking control section 36 performs control to cause the vehicle 40 to park from the second position P2 to the parking space PO based on the position and posture estimation information (x ν1 , y ν1 , θ ν1 ). In this case, even in an environment in which the lighting condition is poor, since the position and posture of the vehicle 40 with respect to the parking space PO can be estimated, it is possible to automatically park from the second position P2 to the parking space PO.

[0102] Next, the operation of the parking assistance device 10 of the present embodiment will be described.

[0103] Figure 9 is a flowchart showing one example of a flow of the parking assistance processing by the parking assistance program 15A of the present embodiment.

[0104] First, if the parking assistance device 10 receives an instruction to start the parking assistance processing, the parking assistance program 15A is started by the CPU 11, and the following steps are executed.

[0105] In step S10, the CPU 11 acquires, from the storage section 15, the odometry information (x ν0 , y ν0 , θ ν0 ) calculated based on the measurement values measured by the wheel speed sensor 20 and the steering angle sensor 21 in the case where the vehicle 40 moves between the parking space PO and the first position PI. Step S10 is one example of an odometry information acquisition step related to the technology of the present application.

[0106] In step S12, the CPU 11 acquires the first image 51 captured by the on-vehicle camera 22 from the first position PI from the storage section 15, and determines the road surface region A of the parking space PO within the acquired first image 51 based on the odometry information (x ν0 , y ν0 , θ ν0 ). Step S12 is one example of a road surface region determination step related to the technology of the present application.

[0107] In step S14, the CPU 11 acquires the second image 52 captured by the on-vehicle camera 22 from the second position P2 in a case where the vehicle 40 is to be automatically parked from the second position P2 to the parking space PO. Next, the position and posture estimation section 34 calculates the relative position and posture information (x νd , y νd , θ νd ) representing the amount of change in position and the amount of change in posture of the vehicle 40 between the first position PI and the second position P2 by comparing the road surface region A within the second image 52 with the road surface region A within the first image 51.

[0108] Specifically, first, the position and posture estimation section 34 calculates a homography matrix between the first image 51 and the second image 52 based on the luminance values of the pixels included in the road surface region A within the first image 51 and the luminance values of the pixels included in the road surface region A within the second image 52. Next, the position and posture estimation section 34 calculates the relative position and posture information (x νd , y νd , θ νd ) by decomposing the homography matrix.

[0109] Then, the position and posture estimation section 34 estimates and outputs the position and posture estimation information (x ν1 , y ν1 , θ ν1 ) of the vehicle 40 relative to the parking space PO when the vehicle 40 is at the second position P2 based on the odometry information (x ν0 , y ν0 , θ ν0 ) and the relative position and posture information (x νd , y νd , θ νd ). Step S14 is one example of a position and posture estimation step related to the technology of the present application.

[0110] In step S16, the CPU 11 performs control to park the vehicle 40 from the second position P2 to the parking space PO based on the position and posture estimation information (x ν1 , y ν1 , θ ν1 ). Step S16 is one example of a parking control step related to the technology of the present application.

[0111] Further, the above-described method explained as the function of the parking assistance device 10 of the present embodiment is one example of a parking assistance method related to the technology of the present application.

[0112] Next, a calculation step of finding the optimal value G opt of the homography matrix will be explained.

[0113] Figure 10 is a flowchart showing one example of a flow of the homogeneous matrix optimum value calculation processing of the present embodiment. Further, here, the numbering i (i = 1, 2 ~ n) is assigned to all pixels (the number of pixels is n) within the road surface region A determined by the end points P1, P2, P3, P4 within the first image 51. Figure 6 is assigned to all pixels within the road surface region A determined by the end points P1, P2, P3, P4 within the second image 52. Although not particularly shown, the numbering i (i = 1, 2 ~ n) is assigned to all pixels within the road surface region A in the same manner as the first image 51.

[0114] In step S40, the CPU 11 specifies a tracking region (identical to the road surface region A within the first image 51) for the image I* (identical to the first image 51) and calculates the luminance gradient matrix J I＊ and the Jacobian matrix J W , J G .

[0115] Specifically, the luminance gradient matrix J I＊ is calculated from the luminance (a value of 0 ~ 255) of each pixel of the tracking region of the image I* by the following equation.

[0116] [Equation 13]

[0117]

[0118] where J I＊i (i = 1, 2 ~ n) is represented by the following equation. J I＊ui represents the luminance gradient in the horizontal direction of the i-th pixel, and J I＊vi represents the luminance gradient in the vertical direction of the i-th pixel.

[0119] [Equation 14]

[0120] J I*i = [J I*ui J I*vi 0] (i = 1 ~ n)

[0121] The Jacobian matrix J W is calculated from the coordinates of each pixel of the tracking region of the image I* by the following equation.

[0122] [Equation 15]

[0123]

[0124] where the coordinates of each pixel of the tracking region are represented by the following equation.

[0125] [Equation 16]

[0126] p i* = [u i * v i * 0] T

[0127] This time, J Wi is represented by the following formula.

[0128] [Equation 17]

[0129]

[0130] Jacobi matrix J G According to the basis A of the Lie algebra i (i = 1 ~ 8) and is calculated by the following formula.

[0131] [Equation 18]

[0132] J G = [[A1] V [A2] V … [A8] V ]

[0133] Here, [A i ] v is a nine-row one-column vector in which each row is rearranged as represented by the following formula.

[0134] [Equation 19]

[0135]

[0136] In step S42, the CPU 11 substitutes the initial value G0 (the unit matrix) into the estimated value G of the homomorphism matrix (G is immediately above G, and the same applies hereafter.) and substitutes 1 into the number of iterations (the number of repetitions) n ite .

[0137] In step S44, the CPU 11 calculates the luminance gradient matrix J I of the tracking region of the image I (identical to the second image 52).

[0138] Specifically, the CPU 11 calculates the coordinates of the image I by the following formula.

[0139] [Equation 20]

[0140]

[0141] where the coordinates pi of the image I are represented by the following formula.

[0142] [Equation 21]

[0143] p i=[u i v i 0] T (i=1~n)

[0144] Brightness gradient matrix J I The brightness of each pixel in the tracking area of ​​image I is calculated using the following formula.

[0145] [Mathematical formula 22]

[0146]

[0147] Among them, J Ii It is represented by the following formula. Iui represents the horizontal brightness gradient of the i-th pixel, J Ivi Represents the vertical brightness gradient of the i-th pixel.

[0148] [Mathematical formula 23]

[0149] J Ii =[J Iui J Ivi 0](i=1~n)

[0150] In step S46 , the CPU 11 calculates the parameter x (a vector of eight rows and one column) of the isomorphic matrix.

[0151] Specifically, the CPU 11 calculates the parameter x using the following formula.

[0152] [Mathematical formula 24]

[0153]

[0154] Here, J esm is the Jacob matrix and is calculated as follows.

[0155] [Mathematical formula 25]

[0156]

[0157] On the other hand, y is a brightness difference vector and is expressed by the following equation.

[0158] [Mathematical formula 26]

[0159] y=[y1 y2 … y n ] r

[0160] Here, y i According to the brightness I of the i-th pixel of image I i , and the brightness I of the i-th pixel of image I* i *And calculated by the following formula.

[0161] [Math. 27]

[0162]

[0163] In step S48, the CPU 11 updates the estimated value G of the homography matrix by the following equation.

[0164] [Math. 28]

[0165]

[0166] G = exp(A(x))

[0167] Further, the CPU 11 sets the above G as a new G.

[0168] In step S50, the CPU 11 determines whether or not the ending condition, i.e., whether or not iteration (repetition) is needed, is satisfied. In the case where it is determined that the ending condition, i.e., iteration (repetition) is not needed, is satisfied (in the case of affirmative determination), the processing moves to step S52, and in the case where it is determined that the ending condition, i.e., iteration (repetition) is needed, is not satisfied (in the case of negative determination), the processing returns to step S44, and the processing is repeated.

[0169] Specifically, if the root mean square of the luminance difference this time is set as y curr , then y curr is represented by the following equation.

[0170] [Math. 29]

[0171]

[0172] Here, the upper limit number of iterations is set as n max (100, for example), and the threshold value of the convergence determination is set as ε (10 -5 , for example).

[0173] In the case where n ite = 1, the root mean square of the luminance difference this time y curr is substituted for the root mean square of the luminance difference of the previous time y prev , the number of iterations n ite is incremented by 1, and the processing returns to step S44.

[0174] In the case where 1 < n ite < n max , if y prev - y curr > ε, it is determined that there is no convergence, the root mean square of the luminance difference this time y curr is substituted for the root mean square of the luminance difference of the previous time y prev , the number of iterations n prev-y curr ≤ε, it is determined that convergence has occurred and the process moves to step S52.

[0175] In n ite =n max In the case of , go to step S52.

[0176] In step S52, the CPU 11 determines the optimal value G OPT The inferred value G ̂ of the isomorphic matrix is ​​adopted, and the processing ends.

[0177] As described above in detail, according to this embodiment, the CPU 11 obtains the odometer information (x ν0 、y ν0 ,θ ν0 ), based on odometer information (x ν0 、y ν0 ,θ ν0 ) to determine the road surface area A of the parking space P0 in the first image 51 captured by the vehicle-mounted camera 22 from the first position P1. Therefore, it is possible to use a computationally intensive method such as semantic segmentation based on the odometer information (x ν0 、y ν0 ,θ ν0 ) is used to determine the road surface area A of the parking space P0 in the first image 51, so that computing resources can be reduced and costs can be reduced.

[0178] Furthermore, the CPU 11 compares the road surface area A in the second image 52 captured by the vehicle-mounted camera 22 from the second position P2 outside the parking space P0 with the road surface area A in the first image 51, thereby calculating relative position and posture information (x) indicating the amount of position change and posture change of the vehicle 40 between the first position P1 and the second position P2. νd 、y νd ,θ νd Here, as an example, the CPU 11 calculates the isomorphism matrix between the first image 51 and the second image 52 based on the brightness values ​​of the pixels contained in the road surface area A in the first image 51 and the brightness values ​​of the pixels contained in the road surface area A in the second image 52, and decomposes the isomorphism matrix to calculate the relative position and posture information (x νd 、y νd ,θ νd ).

[0179] Furthermore, the CPU 11 calculates the odometer information (x ν0 、y ν0 ,θ ν0 ) and relative position and posture information (x νd 、y νd ,θνd ), to infer position and posture inference information (x ν1 , y ν1 , θ ν1 ) representing the position and posture of the vehicle 40 with respect to the parking stall PO when the vehicle 40 is located at the second position P2. Thus, in a case where the relative position and posture information (x νd , y νd , θ νd ) is obtained, since the luminance values of the pixels included in the road surface region A within the first image 51 and the second image 52 are used, the position and posture of the vehicle 40 with respect to the parking stall PO can be inferred even in an environment where the lighting condition of the parking stall PO is poor.

[0180] Further, the CPU 11 performs control to cause the vehicle 40 to park from the second position P2 to the parking stall PO based on the position and posture inference information (x ν1 , y ν1 , θ ν1 ). Thus, even in an environment where the lighting condition is poor, the position and posture of the vehicle 40 with respect to the parking stall PO can be inferred, so the vehicle 40 can be automatically parked from the second position P2 to the parking stall PO.

[0181] Further, in the above-described embodiment, as one example, the CPU 11 calculates the relative position and posture information (x νd , y νd , θ νd ) by calculating the homography matrix between the first image 51 and the second image 52 based on the luminance values of the pixels included in the road surface region A within the first image 51 and the luminance values of the pixels included in the road surface region A within the second image 52, and decomposing the homography matrix. However, the CPU 11 can calculate the relative position and posture information (x νd , y νd , θ νd ) by comparing the road surface region A within the second image 52 with the road surface region A within the first image 51 using a method other than the calculation method of the homography matrix.

[0182] Further, in the above-described embodiment, the CPU 11 obtains the measurement values measured by the wheel speed sensor 20 and the steering angle sensor 21 in a case where the vehicle 40 moves between the parking stall PO and the first position PI, and calculates the odometry information (x ν0 , y ν0 , θ ν0 ) based on the obtained measurement values. However, the CPU 11 can obtain, for example, measurement values measured by a sensor provided at a position other than the vehicle 40, and calculate the odometry information (x ν0 , y ν0 , θν0 ). In addition, the CPU 11 can also acquire odometer information (x ν0 , y ν0 , θ ν0 ) input from the outside of the vehicle to the parking assistance device 10.

[0183] In addition, in the above-described embodiments, the processor refers to a processor in a broad sense, and can be a general-purpose processor such as a CPU, or a component including a dedicated processor such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a FPGA (Field Programmable Gate Array).

[0184] In addition, the operation of the processor of the above-described embodiments can be performed not only by one processor, but also by a plurality of processors existing in physically separated locations in cooperation. In addition, the order of each operation of the processor is not limited to the order described in the above-described embodiments, and can be changed as appropriate.

[0185] The above describes and illustrates the parking assistance device of the embodiments. The embodiments can also be in the form of a program for causing a computer to execute the functions of each part possessed by the parking assistance device. The embodiments can also be in the form of a non-transitory storage medium that is readable by a computer and in which the above-described program is stored.

[0186] Further, the structure of the parking assistance device described in the above-described embodiments is one example, and can be changed as appropriate without departing from the gist.

[0187] In addition, the flow of the processing of the program described in the above-described embodiments is one example. Therefore, in the above-described embodiments, unnecessary steps can be deleted, new steps can be added, or the order of processing can be changed, without departing from the gist.

[0188] In addition, in the above-described embodiments, although a case where the processing of the embodiments is implemented by a computer and by a software structure by executing a program is described, the embodiments are not limited thereto. The embodiments can also be implemented by a hardware structure, a combination of a hardware structure and a software structure, for example.

[0189] To the extent not specifically described herein, all literature, patent applications, and technical standards referred to herein are incorporated by reference in their entirety. In addition, the disclosure of Japanese Application No. 2023-045962, filed on March 22, 2023, is incorporated by reference herein in its entirety.

Claims

1. A parking assist device, wherein: have: an odometer information acquisition unit that acquires odometer information indicating a position change and a posture change between the vehicle in the parking space and a first position outside the parking space; a road surface area specifying unit for specifying, based on the odometer information, a road surface area of ​​the parking space within a first image captured from the first position by a vehicle-mounted camera mounted on the vehicle; as well as A position and posture inference unit calculates relative position and posture information indicating a position change and a posture change of the vehicle between the first position and the second position by comparing the road surface area within a second image captured by the vehicle-mounted camera from a second position different from the first position outside the parking space with the road surface area within the first image, and outputs position and posture inference information indicating the position and posture of the vehicle relative to the parking space when the vehicle is located at the second position based on the odometer information and the relative position and posture information.

2. The parking assist device according to claim 1, wherein: The position and posture inference unit calculates the relative position and posture information by calculating the isomorphic matrix between the first image and the second image based on the brightness values ​​of the pixels contained in the road surface area in the first image and the brightness values ​​of the pixels contained in the road surface area in the second image, and decomposing the isomorphic matrix.

3. The parking assist device according to claim 1 or 2, wherein: have: A parking control unit controls parking of the vehicle from the second position to the parking space based on the position and posture estimation information.

4. A parking assistance method, wherein: The following steps are required: an odometer information obtaining step of obtaining odometer information indicating a position change and a posture change between the vehicle in the parking space and a first position outside the parking space; a road surface area determining step of determining, based on the odometer information, a road surface area of ​​the parking space within a first image captured from the first position by a vehicle-mounted camera mounted on the vehicle; as well as The position and posture inference step calculates relative position and posture information representing the position change and posture change of the vehicle between the first position and the second position by comparing the road surface area in a second image captured by the vehicle-mounted camera from a second position different from the first position outside the parking space with the road surface area in the first image, and outputs position and posture inference information of the position and posture of the vehicle relative to the parking space when the vehicle is at the second position based on the odometer information and the relative position and posture information.

5. A parking assistance program, wherein: Causes the computer to perform the following steps: an odometer information obtaining step of obtaining odometer information indicating a position change and a posture change between the vehicle in the parking space and a first position outside the parking space; a road surface area determining step of determining a road surface area of ​​the parking space within a first image captured from the first position by a vehicle-mounted camera mounted on the vehicle based on the odometer information; as well as The position and posture inference step calculates relative position and posture information representing the position change and posture change of the vehicle between the first position and the second position by comparing the road surface area in a second image captured by the vehicle-mounted camera from a second position different from the first position outside the parking space with the road surface area in the first image, and outputs position and posture inference information of the position and posture of the vehicle relative to the parking space when the vehicle is at the second position based on the odometer information and the relative position and posture information.

Images