Trailer bottom end coordinate calculation device, trailer bottom end coordinate calculation method, and program

Abstract

To allow world coordinate system coordinates of the bottom left end and bottom right end of a trailer to be presented.SOLUTION: A trailer bottom end coordinate calculation device 15 removes coordinates that satisfy coordinate removal conditions among world coordinate system coordinates of the bottom left end and bottom right end of a trailer and a hitch ball contained in each of multiple images taken by a camera 11 mounted on a vehicle 1 at multiple points in time during the calibration travel of the vehicle 1 towing the trailer via a tow bar, and uses those that are not removed to calculate the average value of the world coordinate system distance between the bottom left or right end of the trailer and the hitch ball, the average value of an angle formed in the world coordinate system by the bottom left end of the trailer, the hitch ball, and the bottom right end of the trailer, and the average value of world coordinate system coordinates of the hitch ball, and based on these and the trailer hitch angle at a predetermined point in time after the vehicle's calibration travel, calculates the world coordinate system coordinates of the bottom left and right ends of the trailer at that time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a trailer bottom end coordinate calculation device, a trailer bottom end coordinate calculation method, and a program. [Background technology]

[0002] Patent Document 1 describes a technology for accurately setting the center position of a marker when the towing vehicle and towed vehicle are coupled and traveling straight, regardless of the marker attachment position on the towed vehicle. The technology described in Patent Document 1 performs offset correction when the marker is attached displaced (offset) from the vehicle longitudinal center axis of the coupled member that connects the towing vehicle and towed vehicle. As a result, even when the marker is attached offset, the hitch angle (bend angle) of the towed vehicle can be accurately determined based on an image of the marker. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-096390 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, a driver of a vehicle towing a trailer needs to drive the vehicle so that the trailer does not come into contact with obstacles, etc. Therefore, simply presenting the trailer hitch angle obtained by the technology described in Patent Document 1 does not result in appropriate driving assistance being provided for the vehicle towing the trailer. In order to provide driving assistance to prevent the trailer from coming into contact with obstacles, it is necessary to provide, for example, the coordinates of the bottom left and bottom right ends of the trailer in a world coordinate system. However, the technology described in Patent Document 1 cannot provide the coordinates of the bottom left and bottom right ends of the trailer in a world coordinate system to provide driving assistance for the vehicle towing the trailer.

[0005] In view of the above, the present disclosure aims to provide a trailer bottom end coordinate calculation device, a trailer bottom end coordinate calculation method, and a program that can present the coordinates of the world coordinate system of the bottom left and bottom right ends of a trailer, thereby enabling driving assistance for a vehicle towing a trailer. is. [Means for solving the problem]

[0006] (1) One aspect of the present disclosure includes an acquisition unit that acquires a plurality of images taken by a camera mounted on a vehicle towing a trailer via a tow bar at a plurality of time points during a calibration run of the vehicle; a first calculation unit that calculates the coordinates of a lower left end of the trailer, a lower right end of the trailer, and an image of a hitch ball included in each of the plurality of images acquired by the acquisition unit; a conversion unit that converts the coordinates of the image of the lower left end calculated by the first calculation unit into coordinates of a world coordinate system of the lower left end, converts the coordinates of the image of the lower right end calculated by the first calculation unit into coordinates of a world coordinate system of the lower right end, and converts the coordinates of the image of the hitch ball calculated by the first calculation unit into coordinates of a world coordinate system of the hitch ball; and a conversion unit that calculates a first average value that is an average value of the distance in the world coordinate system between the lower left end and the hitch ball during the calibration run of the vehicle based on the coordinates of the world coordinate system of the lower left end and the hitch ball at a plurality of time points during the calibration run of the vehicle, and converts the world coordinate system of the lower right end and the hitch ball at a plurality of time points during the calibration run of the vehicle. a second calculation unit that calculates a second average value, which is an average value of the distance in the world coordinate system between the lower right end portion and the hitch ball during the calibration run of the vehicle, based on coordinates in a world coordinate system; a removal unit that, if the world coordinate system coordinates of the lower left end portion at a plurality of time points during the calibration run of the vehicle include coordinates in the world coordinate system of the lower left end portion that satisfy a first coordinate removal condition, removes coordinates in the world coordinate system of the lower left end portion that satisfy the first coordinate removal condition, if the world coordinate system coordinates of the lower right end portion at a plurality of time points during the calibration run of the vehicle include coordinates in the world coordinate system of the lower right end portion that satisfy a second coordinate removal condition, removes coordinates in the world coordinate system of the lower right end portion that satisfy the second coordinate removal condition, if the world coordinate system coordinates of the lower right end portion at a plurality of time points during the calibration run of the vehicle include coordinates in the world coordinate system of the lower right end portion that satisfy a third coordinate removal condition, removes coordinates in the world coordinate system of the hitch ball that satisfy the third coordinate removal condition; and the lower left end portion at a plurality of time points during the calibration run of the vehicle that were not removed by the removal unit.a third calculation unit that calculates, by using the coordinates of the right lower end portion and the hitch ball in the world coordinate system, an average value of the distance between the left lower end portion and the hitch ball in the world coordinate system or an average value of the distance between the right lower end portion and the hitch ball in the world coordinate system, an average value of the angle formed by the left lower end portion, the hitch ball, and the right lower end portion in the world coordinate system, and an average value of the coordinates of the hitch ball in the world coordinate system; and a fourth calculation unit that calculates the world coordinate system coordinates of the lower left end at the specified time point and the world coordinate system coordinates of the lower right end at the specified time point based on the average value of the distance in the world coordinate system between the lower left end and the hitch ball or the average value of the distance in the world coordinate system between the lower right end and the hitch ball, the average value of the angle in the world coordinate system formed by the lower left end, the hitch ball, and the lower right end calculated by the third calculation unit, and the average value of the world coordinate system coordinates of the hitch ball.

[0007] (2) In the trailer lower end coordinate calculation device of (1), the removal unit may determine that the first coordinate removal condition is satisfied when the world coordinate system coordinate of the lower left end during the vehicle's calibration run is located outside a predetermined first region, and remove the world coordinate system coordinate of the lower left end at the time when it is located outside the first region; determine that the second coordinate removal condition is satisfied when the world coordinate system coordinate of the lower right end during the vehicle's calibration run is located outside the first region, and remove the world coordinate system coordinate of the lower right end at the time when it is located outside the first region; and determine that the third coordinate removal condition is satisfied when the world coordinate system coordinate of the hitch ball during the vehicle's calibration run is located outside a predetermined second region, and remove the world coordinate system coordinate of the hitch ball at the time when it is located outside the second region.

[0008] (3) In the trailer bottom end coordinate calculation device of (1) or (2), the removal unit calculates a first standard deviation, which is the standard deviation of the difference between the distance in the world coordinate system between the lower left end and the hitch ball during the calibration run of the vehicle and the first average value, and determines that the first coordinate removal condition is satisfied when the difference between the distance in the world coordinate system between the lower left end and the hitch ball during the calibration run of the vehicle and the first average value is twice or more the first standard deviation, and determines that the lower left end at the time when the difference between the distance in the world coordinate system between the lower left end and the hitch ball and the first average value is twice or more the first standard deviation. and calculate a second standard deviation which is the standard deviation of the difference between the second average value and the distance in the world coordinate system between the lower right end and the hitch ball during the calibration run of the vehicle, and determine that the second coordinate removal condition is satisfied when the difference between the second average value and the distance in the world coordinate system between the lower right end and the hitch ball during the calibration run of the vehicle is equal to or greater than twice the second standard deviation, and remove the world coordinate system coordinate of the lower right end at the time when the difference between the second average value and the distance in the world coordinate system between the lower right end and the hitch ball becomes equal to or greater than twice the second standard deviation.

[0009] (4) One aspect of the present disclosure includes an acquisition step in which a trailer bottom end coordinate calculation device acquires a plurality of images taken by a camera mounted on a vehicle towing a trailer via a tow bar at a plurality of time points during a calibration run of the vehicle; a first calculation step in which the trailer bottom end coordinate calculation device calculates coordinates on an image of a bottom left end of the trailer, a bottom right end of the trailer, and a hitch ball included in each of the plurality of images acquired in the acquisition step; a conversion step in which the trailer bottom end coordinate calculation device converts the coordinates on the image of the bottom left end calculated in the first calculation step into coordinates in a world coordinate system of the bottom left end, converts the coordinates on the image of the bottom right end calculated in the first calculation step into coordinates in a world coordinate system of the bottom right end, and converts the coordinates on the image of the hitch ball calculated in the first calculation step into coordinates in a world coordinate system of the hitch ball; and a conversion step in which the trailer bottom end coordinate calculation device calculates the coordinates of the bottom left end and the hitch ball at a plurality of time points during the calibration run of the vehicle. a second calculation step of calculating a first average value, which is an average value of the distance in the world coordinate system between the lower left end and the hitch ball during the calibration run of the vehicle, based on the coordinates of the world coordinate system of the lower right end and the hitch ball at a plurality of time points during the calibration run of the vehicle; and a second calculation step of calculating a second average value, which is an average value of the distance in the world coordinate system between the lower right end and the hitch ball during the calibration run of the vehicle, based on the coordinates of the world coordinate system of the lower right end and the hitch ball at a plurality of time points during the calibration run of the vehicle; and a second calculation step of calculating a second average value, which is an average value of the distance in the world coordinate system between the lower right end and the hitch ball during the calibration run of the vehicle, based on the coordinates of the world coordinate system of the lower left ...a removal step of removing the coordinates of the world coordinate system of the hitch ball that satisfy a third coordinate removal condition when the coordinates of the world coordinate system of the hitch ball at a plurality of times during the calibration running of the vehicle include coordinates of the world coordinate system of the hitch ball that satisfy the third coordinate removal condition; and a trailer lower end coordinate calculation device that uses the coordinates of the world coordinate system of the left lower end, the right lower end, and the hitch ball at a plurality of times during the calibration running of the vehicle that were not removed in the removal step to calculate an average value of the distance in the world coordinate system between the left lower end and the hitch ball or an average value of the distance in the world coordinate system between the right lower end and the hitch ball, an average value of the angle formed in the world coordinate system by the left lower end, the hitch ball, and the right lower end, and a coordinate of the hitch ball. and a fourth calculation step in which the trailer bottom end coordinate calculation device calculates the world coordinate system coordinates of the left lower end at the predetermined time point and the world coordinate system coordinates of the right lower end at the predetermined time point based on the hitch angle of the trailer at the predetermined time point after the vehicle has run a calibration, the average value of the distance in the world coordinate system between the left lower end and the hitch ball or the average value of the distance in the world coordinate system between the right lower end and the hitch ball calculated in the third calculation step, the average value of the angle in the world coordinate system formed by the left lower end, the hitch ball, and the right lower end calculated in the third calculation step, and the average value of the coordinates in the world coordinate system of the hitch ball.

[0010] (5) One aspect of the present disclosure is a method for providing a processor with an acquisition step of acquiring a plurality of images taken by a camera mounted on a vehicle towing a trailer via a tow bar at a plurality of time points during a calibration run of the vehicle; a first calculation step of calculating the coordinates of the image of a lower left end of the trailer, a lower right end of the trailer, and a hitch ball included in each of the plurality of images acquired in the acquisition step; a transformation step of converting the coordinates of the image of the lower left end calculated in the first calculation step into coordinates of a world coordinate system of the lower left end, converting the coordinates of the image of the lower right end calculated in the first calculation step into coordinates of a world coordinate system of the lower right end, and converting the coordinates of the image of the hitch ball calculated in the first calculation step into coordinates of a world coordinate system of the hitch ball; and a calculation step of calculating a first average value that is an average value of the distance in the world coordinate system between the lower left end and the hitch ball during the calibration run of the vehicle based on the coordinates of the world coordinate system of the lower left end and the hitch ball at a plurality of time points during the calibration run of the vehicle, and providing a carrier of the vehicle. a second calculation step of calculating a second average value, which is an average value of the distance in the world coordinate system between the lower right end portion and the hitch ball during the calibration running of the vehicle, based on the world coordinate system coordinates of the lower right end portion and the hitch ball at multiple time points during the calibration running of the vehicle; and a removal step of removing the world coordinate system coordinate of the lower left end portion that satisfies the first coordinate removal condition when the world coordinate system coordinate of the lower left end portion at multiple time points during the calibration running of the vehicle includes the world coordinate system coordinate of the lower left end portion that satisfies the first coordinate removal condition, removing the world coordinate system coordinate of the lower right end portion that satisfies the second coordinate removal condition when the world coordinate system coordinate of the lower right end portion at multiple time points during the calibration running of the vehicle includes the world coordinate system coordinate of the lower right end portion that satisfies the second coordinate removal condition, and removing the world coordinate system coordinate of the hitch ball that satisfies the third coordinate removal condition when the world coordinate system coordinate of the hitch ball at multiple time points during the calibration running of the vehicle includes the world coordinate system coordinate of the hitch ball that satisfies the third coordinate removal condition.a third calculation step of calculating an average value of the distance between the lower left end and the hitch ball in the world coordinate system or an average value of the distance between the lower right end and the hitch ball in the world coordinate system, an average value of the angle formed by the lower left end, the hitch ball, and the lower right end in the world coordinate system, and an average value of the coordinates of the hitch ball in the world coordinate system by using the coordinates of the lower left end, the lower right end, and the hitch ball in the world coordinate system at a plurality of points in time during the calibration run of the vehicle that were not removed in the removal step; and a fourth calculation step of calculating the coordinates of the left lower end at the predetermined time point in the world coordinate system and the coordinates of the right lower end at the predetermined time point in the world coordinate system based on the hitch angle, the average value of the distance in the world coordinate system between the left lower end and the hitch ball or the average value of the distance in the world coordinate system between the right lower end and the hitch ball calculated in the third calculation step, the average value of the angle formed by the left lower end, the hitch ball, and the right lower end in the world coordinate system calculated in the third calculation step, and the average value of the coordinates of the hitch ball in the world coordinate system. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide the coordinates of the bottom left and bottom right ends of the trailer in the world coordinate system, thereby enabling driving assistance for a vehicle towing a trailer. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing an example of a vehicle 1 to which a trailer bottom end coordinate calculation device 15 according to a first embodiment is applied. [Figure 2] 2 is a diagram showing an example of the relationship between the vehicle 1, trailer TR, and tow bar DB shown in FIG. 1. FIG. [Figure 3] 3A is a diagram showing an example of conversion from coordinates on the image IM shown in FIG. 2B to coordinates in the world coordinate system. FIG. [Figure 4] 3A to 3C are diagrams showing an example of a first area AR1. [Figure 5] FIG. 10 is a diagram for explaining the coordinates (xl, yl) in the world coordinate system of the bottom left end TRL of the trailer TR and the coordinates (xr, yr) in the world coordinate system of the bottom right end TRR of the trailer TR at a predetermined point in time calculated by the fourth calculation unit 3G. [Figure 6] 10 is a flowchart illustrating an example of processing executed by a processor 153 of the trailer bottom end coordinate calculation device 15 of the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, with reference to the drawings, embodiments of a trailer bottom end coordinate calculation device, a trailer bottom end coordinate calculation method, and a program according to the present disclosure will be described.

[0014] First Embodiment FIG. 1 is a diagram showing an example of a vehicle 1 to which a trailer bottom coordinate calculation device 15 according to the first embodiment is applied. FIG. 2 is a diagram showing an example of the relationship between the vehicle 1, trailer TR, and tow bar DB shown in FIG. 1. In detail, FIG. 2(A) is a diagram showing the vehicle 1, trailer TR, and tow bar DB from above, and FIG. 2(B) is a diagram showing an example of an image IM including the trailer TR and tow bar DB captured by a camera 11 mounted on the vehicle 1. FIG. 2(C) is a diagram showing an example of a calibration run of the vehicle 1. In the example shown in FIGS. 1 and 2, the vehicle 1 tows the trailer TR via the tow bar DB. The vehicle 1 includes a camera 11, an HMI (Human Machine Interface) 12, a vehicle control device 13, a steering actuator 13A, a braking actuator 13B, a driving actuator 13C, a hitch angle estimation device 14, and a trailer bottom coordinate calculation device 15. The camera 11 is disposed, for example, at the rear end 1R of the vehicle 1. The camera 11 photographs the rear of the vehicle 1 (the right side of FIG. 2(A)) and transmits an image (e.g., a fisheye lens image) IM (see FIG. 2(B)) including the trailer TR and the tow bar DB to the hitch angle estimation device 14 and the trailer bottom end coordinate calculation device 15. As shown in FIGS. 2(A) and 2(B), the tow bar DB is fixed to the trailer TR and is connected to the vehicle 1 so as to be rotatable about the hitch ball HB.

[0015] The HMI 12 has functions such as accepting various operations by the driver of the vehicle 1, and transmits signals indicating the operations by the driver of the vehicle 1 to the vehicle control device 13. The vehicle control device 13 controls the steering actuator 13A, the braking actuator 13B, and the drive actuator 13C based on the signals transmitted from the HMI 12.

[0016] The hitch angle estimation device 14 estimates the hitch angle θa (see FIG. 2(A)) of the trailer TR based on an image IM (see FIG. 2(B)) including the trailer TR and tow bar DB captured by the camera 11. In detail, the hitch angle estimation device 14 estimates the hitch angle θa of the trailer TR based on the image IM including the trailer TR and tow bar DB captured by the camera 11 by using a model obtained by learning using training data, which is a data set of training images captured by a training camera (not shown) mounted on a training vehicle (not shown) that tows the training trailer (not shown) via a training tow bar (not shown) and labels included in the training images that indicate the hitch angle of the training trailer.

[0017] The trailer bottom end coordinate calculation device 15 is configured by a microcomputer equipped with a communication interface (I / F) 151, memory 152, and processor 153. The communication interface 151 has an interface circuit for connecting the trailer bottom end coordinate calculation device 15 to the camera 11, HMI 12, vehicle control device 13, and hitch angle estimation device 14. The memory 152 stores programs and various data used in the processing executed by the processor 153. The processor 153 functions as an acquisition unit 3A, a first calculation unit 3B, a conversion unit 3C, a second calculation unit 3D, a removal unit 3E, a third calculation unit 3F, and a fourth calculation unit 3G. The acquisition unit 3A acquires an image IM including the trailer TR and tow bar DB captured by the camera 11. Specifically, the acquisition unit 3A acquires multiple images IM, ... captured by the camera 11 at multiple points in time during the calibration run of the vehicle 1. Furthermore, the acquisition unit 3A acquires the hitch angle θa of the trailer TR at a predetermined time point after the calibration run of the vehicle 1, which is estimated by the hitch angle estimation device 14. The first calculation unit 3B calculates the coordinates on the image of the bottom left end TRL of the trailer TR (see FIG. 2(B)), the bottom right end TRR of the trailer TR (see FIG. 2(B)), and the hitch ball HB (see FIG. 2(B)) included in each of the multiple images IM, ... acquired by the acquisition unit 3A.

[0018] The conversion unit 3C converts the coordinates on the image of the lower left end TRL of the trailer TR calculated by the first calculation unit 3B into coordinates in the world coordinate system of the lower left end TRL of the trailer TR, for example, by using a known technique called coordinate conversion. The conversion unit 3C also converts the coordinates on the image of the lower right end TRR of the trailer TR calculated by the first calculation unit 3B into coordinates in the world coordinate system of the lower right end TRR of the trailer TR. Furthermore, the conversion unit 3C converts the coordinates on the image of the hitch ball HB calculated by the first calculation unit 3B into coordinates in the world coordinate system of the hitch ball HB. Specifically, the conversion unit 3C converts the coordinates on the image IM of the lower left end TRL of the trailer TR shown in Figure 2(B) into coordinates in the world coordinate system of the lower left end TRL of the trailer TR shown in Figure 3(A), converts the coordinates on the image IM of the lower right end TRR of the trailer TR shown in Figure 2(B) into coordinates in the world coordinate system of the lower right end TRR of the trailer TR shown in Figure 3(A), and converts the coordinates on the image IM of the hitch ball HB shown in Figure 2(B) into coordinates in the world coordinate system of the hitch ball HB shown in Figure 3(A).

[0019] Fig. 3 shows an example of conversion from coordinates on the image IM shown in Fig. 2(B) to coordinates in the world coordinate system. In detail, Fig. 3(A) shows an example of conversion from coordinates on the image IM shown in Fig. 2(B) to coordinates in the world coordinate system. Fig. 3(B) shows an example of an average value (first average value) R1 of the distance in the world coordinate system between the lower left end TRL of the trailer TR and the hitch ball HB during the calibration run of the vehicle 1, calculated by the second calculation unit 3D, and Fig. 3(C) shows an example of an average value (second average value) R2 of the distance in the world coordinate system between the lower right end TRR of the trailer TR and the hitch ball HB during the calibration run of the vehicle 1, calculated by the second calculation unit 3D.

[0020] 1 to 3, the second calculation unit 3D calculates an average value (first average value) R1 (see FIG. 3(B)) of the distance in the world coordinate system between the lower left end portion TRL of the trailer TR and the hitch ball HB during the calibration running of the vehicle 1, based on the coordinates in the world coordinate system of the lower left end portion TRL of the trailer TR and the hitch ball HB at multiple points in time during the calibration running of the vehicle 1. Furthermore, the second calculation unit 3D calculates an average value (second average value) R2 (see FIG. 3(C)) of the distance in the world coordinate system between the lower right end portion TRR of the trailer TR and the hitch ball HB during the calibration running of the vehicle 1, based on the coordinates in the world coordinate system of the lower right end portion TRR of the trailer TR and the hitch ball HB at multiple points in time during the calibration running of the vehicle 1.

[0021] The multiple images IM, ... captured by the camera 11 at multiple times during the calibration travel of the vehicle 1 may include images captured by the camera 11 under adverse image recognition conditions (for example, bad weather, nighttime, or when the vehicle 1 is traveling on a road with a steep gradient). In consideration of this, in the example shown in FIGS. 1 to 3, if the world coordinate system coordinates of the lower left end TRL of the trailer TR at multiple times during the calibration travel of the vehicle 1 include the world coordinate system coordinates of the lower left end TRL of the trailer TR that satisfy the first coordinate removal condition, the removal unit 3E removes the world coordinate system coordinates of the lower left end TRL of the trailer TR that satisfy the first coordinate removal condition. In detail, if the world coordinate system coordinates of the lower left end TRL of the trailer TR during the calibration travel of the vehicle 1 are located outside a predetermined first area AR1 (see FIG. 4(A)), the removal unit 3E determines that the first coordinate removal condition is satisfied, and removes the world coordinate system coordinates of the lower left end TRL of the trailer TR at the time when the world coordinate system coordinates are located outside the first area AR1. The removal unit 3E also calculates the standard deviation (first standard deviation) of the difference between the first average value R1 (see FIG. 3(B)) and the distance in the world coordinate system between the lower left end TRL of the trailer TR and the hitch ball HB during the calibration running of the vehicle 1. Furthermore, when the difference between the first average value R1 and the distance in the world coordinate system between the lower left end TRL of the trailer TR and the hitch ball HB during the calibration running of the vehicle 1 is equal to or greater than twice the first standard deviation, the removal unit 3E determines that the first coordinate removal condition is satisfied, and removes the coordinate in the world coordinate system of the lower left end TRL of the trailer TR at the time when the difference between the first average value R1 and the distance in the world coordinate system between the lower left end TRL of the trailer TR and the hitch ball HB becomes equal to or greater than twice the first standard deviation.

[0022] Furthermore, the removal unit 3E removes the world coordinate system coordinates of the lower right end portion TRR of the trailer TR that satisfy the second coordinate removal condition when the world coordinate system coordinates of the lower right end portion TRR of the trailer TR at multiple points in time during the calibration running of the vehicle 1 include the world coordinate system coordinates of the lower right end portion TRR of the trailer TR that satisfy the second coordinate removal condition. In detail, when the world coordinate system coordinates of the lower right end portion TRR of the trailer TR at the time when the vehicle 1 is running during the calibration running are located outside the first area AR1 (see FIG. 4(A)), the removal unit 3E determines that the second coordinate removal condition is satisfied, and removes the world coordinate system coordinates of the lower right end portion TRR of the trailer TR at the time when the world coordinate system coordinates are located outside the first area AR1. Furthermore, the removal unit 3E calculates the standard deviation (second standard deviation) of the difference between the world coordinate system distance between the lower right end portion TRR of the trailer TR and the hitch ball HB at the time when the vehicle 1 is running during the calibration running of the vehicle 1 and the second average value R2 (see FIG. 3(C)). Furthermore, the removal unit 3E determines that the second coordinate removal condition is satisfied when the difference between the world coordinate system distance between the lower right end TRR of the trailer TR and the hitch ball HB during the calibration run of the vehicle 1 and the second average value R2 is more than twice the second standard deviation, and removes the world coordinate system coordinate of the lower right end TRR of the trailer TR at the time when the difference between the world coordinate system distance between the lower right end TRR of the trailer TR and the hitch ball HB and the second average value R2 is more than twice the second standard deviation.

[0023] Furthermore, the removal unit 3E removes the world coordinate system coordinates of the hitch ball HB that satisfy the third coordinate removal condition when the world coordinate system coordinates of the hitch ball HB at multiple points in time during the calibration travel of the vehicle 1 include the world coordinate system coordinates of the hitch ball HB. In detail, the removal unit 3E determines that the third coordinate removal condition is satisfied when the world coordinate system coordinates of the hitch ball HB during the calibration travel of the vehicle 1 are located outside a predetermined second area AR2 (see FIG. 4(B)), and removes the world coordinate system coordinates of the hitch ball HB of the trailer TR at points in time when they are located outside the second area AR2.

[0024] Fig. 4 is a diagram showing an example of the first area AR1 etc. In detail, Fig. 4(A) shows an example of the first area AR1, and Fig. 4(B) shows an example of the second area AR2.

[0025] In the example shown in Figures 1 to 4, the third calculation unit 3F uses the world coordinate system coordinates of the lower left end TRL of the trailer TR, the lower right end TRR of the trailer TR, and the hitch ball HB at multiple points in time during the calibration run of the vehicle 1 that were not removed by the removal unit 3E to calculate the learned radius R, which is the average value (first average value) R1 of the world coordinate system distance between the lower left end TRL of the trailer TR and the hitch ball HB at multiple points in time during the calibration run of the vehicle 1, or the average value (second average value) R2 of the world coordinate system distance between the lower right end TRR of the trailer TR and the hitch ball HB at multiple points in time during the calibration run of the vehicle 1. In addition, the third calculation unit 3F uses the world coordinate system coordinates of the lower left end TRL of the trailer TR, the lower right end TRR of the trailer TR, and the hitch ball HB at multiple points in time during the calibration run of the vehicle 1 that were not removed by the removal unit 3E to calculate the average value θbav (see Figure 5) of the angle θb (see Figure 4(B)) formed in the world coordinate system by the lower left end TRL of the trailer TR, the hitch ball HB, and the lower right end TRR of the trailer TR at multiple points in time during the calibration run of the vehicle 1, and the average value (xh, yh) of the world coordinate system coordinates of the hitch ball HB at multiple points in time during the calibration run of the vehicle 1.

[0026] The fourth calculation unit 3G calculates the hitch angle θa (see FIGS. 4A and 5A) of the trailer TR at a predetermined time point after the calibration run of the vehicle 1 estimated by the hitch angle estimation device 14, the post-learning radius R (see FIG. 5A) (first average value R1 or second average value R2) calculated by the third calculation unit 3F, and the average value of the angle θb formed in the world coordinate system by the lower left end TRL of the trailer TR, the hitch ball HB, and the lower right end TRR of the trailer TR at multiple times during the calibration run of the vehicle 1 calculated by the third calculation unit 3F. Based on θbav (see Figure 5), the average value (xh, yh) (see Figure 5) of the world coordinate system coordinates of the hitch ball HB at multiple points in time during the calibration run of vehicle 1, and the following equations (1) and (2), the world coordinate system coordinates (xl, yl) (see Figure 5) of the lower left end TRL of the trailer TR at a predetermined point in time after the calibration run of vehicle 1, and the world coordinate system coordinates (xr, yr) (see Figure 5) of the lower right end TRR of the trailer TR at a predetermined point in time after the calibration run of vehicle 1 are calculated.

[0027] (xl,yl)=(R×cosθl+xh,R×sinθl+yh)…(1)

[0028] (xr,yr)=(R×cosθr+xh,R×sinθr+yh)…(2)

[0029] In equation (1), θl=180+θa+(θbav / 2), and in equation (2), θr=180+θa-(θbav / 2).

[0030] Figure 5 is a diagram for explaining the coordinates (xl, yl) in the world coordinate system of the bottom left end TRL of the trailer TR and the coordinates (xr, yr) in the world coordinate system of the bottom right end TRR of the trailer TR at a predetermined point in time calculated by the fourth calculation unit 3G.

[0031] 1 to 5, the world coordinate system coordinates of the bottom left end TRL of the trailer TR, the bottom right end TRR of the trailer TR, and the hitch ball HB calculated by the first calculation unit 3B based on the image captured by the camera 11 under poor image recognition conditions during the calibration run of the vehicle 1 and converted by the conversion unit 3C are removed by the removal unit 3E. Furthermore, based on the world coordinate system coordinates of the bottom left end TRL of the trailer TR, the bottom right end TRR of the trailer TR, and the hitch ball HB during the calibration run of the vehicle 1 that were not removed, the world coordinate system coordinates (xl, yl) of the bottom left end TRL of the trailer TR and the world coordinate system coordinates (xr, yr) of the bottom right end TRR of the trailer TR at a predetermined time point after the calibration run of the vehicle 1. Therefore, in the examples shown in Figures 1 to 5, even if the calibration run of vehicle 1 includes times when poor conditions for image recognition are present, the coordinates (xl, yl) in the world coordinate system of the lower left end TRL of trailer TR and the coordinates (xr, yr) in the world coordinate system of the lower right end TRR of trailer TR at a predetermined point in time after the calibration run of vehicle 1 can be properly calculated.

[0032] FIG. 6 is a flowchart for explaining an example of processing executed by the processor 153 of the trailer bottom end coordinate calculation device 15 of the first embodiment. In the example shown in FIG. 6, in step S10, the acquisition unit 3A acquires a plurality of images IM, . . . captured by the camera 11 at a plurality of time points while the vehicle 1 is traveling for calibration. In step S11, the first calculation unit 3B calculates the coordinates on the images IM, ... of the lower left end TRL of the trailer TR, the lower right end TRR of the trailer TR, and the hitch ball HB contained in each of the multiple images IM, ... acquired in step S10. In step S12, the conversion unit 3C converts the coordinates on the image IM of the lower left end TRL of the trailer TR, the lower right end TRR of the trailer TR, and the hitch ball HB calculated in step S11 into coordinates in the world coordinate system. In step S13, the second calculation unit 3D calculates, based on the coordinates of the world coordinate system converted in step S12, the average value (first average value) R1 of the distance in the world coordinate system between the lower left end TRL of the trailer TR and the hitch ball HB at multiple points in time during the calibration run of the vehicle 1, and the average value (second average value) R2 of the distance in the world coordinate system between the lower right end TRR of the trailer TR and the hitch ball HB. In step S14, the removal unit 3E removes coordinates from the world coordinate system coordinates of the lower left end TRL of the trailer TR, the lower right end TRR of the trailer TR, and the hitch ball HB at multiple points during the calibration run of the vehicle 1 that satisfy the coordinate removal condition (first coordinate removal condition, second coordinate removal condition, or third coordinate removal condition).

[0033] 6, after step S14, the processor 153 determines whether learning using the multiple images IM, ... captured during the calibration run of the vehicle 1 was successful, based on the number of coordinates in the world coordinate system of the lower left end TRL of the trailer TR during the calibration run of the vehicle 1 that was not removed in step S14, the number of coordinates in the world coordinate system of the lower right end TRR of the trailer TR during the calibration run of the vehicle 1 that was not removed in step S14, the number of coordinates in the world coordinate system of the hitch ball HB during the calibration run of the vehicle 1 that was not removed in step S14, and the first standard deviation and the second standard deviation calculated by the removal unit 3E. If learning was successful, the process proceeds to step S15, and if learning was not successful, the process returns to step S11.

[0034] In step S15, the third calculation unit 3F calculates the learned radius R (first average value R1 or second average value R2), the average value θbav of the angle θb formed, and the average value (xh, yh) of the world coordinate system coordinates of the hitch ball HB by using the world coordinate system coordinates of the lower left end TRL of the trailer TR, the lower right end TRR of the trailer TR, and the hitch ball HB at multiple points during the calibration run of the vehicle 1 that were not removed in step S14. In step S16, the fourth calculation unit 3G calculates the coordinates (xl, yl) in the world coordinate system of the lower left end TRL of the trailer TR at a predetermined time point after the calibration run of the vehicle 1 and the coordinates (xr, yr) in the world coordinate system of the lower right end TRR of the trailer TR at a predetermined time point after the calibration run of the vehicle 1 based on the hitch angle θa of the trailer TR at a predetermined time point after the calibration run of the vehicle 1, the post-learning radius R (first average value R1 or second average value) R2), the average value θbav of the angle θb formed, the average value (xh, yh) of the world coordinate system coordinates of the hitch ball HB, and the above-mentioned equations (1) and (2).

[0035] Second Embodiment The vehicle 1 to which the trailer bottom end coordinate calculation device 15 of the second embodiment is applied is configured in the same manner as the vehicle 1 to which the trailer bottom end coordinate calculation device 15 of the first embodiment described above is applied, except for the points described below.

[0036] As described above, in the vehicle 1 to which the trailer bottom end coordinate calculation device 15 of the first embodiment is applied, the conversion unit 3C performs conversion from coordinates on the images IM, ... to coordinates in the world coordinate system by using known technology. On the other hand, in a vehicle 1 to which the trailer bottom end coordinate calculation device 15 of the second embodiment is applied, the conversion unit 3C performs conversion from coordinates on the image IM, ... to coordinates in the world coordinate system by using a technology other than publicly known technology (for example, a technology specific to the manufacturer of the vehicle 1).

[0037] As described above, embodiments of the trailer bottom edge coordinate calculation device, trailer bottom edge coordinate calculation method, and program of the present disclosure have been described with reference to the drawings. However, the trailer bottom edge coordinate calculation device, trailer bottom edge coordinate calculation method, and program of the present disclosure are not limited to the above-described embodiments and may be modified as appropriate without departing from the spirit of the present disclosure. The configurations of the above-described embodiments may be combined as appropriate. In the above-described embodiments, the processing performed by the trailer bottom edge coordinate calculation device 15 has been described as software processing performed by executing a program. However, the processing performed by the trailer bottom edge coordinate calculation device 15 may also be hardware processing. Alternatively, the processing performed by the trailer bottom edge coordinate calculation device 15 may be a combination of both software and hardware. Furthermore, the program stored in the memory 152 of the trailer bottom edge coordinate calculation device 15 (the program that realizes the functions of the processor 153 of the trailer bottom edge coordinate calculation device 15) may be recorded on a computer-readable storage medium such as a semiconductor memory, a magnetic recording medium, an optical recording medium, etc., and provided, distributed, etc. [Explanation of symbols]

[0038] 1...vehicle, 1R...rear end, 11...camera, 12...HMI, 13...vehicle control device, 13A...steering actuator, 13B...braking actuator, 13C...driving actuator, 14...hitch angle estimation device, 15...trailer bottom end coordinate calculation device, 151...communication interface, 152...memory, 153...processor, 3A...acquisition unit, 3B...first calculation unit, 3C...conversion unit, 3D...second calculation unit, 3E...removal unit, 3F...third calculation unit, 3G...fourth calculation unit

Claims

1. an acquisition unit that acquires a plurality of images taken by a camera mounted on the vehicle at a plurality of time points during a calibration run of the vehicle towing a trailer via a tow bar; a first calculation unit that calculates the coordinates on the image of a bottom left edge of the trailer, a bottom right edge of the trailer, and a hitch ball included in each of the plurality of images acquired by the acquisition unit; a conversion unit that converts the coordinates on the image of the lower left edge calculated by the first calculation unit into coordinates of the world coordinate system of the lower left edge, converts the coordinates on the image of the lower right edge calculated by the first calculation unit into coordinates of the world coordinate system of the lower right edge, and converts the coordinates on the image of the hitch ball calculated by the first calculation unit into coordinates of the world coordinate system of the hitch ball; a second calculation unit that calculates a first average value, which is the average value of the distance in the world coordinate system between the lower left end portion and the hitch ball during the calibration run of the vehicle, based on the coordinates in the world coordinate system of the lower left end portion and the hitch ball at multiple points in time during the calibration run of the vehicle, and calculates a second average value, which is the average value of the distance in the world coordinate system between the lower right end portion and the hitch ball during the calibration run of the vehicle, based on the coordinates in the world coordinate system of the lower right end portion and the hitch ball at multiple points in time during the calibration run of the vehicle; a removal unit that removes the coordinates of the world coordinate system of the lower left end that satisfy a first coordinate removal condition when the coordinates of the world coordinate system of the lower left end at multiple time points during the calibration running of the vehicle include coordinates of the world coordinate system of the lower left end that satisfy the first coordinate removal condition, removes the coordinates of the world coordinate system of the lower right end that satisfy the second coordinate removal condition when the coordinates of the world coordinate system of the lower right end at multiple time points during the calibration running of the vehicle include coordinates of the world coordinate system of the lower right end that satisfy a second coordinate removal condition, and removes the coordinates of the world coordinate system of the hitch ball that satisfy a third coordinate removal condition when the coordinates of the world coordinate system of the hitch ball at multiple time points during the calibration running of the vehicle include coordinates of the world coordinate system of the hitch ball that satisfy the third coordinate removal condition; a third calculation unit that calculates an average value of the distance between the lower left end and the hitch ball in the world coordinate system or an average value of the distance between the lower right end and the hitch ball in the world coordinate system, an average value of the angle formed by the lower left end, the hitch ball, and the lower right end in the world coordinate system, and an average value of the coordinates of the hitch ball in the world coordinate system by using the coordinates of the lower left end, the lower right end, and the hitch ball in the world coordinate system at multiple points in time during the calibration run of the vehicle that were not removed by the removal unit; a fourth calculation unit that calculates the world coordinate system coordinates of the lower left end at a predetermined time point and the world coordinate system coordinates of the lower right end at a predetermined time point based on the hitch angle of the trailer at a predetermined time point after the vehicle has been driven for calibration, the average value of the distance in the world coordinate system between the lower left end and the hitch ball or the average value of the distance in the world coordinate system between the lower right end and the hitch ball calculated by the third calculation unit, the average value of the angle in the world coordinate system formed by the lower left end, the hitch ball, and the lower right end calculated by the third calculation unit, and the average value of the coordinates in the world coordinate system of the hitch ball.

2. The removal unit determining that the first coordinate removal condition is satisfied when the coordinates of the world coordinate system of the lower left end portion during calibration travel of the vehicle are outside a predetermined first region, and removing the coordinates of the world coordinate system of the lower left end portion at the time when the coordinates are outside the first region; determining that the second coordinate removal condition is satisfied when the coordinates of the world coordinate system of the lower right end portion during calibration travel of the vehicle are outside the first region, and removing the coordinates of the world coordinate system of the lower right end portion at the time when the coordinates are outside the first region; 2. The trailer lower end coordinate calculation device according to claim 1, wherein when the coordinates of the world coordinate system of the hitch ball during calibration driving of the vehicle are located outside a predetermined second region, it is determined that the third coordinate removal condition is satisfied, and the coordinates of the world coordinate system of the hitch ball at the time when it is located outside the second region are removed.

3. The removal unit calculating a first standard deviation that is a standard deviation of the difference between the distance in the world coordinate system between the lower left end portion and the hitch ball during calibration traveling of the vehicle and the first average value; determining that the first coordinate removal condition is satisfied when a difference between the distance in the world coordinate system between the lower left end portion and the hitch ball and the first average value during calibration travel of the vehicle is equal to or greater than twice the first standard deviation, and removing the world coordinate system coordinate of the lower left end portion at the time when the difference between the distance in the world coordinate system between the lower left end portion and the hitch ball and the first average value is equal to or greater than twice the first standard deviation; calculating a second standard deviation that is the standard deviation of the difference between the distance in the world coordinate system between the lower right end portion and the hitch ball during calibration traveling of the vehicle and the second average value; 3. The trailer bottom end coordinate calculation device according to claim 1, wherein the second coordinate removal condition is determined to be satisfied when the difference between the world coordinate system distance between the lower right end and the hitch ball and the second average value during calibration driving of the vehicle is equal to or greater than twice the second standard deviation, and the world coordinate system coordinate of the lower right end is removed at the time when the difference between the world coordinate system distance between the lower right end and the hitch ball and the second average value is equal to or greater than twice the second standard deviation.

4. an acquisition step in which the trailer bottom end coordinate calculation device acquires a plurality of images taken by a camera mounted on the vehicle at a plurality of points in time during calibration travel of the vehicle towing the trailer via a tow bar; a first calculation step in which the trailer bottom end coordinate calculation device calculates coordinates on the image of the bottom left end of the trailer, the bottom right end of the trailer, and the hitch ball included in each of the plurality of images acquired in the acquisition step; a conversion step in which the trailer bottom end coordinate calculation device converts the coordinates on the image of the bottom left end calculated in the first calculation step into coordinates of the world coordinate system of the bottom left end, converts the coordinates on the image of the bottom right end calculated in the first calculation step into coordinates of the world coordinate system of the bottom right end, and converts the coordinates on the image of the hitch ball calculated in the first calculation step into coordinates of the world coordinate system of the hitch ball; a second calculation step in which the trailer lower end coordinate calculation device calculates a first average value, which is the average value of the distance in the world coordinate system between the lower left end and the hitch ball during the calibration run of the vehicle, based on the coordinates in the world coordinate system of the lower left end and the hitch ball at multiple points in time during the calibration run of the vehicle, and calculates a second average value, which is the average value of the distance in the world coordinate system between the lower right end and the hitch ball during the calibration run of the vehicle, based on the coordinates in the world coordinate system of the lower right end and the hitch ball at multiple points in time during the calibration run of the vehicle; a removal step in which the trailer bottom end coordinate calculation device removes the world coordinate system coordinates of the bottom left end that satisfy a first coordinate removal condition when the world coordinate system coordinates of the bottom left end at multiple points in time during the calibration running of the vehicle include the world coordinate system coordinates of the bottom left end that satisfy the first coordinate removal condition, removes the world coordinate system coordinates of the bottom right end that satisfy the second coordinate removal condition when the world coordinate system coordinates of the bottom right end at multiple points in time during the calibration running of the vehicle include the world coordinate system coordinates of the bottom right end that satisfy the second coordinate removal condition, and removes the world coordinate system coordinates of the hitch ball that satisfy the third coordinate removal condition when the world coordinate system coordinates of the hitch ball at multiple points in time during the calibration running of the vehicle include the world coordinate system coordinates of the hitch ball that satisfy the third coordinate removal condition; a third calculation step in which the trailer lower end coordinate calculation device calculates an average value of the distance in the world coordinate system between the lower left end and the hitch ball or an average value of the distance in the world coordinate system between the lower right end and the hitch ball, an average value of the angle formed in the world coordinate system by the lower left end, the hitch ball, and the lower right end, and an average value of the coordinates in the world coordinate system of the hitch ball, by using the world coordinate system coordinates of the lower left end, the lower right end, and the hitch ball at multiple points in time during the calibration run of the vehicle that were not removed in the removal step; a fourth calculation step in which the trailer bottom end coordinate calculation device calculates the world coordinate system coordinate of the lower left end at the predetermined time point and the world coordinate system coordinate of the lower right end at the predetermined time point based on the hitch angle of the trailer at the predetermined time point after the vehicle has run for calibration, the average value of the distance in the world coordinate system between the lower left end and the hitch ball or the average value of the distance in the world coordinate system between the lower right end and the hitch ball calculated in the third calculation step, the average value of the angle in the world coordinate system formed by the lower left end, the hitch ball, and the lower right end calculated in the third calculation step, and the average value of the coordinates in the world coordinate system of the hitch ball.

5. The processor an acquisition step of acquiring a plurality of images taken by a camera mounted on a vehicle towing a trailer via a tow bar at a plurality of time points during a calibration run of the vehicle; a first calculation step of calculating the coordinates on the image of the bottom left edge of the trailer, the bottom right edge of the trailer, and the hitch ball included in each of the plurality of images acquired in the acquisition step; a transformation step of transforming the coordinates on the image of the lower left edge calculated in the first calculation step into coordinates of the world coordinate system of the lower left edge, transforming the coordinates on the image of the lower right edge calculated in the first calculation step into coordinates of the world coordinate system of the lower right edge, and transforming the coordinates on the image of the hitch ball calculated in the first calculation step into coordinates of the world coordinate system of the hitch ball; a second calculation step of calculating a first average value, which is the average value of the distance in the world coordinate system between the lower left end portion and the hitch ball during the calibration run of the vehicle, based on the coordinates in the world coordinate system of the lower left end portion and the hitch ball at multiple points in time during the calibration run of the vehicle, and calculating a second average value, which is the average value of the distance in the world coordinate system between the lower right end portion and the hitch ball during the calibration run of the vehicle, based on the coordinates in the world coordinate system of the lower right end portion and the hitch ball at multiple points in time during the calibration run of the vehicle; a removal step of removing the coordinates of the world coordinate system of the lower left end that satisfy a first coordinate removal condition when the coordinates of the world coordinate system of the lower left end at multiple time points during the calibration run of the vehicle include coordinates of the world coordinate system of the lower left end that satisfy the first coordinate removal condition, removing the coordinates of the world coordinate system of the lower right end that satisfy the second coordinate removal condition when the coordinates of the world coordinate system of the lower right end at multiple time points during the calibration run of the vehicle include coordinates of the world coordinate system of the lower right end that satisfy a second coordinate removal condition, and removing the coordinates of the world coordinate system of the hitch ball that satisfy a third coordinate removal condition when the coordinates of the world coordinate system of the hitch ball at multiple time points during the calibration run of the vehicle include coordinates of the world coordinate system of the hitch ball that satisfy the third coordinate removal condition; a third calculation step of calculating an average value of the distance between the lower left end and the hitch ball in the world coordinate system or an average value of the distance between the lower right end and the hitch ball in the world coordinate system, an average value of the angle formed by the lower left end, the hitch ball, and the lower right end in the world coordinate system, and an average value of the coordinates of the hitch ball in the world coordinate system, by using the coordinates of the lower left end, the lower right end, and the hitch ball in the world coordinate system at multiple points in time during the calibration run of the vehicle that were not removed in the removal step; a fourth calculation step of calculating the world coordinate system coordinates of the lower left end portion at a predetermined time point and the world coordinate system coordinates of the lower right end portion at the predetermined time point based on the hitch angle of the trailer at the predetermined time point after the vehicle has been driven for calibration, the average value of the world coordinate system distance between the lower left end portion and the hitch ball or the average value of the world coordinate system distance between the lower right end portion and the hitch ball calculated in the third calculation step, the average value of the angle formed by the lower left end portion, the hitch ball, and the lower right end portion in the world coordinate system calculated in the third calculation step, and the average value of the world coordinate system coordinates of the hitch ball.

Images