Traction bar estimation device, traction bar estimation method, and program

Abstract

To properly estimate an angle of a traction bar included in an image captured by a camera on the basis of a line fitting result even if the camera is stained.SOLUTION: A traction bar estimation device 14 includes a traction bar estimation unit 3B that estimates a traction bar included in an image captured by a camera 11 mounted on a vehicle 1 towing a trailer via the traction bar, an extraction unit 3C that extracts a sequence of points on an image indicating the traction bar, a line fitting unit 3D that performs line fitting on the sequence of points, and a stain estimation unit 3E that estimates whether or not a stain on the camera 11 is included in the image. In a case where a portion of the sequence of points is positioned in an image region indicating the stain, the line fitting unit 3D performs the line fitting on the remaining portion of the sequence of points obtained by excluding the portion of the sequence positioned in the region from the sequence extracted by the extraction unit 3C.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a tow bar estimation device, a tow bar estimation method, and a program. [Background technology]

[0002] Patent document 1 describes a technique for capturing a first image of an object that is optically occluded by contamination, capturing a second image of the same object from a different viewpoint, and reconstructing the optically occluded portion of the first image using information from the second image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2010 / 084707 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 is applied to images captured by a camera mounted on a vehicle towing a trailer via a tow bar, and is not applied to images used to estimate the angle of a linear tow bar (trailer hitch angle). In other words, the technology described in Patent Document 1 does not extract the sequence of points necessary to estimate the angle of a linear tow bar, nor does it perform straight-line fitting on the extracted sequence of points. Therefore, with the technology described in Patent Document 1, if dirt adheres to the camera mounted on the vehicle towing a trailer via a tow bar, there is a risk that the tow bar angle contained in the image captured by the camera will not be accurately estimated.

[0005] In view of the above, an object of the present disclosure is to provide a tow bar estimation device, a tow bar estimation method, and a program that can appropriately estimate the angle of the tow bar (trailer hitch angle) included in an image captured by a camera based on the results of straight line fitting, even if dirt adheres to the camera mounted on a vehicle towing a trailer via a tow bar. is. [Means for solving the problem]

[0006] (1) One aspect of the present disclosure is a tow bar estimation device that includes a tow bar estimation unit that estimates a tow bar included in an image captured by a camera mounted on a vehicle that tows a trailer via a tow bar; an extraction unit that extracts a sequence of points on the image that indicates the tow bar estimated by the tow bar estimation unit; a straight line fitting unit that performs straight line fitting on the sequence of points extracted by the extraction unit; and a dirt estimation unit that estimates whether dirt adhering to the camera is included in the image, wherein when a portion of the sequence of points extracted by the extraction unit is located within an area on the image that indicates the dirt estimated by the dirt estimation unit, the straight line fitting unit performs the straight line fitting on another portion of the sequence of points that is the sequence of points extracted by the extraction unit, excluding the portion of the sequence of points that is located within the area.

[0007] (2) In the tow bar estimation device of (1), the dirt estimation unit may estimate whether the dirt is included in the image by using semantic segmentation.

[0008] (3) In the tow bar estimation device of (1) or (2), the dirt on the camera estimated by the dirt estimation unit includes rain on the camera, snow on the camera, and mud on the camera, and the dirt estimation unit includes a rain estimation unit that estimates whether the rain on the camera is included in the image, a snow estimation unit that estimates whether the snow on the camera is included in the image, and a mud estimation unit that estimates whether the mud on the camera is included in the image, and the rain estimation unit generates first teacher data that is a data set of training images taken by a training camera mounted on a training vehicle that tows a training trailer via a training tow bar and a first label that indicates whether the rain on the training camera is included in the training image. The snow estimation unit may estimate whether or not rain adhering to the camera is included in the image by using a first model obtained by learning using second teacher data, which is a data set of the training images and second labels indicating whether or not snow adhering to the training camera is included in the training images, and the mud estimation unit may estimate whether or not mud adhering to the camera is included in the image by using a third model obtained by learning using third teacher data, which is a data set of the training images and third labels indicating whether or not mud adhering to the training camera is included in the training images.

[0009] (4) One aspect of the present disclosure is a tow bar estimation method including: a tow bar estimation step in which a tow bar estimation device estimates a tow bar included in an image captured by a camera mounted on a vehicle towing a trailer via a tow bar; an extraction step in which the tow bar estimation device extracts a sequence of points on the image that indicates the tow bar estimated in the tow bar estimation step; a straight line fitting step in which the tow bar estimation device performs straight line fitting on the sequence of points extracted in the extraction step; and a dirt estimation step in which the tow bar estimation device estimates whether dirt adhering to the camera is included in the image, wherein, if a portion of the sequence of points extracted in the extraction step is located within a region on the image that indicates the dirt estimated in the dirt estimation step, in the straight line fitting step, the straight line fitting is performed on another portion of the sequence of points that is obtained by excluding the portion of the sequence of points extracted in the extraction step from the sequence of points extracted in the extraction step and that is located within the region.

[0010] (5) One aspect of the present disclosure is a program for causing a processor to execute a tow bar estimation step of estimating a tow bar included in an image captured by a camera mounted on a vehicle towing a trailer via a tow bar; an extraction step of extracting a sequence of points on the image that indicates the tow bar estimated in the tow bar estimation step; a straight line fitting step of performing straight line fitting on the sequence of points extracted in the extraction step; and a dirt estimation step of estimating whether dirt adhering to the camera is included in the image, wherein if a portion of the sequence of points extracted in the extraction step is located within an area on the image that indicates the dirt estimated in the dirt estimation step, in the straight line fitting step, the straight line fitting is performed on another portion of the sequence of points that is the sequence of points extracted in the extraction step minus the portion of the sequence of points that is located within the area. [Effects of the Invention]

[0011] According to the present disclosure, even if dirt adheres to a camera mounted on a vehicle towing a trailer via a tow bar, the angle of the tow bar (trailer hitch angle) contained in the image captured by the camera can be appropriately estimated based on the results of straight line fitting. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing an example of a vehicle 1 to which a tow bar estimation device 14 of a first embodiment is applied. [Figure 2] 2 is a diagram showing an example of the relationship between the vehicle 1, trailer TR, and tow bar DB shown in FIG. 1. FIG. [Figure 3] 2(B) is a diagram showing an example of a sequence of points PDB indicating a tow bar DB extracted by an extraction unit 3C from the image IM shown in FIG. 2(B). FIG. [Figure 4] FIG. 10 is a diagram showing a comparison between an example of a straight line LDB generated by the straight line fitting unit 3D of the tow bar estimation device 14 of the first embodiment when dirt is attached to the camera 11, and an example of a straight line LDB-r generated by the straight line fitting unit of the comparative example when dirt is attached to the camera 11. [Figure 5] 5 is a flowchart illustrating an example of processing executed by a processor 143 of the drawbar estimation device 14 of the first embodiment. [Figure 6] 1 is a diagram showing an example of a vehicle 1 to which a tow bar estimation device 14 of a second embodiment is applied. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of a tow bar estimation device, a tow bar estimation method, and a program according to the present disclosure will be described with reference to the drawings.

[0014] First Embodiment Fig. 1 is a diagram showing an example of a vehicle 1 to which a tow bar estimation device 14 of the first embodiment is applied. Fig. 2 is a diagram showing an example of the relationship between the vehicle 1, trailer TR, and tow bar DB shown in Fig. 1. In detail, Fig. 2(A) is a diagram showing the vehicle 1, trailer TR, and tow bar DB as viewed from above, and Fig. 2(B) is a diagram showing an example of an image IM including the trailer TR and tow bar DB captured by a camera 11 mounted on the vehicle 1. In the example shown in FIGS. 1 and 2, a vehicle 1 tows a trailer TR via a tow bar DB. The vehicle 1 is equipped with a camera 11, an HMI (Human Machine Interface) 12, a vehicle control device 13, a steering actuator 13A, a braking actuator 13B, a driving actuator 13C, and a tow bar estimation device 14. The camera 11 is disposed, for example, at the rear end 1R of the vehicle 1. The camera 11 captures an image of the rear of the vehicle 1 (the right side in FIG. 2(A)) and transmits an image (for example, a fisheye lens image) IM (see FIG. 2(B)) including the trailer TR and tow bar DB to the tow bar estimation device 14. As shown in FIG. 2, the tow bar DB is fixed to the trailer TR and is connected to the vehicle 1 so as to be rotatable about a hitch ball HB.

[0015] 1 and 2, the HMI 12 has a function of receiving various operations by the driver of the vehicle 1, and transmits a signal indicating the operation by the driver of the vehicle 1 to the vehicle control device 13. The vehicle control device 13 controls the steering actuator 13A, the braking actuator 13B, and the drive actuator 13C based on the signal transmitted from the HMI 12.

[0016] The drawbar estimation device 14 is configured by a microcomputer equipped with a communication interface (I / F) 141, a memory 142, and a processor 143. The communication interface 141 has an interface circuit for connecting the drawbar estimation device 14 to the camera 11, the HMI 12, and the vehicle control device 13. The memory 142 stores programs and various data used in the processing executed by the processor 143. The processor 143 functions as an acquisition unit 3A, a drawbar estimation unit 3B, an extraction unit 3C, a straight line fitting unit 3D, and a dirt estimation unit 3E.

[0017] The acquisition unit 3A acquires an image IM (see FIG. 2(B)) captured by the camera 11 and including the trailer TR and the tow bar DB. The tow bar estimation unit 3B estimates the tow bar DB included in the image IM acquired by the acquisition unit 3A. In detail, the tow bar estimation unit 3B estimates the tow bar DB included in the image IM based on the image IM acquired by the acquisition unit 3A by using a model obtained by performing learning using training data, which is a data set of training images taken by a training camera (not shown) mounted on a training vehicle (not shown) that tows a training trailer (not shown) via a training tow bar (not shown), and labels indicating the training tow bar included in the training images. The extraction unit 3C extracts a sequence of points PDB (see FIG. 3(A)) on the image IM that indicates the tow bar DB estimated by the tow bar estimation unit 3B. The straight line fitting unit 3D performs straight line fitting on the point sequence PDB extracted by the extraction unit 3C to generate a straight line LDB (see FIG. 3B) indicating the tow bar DB. The straight line LDB generated by the straight line fitting unit 3D is used to estimate, for example, the angle of the tow bar DB included in the image IM (the hitch angle θ of the trailer TR (see FIG. 2A)).

[0018] Fig. 3 shows an example of a point sequence PDB indicating the tow bar DB extracted by the extraction unit 3C from the image IM shown in Fig. 2(B). In detail, Fig. 3(A) shows an example of a point sequence PDB indicating the tow bar DB extracted by the extraction unit 3C from the image IM shown in Fig. 2(B), and Fig. 3(B) shows an example of a straight line LDB indicating the tow bar DB generated by the straight line fitting unit 3D from the point sequence PDB shown in Fig. 3(A). 2(B), 3(A), and 3(B), no dirt adheres to the camera 11 (specifically, the lens of the camera 11). On the other hand, for example, during rainfall, snowfall, or when the vehicle 1 is traveling on an unpaved road, dirt such as rain, snow, or mud may adhere to the camera 11.

[0019] Therefore, in the examples shown in Figures 1 to 3, measures described below are taken so that even if the camera 11 is covered with dirt such as rain, snow, or mud, the angle of the tow bar DB (hitch angle θ of the trailer TR) contained in the image IM can be appropriately (accurately) estimated based on the straight line LDB generated by the straight line fitting unit 3D. The dirt estimation unit 3E estimates whether or not dirt adhering to the camera 11 is included in the image IM captured by the camera 11. In particular, the dirt estimation unit 3E estimates whether or not dirt adhering to the camera 11 is included in the image IM captured by the camera 11, based on the image IM acquired by the acquisition unit 3A, by using a model obtained by performing learning using training data, which is a data set of training images captured by a training camera (not shown) mounted on a training vehicle (not shown) that tows a training trailer (not shown) via a training tow bar (not shown), and labels indicating whether or not dirt adhering to the training camera is included in the training image. Specifically, the dirt estimation unit 3E estimates whether or not dirt adhering to the camera 11 is included in the image IM captured by the camera 11, for example, by using semantic segmentation.

[0020] 4A and 4B are diagrams showing a comparison between an example of a straight line LDB generated by the straight line fitting unit 3D of the tow bar estimation device 14 of the first embodiment when dirt is attached to the camera 11, and an example of a straight line LDB-r generated by the straight line fitting unit of the comparative example when dirt is attached to the camera 11. In detail, FIG. 4A shows an example of a straight line LDB generated by the straight line fitting unit 3D of the tow bar estimation device 14 of the first embodiment when dirt is attached to the camera 11, and FIG. 4B shows an example of a straight line LDB-r generated by the straight line fitting unit of the comparative example when dirt is attached to the camera 11.

[0021] In the comparative example shown in FIG. 4(B), a portion PDB1-r of the sequence of points PDB-r extracted by the extraction unit is located within an area AR on the image IM showing dirt (raindrops) attached to the camera 11. As shown in FIG. 4(B), due to refraction of light passing through the raindrops, the portion PDB1-r of the sequence of points PDB-r is located at a position deviating from a straight line LDB2-r (a straight line LDB generated by the straight line fitting unit 3D of the tow bar estimation device 14 of the first embodiment when there is no dirt on the camera 11 (see FIG. 3(B))) including another portion PDB2-r of the sequence of points PDB-r located outside the area AR showing the dirt. Therefore, in the comparative example shown in FIG. 4(B), a straight line LDB-r located at a position deviating from the straight line LDB2-r including the other portion PDB2-r of the sequence of points PDB-r is generated by the straight line fitting unit 3D based on the portion PDB1-r and the other portion PDB2-r. As a result, in the comparative example shown in FIG. 4(B), the angle of the tow bar DB (hitch angle θ of the trailer TR) included in the image IM captured by the camera 11 may be inappropriately estimated.

[0022] On the other hand, in the example shown in FIG. 4(A) (an example of a straight line LDB generated by the straight line fitting unit 3D of the tow bar estimation device 14 of the first embodiment), a portion PDB1 of the point sequence PDB extracted by the extraction unit 3C is located within an area AR on the image IM showing the dirt estimated by the dirt estimation unit 3E. Therefore, the straight line fitting unit 3D performs straight line fitting on a portion PDB2 of the point sequence PDB, which is obtained by excluding the portion PDB1 of the point sequence PDB located within the area AR from the point sequence PDB extracted by the extraction unit 3C, to generate a straight line LDB used to estimate the angle of the tow bar DB included in the image IM (the hitch angle θ of the trailer TR). Therefore, in the example shown in FIG. 4(A), even if the portion PDB1 of the point sequence PDB extracted by the extraction unit 3C is located within an area AR on the image IM showing the dirt adhering to the camera 11, the angle of the tow bar DB (the hitch angle θ of the trailer TR) can be appropriately estimated.

[0023] FIG. 5 is a flowchart for explaining an example of processing executed by the processor 143 of the drawbar estimation device 14 of the first embodiment. In the example shown in FIG. 5, the acquisition unit 3A acquires an image IM including the trailer TR and the tow bar DB photographed by the camera 11 in step S10. In step S11, the tow bar estimation unit 3B estimates the tow bar DB included in the image IM acquired in step S10. In step S12, the extraction unit 3C extracts a sequence of points PDB on the image IM that indicates the tow bar DB estimated in step S11. In step S13, the dirt estimation unit 3E estimates whether or not the dirt adhering to the camera 11 is included in the image IM acquired in step S10. If YES, the process proceeds to step S14, and if NO, the process proceeds to step S16.

[0024] In step S14, for example, the straight line fitting unit 3D determines whether the part PDB1 of the point sequence PDB extracted in step S12 is located within the area AR on the image IM indicating the dirt estimated in step S13. If the answer is YES, the process proceeds to step S15, and if the answer is NO, the process proceeds to step S16. In step S15, the line fitting unit 3D performs line fitting on another part PDB2 of the point sequence PDB. In step S16, the line fitting unit 3D performs line fitting on the entire point sequence PDB.

[0025] Second Embodiment The vehicle 1 to which the tow bar estimation device 14 of the second embodiment is applied is configured in the same manner as the vehicle 1 to which the tow bar estimation device 14 of the first embodiment described above is applied, except for the points described below.

[0026] Fig. 6 is a diagram showing an example of a vehicle 1 to which a tow bar estimation device 14 of the second embodiment is applied. In the example shown in Fig. 6, the dirt estimation unit 3E includes a rain estimation unit 3E1 that estimates whether rain adhering to the camera 11 is included in the image IM, a snow estimation unit 3E2 that estimates whether snow adhering to the camera 11 is included in the image IM, and a mud estimation unit 3E3 that estimates whether mud adhering to the camera 11 is included in the image IM. 6, the dirt estimation unit 3E has a function of identifying whether the dirt adhering to the camera 11 is rain, snow, or mud. The dirt adhering to the camera 11 estimated by the dirt estimation unit 3E includes rain adhering to the camera 11, snow adhering to the camera 11, and mud adhering to the camera 11.

[0027] The rain estimation unit 3E1 estimates whether or not rain adhering to the camera 11 is included in the image IM by using a first model (rain estimation model) obtained by learning using first training data, which is a data set of training images taken by a training camera (not shown) mounted on a training vehicle (not shown) towing a training trailer (not shown) via a training tow bar (not shown), and a first label indicating whether or not rain adhering to the training camera is included in the training image. The snow estimation unit 3E2 estimates whether or not snow adhering to the camera 11 is included in the image IM by using a second model (snow estimation model) obtained by learning using second training data, which is a data set of training images taken by a training camera and second labels indicating whether or not snow adhering to the training camera is included in the training image. The mud estimation unit 3E3 estimates whether or not mud adhering to the camera 11 is included in the image IM by using a third model (mud estimation model) obtained by learning using third training data, which is a data set of training images taken by a training camera and a third label indicating whether or not mud adhering to the training camera is included in the training image.

[0028] As described above, embodiments of the towbar estimation device, towbar estimation method, and program of the present disclosure have been described with reference to the drawings. However, the towbar estimation device, towbar estimation method, and program of the present disclosure are not limited to the above-described embodiments, and appropriate modifications can be made without departing from the spirit of the present disclosure. The configurations of the examples of the above-described embodiments may be combined as appropriate. In each of the examples of the above-described embodiments, the processing performed in the towbar estimation device 14 has been described as software processing performed by executing a program. However, the processing performed in the towbar estimation device 14 may also be processing performed by hardware. Alternatively, the processing performed in the towbar estimation device 14 may be processing that combines both software and hardware. Furthermore, the program stored in the memory 142 of the towbar estimation device 14 (a program that realizes the functions of the processor 143 of the towbar estimation device 14) may be recorded on a computer-readable storage medium such as a semiconductor memory, a magnetic recording medium, an optical recording medium, or the like, and provided, distributed, etc. [Explanation of symbols]

[0029] 1...vehicle, 1R...rear end, 11...camera, 12...HMI, 13...vehicle control device, 13A...steering actuator, 13B...braking actuator, 13C...driving actuator, 14...tow bar estimation device, 141...communication interface, 142...memory, 143...processor, 3A...acquisition unit, 3B...tow bar estimation unit, 3C...extraction unit, 3D...straight line fitting unit, 3E...dirt estimation unit, 3E1...rain estimation unit, 3E2...snow estimation unit, 3E3...mud estimation unit

Claims

1. a tow bar estimation unit that estimates the tow bar included in an image captured by a camera mounted on a vehicle towing a trailer via the tow bar; an extraction unit that extracts a sequence of points on the image that indicate the tow bar estimated by the tow bar estimation unit; a straight line fitting unit that performs straight line fitting on the sequence of points extracted by the extraction unit; a dirt estimation unit that estimates whether dirt attached to the camera is included in the image, When a portion of the sequence of points extracted by the extraction unit is located within an area on the image that indicates the dirt estimated by the dirt estimation unit, the straight line fitting unit performs the straight line fitting on another portion of the sequence of points that is obtained by excluding the portion of the sequence of points that is located within the area from the sequence of points extracted by the extraction unit.

2. The tow bar estimation device according to claim 1 , wherein the dirt estimation unit estimates whether the dirt is included in the image by using semantic segmentation.

3. The dirt adhering to the camera estimated by the dirt estimation unit includes rain adhering to the camera, snow adhering to the camera, and mud adhering to the camera, The dirt estimation unit includes a rain estimation unit that estimates whether rain adhering to the camera is included in the image, a snow estimation unit that estimates whether snow adhering to the camera is included in the image, and a mud estimation unit that estimates whether mud adhering to the camera is included in the image, the rain estimation unit estimates whether rain adhering to the camera is included in the image by using a first model obtained by learning using first teacher data, which is a data set of training images taken by a training camera mounted on a training vehicle towing a training trailer via a training tow bar and first labels indicating whether rain adhering to the training camera is included in the training image; the snow estimation unit estimates whether or not snow adhering to the camera is included in the image by using a second model obtained by performing learning using second teacher data, which is a data set of the training image and a second label indicating whether or not snow adhering to the training camera is included in the training image; 3. The tow bar estimation device according to claim 1, wherein the mud estimation unit estimates whether mud adhering to the camera is included in the image by using a third model obtained by learning using third teacher data, which is a data set of the training image and a third label indicating whether mud adhering to the training camera is included in the image.

4. a tow bar estimation step in which a tow bar estimation device estimates the tow bar included in an image captured by a camera mounted on a vehicle towing a trailer via the tow bar; an extraction step in which the tow bar estimation device extracts a sequence of points on the image indicating the tow bar estimated in the tow bar estimation step; a line fitting step in which the drawbar estimation device performs line fitting of the sequence of points extracted in the extraction step; a dirt estimation step in which the tow bar estimation device estimates whether dirt attached to the camera is included in the image, A tow bar estimation method, wherein, when a portion of the sequence of points extracted in the extraction step is located within an area on the image that indicates the dirt estimated in the dirt estimation step, in the straight line fitting step, the straight line fitting is performed on another portion of the sequence of points that is obtained by excluding the portion of the sequence of points extracted in the extraction step that is located within the area.

5. The processor a tow bar estimation step of estimating a tow bar included in an image captured by a camera mounted on a vehicle towing a trailer via the tow bar; an extraction step of extracting a sequence of points on the image indicating the tow bar estimated in the tow bar estimation step; a straight line fitting step of performing straight line fitting of the sequence of points extracted in the extraction step; a dirt estimation step of estimating whether dirt attached to the camera is included in the image, A program in which, when a portion of the sequence of points extracted in the extraction step is located within an area on the image that indicates the dirt estimated in the dirt estimation step, in the straight line fitting step, the straight line fitting is performed on another portion of the sequence of points that is obtained by excluding the portion of the sequence of points extracted in the extraction step that is located within the area.

Images