Hitch angle detection system, hitch angle detector, and program

Abstract

To inhibit an increase in a load of preparing learning data, and upgrade the precision in detecting a hitch angle.SOLUTION: A hitch angle detector 15 includes a detection unit 3B that uses a model, which is obtained through learning performed using learning data that is a set of a learning trailer image taken by a learning camera of a learning vehicle and a label indicating a hitch angle of a learning trailer, to detect a hitch angle of a trailer SY1 on the basis of an image of the trailer SY1 taken by an onboard camera 11. A server SY2 performs additional learning of a model using additional learning data that is a set of an additional learning image of the trailer SY1 taken by the onboard camera 11 and a label indicating a hitch angle of the trailer SY1 which is obtained at a time of taking the additional learning image of the trailer SY1 and inputted via an onboard HMI 12. The detection unit 3B uses the model, which has undergone additional learning, to detect the hitch angle of the trailer SY1 on the basis of the image of the trailer SY1 taken by the onboard camera 11.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a hitch angle detection system, a hitch angle detection device, and a program. [Background technology]

[0002] Patent document 1 describes a technology in which multiple distances and multiple reference points associated with the shape of a vehicle and trailer are learned based on camera input, multiple reference distances associated with the multiple reference points are determined, and the current hitch angle is determined using the multiple reference distances and the shape distance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-509204 Summary of the Invention [Problem to be solved by the invention]

[0004] One possible method is to attach a separate mechanical sensor to the trailer to measure the hitch angle, but due to the cost and the hassle of attaching a sensor to each individual trailer owned by the user, the hitch angle may be detected using images from the rear camera of the vehicle towing the trailer. When detecting the hitch angle using images, another possible method is to attach a distinctive marker to the trailer, but as with the method of attaching a mechanical sensor to the trailer, this involves the hassle of attaching a marker to each individual trailer and setting it up to detect (calculate) the hitch angle. Another possible method is to train a model using images of trailers labeled with angle values ​​(or similar) as training data, and then use the trained model to detect the hitch angle, without using any special distinctive markers. However, because there are many types of trailers on the market, preparing training data for all types of trailers would require an excessively large amount of effort and would be difficult to achieve. Therefore, the only option is to prepare training data using representative types of trailers, train a model using that training data, and then use the trained model to detect the hitch angle. However, when a model is trained using training data prepared using representative types of trailers and the trained model is used to detect the hitch angle, the reality is that it is not possible to sufficiently improve the accuracy of hitch angle detection for trailers with distinctive designs or trailers whose appearance has been customized by the user. There is a need for a technology that can sufficiently improve the accuracy of detecting the hitch angle of trailers with distinctive designs, for example, while suppressing the increase in the burden of preparing learning data.

[0005] In view of the above, the present disclosure aims to provide a hitch angle detection system, a hitch angle detection device, and a program that can sufficiently improve the detection accuracy of the hitch angle of, for example, a trailer with a distinctive design while suppressing an increase in the load of preparing learning data. is. [Means for solving the problem]

[0006] (1) One aspect of the present disclosure is a hitch angle detection system including: a trailer; a vehicle towing the trailer; a camera mounted on the vehicle; a hitch angle detection device that detects the hitch angle of the trailer based on images of the trailer taken by the camera mounted on the vehicle by using a model obtained by performing learning using learning data that is a dataset of images of the training trailer taken by the training camera mounted on the training vehicle and labels indicating the hitch angle of the training trailer; and a server that performs additional learning of the model, wherein the vehicle has an HMI that accepts input of the trailer hitch angle used for additional learning of the model, and a communication device that communicates with the server. the camera mounted on the vehicle takes images of the trailer for additional training to be used for additional training of the model, the HMI accepts input of the trailer hitch angle at the time the trailer image for additional training was taken, the server performs additional training of the model using additional training data which is a dataset of the trailer image for additional training and a label indicating the trailer hitch angle accepted as input by the HMI, and the hitch angle detection device detects the trailer hitch angle based on the trailer image taken by the camera mounted on the vehicle by using the model for which additional training has been performed in the server.

[0007] (2) One aspect of the present disclosure is a hitch angle detection device that includes a detection unit that detects the hitch angle of a trailer based on an image of a trailer towed by a vehicle taken by a camera mounted on the vehicle, by using a model obtained by performing training using training data that is a dataset of images of a training trailer taken by a training camera mounted on the vehicle and labels indicating the hitch angle of the training trailer, wherein additional training of the model using additional training data that is a dataset of additional training images of the trailer taken by the camera mounted on the vehicle and labels indicating the hitch angle of the trailer at the time the additional training images of the trailer were taken, input via an HMI mounted on the vehicle, is performed on a server that communicates with a communication device mounted on the vehicle, and the detection unit detects the hitch angle of the trailer based on the image of the trailer taken by the camera mounted on the vehicle by using the model that has been additionally trained on the server.

[0008] (3) One aspect of the present disclosure is a program for causing a processor having the function of a detection unit that detects the hitch angle of a trailer based on an image of a trailer towed by a vehicle taken by a camera mounted on a vehicle, by using a model obtained by learning using learning data, which is a dataset of images of a training trailer taken by a training camera mounted on the training vehicle and labels indicating the hitch angle of the training trailer, to execute the following steps: acquiring the model that has been additionally trained in a server using additional learning data, which is a dataset of additional training images of the trailer taken by the camera mounted on the vehicle and labels indicating the hitch angle of the trailer at the time the additional training images of the trailer were taken, which are input via an HMI mounted on the vehicle; and detecting the hitch angle of the trailer based on the image of the trailer taken by the camera mounted on the vehicle, by using the model that has been additionally trained in the server, wherein the server communicates with a communication device mounted on the vehicle. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to sufficiently improve the detection accuracy of the hitch angle of, for example, a trailer with a distinctive design while suppressing an increase in the load of preparing learning data. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing an example of a vehicle 1 to which a hitch angle detection device 15 of a first embodiment is applied. [Figure 2] 2 is a diagram showing an example of a hitch angle detection system SY including the vehicle 1 shown in FIG. 1, a trailer SY1 towed by the vehicle 1, a server SY2, and the like. [Figure 3] FIG. 10 is a sequence diagram illustrating an example of processing executed in a hitch angle detection system SY including a vehicle 1 to which a hitch angle detection device 15 of the first embodiment is applied when additional learning of a model is performed. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of a hitch angle detection system, a hitch angle detection device, and a program according to the present disclosure will be described with reference to the drawings.

[0012] First Embodiment FIG. 1 is a diagram showing an example of a vehicle 1 to which a hitch angle detection device 15 according to the first embodiment is applied. In the example shown in Figure 1, a vehicle 1 is equipped with a camera 11, an HMI (Human Machine Interface) 12, a communication device 13, a vehicle control device 14, a steering actuator 14A, a braking actuator 14B, a driving actuator 14C, and a hitch angle detection device 15. The camera 11 is disposed, for example, at the rear of the vehicle 1. The camera 11 photographs the area behind the vehicle 1 and transmits to the hitch angle detection device 15 an image (for example, a fisheye lens image) including the trailer SY1 (see FIG. 2) being towed by the vehicle 1. The HMI 12 has functions such as accepting various operations by the driver of the vehicle 1, and transmits a signal indicating the operation by the driver of the vehicle 1 to the vehicle control device 14. The communication device 13 communicates with the outside of the vehicle 1 (for example, the server SY2 (see FIG. 2)).

[0013] FIG. 2 is a diagram showing an example of a hitch angle detection system SY including the vehicle 1 shown in FIG. 1, a trailer SY1 towed by the vehicle 1, a server SY2, and the like. In the example shown in FIG. 2, the hitch angle detection system SY is composed of a vehicle 1, a trailer SY1 towed by the vehicle 1, a server SY2, etc. The vehicle 1 (communication device 13) and the server SY2 are configured to be able to communicate via a network NW. To obtain a model used when the hitch angle detection device 15 detects (infers) the hitch angle of the trailer SY1, the server SY2 performs learning using learning data, which is a data set of images of a training trailer (not shown) taken by a training camera (not shown) mounted on a training vehicle (not shown) and labels indicating the hitch angle of the training trailer. Furthermore, the server SY2 transmits the learned model to the vehicle 1 via the network NW. In another example, to obtain a model used when the hitch angle detection device 15 detects (infers) the hitch angle of the trailer SY1, learning may be performed in a location other than the server SY2 (such as a manufacturing plant for the hitch angle detection device 15) using learning data, which is a data set of images of the training trailer taken by a training camera mounted on the training vehicle and labels indicating the hitch angle of the training trailer. In this example, when the hitch angle detection device 15 is installed in the vehicle 1, the trained model is also installed in the vehicle 1.

[0014] In the example shown in FIGS. 1 and 2, the communication device 13 receives the trained model transmitted from the server SY2. The vehicle control device 14 controls the steering actuator 14A, the braking actuator 14B, and the drive actuator 14C based on signals transmitted from the HMI 12, etc. The hitch angle detection device 15 is configured by a microcomputer equipped with a communication interface (I / F) 151, a memory 152, and a processor 153. The communication interface 151 has an interface circuit for connecting the hitch angle detection device 15 to the camera 11, the HMI 12, the communication device 13, and the vehicle control device 14. The memory 152 stores programs and various data used in the processing executed by the processor 153. The processor 153 functions as an acquisition unit 3A and a detection unit 3B.

[0015] 1 and 2, the acquisition unit 3A acquires a trained model that has been trained in the server SY2. In detail, the acquisition unit 3A acquires a trained model that has been received by the communication device 13. In the other example described above (an example in which the trained model is also incorporated into the vehicle 1 when the hitch angle detection device 15 is incorporated into the vehicle 1), the trained model is stored in the memory 152 and is pre-integrated into the hitch angle detection device 15, and the acquisition unit 3A acquires the trained model from the memory 152.

[0016] In the example shown in FIGS. 1 and 2, the acquisition unit 3A acquires an image of the trailer SY1 captured by the camera 11 (an image of the trailer SY1 transmitted from the camera 11). The detection unit 3B detects (infers) the hitch angle of the trailer SY1 based on the image of the trailer SY1 captured by the camera 11, by using the trained model acquired by the acquisition unit 3A. The hitch angle of the trailer SY1 detected by the detection unit 3B is used by the driver of the vehicle 1, the vehicle control device 14, etc., for example, during reverse control of the vehicle 1 and the trailer SY1, during lane keeping control of the vehicle 1 and the trailer SY1, etc.

[0017] As mentioned above, there are many types of trailers on the market. Therefore, if the trailer SY1 is, for example, a trailer with a distinctive design or a trailer whose appearance has been customized by the user (and the training trailer is not such a trailer), it may not be possible to accurately detect (infer) the hitch angle of the trailer SY1 by using a trained model that has been trained using training data, which is a dataset of images of the training trailer and labels indicating the hitch angle of the training trailer. Therefore, in the example shown in FIGS. 1 and 2, additional learning of the already-learned model is performed in such a case.

[0018] Specifically, in the example shown in Figures 1 and 2, the camera 11 captures an additional learning image of the trailer SY1 used for additional learning of the trained model, and transmits the additional learning image of the trailer SY1 to the communication device 13. The HMI 12 receives input of the hitch angle of the trailer SY1 (for example, measured by the user) at the time the additional learning image of the trailer SY1 was captured, and transmits to the communication device 13 the hitch angle of the trailer SY1 at the time the additional learning image of the trailer SY1 was captured. In detail, the additional learning images of the trailer SY1 are captured and the hitch angle of the trailer SY1 is input when the additional learning images of the trailer SY1 are captured, with multiple sets of different hitch angle values ​​for the trailer SY1, in order to improve the performance of the model after additional learning has been performed. The communication device 13 transmits a set of the additional learning image of the trailer SY1 captured by the camera 11 and the hitch angle of the trailer SY1 input received by the HMI 12 to the server SY2 via the network NW.

[0019] The server SY2 performs additional learning of the trained model using additional learning data, which is a data set of additional learning images of the trailer SY1 captured by the camera 11 and labels indicating the hitch angle of the trailer SY1 received as input by the HMI 12. Furthermore, the server SY2 transmits the additionally trained model to the vehicle 1 via the network NW. The communication device 13 receives the additionally trained model transmitted from the server SY2. The acquisition unit 3A acquires a model that has undergone additional training in the server SY2. In detail, the acquisition unit 3A acquires a model that has undergone additional training and that has been received by the communication device 13. The camera 11 captures an image of the trailer SY1 to be used for detecting (inferring) the hitch angle of the trailer SY1, which is performed by the detection unit 3B using a model that has undergone additional learning, and transmits the image of the trailer SY1 to the hitch angle detection device 15. The acquisition unit 3A acquires an image of the trailer SY1 captured by the camera 11 (an image of the trailer SY1 used for detecting (inferring) the hitch angle of the trailer SY1 using a model that has undergone additional learning). The detection unit 3B detects (infers) the hitch angle of the trailer SY1 based on the image of the trailer SY1 captured by the camera 11, by using the model that has undergone additional learning and that has been acquired by the acquisition unit 3A.

[0020] Therefore, in the example shown in Figures 1 and 2, it is possible to suppress an increase in the load of preparing learning data (more specifically, learning data that is a data set of images of the learning trailer and labels indicating the hitch angle of the learning trailer described above), while sufficiently improving the detection accuracy of the hitch angle of trailer SY1, even if trailer SY1 is, for example, a trailer with a distinctive design.

[0021] 1 and 2, the user of trailer SY1 (the driver of vehicle 1) first attempts to detect the hitch angle of trailer SY1 using detection unit 3B, which uses a trained model obtained by performing training using training data, which is a data set of images of a training trailer and labels indicating the hitch angle of the training trailer. Next, if the detection performance of detection unit 3B using the trained model is insufficient for the hitch angle of trailer SY1, the user of trailer SY1 causes camera 11 to capture additional training images of trailer SY1, measures the hitch angle of trailer SY1 at the time the additional training images were captured, and inputs the measurement results of the hitch angle of trailer SY1 via HMI 12. As a result, as described above, additional learning of the trained model is performed on server SY2 using additional learning data, which is a data set of additional learning images of trailer SY1 taken by camera 11 and labels indicating the hitch angle of trailer SY1 accepted as input by HMI 12. Furthermore, as described above, the detection unit 3B detects (infers) the hitch angle of the trailer SY1 based on the image of the trailer SY1 captured by the camera 11 by using a model that has undergone additional learning in the server SY2. As a result, the user of the trailer SY1 can obtain highly accurate detection results of the hitch angle of the trailer SY1 even if the trailer SY1 is, for example, a trailer with a distinctive design.

[0022] FIG. 3 is a sequence diagram illustrating an example of processing executed in the hitch angle detection system SY including the vehicle 1 to which the hitch angle detection device 15 of the first embodiment is applied when additional learning of a model is performed. In the example shown in Figure 3, in step S10, server SY2 uses learning data, which is a data set of images of the training trailer taken by a training camera mounted on the training vehicle and labels indicating the hitch angle of the training trailer, to perform learning of a model used to detect (infer) the hitch angle of trailer SY1 in step S15. In step S11, the server SY2 transmits the trained model (the model trained in step S10) to the vehicle 1 via the network NW, and the communication device 13 of the vehicle 1 receives the trained model. In step S12, the acquisition unit 3A acquires the trained model from the communication device 13.

[0023] In step S13, camera 11 captures an image of trailer SY1 that is used to detect (infer) the hitch angle of trailer SY1 in step S15. In step S14, the camera 11 transmits an image of the trailer SY1 to the hitch angle detection device 15, and the acquisition unit 3A acquires the image of the trailer SY1. In step S15, the detection unit 3B uses the trained model acquired in step S12 to detect (infer) the hitch angle of the trailer SY1 based on the image of the trailer SY1 acquired in step S14 (the image of the trailer SY1 photographed in step S13).

[0024] In the example shown in Figure 3, step S15 is executed when the user of trailer SY1 (the driver of vehicle 1) is actually using trailer SY1, and because the user desires to improve the performance of hitch angle detection device 15 (for example, because the user is not satisfied with the detection (inference) result of the hitch angle of trailer SY1 in step S15), processing from step S16 onwards is executed.

[0025] In step S16, the camera 11 captures an image for additional learning of the trailer SY1, which is used for additional learning of the trained model in step S21. In step S17, the camera 11 transmits the additional learning image of the trailer SY1 to the communication device 13. In step S18, the HMI 12 receives an input (input by the user) of the hitch angle of the trailer SY1 at the time of capturing the additional learning image of the trailer SY1. In step S19, the HMI 12 transmits to the communication device 13 the hitch angle of the trailer SY1 at the time the additional learning image of the trailer SY1 was captured (the hitch angle of the trailer SY1 that the HMI 12 accepted as input in step S18). As described above, the photographing of the additional learning image of the trailer SY1 in step S16 and the input of the hitch angle of the trailer SY1 when photographing the additional learning image of the trailer SY1 in step S18 are performed multiple times with different values ​​of the hitch angle of the trailer SY1.

[0026] In step S20, the communication device 13 transmits a set of the additional learning image of the trailer SY1 taken in step S16 (the additional learning image of the trailer SY1 transmitted from the camera 11 in step S17) and the hitch angle of the trailer SY1 accepted as input by the HMI 12 in step S18 (the hitch angle of the trailer SY1 transmitted from the HMI 12 in step S19) to the server SY2 via the network NW. In step S21, server SY2 performs additional learning of the trained model using additional learning data, which is a data set of additional learning images of trailer SY1 taken in step S16 (additional learning images of trailer SY1 transmitted from communication device 13 in step S20) and labels indicating the hitch angle of trailer SY1 accepted as input by HMI 12 in step S18 (hitch angle of trailer SY1 transmitted from communication device 13 in step S20). In step S22, the server SY2 transmits the model on which additional learning has been performed (the model on which additional learning has been performed in step S21) to the vehicle 1 via the network NW, and the communication device 13 of the vehicle 1 receives the model on which additional learning has been performed. In step S23, the acquisition unit 3A acquires from the communication device 13 the model on which the additional learning has been performed.

[0027] In step S24, camera 11 captures an image of trailer SY1 that is used to detect (infer) the hitch angle of trailer SY1 in step S26. In step S25, the camera 11 transmits the image of the trailer SY1 taken in step S24 to the hitch angle detection device 15, and the acquisition unit 3A acquires the image of the trailer SY1. In step S26, the detection unit 3B uses the model that underwent additional learning acquired in step S23 to detect (infer) the hitch angle of the trailer SY1 based on the image of the trailer SY1 acquired in step S25 (the image of the trailer SY1 photographed in step S24).

[0028] Second Embodiment The vehicle 1 to which the hitch angle detection device 15 of the second embodiment is applied is configured in the same manner as the vehicle 1 to which the hitch angle detection device 15 of the first embodiment described above is applied, except for the points described below.

[0029] As described above, in one example of a hitch angle detection system SY (the example shown in Figure 3) including a vehicle 1 to which the hitch angle detection device 15 of the first embodiment is applied, step S15 of Figure 3 (detection (inference) of the hitch angle of the trailer SY1) is executed when the user of the trailer SY1 (the driver of the vehicle 1) is actually using the trailer SY1, and if the user desires to improve the performance of the hitch angle detection device 15 (the detection performance of the hitch angle of the trailer SY1), processing from step S16 onwards of Figure 3 is executed. On the other hand, in an example of a hitch angle detection system SY including a vehicle 1 to which the hitch angle detection device 15 of the second embodiment is applied, step S15 of Figure 3 (detection (inference) of the hitch angle of the trailer SY1) is executed during a calibration run before the user of the trailer SY1 starts using the trailer SY1, and if the user wishes to improve the performance of the hitch angle detection device 15 (detection performance of the hitch angle of the trailer SY1), processing from step S16 onwards in Figure 3 is executed.

[0030] As described above, embodiments of the hitch angle detection system, hitch angle detection device, and program of the present disclosure have been described with reference to the drawings. However, the hitch angle detection system, hitch angle detection device, and program of the present disclosure are not limited to the above-described embodiments and may be modified as appropriate without departing from the spirit and scope of the present disclosure. The configurations of the above-described embodiments may be combined as appropriate. In the above-described embodiments, the processing performed by the hitch angle detection device 15 has been described as software processing performed by executing a program. However, the processing performed by the hitch angle detection device 15 may also be hardware processing. Alternatively, the processing performed by the hitch angle detection device 15 may be a combination of both software and hardware. Furthermore, the program stored in the memory 152 of the hitch angle detection device 15 (the program that realizes the functions of the processor 153 of the hitch angle detection device 15) may be provided, distributed, etc., by being recorded on a computer-readable storage medium such as a semiconductor memory, a magnetic recording medium, an optical recording medium, etc. [Explanation of symbols]

[0031] 1...vehicle, 11...camera, 12...HMI, 13...communication device, 14...vehicle control device, 14A...steering actuator, 14B...braking actuator, 14C...driving actuator, 15...hitch angle detection device, 151...communication interface, 152...memory, 153...processor, 3A...acquisition unit, 3B...detection unit, SY...hitch angle detection system, SY1...trailer, SY2...server

Claims

1. A hitch angle detection system including: a trailer; a vehicle towing the trailer; a camera mounted on the vehicle; a hitch angle detection device that detects the hitch angle of the trailer based on images of the trailer taken by the camera mounted on the vehicle by using a model obtained by performing learning using learning data that is a data set of images of a training trailer taken by a training camera mounted on a training vehicle and labels indicating the hitch angle of the training trailer; and a server that performs additional learning of the model, the vehicle includes an HMI that receives an input of the trailer hitch angle used for additional learning of the model, and a communication device that communicates with the server; the camera mounted on the vehicle captures an additional learning image of the trailer to be used for additional learning of the model; The HMI receives an input of a hitch angle of the trailer at the time of capturing an additional learning image of the trailer, the server performs additional learning of the model using additional learning data, which is a data set of additional learning images of the trailer and labels indicating the hitch angle of the trailer that the HMI has accepted as input; A hitch angle detection system in which the hitch angle detection device detects the hitch angle of the trailer based on an image of the trailer taken by the camera mounted on the vehicle by using the model that has undergone additional learning in the server.

2. A hitch angle detection device including a detection unit that detects the hitch angle of a trailer based on an image of a trailer towed by a vehicle taken by a camera mounted on the vehicle, by using a model obtained by performing learning using learning data, which is a data set of images of a training trailer taken by a training camera mounted on the training vehicle and labels indicating the hitch angle of the training trailer, additional learning of the model using additional learning data, which is a data set of additional learning images of the trailer taken by the camera mounted on the vehicle and labels indicating the hitch angle of the trailer at the time the additional learning images of the trailer were taken and input via an HMI mounted on the vehicle, is performed in a server that communicates with a communication device mounted on the vehicle; A hitch angle detection device in which the detection unit detects the hitch angle of the trailer based on an image of the trailer taken by the camera mounted on the vehicle by using the model that has undergone additional learning in the server.

3. A model obtained by performing learning using learning data, which is a data set of images of a training trailer taken by a training camera mounted on a training vehicle and labels indicating the hitch angle of the training trailer, is used to cause a processor having a function as a detection unit that detects the hitch angle of the trailer based on images of the trailer towed by the vehicle taken by a camera mounted on the vehicle, an acquisition step of acquiring the model that has undergone additional training in a server using additional training data, which is a data set of additional training images of the trailer taken by the camera mounted on the vehicle and labels indicating the hitch angle of the trailer at the time the additional training images of the trailer were taken, input via an HMI mounted on the vehicle; a detection step of detecting a hitch angle of the trailer based on an image of the trailer taken by the camera mounted on the vehicle by using the model that has undergone additional learning in the server, The server communicates with a communication device mounted on the vehicle.

Images