Driving assistance devices

Abstract

To provide a drive support device which can acquire, through image recognition, the accurate distance from a towing vehicle to a coupler of a towing object vehicle, and can display a bird's-eye view image at a more appropriate and stable timing.SOLUTION: A drive support device is configured to: acquire a photographed image obtained by imaging a periphery of a towing vehicle 2, using a rear camera 9 installed on the towing vehicle; generate a bird's-eye view image overlooking the periphery of the towing vehicle 2 on the basis of the photographed image; detect a coupler 7 which is provided in a towing object vehicle 3 being an object to be towed by the towing vehicle 2 for connection with the towing vehicle 2 through image recognition performed on the photographed image; acquire a distance from the towing vehicle 2 to a tip of the detected coupler 7; display a support image in a first display mode that does not include the bird's-eye view image in the display object when the distance from the towing vehicle 2 to the tip of the coupler 7 is equal to or greater than a threshold; and display a support image in a second display mode that includes the bird's-eye view image in the display object when the distance from the towing vehicle 2 to the tip of the coupler 7 is less than the threshold.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device that assists driving of a vehicle. [Background technology]

[0002] Tractors that tow trailers have been known for some time. When towing a trailer with a trailer, the first step is to physically connect the hitch ball located at the rear of the trailer with the coupler located at the front of the trailer. This connection between the hitch ball and the coupler is typically accomplished by the driver of the trailer backing the trailer and aligning the hitch ball with the coupler of the parked trailer.

[0003] However, the rear of the vehicle is a blind spot for the driver, and aligning the hitch ball and coupler while backing up the towing vehicle requires advanced driving skills. For example, Japanese Patent Application Laid-Open No. 2018-144526 proposes a technology that displays a rearward image captured by a camera installed in the towing vehicle on an in-vehicle display, identifies a reverse route from the vehicle's current position to a position where the hitch ball and coupler can be connected, and automatically steers the vehicle to reverse along the identified reverse route. When the distance from the hitch ball to the coupler becomes less than a predetermined distance, the rearward image is switched to an overhead image viewed from above the vehicle, making it easier to align the hitch ball and coupler. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-144526 A (paragraphs 0060-0065, 0071-0074, 0087-0096) Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, the timing at which the rear image switches to the overhead image is when the distance from the hitch ball to the coupler along the reversing route becomes less than a predetermined distance. However, the reversing route varies depending on the relative positions and angles of the towing vehicle and the towed vehicle at the start of reversing, and can result in a significantly meandering or curved trajectory. As a result, the timing at which the overhead image is displayed is not consistent and varies greatly depending on the situation, making it difficult for users to use. Furthermore, in Patent Document 1, the positions of the hitch ball and coupler in the rearward image are identified based on the user's touch operation on the rearward image, which makes it difficult to calculate the accurate distance from the hitch ball to the coupler.

[0006] The present invention has been made to solve the above-mentioned problems in the past, and aims to provide a driving assistance device that can obtain the accurate distance from the towing vehicle to the coupler on the towed vehicle through image recognition, and that can display an overhead image in a stable and more appropriate manner. [Means for solving the problem]

[0007] In order to achieve the above object, the driving assistance device according to the present invention is a driving assistance device that assists a driver in vehicle operation in a towing vehicle, and includes an image acquisition unit that acquires an image of the periphery of the towing vehicle using an imaging device installed in the towing vehicle, an overhead image generation unit that generates an overhead image of the periphery of the towing vehicle from above based on the image, an image display unit that displays at least one of the image acquisition unit and the overhead image on an image display device installed in the towing vehicle as an assistance image that assists vehicle operation, and a towed vehicle that is to be towed by the towing vehicle. The device includes an image recognition unit that detects a coupler provided on a towing vehicle for connecting with the towing vehicle by image recognition of the captured image, and a distance acquisition unit that acquires the distance from the towing vehicle to a specific point of the detected coupler, wherein the image display unit displays the support image in a first display mode that does not include the overhead image as a display target when the distance from the towing vehicle to the specific point of the coupler is equal to or greater than a threshold, and displays the support image in a second display mode that includes the overhead image as a display target when the distance from the towing vehicle to the specific point of the coupler is less than the threshold. On the other hand, when the image recognition unit cannot detect the coupler, the support image is displayed in the first display mode. . The "specific point of the coupler" may be, for example, the tip of the coupler, or the point where the hitch ball connects to the coupler (more specifically, the point where the ball-shaped part of the hitch ball is inserted). [Effects of the Invention]

[0008] The driving assistance device according to the present invention having the above configuration performs image recognition on captured images of the towing vehicle's surroundings, thereby making it possible to obtain the accurate distance from the towing vehicle to a specific point on the coupler of the towed vehicle. Since the obtained distance is less than a threshold, the overhead image is displayed, making it possible to display the overhead image more stably and at a more appropriate timing than in the past. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing a towing vehicle and a towed vehicle according to an embodiment of the present invention; [Figure 2]FIG. 2 is an enlarged view of the vicinity of the towing device of the towing vehicle. [Figure 3] 10 is a diagram showing the movement of the towing vehicle and the towed vehicle when the hitch ball and the coupler are connected. FIG. [Figure 4] 1 is a block diagram showing a configuration of a driving assistance device according to an embodiment of the present invention. [Figure 5] 4 is a flowchart of a driving assistance processing program according to the present embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a support image displayed on the liquid crystal display in S2. [Figure 7] FIG. 10 is a diagram showing the distance from the towing vehicle to the coupler, which is a display condition for the overhead image. [Figure 8] 10A and 10B are diagrams illustrating a method for converting a captured image into a bird's-eye view image. [Figure 9] FIG. 10 is a diagram showing an example of a support image displayed on the liquid crystal display in S8. DETAILED DESCRIPTION OF THE INVENTION

[0010] A detailed description of one embodiment of a driving assistance device according to the present invention will be given below with reference to the drawings. First, a description will be given of a towing vehicle (tractor) 2 equipped with a driving assistance device 1 according to this embodiment and a towed vehicle (trailer) 3 towed by the towing vehicle 2. Fig. 1 shows the towing vehicle 2 and the towed vehicle 3, with the upper drawing particularly showing the towing vehicle 2 and the towed vehicle 3 coupled together and the lower drawing showing the towing vehicle 2 and the towed vehicle 3 separated before being coupled together.

[0011] Here, the towing vehicle 2 is also called a tractor, and is configured to be able to travel while towing a towed vehicle 3. The towing vehicle 2 may be, for example, a vehicle (internal combustion engine vehicle) powered by an internal combustion engine (engine, etc.), a vehicle (electric vehicle, fuel cell vehicle, etc.) powered by an electric motor (motor, etc.), or a vehicle powered by both of these (hybrid vehicle). Furthermore, the vehicle type is not limited, and as long as it is equipped with a towing device 4 described below, it may be a standard car or a large commercial tractor (trailer head).

[0012] 1, a towing device 4 (hitch) for towing a towed vehicle 3 is disposed so as to protrude from, for example, the lower part of the center part in the vehicle width direction of the rear bumper of the towing vehicle 2. FIG. 2 is an enlarged view showing the towing device 4 and its vicinity in particular.

[0013] As shown in Figure 2, the towing device 4 is fixed to, for example, the frame of the towing vehicle 2. As an example, the towing device 4 is equipped with a hitch ball 5 that stands upright vertically (in the vehicle's up-and-down direction) and has a spherical tip. A coupler 7 provided at the tip of a connecting member 6 fixed to the towed vehicle 3 fits over the hitch ball 5, connecting the hitch ball 5 and the coupler 7, thereby connecting the towing vehicle 2 and the towed vehicle 3. However, the shapes of the hitch ball 5 and the coupler 7 are not limited to those shown in Figure 2, and any shape is acceptable as long as the towing vehicle 2 and the towed vehicle 3 can be connected.

[0014] When the hitch ball 5 and coupler 7 are connected, the hitch ball 5 transmits forward / backward and left / right movement to the towed vehicle 3 (connecting member 6) in accordance with the movement of the towing vehicle 2. Also, as shown in Figure 3, even when the coupler 7 is connected to the hitch ball 5, the angle of the coupler 7 relative to the hitch ball 5 can be freely changed (up to an upper limit), allowing the towed vehicle 3 to swing (turn) in the width direction relative to the towing vehicle 2.

[0015] Meanwhile, as shown in FIG. 1, a rear camera (imaging device) 9 is installed on the wall of the rear hatch at the rear of the towing vehicle 2. The rear camera 9 is, for example, a digital camera with a built-in imaging element such as a CCD or CIS. The rear camera 9 can output video data (captured image data) at a predetermined frame rate. The rear camera 9 has a wide-angle lens or a fisheye lens, and its optical axis is set to face diagonally downward, allowing it to capture images over a horizontal range of, for example, 140° to 220°. Therefore, when the towed vehicle 3 is not coupled, the rear camera 9 can output captured image data of a wide area around the rear of the towing vehicle 2, including the road surface.

[0016] The imaging range of the rear camera 9 includes at least the towing device 4 and hitch ball 5 at the rear end of the towing vehicle 2. Even if the towed vehicle 3 is not coupled, if there is a towed vehicle 3 near the rear of the towing vehicle 2, the towed vehicle 3 and the coupling member 6 and coupler 7 equipped on the towed vehicle 3 will also be imaged. In addition to recognizing the towed vehicle 3, the image data captured by the rear camera 9 can be used to detect the coupling state between the towing vehicle 2 and the towed vehicle 3 (e.g., coupling angle, whether or not they are coupled, etc.). In particular, in this embodiment, when the system is in a connection assistance mode that assists in coupling between the towing vehicle 2 and the towed vehicle 3, as described below, the image data captured by the rear camera 9 is also used to detect the distance from the towing vehicle 2 to the coupler 7 equipped on the towed vehicle 3. When the system is in the connection assistance mode, the image data captured by the rear camera 9 is displayed on a display inside the towing vehicle 2 so that the occupants can see it. Furthermore, the image data captured by the rear camera 9 is also used to generate a bird's-eye view image of the area around the towing vehicle from above. The generation of the bird's-eye view image will be described in detail later.

[0017] On the other hand, the towed vehicle 3 is also called a trailer, and travels while being towed by the towing vehicle 2. Therefore, unlike the towing vehicle 2, it basically does not have an engine or motor as a drive source. For example, it is a camper trailer with a living space inside, or a light trailer for carrying cars or boats. The towed vehicle 3 comprises a main body, a plurality of trailer wheels (two in this embodiment), a connecting member 6, and a coupler 7.

[0018] Here, as shown in Figure 1, the connecting member 6 is provided at the lower part of the center of the body of the towed vehicle 3 in the vehicle width direction, and is arranged so as to protrude forward (in the direction of travel) from the front end of the body.

[0019] As shown in Figure 2, the coupler 7 is provided at the front end of the connecting member 6 and has a spherical recess formed therein that covers the hitch ball 5. When the coupler 7 covers the hitch ball 5, the towed vehicle 3 is rotatably connected to the towing vehicle 2 as described above (see Figure 3). The length and height from the ground surface of the connecting member 6 (i.e., the position of the coupler 7 on the towed vehicle 3) vary depending on the type of towed vehicle 3. However, in this embodiment, the coupler 7 is positioned so that it can be connected to at least the hitch ball 5 provided on the towing vehicle 2.

[0020] Next, we will explain the driving assistance device 1 provided in the towing vehicle 2. The driving assistance device 1 is a device that assists the driver in vehicle operation when backing up the towing vehicle 2 in order to couple the towing vehicle 2 and towed vehicle 3 described above. Figure 4 is a block diagram showing the configuration of the driving assistance device 1 according to this embodiment.

[0021] 4, the driving assistance device 1 according to this embodiment includes an operation unit 14 that accepts operations from the occupant of the towing vehicle 2, an LCD display 15 that displays images captured by the rear camera 9 and other information related to driving assistance to the occupant of the towing vehicle 2, a speaker 16 that outputs audio guidance related to driving assistance, a vehicle information DB 21 that stores various data related to the towing vehicle 2 and the towed vehicle 3, and a driving assistance ECU 23 that performs various calculations based on the input information. The driving assistance device 1 is also connected via an in-vehicle network such as a CAN to various sensors installed on the towing vehicle 2, a vehicle control ECU 24 that performs various controls on the towing vehicle 2, a vehicle speed sensor 25, a steering sensor 26, a shift position sensor 27, etc.

[0022] The operation unit 14 is provided on the instrument panel or steering wheel of the towing vehicle 2, and has a plurality of operation switches (not shown), such as various keys and buttons, which are operated, for example, to switch to a connection assistance mode (described below) or to input various parameters related to the towing vehicle 2 and towed vehicle 3. The driving assistance ECU 23 controls the execution of various corresponding operations based on switch signals output by pressing each switch. The operation unit 14 may also have a touch panel provided on the front surface of the liquid crystal display 15. It may also have a microphone and a voice recognition device.

[0023] The LCD display 15 is provided on the instrument panel of the towing vehicle 2 and displays image data captured by the rear camera 9 when the towing vehicle 2 switches to a connection assistance mode that assists in coupling the towing vehicle 2 and towed vehicle 3. When predetermined conditions are met, an overhead image generated based on the captured image data is also displayed. The LCD display 15 may also be used for a navigation device.

[0024] The speaker 16 also outputs voice guidance and the like that guides driving when the vehicle transitions to the connection assistance mode based on an instruction from the driving assistance ECU 23. The speaker 16 may also be used for a navigation device.

[0025] The vehicle information DB 21 is a storage means for storing various information related to the towing vehicle 2 and the towed vehicle 3. For example, the following information is stored for the towing vehicle 2: the installation position of the rear camera 9 (height from the ground, lateral position, distance from the rear end of the vehicle), the optical axis direction, the installation position of the hitch ball 5 (height from the ground, lateral position, distance from the rear end of the vehicle), the overall length, vehicle width, wheelbase, minimum turning radius, etc. For the towed vehicle 3, the following information is stored: the installation position of the coupler 7 (height from the ground, lateral position, distance from the front end of the vehicle), the overall length, vehicle width, wheelbase, minimum turning radius, etc. This information may be input in advance by a driver or a person from the vehicle manufacturer using the operation unit 14, or values ​​detected by the rear camera 9 or various sensors may be input automatically. Note that the towed vehicle 3 to be towed is not necessarily fixed, so the above parameters must be changed when the towed vehicle 3 to be towed changes. A memory card, for example, can be used as a storage medium for the vehicle information DB 21. Furthermore, it may be provided in a storage area (for example, RAM or flash memory) within the driving assistance ECU 23.

[0026] Meanwhile, the driving assistance ECU (electronic control unit) 23 is an electronic control unit that controls the entire driving assistance device 1. It includes internal storage devices such as a CPU 31, which functions as a calculation device and control device; a RAM 32, which is used as working memory when the CPU 31 performs various calculation processes and stores route data and other information used when a route is searched; a ROM 33, which stores control programs as well as a driving assistance program (see FIG. 5 ), which will be described later; and a flash memory 34, which stores programs read from the ROM 33. The driving assistance ECU 23 also includes various control units that function as processing algorithms. For example, the captured image acquisition unit acquires captured images of the area around the towing vehicle 2 using the rear camera 9 installed on the towing vehicle 2. The overhead image generation unit generates an overhead image of the area around the towing vehicle 2 from above based on the captured images. The image display unit displays at least one of the captured image and the overhead image on the LCD display 15 of the towing vehicle 2 as an assistance image that supports vehicle operation. The image recognition unit detects the coupler 7 provided on the towed vehicle 3 to be towed by the towing vehicle 2 so that the towed vehicle 3 can connect to the towing vehicle 2 by image recognition of the captured image. The distance acquisition unit acquires the distance from the towing vehicle 2 to the specific point on the detected coupler 7.

[0027] The vehicle control ECU 24 is an electronic control unit that controls the towing vehicle 2. The vehicle control ECU 24 is connected to each drive unit of the vehicle, such as the steering, brakes, and accelerator. In this embodiment, the vehicle control ECU 24 can control each drive unit to provide automatic driving assistance for the towing vehicle 2, for example, when transitioning to a connection assistance mode (described later) or a parking assistance mode (for example, when transitioning to a parking assistance mode that assists in parking into a parking space). Specifically, the driving assistance ECU 23 transmits various types of assistance information related to automatic driving assistance generated by the driving assistance device 1 to the vehicle control ECU 24 via the CAN when the connection assistance mode or parking assistance mode is being executed. The vehicle control ECU 24 then uses the received various types of assistance information to provide automatic driving assistance after the vehicle starts traveling. Examples of assistance information include recommended travel trajectories for the towing vehicle 2 and towed vehicle 3, and information indicating the vehicle speed and steering angle when traveling along the trajectory. Note that automatic driving assistance may involve automatic steering operation alone, or automatic control of the drive source and brakes as well. On the other hand, it is not essential that the towing vehicle 2 be equipped with the above-mentioned automatic driving assistance, and the towing vehicle 2 may be a vehicle that can only be driven manually.

[0028] The vehicle speed sensor 25 is made up of an active wheel speed sensor attached to the wheels of the towing vehicle 2, and detects the rotational speed of the wheels and outputs a speed signal. The steering sensor 26 is attached inside the steering device, and detects the steering angle when the steering wheel is turned and outputs a steering angle signal. Furthermore, the shift position sensor 27 is built into the shift lever, and detects whether the shift position is "P (parking)", "N (neutral)", "R (reverse)", "D (drive)", "2 (second speed)", or "L (low)".

[0029] The driving assistance ECU 23 can acquire the current vehicle speed, travel distance, steering angle, shift position, etc. of the towing vehicle 2 based on the output signals from the various sensors described above.

[0030] Next, a driving assistance processing program executed by the driving assistance ECU 23 in the driving assistance device 1 having the above configuration will be described with reference to Fig. 5. Fig. 5 is a flowchart of the driving assistance processing program according to this embodiment. Here, the driving assistance processing program is executed after the ACC power supply (accessory power supply) of the towing vehicle 2 is turned on, and is a program that assists the driver in operating the vehicle when backing the towing vehicle 2 and coupling it to the towed vehicle 3. The program shown in the flowchart in Fig. 5 below is stored in the RAM 32 and ROM 33 provided in the driving assistance device 1, and is executed by the CPU 31.

[0031] First, in step (hereinafter abbreviated as S) 1, the CPU 31 determines whether the system has entered a connection assistance mode that assists in coupling between the towing vehicle 2 and the towed vehicle 3. Here, the system enters the connection assistance mode, for example, when the shift position is in "P" or "R" and the occupant of the towing vehicle 2 performs a predetermined mode transition operation on the operation unit 14. However, the shift position may be excluded from the conditions for mode transition. Also, the condition for mode transition may include the condition that the towed vehicle 3 is not currently coupled. Note that, for example, an image captured by the rear camera 9 can be used to detect whether the towing vehicle 2 and the towed vehicle 3 are coupled. Furthermore, the system may automatically enter the connection assistance mode when the vehicle begins to back up with the towed vehicle 3 not coupled.

[0032] If it is determined that the mode has shifted to a connection assistance mode that assists in coupling between the towing vehicle 2 and the towed vehicle 3 (S1: YES), the process proceeds to S2. On the other hand, if it is determined that the mode has not shifted to the connection assistance mode (S1: NO), the driving assistance processing program is terminated.

[0033] In S2, the CPU 31 outputs real-time images captured by the rear camera 9 (more specifically, video data (captured image data) captured at a predetermined frame rate) to the LCD display 15 as support images for assisting vehicle operation (first display mode). Here, the rear camera 9 is installed on the wall of the rear hatch at the rear of the towing vehicle 2 as shown in FIG. 1, with its optical axis facing diagonally downward and using a wide-angle or fisheye lens, and is able to capture images over a horizontal range of, for example, 140° to 220°. Therefore, when the towed vehicle 3 is not coupled, the rear camera 9 can output captured image data capturing a wide range of the area behind the towing vehicle 2, including the road surface. Thereafter, images captured by the rear camera 9 are continuously displayed until the connection support mode is ended (S4: YES or S9: YES).

[0034] 6 is a diagram showing an example of the support image 51 displayed on the LCD display 15 in S2. As shown in FIG. 6, the support image 51 includes at least the towing device 4 and hitch ball 5 at the rear end of the towing vehicle 2. If a towed vehicle 3 is not coupled to the towing vehicle 2 and is located near the rear of the towing vehicle 2, the towed vehicle 3 and the coupling member 6 and coupler 7 equipped on the towed vehicle 3 will also be included in the support image 51, as shown in FIG. 6. Note that only a specific area (for example, only the area around the hitch ball 5) of the image captured by the rear camera 9 may be cropped and displayed as the support image 51.

[0035] By visually checking the support image 51 displayed on the LCD display 15, the driver of the towing vehicle 2 can easily grasp the relative positions of the hitch ball 5 of his or her own vehicle and the coupler 7 of the towed vehicle 3 to be towed. Then, by operating the steering wheel to align the positions of the hitch ball 5 and the coupler 7 and backing up the towing vehicle 2, the hitch ball 5 and the coupler 7 can be brought closer to each other.

[0036] The current steering angle of the towing vehicle 2 may be obtained based on a detection signal from the steering sensor 26, and the future movement trajectory of the towing vehicle 2 may be calculated and displayed superimposed on the support image 51. The movement trajectory to be displayed superimposed on the support image 51 may be the movement trajectory of the rear tires of the towing vehicle 2, or the movement trajectory of the left and right ends of the vehicle body, or the movement trajectory of the hitch ball 5.

[0037] Next, in S3, the CPU 31 performs image recognition processing on the captured image taken by the rear camera 9 to determine whether the captured image includes a towed vehicle 3 to be towed. For example, to detect the towed vehicle 3 in the captured image taken by the rear camera 9, the CPU 31 performs brightness correction on the road surface and structures on the road surface based on the brightness difference. Then, it performs binarization processing to separate the structures from the image, geometric processing to correct distortion, smoothing processing to remove noise from the image, and other processing to detect the boundary between the road surface and the structures. Then, it determines whether the structures on the road surface are towed vehicles 3 to be towed, for example, using well-known template matching processing or feature point detection processing. However, the image recognition processing on the captured image is not limited to the above example, and may be performed using machine learning, for example.

[0038] If it is determined that the captured image contains the towed vehicle 3 to be towed (S3: YES), the process proceeds to S5. On the other hand, if it is determined that the captured image does not contain the towed vehicle 3 to be towed (S3: NO), the process proceeds to S4.

[0039] In S4, the CPU 31 determines whether or not to end the connection assistance mode. The connection assistance mode may be ended, for example, when an occupant of the towing vehicle 2 performs a predetermined mode ending operation on the operation unit 14, or when the shift position is shifted to "P" or the engine is turned off. The condition for ending the mode may also be when the towed vehicle 3 is coupled. Note that, to detect whether the towing vehicle 2 and towed vehicle 3 are coupled, an image captured by the rear camera 9, for example, may be used.

[0040] If it is determined that the connection assistance mode has ended (S4: YES), the driving assistance processing program is terminated. On the other hand, if it is determined that the connection assistance mode has not ended (S4: NO), the process returns to S2, and the assistance image continues to be displayed in the first display mode. That is, in this embodiment, if the towed vehicle 3 (including the coupler installed on the towed vehicle 3) cannot be detected based on the captured image, the assistance image continues to be displayed in the first display mode without transitioning to the second display mode, which will be described later.

[0041] Meanwhile, in S5, the CPU 31 performs image recognition processing on the captured image taken by the rear camera 9 to detect the coupler 7 of the towed vehicle 3 included in the captured image. The coupler 7 is detected using, for example, well-known template matching processing or feature point detection processing. If the vehicle information DB 21 contains information specifying the installation position of the coupler 7 on the towed vehicle 3, it is desirable to also refer to this information. However, the image recognition processing on the captured image is not limited to the above example, and may also be performed using machine learning, for example.

[0042] Furthermore, in S5, the CPU 31 acquires the distance L from the towing vehicle 2 to the detected coupler 7. In this embodiment, the distance from the rear end of the towing vehicle 2 to the tip of the coupler 7 is acquired, as shown in FIG. 7. However, the criteria for distance L can be changed as appropriate, and it may be the distance from the tip of the towing device 4 of the towing vehicle 2 to the tip of the coupler 7, or the distance from the hitch ball 5 of the towing vehicle 2 to the tip of the coupler 7. Also, instead of the distance to the tip of the coupler 7, it may be the distance to a spherical recess (see FIG. 2) in the coupler 7 that covers the hitch ball 5. The positions of the tip and recess of the coupler 7 can also be identified by the image recognition process described above.

[0043] Even if the rear camera 9 is a monocular camera, the distance L from the towing vehicle 2 to the coupler 7 can be detected from the position of the coupler 7 (the area occupied by the pixel displaying the coupler 7) included in the captured image captured by the rear camera 9 using, for example, the angle of view, lens characteristics, and information about the coupler 7 of the rear camera 9. For example, by utilizing the fact that aberration (blur) is characteristic of the distance to the subject and the pixel position of the captured image, it is possible to learn the change in the shape of the blur of the image depending on the distance through machine learning and estimate the distance L to the coupler 7 in the captured image. Furthermore, the distance L to the coupler 7 in the captured image can be estimated using information identifying the optical axis and position of the rear camera 9 and the position of the coupler 7 (particularly the height from the ground surface) stored in the vehicle information DB 21. Alternatively, detection is also possible by using a stereo camera for the rear camera 9. Furthermore, a sensor capable of detecting the distance to an object, such as a millimeter-wave radar or a laser sensor, may be combined and used.

[0044] Thereafter, in S6, the CPU 31 determines whether the distance L from the towing vehicle 2 to the coupler 7 is less than a threshold value. The threshold value can be set appropriately, for example, to 3 m. The user may also be able to set an arbitrary value for the threshold value using the operation unit 14.

[0045] If it is determined that the distance L from the towing vehicle 2 to the coupler 7 is less than the threshold value (S6: YES), the process proceeds to S7. On the other hand, if it is determined that the distance L from the towing vehicle 2 to the coupler 7 is equal to or greater than the threshold value (S6: NO), the process proceeds to S4, where a determination is made as to whether to terminate the connection assistance mode described above.

[0046] However, even if it is determined that the distance L from the towing vehicle 2 to the coupler 7 is equal to or greater than the threshold value (S6: NO), if a predetermined operation to display an overhead image is received from the occupant of the towing vehicle 2, it is desirable to transition to S7 in order to display the overhead image, giving priority to the occupant's wishes. For example, a dedicated button for switching the support image displayed on the LCD display 15 to an overhead image may be provided on the operation unit 14, and if that button is operated, the process may be forced to transition to S7 thereafter, even if the distance L from the towing vehicle 2 to the coupler 7 is equal to or greater than the threshold value.

[0047] 8, the CPU 31 generates an overhead image by projecting a real-time image captured by the rear camera 9 onto a virtual projection surface, which is a horizontal plane corresponding to the height of the ground, and converting the image projected onto the virtual projection surface into an image viewed from a virtual viewpoint looking vertically down from above the towing vehicle 2. However, the height of the virtual projection surface may be a horizontal plane corresponding to the height of the hitch ball 5 or the coupler 7 instead of the ground. The height of the hitch ball 5 or the coupler 7 is stored in the vehicle information DB 21.

[0048] The conversion to an image viewed from a virtual viewpoint in S7 is performed by first converting each coordinate in the captured image coordinate system set along a plane perpendicular to the optical axis of the rear camera 9 into each coordinate in the ground coordinate system set along the ground surface, and then converting it into each coordinate in the overhead image coordinate system. Note that the conversion formulas used for each coordinate conversion are already known, so their explanation will be omitted.

[0049] Next, in S8, the CPU 31 outputs the overhead image generated in S7 to the liquid crystal display 15 as an assistance image for assisting vehicle operation (second display mode). Note that the image captured by the rear camera 9 that was displayed in the first display mode in a dual-screen configuration may continue to be displayed on one of the screens, or the image captured by the rear camera 9 may be hidden (the display may be switched to the overhead image) as the overhead image is displayed. Furthermore, the overhead image may not be displayed as is as a real-life image, but may be converted into an illustrated image that schematically shows the towing vehicle 2 and towed vehicle 3 and then displayed. Thereafter, the overhead image of S7 based on the real-time captured image and the generated overhead image are continuously generated and displayed until the connection assistance mode is terminated (S9: YES).

[0050] 9 shows an example of the support image displayed on the LCD display 15 in S8. As shown in FIG. 9, the support image is divided into a first support image 52 displayed on the left screen and a second support image 53 displayed on the right screen. The first support image 52 continues to display an image captured by the rear camera 9. Meanwhile, the second support image 53 displays an overhead image. The overhead image includes at least the towing device 4 and hitch ball 5 at the rear end of the towing vehicle 2. As mentioned above, the overhead image is displayed only when the distance L from the towing vehicle 2 to the coupler 7 is less than the threshold value. Therefore, as shown in FIG. 9, the towed vehicle 3 and the coupling member 6 and coupler 7 equipped on the towed vehicle 3 are also basically included in the overhead image.

[0051] The driver of the towing vehicle 2 can easily grasp the relative positions of the hitch ball 5 of his or her own vehicle and the coupler 7 of the towed vehicle 3 by visually checking the first support image 52 and second support image 53 displayed on the LCD display 15. In particular, the second support image 53 provides a bird's-eye view of the hitch ball 5 and coupler 7 from directly above, making the relative positions even clearer. Then, by operating the steering wheel to align the positions of the hitch ball 5 and coupler 7 and backing up the towing vehicle 2, it becomes possible to connect the hitch ball 5 and coupler 7 as shown in Figure 2.

[0052] When displaying the overhead image as the second support image 53, it is desirable to crop only a specific area of ​​the overhead image generated in S7 and then enlarge and display the cropped image. For example, a minimum area including at least the hitch ball 5 and the vicinity of the tip of the coupler 7 is cropped and enlarged for display. As a result, the overhead image is enlarged and displayed according to the distance from the towing vehicle 2 to the tip of the coupler 7 (the shorter the distance from the towing vehicle 2 to the tip of the coupler 7, the more enlarged the overhead image displayed). When the hitch ball 5 and the coupler 7 are close to each other, the positional relationship between the hitch ball 5 and the coupler 7 can be identified in more detail based on the overhead image. The overhead image may be enlarged or reduced based on a user operation.

[0053] The current steering angle of the towing vehicle 2 may be obtained based on a detection signal from the steering sensor 26, and the future movement trajectory of the towing vehicle 2 may be calculated and displayed superimposed on one or both of the first support image 52 and the second support image 53. The movement trajectory displayed superimposed on the first support image 52 or the second support image 53 may be the movement trajectory of the rear tires of the towing vehicle 2, or the movement trajectory of the left and right ends of the vehicle body, or the movement trajectory of the hitch ball 5.

[0054] Thereafter, in S9, the CPU 31 determines whether or not to end the connection assistance mode. Here, the connection assistance mode may be ended, for example, when an occupant of the towing vehicle 2 performs a predetermined mode ending operation on the operation unit 14, or when the shift position is shifted to "P" or the engine is turned off. Alternatively, the condition for ending the mode may be when the towed vehicle 3 is coupled. Note that, to detect whether the towing vehicle 2 and towed vehicle 3 are coupled, for example, an image captured by the rear camera 9 may be used.

[0055] If it is determined that the connection assistance mode has ended (S9: YES), the driving assistance processing program is terminated. On the other hand, if it is determined that the connection assistance mode has not ended (S9: NO), the process returns to S7, and the display of the assistance image in the second display mode is continued.

[0056] As explained in detail above, the driving assistance device 1 and the computer program executed by the driving assistance device 1 according to this embodiment acquire an image of the area around the towing vehicle using the rear camera 9 installed on the towing vehicle 2, and generate an overhead image of the area around the towing vehicle 2 from above based on the acquired image (S7). At the same time, the coupler 7 equipped on the towed vehicle 3 to be towed by the towing vehicle 2, which connects the towed vehicle 3 to the towing vehicle 2, is detected by image recognition of the acquired image (S5). Furthermore, the distance from the towing vehicle 2 to the tip of the detected coupler 7 is acquired (S5). If the distance from the towing vehicle 2 to the tip of the coupler 7 is equal to or greater than a threshold, the assistance image is displayed in a first display mode that does not include the overhead image as a display object. If the distance from the towing vehicle 2 to the tip of the coupler 7 is less than the threshold, the assistance image is displayed in a second display mode that includes the overhead image as a display object (S8). Therefore, by performing image recognition on the acquired image of the area around the towing vehicle, it is possible to acquire the accurate distance from the towing vehicle to the tip of the coupler equipped on the towed vehicle. Then, the overhead image is displayed on the condition that the acquired distance is less than the threshold value, so that the overhead image can be displayed at a more stable and appropriate timing than in the past. In addition, when the coupler cannot be detected, the support image is displayed in the first display mode (S2), so that the overhead image can be displayed only in situations where the positional relationship between the hitch ball and the coupler can be identified using the overhead image. In addition, the device has an operation unit 14 that accepts operations from the driver, and if a specified operation from the driver is accepted even if the distance from the towing vehicle 2 to the tip of the coupler 7 is equal to or greater than a threshold value, the display mode of the support image is switched from the first display mode to the second display mode (S8).Therefore, while the display mode of the support image is switched based on the distance from the towing vehicle 2 to the tip of the coupler 7, it is also possible to switch the support image in accordance with the driver's wishes when necessary. Furthermore, when the support image is displayed in the second display mode, the overhead image is enlarged and displayed according to the distance from the towing vehicle 2 to the tip of the coupler 7 (S8), so that the positional relationship between the hitch ball and the coupler can be identified in more detail based on the enlarged overhead image as needed.

[0057] The present invention is not limited to the above-described embodiment, and it goes without saying that various improvements and modifications are possible within the scope of the present invention. For example, in this embodiment, the driving assistance processing program (FIG. 5) displays assistance images (FIGS. 6 and 9) on the LCD display 15. However, in addition to displaying the assistance images, vehicle control using automatic driving assistance may also be performed. For example, when the connection assistance mode is being executed, the CPU 31 transmits various assistance information related to automatic driving assistance generated by the driving assistance device 1 to the vehicle control ECU 24 via the CAN. The vehicle control ECU 24 then uses the received various assistance information to perform automatic driving assistance after starting reverse. Examples of assistance information include a recommended driving trajectory for connecting the hitch ball 5 and the coupler 7, and information indicating the vehicle speed and steering angle when driving along the trajectory. Note that automatic driving assistance may involve automatic steering operation alone, or automatic control of the drive source and brakes. Note that the towing vehicle 2 does not necessarily need to be equipped with the above-mentioned automatic driving assistance; the towing vehicle 2 may be a vehicle that can only be driven manually.

[0058] Furthermore, in this embodiment, the distance from the towing vehicle 2 to the tip of the coupler 7 is detected based on the image captured by the rear camera 9, but a sensor capable of detecting the distance to an object, such as a millimeter-wave radar or a laser sensor, may be installed on the towing vehicle 2, and the distance from the towing vehicle 2 to the tip of the coupler 7 may be detected using the sensor.

[0059] In this embodiment, the driving assistance processing program (FIG. 5) is executed by the driving assistance ECU 23 of the driving assistance device 1, but the execution entity can be changed as appropriate. For example, the program may be executed by the control unit of the liquid crystal display 15, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle device. [Explanation of symbols]

[0060] 1...driving assistance device, 2...towing vehicle, 3...towed vehicle, 4...towing device, 5...hitch ball, 6...connecting member, 7...coupler, 9...rear camera, 15...liquid crystal display, 31...CPU, 32...RAM, 33...ROM, 51...assistance image, 52...first support image, 53...second support image

Claims

1. A driving assistance device that assists a driver in operating a towing vehicle, an image acquisition unit that acquires an image of the periphery of the towing vehicle using an imaging device installed in the towing vehicle; an overhead image generating unit that generates an overhead image of the periphery of the towing vehicle from above based on the captured image; an image display unit that displays at least one of the captured image and the overhead image on an image display device provided in the towing vehicle as an assistance image that assists in vehicle operation; an image recognition unit that detects a coupler provided on a towed vehicle to be towed by the towing vehicle in order to connect the towing vehicle to the towing vehicle by image recognition of the captured image; a distance acquisition unit that acquires a distance from the towing vehicle to a specific point of the detected coupler, The image display unit When the distance from the towing vehicle to the specific point of the coupler is equal to or greater than a threshold, the support image is displayed in a first display mode in which the overhead image is not included in the display targets; When the distance from the towing vehicle to the specific point of the coupler is less than a threshold, the support image is displayed in a second display mode that includes the overhead image as a display object, A driving assistance device that displays the assistance image in the first display mode when the image recognition unit cannot detect the coupler.

2. an operation reception unit that receives an operation by the driver, 2. The driving assistance device according to claim 1, wherein the image display unit switches the display mode of the assistance image from the first display mode to the second display mode when a predetermined operation by the driver is received even if the distance from the towing vehicle to the specific point of the coupler is equal to or greater than a threshold.

3. 2. The driving assistance device according to claim 1, wherein when the image display unit displays the assistance image in the second display mode, the image display unit enlarges and displays the overhead image in accordance with the distance from the towing vehicle to the specific point of the coupler.

Images