System and method for detecting trailer coupling and planning trajectory

Abstract

A system and method for positioning a trailer coupler on a trailer to assist in coupling the trailer coupler to a vehicle hook on a vehicle. The system includes a human machine interface, a vehicle camera and a trailer camera, a mobile device application (App) that can access the cameras and a mobile network, a vehicle-to-infrastructure communication network, a controller, a memory, a sensor, and a trailer detection and trajectory planning application. The application includes initializing data streams from a vehicle camera and a trailer camera, capturing the data streams from the vehicle camera and the trailer camera, determining key landmarks in the data streams, combining the key landmarks, determining a coupler position, calculating a joint reference trajectory using the coupler position, and outputting the joint reference trajectory. And providing navigation assistance in real time according to the joint reference trajectory.

Description

Technical Field

[0001] The present disclosure relates to advanced driver assistance systems (ADAS), and more particularly to driver assistance systems that detect a trailer coupler for coupling a trailer to a vehicle and provide trajectory planning and navigation assistance. Background Art

[0002] A carrier is typically used to tow a trailer to carry equipment, cargo, items, animals, other vehicles (including boats and off-road vehicles) from one location to another along a road. The carrier is equipped with a hitch that removably couples to a trailer coupler of the trailer. Coupling the trailer coupler to the carrier hitch can be difficult and requires backing up with limited operator visibility.

[0003] Therefore, trailer coupler identification systems have been developed to help locate the trailer coupler on the trailer. For example, some ADAS use electronic tags, markers, or IDs placed on the trailer coupler that are easily recognized by the ADAS's vehicle computer vision system. ADAS that assist in backing up are typically only effective at close ranges (e.g., <3 feet), require markers, and typically use only one camera in the computer vision system. Although current systems and methods for trailer coupler detection and navigation assistance achieve their intended purposes, there is a need for a system and method for trailer coupler detection and navigation assistance that improves the positioning accuracy of locating the trailer coupler on the trailer when backing up, reduces the computational burden, reduces component complexity, and reduces the difficulty of trailer coupling, thereby increasing operator confidence and satisfaction. Summary of the invention

[0004] According to several aspects of the present disclosure, a system for detecting a trailer coupler on a trailer to assist in connecting the trailer coupler to a vehicle hook is provided. The system includes at least one human-machine interface (HMI) disposed in a vehicle, a vehicle camera disposed on the vehicle and detecting the vehicle's surroundings, a trailer camera disposed on the trailer and detecting the trailer's surroundings, a vehicle-to-infrastructure (V2I) communication network, a mobile device application (App) that can access the camera and the mobile network, and one or more controllers. Each of the one or more controllers has a processor, a memory, and one or more input / output (I / O) ports that communicate with the vehicle camera and the trailer camera and the one or more HMIs via the V2I communication network. The memory stores instructions including a trailer detection and trajectory planning application (DTPA). The DTPA includes the following instructions: initialize data streams from a vehicle camera and a trailer camera, which can be a mobile device with a camera located in a trailer and running an OEM application; simultaneously capture data streams from the vehicle camera and the trailer camera; determine key landmarks in the data stream from the vehicle camera and the data stream from the trailer camera; combine the key landmarks; determine the position of the coupler relative to the vehicle hitch using the combined key landmarks; calculate a joint reference trajectory using the coupler position relative to the hitch position; and provide navigation assistance in real time based on the joint reference trajectory.

[0005] In another aspect of the present disclosure, the trailer camera is a cell phone camera that communicates with a mobile application.

[0006] In yet another aspect of the present disclosure, one or more controllers are located in the vehicle and at the trailer.

[0007] In yet another aspect of the present disclosure, one or more controllers are located at a remote computing system.

[0008] In yet another aspect of the present disclosure, the system includes instructions further requesting an operator to provide input to the HMI to confirm initialization of the DTPA.

[0009] In yet another aspect of the present disclosure, the system includes instructions for performing key landmark detection and spatial calculations on the captured data stream to determine a position of a trailer coupler in real time.

[0010] In yet another aspect of the present disclosure, the system further includes instructions for presenting to the HMI a navigation aid including at least one of a verbal or visual navigation aid to guide the vehicle hitch to the trailer coupler.

[0011] In yet another aspect of the present disclosure, the system further includes instructions for providing navigation assistance including automatic steering assistance.

[0012] In yet another aspect of the present disclosure, a method for detecting the position of a trailer coupler on a trailer and assisting in coupling the trailer coupler to a vehicle hitch on a vehicle is provided. The method includes initializing data streams from a vehicle camera and a trailer camera, capturing data streams from the vehicle camera and from the trailer camera, determining key landmarks in the data streams from the vehicle camera and from the trailer camera, combining the key landmarks, determining a coupler position relative to the vehicle hitch using the combined key landmarks, calculating a joint reference trajectory using the coupler position relative to the hitch position, and providing navigation assistance in real time based on the joint reference trajectory.

[0013] In yet another aspect of the present disclosure, the method further includes requesting an operator to provide input to the HMI.

[0014] In yet another aspect of the present disclosure, the method further includes providing a remote computing system, wherein the I / O port communicates with the remote computing system via a V2I communication network.

[0015] In yet another aspect of the present disclosure, the method further includes capturing the data stream while capturing the video streams from the vehicle camera and the trailer camera.

[0016] In yet another aspect of the present disclosure, the method includes presenting the navigation assistance in real time using one or more human-machine interfaces and presenting at least one of verbal or visual assistance.

[0017] In yet another aspect of the present disclosure, the method further includes presenting navigation assistance and providing steering assistance in real time.

[0018] In yet another aspect of the present disclosure, the method further includes determining, using a processor in the vehicle, key landmarks in the data stream from the vehicle camera and the data stream from the trailer camera.

[0019] In yet another aspect of the present disclosure, the method further includes determining, using a processor in the vehicle and a processor in the trailer, key landmarks in a data stream from the vehicle camera and a data stream from the trailer camera.

[0020] In yet another aspect of the present disclosure, the method further includes determining, using a controller in the remote computer, key landmarks in the data stream from the vehicle camera and the data stream from the trailer camera.

[0021] In yet another aspect of the present disclosure, the method includes the trailer camera being a mobile device camera located at a trailer coupler.

[0022] In yet another aspect of the present disclosure, a method for detecting the position of a trailer coupler on a trailer and assisting in coupling the trailer coupler to a vehicle hitch on a vehicle is provided. The method includes requesting an operator to provide input to an HMI, initializing data streams from a vehicle camera and a trailer camera, capturing data streams from the vehicle camera and the trailer camera, determining key landmarks in real time from the data stream from the vehicle camera and the data stream from the trailer camera, combining key landmarks determined based on the data stream from the vehicle camera and the data stream from the trailer camera, and determining a coupler position relative to the vehicle hitch using the combined key landmarks. The DTPA also includes control logic for actively and dynamically calculating a joint reference trajectory using the coupler position relative to the hitch position, and actively and continuously presenting navigation assistance in real time based on the joint reference trajectory, wherein the navigation assistance includes automatic steering assistance.

[0023] In yet another aspect of the present disclosure, the method includes navigation assistance with verbal and visual instructions displayed on one or more HMIs.

[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0026] Figure 1 is a schematic diagram of a system for detecting a trailer coupler on a trailer to assist in coupling the trailer coupler to a carrier hitch according to an exemplary embodiment;

[0027] Figure 2 is a perspective interior view of a vehicle according to an exemplary embodiment, including a human machine interface (HMI) displaying a view of the vehicle's surroundings, utilizing Figure 1 a system for detecting a trailer coupler on a trailer to assist in coupling the trailer coupler to a vehicle hitch;

[0028] Figure 3 is a method for using according to an exemplary embodiment Figure 1 A flowchart of a method for implementing a trailer detection and navigation assistance application in a system;

[0029] Figure 4 is a schematic diagram according to an exemplary embodiment, which shows that when a vehicle moves over time, using Figure 3 The camera poses for the method shown; and

[0030] Figure 5is a schematic diagram of images captured from a vehicle camera and a trailer camera for determining key landmarks used by a trailer detection and trajectory planning application (DTPA) according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0032] refer to Figure 1 , shows a schematic diagram of a system 10 for detecting a trailer coupler and planning a trajectory from a vehicle hitch to the trailer coupler. The system 10 includes a vehicle 12 and a trailer 14 that can be attached to the vehicle 12 via a hitch 16. Although the vehicle 12 is shown as a passenger car or a sport utility vehicle (SUV), it should be understood that the vehicle 12 can be any type of vehicle without departing from the scope or intent of the present disclosure. In some examples, the vehicle 12 can be a car, a truck, an SUV, a bus, a semi-tractor, a tractor used for agriculture or construction, etc., a push-behind tractor or tugboat, a luggage tractor or cargo tractor, etc. Likewise, although Figure 1 The trailer 14 shown is a single-axle trailer, but the trailer 14 can be any of a variety of trailer 14 types without departing from the scope or intent of the present disclosure. In some examples, the trailer 14 can be an enclosed or open-air trailer 14, including a flatbed trailer, a dry van, a refrigerated trailer, a low-box trailer, a fifth-wheel camper, a gooseneck trailer, a luggage or cargo trailer, etc. In other examples, the trailer 14 can be another vehicle 12 as described above, or a vehicle 12 such as an airplane, helicopter, or other such aircraft. That is, a vehicle 12 can tow another vehicle 12 as a trailer 14, and multiple trailers 14 can be towed together in a single file or parallel to each other.

[0033] The trailer 14 may be attached or mounted to the vehicle 12 via a hook 16 on the vehicle 12 and a trailer coupler 17 on the trailer 14. The hook 16 and trailer coupler 17 allow rotational movement of the trailer 14 relative to the vehicle 12, thereby providing the trailer 14 and vehicle 12 with a facility to turn while in motion. Figure 1 The hitch 16 shown is a ball hitch, however, it should be understood that other forms of hitch 16 may be used without departing from the scope or intent of the present disclosure. For example, the hitch 16 may be a ball hitch, a receiver hitch, a fifth wheel hitch, a gooseneck hitch, a pivot hitch, a bumper hitch, a weight distribution hitch, etc. In several aspects, the hitch 16 operates as a pivot. It is desirable that when attempting to couple a trailer to the vehicle to control the vehicle 12 so that when the vehicle 12 is reversed and backed toward the trailer 14, the vehicle 12 does not physically contact the trailer or obstacles in the environment 18 surrounding the vehicle 12 and the trailer 14.

[0034] The system 10 includes one or more controllers 20. The controller 20 is a non-general purpose electronic control device having a pre-programmed digital computer or processor 22, a non-transitory computer readable medium or memory 24 for storing data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and one or more input / output (I / O) ports 26. Computer readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. The non-transitory computer readable medium or memory 24 includes media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical disks or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code. The processor 22 is configured to execute code or instructions. In some examples, the controller 20 can be a dedicated Wi-Fi controller or an engine control module, a transmission control module, a body control module, an infotainment control module, etc. The I / O port 26 is configured to communicate via a wired or wireless connection using a Wi-Fi protocol under IEEE 802.11x, a Bluetooth communication protocol, a radio frequency (RF) protocol, etc. In some examples, the controller 20 also includes one or more applications 28, which are software programs configured to perform a specific function or set of functions. The application 28 may include one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or parts thereof adapted to be implemented in appropriate computer-readable program codes. The application 28 may be stored in the memory 24 or in an additional or separate memory. Examples of the application 28 include audio or video streaming services, games, browsers, social media, suspension and engine control programs, body control programs, advanced driver assistance systems (ADAS) programs, etc. In the specific application 28 of the present disclosure, the system 10 includes a trailer detection and trajectory planning application (trailer detection and trajectory planning, DTPA) 30.

[0035] The system 10 also includes one or more sensors 32. The sensors 32 generate real-time positioning and detection information about the position and / or movement of the vehicle 12. In the example provided, the sensors 32 include a vehicle camera 34 and a trailer camera 34'. The vehicle camera 34 is fixed to the rear of the vehicle 12, which has a field of view of the rear of the vehicle 12. Alternatively, the vehicle camera 34 may be located on the side mirror or rearview mirror of the vehicle 12. The trailer camera 34' may be located at the coupling 17 or other locations of the trailer 14. However, the sensor 32 may include any of a variety of different types of sensors 32, including but not limited to: light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, sound navigation and ranging (SONAR) sensors, ultrasonic sensors, or combinations thereof. In addition, the sensor 32 may have the ability to communicate with a global positioning system (GPS), and specifically, communicate with image data collected by satellites 33 in orbit around the earth to more accurately and precisely report the position of the vehicle 12. In further examples, the sensors 32 may include wheel speed sensors disposed on one or more of the vehicle 12 and the trailer 14 , or hitch angle estimation sensors that may be used in conjunction with a hitch angle estimation algorithm.

[0036] In another example, the sensor 32 may include an inertial measurement unit (IMU). The IMU uses a combination of some or all of the following to measure and report the pose or position, linear velocity, acceleration, and angular rate relative to a global reference system: an accelerometer, a gyroscope, and a magnetometer. In some examples, the IMU may also use global positioning system (GPS) data to indirectly measure the pose or position, velocity, acceleration, and angular rate. When used in the system 10 of the present disclosure, the IMU measures and reports the pose or position, linear velocity, acceleration, and angular rate of the vehicle 12 and the vehicle camera 34. The controller 20 and the sensor 32 may be linked to and communicate on one or more communication networks 40, including one or more of a vehicle-to-vehicle (V2V) communication network, a vehicle-to-infrastructure (V2I) communication network 44 (remote computing system), etc. For the sake of brevity, the term "infrastructure" is used to collectively refer to network entities capable of self-organizing wireless communications in direct links or side links, as well as cellular communication networks owned by service providers, so vehicle-to-network (V2N) communications are also intended to be included in the scope of V2I.

[0037] Reference now Figure 2 And continue to refer to Figure 1, the system 10 also includes a human machine interface (HMI) 46 disposed within the vehicle 12 to interact with an operator of the vehicle 12. In several aspects, the HMI 46 includes one or more devices capable of interacting with the operator, such as a screen disposed within the vehicle 12, such as an instrument panel 48, an infotainment screen 50, a head-up display (HUD) 52, an interior rear view screen (e.g., a rear view mirror enhanced by a screen 53), a sound transmission system, speakers, microphones, etc. However, it should be understood that other HMIs 46 are also contemplated herein. For example, the HMI 46 can be a mobile device, such as a tablet, mobile phone, cell phone, etc. that communicates with the DTPA and runs an original equipment manufacturer (OEM) application located at the trailer 14, and the HMI 46 can be provided by the operator and temporarily installed or disposed on an interior passenger compartment component of the vehicle 12. In several aspects, the HMI 46 communicates with the controller 20 via the I / O port 26 , and includes sending and receiving information to and from the I / O port 26 of the controller 20 during operation of the DTPA 30 .

[0038] For example, the DTPA 30 in combination with other vehicle control applications may assist the vehicle 12 operator in maneuvering the vehicle 12 by providing verbal (e.g., via a vehicle speaker) or visual instructions (e.g., via an HMI 46, HUD 52, etc.) to the operator, or by rotating the steering wheel 62 (e.g., automatic steering assistance) and directing the vehicle 12 toward the trailer 14 as the vehicle 12 moves toward the trailer 14 to attach, hitch, or couple the trailer 14 to the vehicle 12. The verbal and / or visual assistance may be configured by the operator in the HMI 46 or in an application in the vehicle 12 (e.g., OnStar) and / or in an application at the trailer 14 (e.g., a mobile phone using OnStar). More specifically, the DTPA 30 includes a plurality of subroutines or instructions that are stored in the memory 24 of the controller 20 and executed by the processor 22 while receiving data from sensors 32 (e.g., vehicle camera 34 and trailer camera 34', hitch angle estimation sensor, IMU, and / or satellites 33 reporting GPS data) via the I / O port 26. The DTPA 30 also includes a plurality of subroutines or instructions that enable data to be transmitted from the controller 20 to the HMI 46.

[0039] refer to Figure 3, a flow chart of a method 100 of a DTPA 30 according to the present disclosure is shown. The DTPA 30 is initialized or started at box 102 when one or more specific conditions occur. Specific conditions include manual initialization by an operator through the HMI 46 or operation of a physical button set inside the vehicle 12. In other examples, these conditions can be automatically satisfied by the sensor 32 and / or the vehicle camera 34 and the trailer camera 34' detecting that the trailer 14 is approaching but not coupled to the vehicle 12. The method 100 then proceeds to box 104.

[0040] At block 104, upon receiving confirmation at block 102 that the operator wishes to perform the DTPA 30, the controller 20 utilizes the HMI 46 to present a prompt to the operator to confirm that the operator wishes to perform the DTPA 30, or to request that the operator provide input to the HMI 46. The prompt may be displayed on the HMI 46, on a screen on the HUD 52, audibly through the vehicle 12 stereo system, etc. The method 100 then proceeds to block 105.

[0041] At block 105, a data stream is initiated and received by the controller 20 from the vehicle camera 34 and the trailer camera 34'. The data stream includes video data or image frames from video data, graphics data, or other data from the vehicle camera 34 and the trailer camera 34'. The controller receives the data streams from the vehicle camera 34 and the trailer camera 34' simultaneously and in real time. The method then proceeds to block 106.

[0042] At box 106, the DTPA 30 uses a visual simultaneous localization and mapping (vSLAM, Co-SLAM, Multi-Co-SLAM) algorithm that uses visual, video and / or optical data to capture data streams from the vehicle camera 34 and the trailer camera 34' and determine key landmarks. The environmental and key landmark scan data received from the vehicle camera 34 and the trailer camera 34' is processed in one or more image or video processing algorithms (e.g., vSLAM). In several aspects, the image or video processing algorithms may include one or more image capture algorithms that capture images of the trailer 14 and trailer coupler 17 from the video data stream and perform image distortion correction as needed to correct for camera 34, 34' lens shape, etc. Feature or landmark detection algorithms, such as a Canny edge algorithm, perform feature or landmark detection within the images of the trailer 14 and trailer coupler 17.

[0043] Continue to refer Figure 3 And now also refer to Figure 4, which shows exemplary camera poses as the vehicle 12 moves over time, the DTPA 30 uses, for example, a vSLAM algorithm to use data streams from the vehicle camera 34 and from the trailer camera 34' to determine and reconstruct structures and key landmarks in the environment 18. Although vSLAM is used throughout the disclosure, it should be understood that other image or video processing algorithms and / or visual simultaneous localization and mapping applications may be utilized. The vSLAM algorithm uses optical and / or video data received at box 105 to perform camera pose calculations. Key landmarks include landmarks (e.g., hook 16, coupler 17, etc.) that are relevant to determining the position of the vehicle 12 relative to the trailer 14. The environment 18 includes structures, key landmarks, and the surrounding environment from the viewpoint of the vehicle camera 34 and the trailer camera 34'.

[0044] refer to Figure 4 , shows a first camera pose 72, a second camera pose 74, and a third camera pose 76 as the vehicle 12 and cameras 34, 34' move over time T1, T2, T3. The camera poses 72, 74, 76 are captured from data streams from the vehicle camera 34 and the trailer camera 34'. The vSLAM algorithm provides an estimate of the position and orientation of the camera 34 on the vehicle 12 relative to the environment 18 and relative to the camera 34' on the trailer 14, while mapping the environment 18. The vehicle camera 34 and the trailer camera 34' capture images of the environment 18 so that the vSLAM algorithm can extract features and key landmarks, such as the trailer coupler 17 and the hook 16. The vSLAM algorithm also performs closed loop detection to recognize that the vehicle camera 34 and the trailer camera 34' are returning to a previously captured environment 18, and corrects for drift and camera poses in the map.

[0045] return Figure 3 Once the key landmarks are determined at box 106, the method proceeds to box 108 where the key landmarks from the data streams captured in box 105 are combined. The key landmarks determined at box 104, such as the hitch 16 and trailer coupler 17, are combined and mapped onto consecutive video frames to estimate the motion of the cameras 34, 34' and the 3D positions of the hitch 16 and trailer coupler 17 and / or other features in the environment 18. The camera poses 72, 74, 76 and 3D features captured by the cameras 34, 34' are combined to form a map of the environment 18, which is updated as the system 10 captures and processes new images in real time (e.g., actively and continuously).

[0046] For example, continue to refer to Figure 3 , and now refer to Figure 5, receiving optical or video data (e.g., images or camera poses from video frames) from a vehicle camera 34 and being shown with identified landmarks u, i. In addition, receiving optical or video data (e.g., images or camera poses from video frames) from a trailer camera 34' and being shown with identified landmarks v, k, j. Landmark u determined from the image from the vehicle camera 34 is mapped to landmark v determined from the image from the trailer camera 34'. Landmark i is not mapped to landmark k because the vSLAM algorithm determines that it is not feasible or possible to map landmark k to landmark i. For example, when a first landmark image (e.g., landmark i) does not match a second landmark image (e.g., landmark k), an infeasible landmark is determined. Instead, landmark j is mapped to landmark i, key landmarks u and v are mapped and combined, and key landmarks i and j are mapped and combined.

[0047] Back to Figure 3 At box 110, the controller 20 uses the combined key landmarks from box 108 to determine the position of the coupler 17 relative to the vehicle hook 16. By measuring the vehicle dynamics (α = pitch, β = roll, γ = yaw and its angular rotation) of the vehicle 12 moving over time, the position of the coupler 17 is estimated using the combined key landmarks in the X, Y, Z coordinate system. For example, the vSLAM algorithm is used to estimate the position of the coupler 17 for each time T1, T2 and T3. Figure 4 The trailer coupler 17 is positioned in each of the camera poses 72, 74, 76 shown in , so that the 3D position of the coupler in each image of each camera pose 72, 74, 76 is projected to the 2D image. Figure 4 As shown, given that both distance "a" and distance "b" are known, the position of the hook 16 relative to the camera 34' and the position of the coupler 17 relative to the vehicle camera 34 are determined using the calculated distances between the cameras 34, 34' and "p1" (e.g., the determined landmark). Distance "a" is the vertical height of the camera 34 in the vehicle, and distance "b" is the horizontal distance from the camera 34, 34' to p1. The method 100 then proceeds to block 112.

[0048] At block 112, DTPA 30 calculates an optimal joint reference trajectory RT using the position of coupler 17 determined at block 110. j Calculate the reference trajectory RT from the perspective of the vehicle camera 34 v , and calculates the reference trajectory RT from the perspective of the coupling 17 and the trailer camera 34' c Then, by using DTPA 30, the reference trajectory RT v Subtract the coupling reference trajectory RT c The absolute value of RT d = RTv -RT c │) to calculate the reference trajectory RT d (Using, for example, Euclidean distance, dynamic time warping algorithms, etc.) Then calculate the best reference trajectory RT from the hook 16 to the coupling 17 j , making RT d Minimum. Best joint reference trajectory RT j The calculation of can be repeated until the vehicle 12 is coupled to the trailer 14. The optimal joint reference trajectory RT j The DTPA 30 is used for calculations, with the controller 20 located in the vehicle 12 , trailer 14 , and / or network 40 , 44 (eg, a remote computer). The method 100 then proceeds to block 114 .

[0049] At block 114 , the DTPA 30 calculates the joint reference trajectory RT based on the joint reference trajectory RT calculated at block 112 . j The DTPA 30 provides navigation assistance in real time. The DTPA 30 provides navigation assistance to the operator in the form of verbal or visual instructions through the HMI 46 or HUD 52 to guide and steer the vehicle 12. For example, visual instructions indicating the direction to turn the steering wheel and the speed at which the operator should operate the vehicle 12 may be displayed on the HMI 46. In another aspect, the DTPA 30 provides navigation assistance by providing navigation instructions to an automatic or semi-automatic steering system having an actuator and directing the system to turn the steering wheel of the passenger vehicle. The method 100 then ends.

[0050] The disclosed system 10 and method 100 for detecting a trailer coupler 17 and thereby providing trajectory planning and navigation assistance provides a number of advantages over prior art systems. These include: detecting a coupler 17 from a range greater than 3 feet, eliminating the need for a coupler 17 with markings by using a trailer camera 34', and using bi-directional detection and trajectory planning, thereby improving the operator's vehicle and trailer coupling.

[0051] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A system for detecting a trailer coupler on a trailer to assist in coupling the trailer coupler to a vehicle hitch, the system comprising: One or more human-machine interfaces HMI, disposed in the vehicle; A vehicle camera, disposed on the vehicle and used to detect the environment around the vehicle; A trailer camera, disposed on the trailer and used to detect the environment around the trailer; Vehicle-to-infrastructure V2I communication network; One or more controllers, each of the one or more controllers having a processor, a memory and one or more input / output I / O ports, the I / O ports communicating with the vehicle camera and the trailer camera, and communicating with the one or more HMIs via the V2I communication network, the memory storing instructions including a trailer detection and trajectory planning application DTPA, which when executed by the processor causes the processor to perform the following operations: Initializing data streams from the vehicle camera and the trailer camera; capturing the data streams from the vehicle camera and the trailer camera simultaneously; determining key landmarks in the data stream from the vehicle camera and the data stream from the trailer camera; combining the key landmarks; determining a coupler position relative to the carrier hitch using the combined key landmarks; calculating a joint reference trajectory using said coupler position relative to the hook position; as well as Navigation assistance is provided in real time based on the joint reference trajectory.

2. The system according to claim 1, wherein: The trailer camera is a cell phone camera that communicates with the mobile application.

3. The system according to claim 1, wherein: The one or more controllers are located within the vehicle and at the trailer.

4. The system according to claim 1, wherein: The one or more controllers are located at a remote computing system.

5. The system of claim 1, further comprising instructions that, when executed by the processor, request an operator to provide input to the HMI to confirm initialization of the DTPA.

6. The system according to claim 1, wherein: Determining key landmarks in the data stream from the vehicle camera and the data stream from the trailer camera includes performing key landmark detection and spatial calculations on the captured data streams to determine the coupler position in real time.

7. The system of claim 1 further comprising instructions that, when executed by the processor, present navigation assistance including at least one of verbal or visual navigation assistance to the HMI to guide the vehicle hitch to the trailer coupler.

8. The system of claim 1 further comprising instructions that, when executed by the processor, provide automatic steering assistance.

9. A method for detecting a position of a trailer coupler on a trailer and facilitating coupling the trailer coupler to a vehicle hitch on a vehicle, the method comprising: Initialize data streams from vehicle camera and trailer camera; capturing the data streams from the vehicle camera and the trailer camera; determining key landmarks in the data stream from the vehicle camera and the data stream from the trailer camera; combining the key landmarks; determining a coupler position relative to the carrier hook using the combined key landmarks; calculating a joint reference trajectory using said coupler position relative to the hook position; as well as Navigation assistance is provided in real time according to the joint reference trajectory.

10. The method according to claim 9, further comprising: An operator is requested to provide input to the HMI.

Images