Method for detecting trailer coupling position using camera-based triangulation, optimization and filtering

Abstract

A method of detecting and utilizing a three-dimensional (3D) position of a coupler of a trailer may include calculating a motion of a camera mounted to a vehicle; determining a two-dimensional (2D) position of the coupler within the input image captured by the camera; estimating a 3D estimated position of the coupler based on the 2D position of the coupler and the motion of the camera; optimizing the 3D estimated position of the coupler to generate an adjusted 3D coupler position; and filtering the adjusted 3D coupler position to detect the 3D position of the coupler of the trailer.

Description

[0001] introduction The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent it is described in this section, and aspects of this description that may not otherwise qualify as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0002] The present disclosure relates to vehicles, and more particularly, to a method for detecting a position of a coupler of a trailer, an apparatus, and a computer-readable medium configured to perform the method.

[0003] The vehicle includes one or more torque-producing devices, such as an internal combustion engine and / or an electric motor.Occupants of the vehicle ride in a passenger cabin (or passenger compartment) of the vehicle.

[0004] The vehicle may include one or more different types of sensors that sense the vehicle's surroundings. One example of a sensor that senses the vehicle's surroundings is a camera that is configured to capture images of the vehicle's surroundings. Examples of such cameras include forward-facing cameras, rear-facing cameras, and side-facing cameras. Another example of a sensor that senses the vehicle's surroundings includes a radar sensor that is configured to capture information about the vehicle's surroundings. Other examples of sensors that sense the vehicle's surroundings include sonar sensors and light detection and ranging (LIDAR) sensors that are configured to capture information about the vehicle's surroundings.

[0005] The vehicle may include a trailer hitch mounted to a structural portion of the vehicle, such as the chassis / frame of the vehicle. The trailer hitch allows the vehicle to tow various types of trailers and items carried by the trailer by connecting the vehicle's trailer hitch to the trailer's coupler. For example, the vehicle may tow a boat on a boat trailer, a travel trailer, one or more bicycles, a utility vehicle, an off-road vehicle (ORV), etc.

[0006] Driver assistance systems have been developed to use the vehicle's sensors to assist the vehicle's operator (or alternatively, the driver) in towing a trailer. However, such systems may not assist the vehicle's operator in coupling the vehicle's trailer hitch to the trailer's coupler. Summary of the invention

[0007] At least some example embodiments relate to methods of detecting and utilizing a three-dimensional (3D) position of a coupler of a trailer.

[0008] In some example embodiments, the method includes: calculating the motion of a camera mounted to a vehicle; determining a two-dimensional (2D) position of a coupler within an input image captured by the camera; estimating a 3D estimated position of the coupler based on the 2D position of the coupler and the motion of the camera; optimizing the 3D estimated position of the coupler to generate an adjusted 3D coupler position; and filtering the adjusted 3D coupler position to detect the 3D position of the coupler of the trailer.

[0009] In some example embodiments, the method further includes: receiving a plurality of input images captured by a camera; and detecting whether the connector is present in one or more of the plurality of input images, wherein determining the 2D position of the connector is performed in response to detecting the presence of the connector in one or more of the plurality of input images.

[0010] In some example embodiments, detecting whether a joiner is present comprises performing object detection on each of the plurality of input images using a trained machine learning or deep learning model.

[0011] In some example embodiments, the method further includes performing supervised machine learning to generate the trained machine learning or deep learning model by iteratively applying a plurality of labeled test images having ground truth values ​​indicating the presence of a trailer coupler or the absence of a trailer coupler to the model to reduce an error between an output of the neural network and the ground truth values ​​of the plurality of labeled test images.

[0012] In some example embodiments, estimating the 3D estimated position of the coupler comprises triangulating 2D positions of the coupler in the plurality of input images captured by a camera mounted to the vehicle to estimate the 3D estimated position of the coupler.

[0013] In some example embodiments, the triangulation includes: calculating a first depth value extending between one of the 3D estimated positions of the coupler and a camera focus with respect to a first of the plurality of input images; calculating a second depth value extending between the one of the 3D estimated positions of the coupler and a camera focus with respect to a second of the plurality of input images; and estimating the one of the 3D estimated positions of the coupler by performing triangulation in 3D space using the first depth value and the second depth value.

[0014] In some example embodiments, the optimization includes: reprojecting the 3D estimated position of the connector back onto a corresponding input image in the input image to determine a reprojected 2D position; determining a reprojection error between a 2D position in the two-dimensional (2D) position of the connector within the corresponding input image in the input image and a corresponding 2D position in the reprojected 2D position; and adjusting the 3D estimated position of the connector to reduce the sum of the reprojection errors.

[0015] In some example embodiments, the filtering includes: averaging the adjusted 3D connector positions to generate an average 3D connector position; removing 3D connector positions in the adjusted 3D connector positions that are outside the defined radius from the average 3D connector position to generate remaining adjusted 3D connector positions; and averaging the remaining adjusted 3D connector positions to determine a final 3D connector position.

[0016] In some example embodiments, the method further comprises directing the vehicle such that a hitch ball of the vehicle approaches the coupler of the trailer based at least on the 3D position of the coupler.

[0017] In some example embodiments, directing includes controlling a graphical user interface (GUI) to output a display to an operator of the vehicle, the display illustrating at least a relative position between a hitch ball of the vehicle and a coupler of the trailer based on the 3D position of the coupler.

[0018] In some example embodiments, the control GUI further includes adjusting one or more of an angle, length, and color of a virtual guide line extending between the vehicle's hitch ball and the trailer's coupler as the relative position between the vehicle's hitch ball and the trailer's coupler changes.

[0019] In some example embodiments, the guidance includes: detecting a height of a coupler of a trailer based on a 3D position of the coupler; determining whether the height of the coupler corresponds to a height of a hitch ball of a vehicle; and in response to the height of the coupler not corresponding to the height of the hitch ball of the vehicle, instructing an operator of the vehicle to adjust the height of the coupler.

[0020] In some example embodiments, directing includes electronically repositioning the vehicle so that a hitch ball of the vehicle approaches a coupler of the trailer based on a 3D position of the coupler.

[0021] In some example embodiments, electronically repositioning includes controlling one or more of steering and braking of the vehicle while reversing the vehicle toward the coupler.

[0022] In some example embodiments, electronically repositioning further comprises instructing a vehicle control system to reverse the vehicle toward a coupler of the trailer based on the 3D position of the coupler while controlling one or more of steering and braking of the vehicle.

[0023] Other example embodiments relate to an electronic device.

[0024] In some example embodiments, the electronic device includes a memory; and a processing circuit configured to detect and utilize a three-dimensional (3D) position of a coupler of a trailer by: calculating the motion of a camera mounted to the vehicle; determining a two-dimensional (2D) position of the coupler within an input image captured by the camera; estimating a 3D estimated position of the coupler based on the 2D position of the coupler and the motion of the camera; optimizing the 3D estimated position of the coupler to generate an adjusted 3D coupler position; and filtering the adjusted 3D coupler position to detect the 3D position of the coupler.

[0025] In some example embodiments, the processing circuit is configured to estimate the 3D estimated position of the coupler by triangulating the 2D position of the coupler in a plurality of input images captured by a camera mounted to the vehicle to estimate the 3D estimated position of the coupler.

[0026] In some example embodiments, the processing circuit is configured to optimize the 3D estimated position by: reprojecting the 3D estimated position of the connector back onto a corresponding input image in the input image to determine a reprojected 2D position; determining a reprojection error between a 2D position in the two-dimensional (2D) position of the connector within the corresponding input image in the input image and a corresponding 2D position in the reprojected 2D position; and adjusting the 3D estimated position of the connector to reduce a sum of the reprojection errors.

[0027] In some example embodiments, the processing circuit is configured to filter the adjusted 3D coupler positions by averaging the adjusted 3D coupler positions to generate an average 3D coupler position; removing 3D coupler positions in the adjusted 3D coupler positions that are outside a defined radius from the average 3D coupler position to generate remaining adjusted 3D coupler positions; and averaging the remaining adjusted 3D coupler positions to determine a final 3D coupler position.

[0028] In some example embodiments, the electronic device is an electronic control unit (ECU) or a vehicle control unit (VCU).The present disclosure provides the following embodiments.

[0029] 1. A method for detecting and utilizing the three-dimensional (3D) position of a coupler of a trailer, comprising: calculating the motion of a camera mounted to the vehicle; determining a two-dimensional (2D) position of a connector within an input image captured by a camera; estimating a 3D estimated position of the connector based on the 2D position of the connector and the motion of the camera; optimizing the estimated 3D position of the coupler to generate an adjusted 3D coupler position; and The adjusted 3D coupler position is filtered to detect the 3D position of the coupler of the trailer.

[0030] 2. The method according to embodiment 1, further comprising: receiving a plurality of input images captured by a camera; and detecting whether a joiner is present in one or more of the plurality of input images, wherein Determining the 2D position of the joiner is performed in response to detecting that the joiner is present in one or more of the plurality of input images.

[0031] 3. The method of embodiment 2, wherein detecting whether the connector is present comprises: Object detection is performed on each of the plurality of input images using a trained machine learning or deep learning model.

[0032] 4. The method according to embodiment 3, further comprising: Supervised machine learning is performed to generate the trained machine learning or deep learning model by iteratively applying a plurality of labeled test images having ground truth values ​​indicating the presence of a trailer coupler or the absence of a trailer coupler to the model to reduce an error between an output of the neural network and the ground truth values ​​of the plurality of labeled test images.

[0033] 5. The method of embodiment 1, wherein estimating the 3D estimated position of the connector comprises: The 2D positions of the coupler in the plurality of input images captured by a camera mounted to the vehicle are triangulated to estimate a 3D estimated position of the coupler.

[0034] 6. The method of embodiment 5, wherein the triangulation comprises: Calculating a first depth value extending between one of the 3D estimated positions of the coupler and a camera focus with respect to a first of the plurality of input images, calculating a second depth value extending between the one of the 3D estimated positions of the coupler and a camera focus with respect to a second of the plurality of input images, and estimating the one of the 3D estimated positions of the coupler by performing triangulation in 3D space using the first depth value and the second depth value.

[0035] 7. The method of embodiment 1, wherein the optimizing comprises: reprojecting the 3D estimated positions of the connectors back onto corresponding ones of the input images to determine reprojected 2D positions; determining a reprojection error between two-dimensional (2D) positions of a coupler within the corresponding input image in the input image and corresponding 2D positions in the reprojected 2D positions; and The estimated 3D position of the coupler is adjusted to reduce the sum of the reprojection errors.

[0036] 8. The method of embodiment 1, wherein the filtering comprises: averaging the adjusted 3D coupler positions to generate an average 3D coupler position; removing 3D coupler positions of the adjusted 3D coupler positions that are outside the defined radius from the average 3D coupler position to generate remaining adjusted 3D coupler positions; and The remaining adjusted 3D coupler positions are averaged to determine a final 3D coupler position.

[0037] 9. The method according to embodiment 1, further comprising: The vehicle is guided based on at least the 3D position of the coupler such that the hitch ball of the vehicle approaches the coupler of the trailer.

[0038] 10. The method of embodiment 9, wherein the directing comprises: A graphical user interface (GUI) is controlled to output a display to an operator of the vehicle that illustrates at least the relative position between the hitch ball of the vehicle and the coupler of the trailer based on the 3D position of the coupler.

[0039] 11. The method of embodiment 10, wherein controlling the GUI further comprises: As the relative position between the hitch ball of the vehicle and the coupler of the trailer changes, one or more of an angle, a length, and a color of a virtual guide line extending between the hitch ball of the vehicle and the coupler of the trailer is adjusted.

[0040] 12. The method of embodiment 11, wherein the directing comprises: detecting a height of a coupler of the trailer based on the 3D position of the coupler; determining whether the height of the coupler corresponds to the height of the hitch ball of the vehicle; and In response to the height of the coupler not corresponding to the height of the hitch ball of the vehicle, an operator of the vehicle is instructed to adjust the height of the coupler.

[0041] 13. The method of embodiment 9, wherein the directing comprises: The vehicle is electronically repositioned so that the vehicle's hitch ball approaches the trailer's coupler based on the coupler's 3D position.

[0042] 14. The method of embodiment 13, wherein electronically repositioning comprises: One or more of steering and braking of the vehicle is controlled while the vehicle is reversed toward the coupler.

[0043] 15. The method of embodiment 14, wherein electronically repositioning further comprises: The vehicle control system is instructed to reverse the vehicle toward the coupler of the trailer based on the 3D position of the coupler while controlling one or more of steering and braking of the vehicle.

[0044] 16. An electronic device, comprising: Memory; and Processing circuitry configured to detect and utilize a three-dimensional (3D) position of a coupler of a trailer by: Calculate the motion of a camera mounted to the vehicle, determining the two-dimensional (2D) position of the connector within an input image captured by a camera, Estimate the 3D estimated position of the coupler based on the coupler’s 2D position and the camera’s motion, optimizing the estimated 3D position of the coupler to generate an adjusted 3D coupler position, and The adjusted 3D coupler position is filtered to detect the 3D position of the coupler of the trailer.

[0045] 17. An electronic device according to embodiment 16, wherein the processing circuit is configured to estimate the 3D estimated position of the connector by triangulating the 2D position of the connector in multiple input images captured by a camera mounted to the vehicle to estimate the 3D estimated position of the connector.

[0046] 18. The electronic device of embodiment 16, wherein the processing circuit is configured to optimize the 3D estimated position by: reprojecting the 3D estimated positions of the connectors back onto corresponding ones of the input images to determine reprojected 2D positions, determining a reprojection error between a two-dimensional (2D) position of a coupler within the corresponding input image in the input image and a corresponding 2D position in the reprojected 2D positions, and The estimated 3D position of the coupler is adjusted to reduce the sum of the reprojection errors.

[0047] 19. The electronic device of embodiment 16, wherein the processing circuit is configured to filter the adjusted 3D coupler position by: averaging the adjusted 3D coupler positions to generate an average 3D coupler position, removing 3D coupler positions of the adjusted 3D coupler positions that are outside the defined radius from the average 3D coupler position to generate remaining adjusted 3D coupler positions, and The remaining adjusted 3D coupler positions are averaged to determine a final 3D coupler position.

[0048] 20. The electronic device of embodiment 16, wherein the electronic device is an electronic control unit (ECU) or a vehicle control unit (VCU).

[0049] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present disclosure will become more fully understood from this detailed description and the accompanying drawings, in which: Figure 1 illustrates a functional block diagram of a vehicle system according to some example embodiments; Figure 2 is a diagram of a functional control system for detecting trailer coupler position according to an example embodiment; Figure 3 is a diagram of a method of detecting a trailer coupler position according to some example embodiments; Figure 4 is a diagram of a method of estimating 3D coupler positions based on triangulation according to some example embodiments; Figure 5A and 5B illustrates performing triangulation to estimate 3D coupler positions according to some example embodiments; Figure 6 illustrates a method of optimizing the 3D position of a trailer coupler according to some example embodiments; and Figure 7 illustrates an example of performing an optimization on a 3D position of a trailer coupler according to some example embodiments; Figure 8 illustrates an example of a method of filtering 3D coupler positions according to some example embodiments; Fig. 9 illustrates an example of performing filtering on a 3D position of a trailer coupler according to some example embodiments; Fig.10 illustrates experimental results of final coupler positions captured at different depths as a vehicle approaches a trailer, according to some example embodiments; Fig.11 illustrates Hitch Guidance according to some example embodiments; and Fig.12 Methods of automatically aligning a vehicle to a final 3D coupler position are illustrated according to some example embodiments.

[0051] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0052] Various types of vehicles can be used to tow trailers, such as cars, hatchbacks, utility vehicles, trucks, etc. The vehicle can include a trailer hitch mounted to the vehicle structure. A towing drawbar can be inserted into the hitch, and a ball can be attached to the end of the drawbar to enable attachment of a coupler of a trailer for towing by the vehicle.

[0053] The present application relates to detecting trailer coupler position. The position of a trailer coupler can be used to assist an operator in attaching a trailer hitch (eg, a trailer hitch ball) to a coupler of a trailer.

[0054] Figure 1 A functional block diagram of a vehicle system is illustrated, according to some example embodiments.

[0055] Reference now Figure 1 , a functional block diagram of an example vehicle system is presented. Although a vehicle system for a hybrid vehicle is shown and will be described, the present application is also applicable to non-hybrid vehicles, electric vehicles, fuel cell vehicles, and other types of vehicles. The present application is applicable to autonomous vehicles, semi-autonomous vehicles, non-autonomous vehicles, shared vehicles, non-shared vehicles, and other types of vehicles.

[0056] The engine 102 may combust an air / fuel mixture to generate drive torque. An engine control module (ECM) 106 controls the engine 102. For example, the ECM 106 may control actuation of engine actuators such as a throttle, one or more spark plugs, one or more fuel injectors, a valve actuator, a camshaft phaser, an exhaust gas recirculation (EGR) valve, one or more boost devices, and other suitable engine actuators. In some types of vehicles (e.g., electric vehicles), the engine 102 may be omitted.

[0057] The engine 102 may output torque to the transmission 110. A transmission control module (TCM) 114 controls operation of the transmission 110. For example, the TCM 114 may control gear selection within the transmission 110 and one or more torque transmitting devices (eg, a torque converter, one or more clutches, etc.).

[0058] The vehicle system may include one or more electric motors. For example, electric motor 118 may be implemented within transmission 110, such as Figure 1 As shown in the example of . At a given time, the electric motor can act as both a generator and a motor. When acting as a generator, the electric motor converts mechanical energy into electrical energy. The electrical energy can be used, for example, to charge the battery 126 via the power control device (PCD) 130. When acting as a motor, the electric motor generates torque that can be used, for example, to supplement or replace the torque output by the engine 102. Although an example of an electric motor is provided, the vehicle can include zero or more than one electric motor.

[0059] A power inverter module (PIM) 134 may control the electric motor 118 and the PCD 130. The PCD 130 applies power from the battery 126 to the electric motor 118 based on a signal from the PIM 134, and the PCD 130 provides the power output by the electric motor 118 to, for example, the battery 126. The PIM 134 may include, for example, an inverter.

[0060] The steering control module 140 controls the steering / rotation of the vehicle's wheels, for example, based on a driver turning a steering wheel within the vehicle and / or steering commands from one or more vehicle control modules. A steering wheel angle (SWA) sensor (not shown) monitors the rotational position of the steering wheel and generates SWA 142 based on the position of the steering wheel. As an example, the steering control module 140 can control vehicle steering via an electronic power steering (EPS) motor 144 based on the SWA 142. However, the vehicle may include another type of steering system.

[0061] The brake control module 150 can selectively control (e.g., rub) brakes 154 of the vehicle based on one or more driver inputs, such as a brake pedal position (BPP) 170. The damper control module 156 controls the damping of dampers 158 of the wheels of the vehicle respectively. The dampers 158 damp the vertical movement of the wheels. The damper control module 156 can control, for example, the damping coefficient of the dampers 158 respectively. For example, the dampers 158 can include magnetorheological dampers, continuous damping control dampers, or another suitable type of adjustable dampers. The dampers 158 include actuators 160 that adjust the damping of the dampers 158 respectively. In the example of magnetorheological dampers, the actuators 160 can adjust the magnetic field of the magnetorheological fluid applied to the dampers 158 respectively to adjust the damping.

[0062] The modules of the vehicle can share parameters via a network 162 such as a controller area network (CAN). CAN may also be referred to as a car area network. For example, the network 162 may include one or more data buses. Various parameters may be made available to other modules via the network 162 by a given module.

[0063] The driver input may include, for example, an accelerator pedal position (APP) 166, which may be provided to the ECM 106. The BPP 170 may be provided to the brake control module 150. The position 174 of the park, reverse, neutral, drive lever (PRNDL) may be provided to the TCM 114. The ignition state 178 may be provided to the body control module (BCM) 180. For example, the ignition state 178 may be input by the driver via an ignition key, button, or switch. At a given time, the ignition state 178 may be one of off, accessory, run, or crank.

[0064] The infotainment module 183 may output various information via one or more output devices 184. The output devices 184 may include, for example, one or more displays (non-touch screen and / or touch screen), one or more other suitable types of video output devices, one or more speakers, one or more haptic devices, and / or one or more other suitable types of output devices.

[0065] The infotainment module 183 may output video via one or more displays. The infotainment module 183 may output audio via one or more speakers. The infotainment module 183 may output other feedback via one or more haptic devices. For example, a haptic device may be included in one or more seats, one or more seat belts, a steering wheel, etc. Examples of displays may include, for example, one or more displays of a vehicle (e.g., on a front console), a head-up display (HUD) that displays information via a substrate (e.g., a windshield), one or more displays that drop down or extend upward to form a panoramic view, and / or one or more other suitable displays.

[0066] The vehicle may include a plurality of external sensors and cameras 186. One or more actions may be taken based on input from the external sensors and cameras 186. For example, during driving, the infotainment module 183 may display video, various views, and / or alerts on a display via input from the external sensors and cameras 186. The cameras 186 may include a rear-facing camera.

[0067] The trailer hitch and ball can be secured to the vehicle, such as via bolts to the frame, to accommodate the mounting of the trailer coupler to the vehicle ball. When the vehicle's trailer hitch and ball are coupled to a trailer, the rear-facing camera 186 can be configured to capture an image frame of the trailer, including the trailer's coupler.

[0068] The vehicle may include one or more additional control modules not shown, such as a chassis control module, a battery pack control module, etc. The vehicle may omit one or more of the control modules shown and discussed. In addition, the vehicle may include one or more other types of sensors.

[0069] Figure 2 is a diagram of a functional control system for detecting a trailer coupler position according to an example embodiment, and Figure 3 is a diagram of a method of detecting a trailer coupler position, according to some example embodiments.

[0070] refer to Figure 2 and 3 The method for detecting the position of a trailer coupler may be initiated manually, for example, by an operator of the vehicle selecting hitch assist on an infotainment module, or may be initiated automatically when the vehicle is placed in reverse.

[0071] In operation S310, once the method is initiated, the image feed generator 210 may continuously receive image frames I1-I n For example, the image feed generator 210 may receive and buffer images from a camera 186 mounted to the rear of the vehicle. The image frames provided by the image feed generator 210 may include pairs of consecutive image frames I1 and I2.

[0072] In operation S320 , the camera position calculator 220 may analyze the image frames I1 and I2 and calculate the three-dimensional (3D) position of the camera to determine the camera translation t and the camera rotation R. For example, the camera translation t and the camera rotation R may be obtained from vehicle movement or simultaneous positioning and mapping techniques.

[0073] In operation S330, the coupler position estimator 230 may determine a two-dimensional (2D) position p1, p2 of a trailer coupler within an image provided by the camera. For example, the coupler position estimator 230 may perform object detection on the image frames I1 and I2 to extract the two-dimensional (2D) position p1, p2 of the trailer coupler in the corresponding image frames of the image frames I1 and I2. The 2D position p1, p2 of the trailer coupler may be one or more pixel coordinates of the trailer coupler in the corresponding image frames of the image frames I1 and I2. For example, the pixel position of the center of the trailer coupler in the vertical and horizontal axes.

[0074] Object detection can be achieved through machine learning or deep learning methods (e.g., supervised machine learning methods). Generally speaking, a machine learning or deep learning model is trained offline, during which the model is iteratively trained using multiple trailer coupler pictures with labeled coupler locations in the image. Although it depends on the exact architecture selected, in general, a neural network consists of multiple layers, each connected to components within the next layer, called neurons. During training, the placement and strength of these logical connections are adjusted based on the error between the output of the neural network in training when compared to the ground truth corresponding to the tested image. In this application, the network will initially see the difference between the estimated 2D pixel coordinates of (one or more) of the trailer coupler and the correct ground truth provided using the test image. After appropriate training iterations, the model can identify the coupler location in a given image with high accuracy. The trained model is then used to identify the coupler.

[0075] In operation S340, the 3D coupler position estimator 240 may estimate the 3D position P of the coupler based on the image frames I1 and I2, the 2D coupler positions p1, p2, and the camera translation t and the camera rotation R. Figure 4 and 5 discuss the 3D estimation of operation S340 in more detail.

[0076] In operation S350, the optimization module 250 may perform optimization on the estimated 3D connector position P to reduce the reprojection error therein, thereby generating an optimized 3D connector position P'. Figure 6 and 7 The optimization of operation S350 is discussed in more detail.

[0077] In operation S360, the filtering module 260 may filter the optimized 3D connector position P' to generate a final 3D connector position Pf. Figure 8 and 9 The filtering of operation S360 is discussed in more detail.

[0078] In operation S370, in some example embodiments, as described below with reference to Fig.11 As discussed, the hitch guidance or automatic hitch module 270 can guide the operator of the vehicle to approach the final 3D coupler position Pf. In other example embodiments, as described below with reference to Fig.12 As discussed, the hitch guidance or automatic hitch module 270 may automatically control the vehicle to approach the final 3D coupler position Pf. However, example embodiments are not limited thereto, and the vehicle may perform other functions based on the final 2D coupler position Pf.

[0079] refer to Figure 2 The modules described may be implemented using hardware, software, and / or a combination thereof. For example, a hardware device may be implemented using a processing circuit such as, but not limited to, a processor, a central processing unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, or any other device capable of responding to instructions and executing instructions in a defined manner.

[0080] The processing circuit can execute program code (e.g., computer readable instructions) and data from the memory, thereby converting the processing circuit into a dedicated processor to perform the functions of the above modules. In addition, although the above modules are used to describe the above functions to increase the clarity of this description, the processing circuit is not intended to be limited to these functional units. For example, in one or more example embodiments, various operations and / or functions of the functional units can be performed by other functional units in the functional unit. In addition, the processing circuit can perform the operations and / or functions of various functional units without subdividing the operations and / or functions of the computer processing unit into these various functional units.

[0081] Figure 4 is a diagram of a method of estimating 3D coupler positions based on triangulation, according to some example embodiments.

[0082] refer to Figure 4 In operation S410 , the 3D coupler position estimator 240 may receive an input image frame I from the image feed generator 210 .

[0083] In operation S420, the 3D coupler position estimator 240 may determine whether a reference frame exists. For example, the 3D coupler position estimator 240 may determine whether a previous image frame is stored in a buffer. The buffer may be included in a memory included in a control module of the vehicle.

[0084] In operation S430 , if the reference frame does not exist, the 3D coupler position estimator 240 may define the received image frame I as a reference image frame I1 and wait until the next image frame is received.

[0085] In operation S440, if the reference frame already exists, the 3D coupler position estimator 240 may receive the 2D positions p1, p2 of the trailer coupler from the reference image I1 and the next image frame I2. For example, the 3D coupler position estimator 240 may receive the two consecutive image frames I1 and I2 received in operation S310, the rotation R and translation t of the camera determined in operation S320, and the 2D coupler positions p1, p2 determined in operation S330, and may determine the 3D position of the trailer coupler by performing triangulation using the data identified above. More specifically, given the images I1 and I2 viewing the same trailer coupler from two known positions, the 3D coupler position estimator 240 may determine the 3D coupler position by calculating the depth of the 2D position p1 or p2 of the trailer coupler.

[0086] Figure 5A and 5B An example of performing triangulation to estimate a 3D coupler position according to some example embodiments is illustrated.

[0087] refer to Figure 5A , the 3D coupler position P may be estimated based on the 2D coupler positions p1 , p2 of the coupler within corresponding ones of the image frames I1 and I2 .

[0088] The 3D connector position estimator 240 may treat the first image frame I1 and the second image frame I2 as planes, and may map the known positions p1, p2 of the 2D connector points into the normalized image planes. Figure 5AAs shown in the figure, P is the 3D position of the connector, I1 is the first image frame, I2 is the second image frame, O1 is the camera focus at the first image frame, O2 is the camera focus at the second image frame, p1 and p2 are the 2D pixel positions of the connector in the first and second frame images respectively, and t and R are the translation and rotation of the camera position between the first frame image and the second frame image. s1 and s2 are unknowns representing the distances from P to O1 and O2 respectively. u1, v1 and u2, v2 are the pixel positions in the first image frame I1 and the second image frame I2 respectively, and X, Y and Z are the actual 3D positions in the three dimensions with respect to the camera. The 3D connector position estimator 240 can determine the unknown distances s1 and s2 by triangulation as follows.

[0089] For example, the position p1 of the 3D connector point in the plane of the first image frame I1, the position p2 of the 3D connector point in the plane of the second image frame I2, and the estimated 3D connector position P can be expressed as: The 3D connector position estimator 240 can project the 3D connector position P onto the image frames I1 and I2 to determine Equation 2: Zp1 = KP and Zp1 = K(RP + t) (1)(2) The 3D connector position estimator 240 can define the 2D of the connector point in each plane in the normalized image plane as x1 and x2 as follows: Some variables in Equations 1 - 3 can be known. For example, the 3D connector position estimator 240 can determine the known variables p1 and p2 from the 2D connector positions, K from the input intrinsic properties of the camera, the rotation R and translation t of the camera in 3D space.

[0090] The 3D connector position estimator 240 can obtain Equation 4 by performing the rotation R and translation t of the 2D connector position in the normalized image plane from I1 to I2 as follows: s1x1 = s2Rx2 + t (4) In Equation 4, t and R, x1 and x2 are all known variables, and thus, the 3D connector position estimator 240 can either calculate the distance s1 from the camera focus O1 at the first image frame to the 3D position P of the connector or calculate the distance s2 from the camera focus O2 at the second image frame to the 3D position P of the connector.

[0091] For example, to calculate the distance s2 from the camera focus O2 at the second image frame to the 3D position of the coupler, the 3D coupler position estimator 240 may convert Equation 4 into Equation 5 and solve for the distance s2 as follows: s2x1^Rx2+x1^t=0 (5) In Equation 5, ^ is the outer product of the two matrices. Once the distance s1 or s2 is determined, the 3D coupler position estimator 240 may generate an estimate of the 3D coupler position P based on the 2D position and the determined distance.

[0092] refer to Figure 5B , the 3D joint position estimator 240 may continuously perform triangulation when a new image frame is received. The new image frame and the previous frame (which are in Figure 5A triangulation is performed using image frames I1 and I2).

[0093] Reference Figure 3 As discussed above, in operation S350, the optimization module 250 may perform optimization on the estimated 3D coupler position P to reduce the reprojection error therein, thereby generating an optimized 3D coupler position P'.

[0094] Figure 6 illustrates a method of optimizing the 3D position of a trailer coupler according to some example embodiments, and Figure 7 An example of performing an optimization on a 3D position of a trailer coupler is illustrated in accordance with some example embodiments.

[0095] refer to Figure 6 and 7 In operation S610, the optimization module 250 may receive the estimated 3D coupler position P, the first and second image frames I1 and I2, the 2D pixel positions p1 and p2 of the coupler position, and the rotation R and translation t of the camera in the 3D space from, for example, a buffer. The buffer may be included in a memory included in a control module of the vehicle.

[0096] In operation S620, the optimization module 250 may reproject the estimate of the 3D connector position P back to the plurality of buffered image frames I1-I n Focus on O1-O n To determine p'1-p' n The optimization module 250 can determine the p'1-p' n The reprojected 2D position is consistent with the connector position p1-p determined by the connector position estimator 230 n The stored 2D pixel positions are compared with the corresponding 2D pixel positions, and the reprojection error E1-E between them can be calculated. n.

[0097] In operation S630, the optimization module 250 may aim to reduce (or alternatively, minimize) the total reprojection error E by adjusting the 3D coupler position P estimated by the optimization module 250. total , where the total reprojection error E total can be expressed as follows: E total =E1+E2+E3+...E n (6) Reference Figure 3 As discussed above, in operation S360 , the filtering module 260 may filter the optimized 3D coupler position P′ to generate a final 3D coupler position Pf.

[0098] Figure 8 illustrates an example of a method of filtering 3D connector positions according to some example embodiments, and Fig. 9 An example of performing filtering on a 3D position of a trailer coupler is illustrated in accordance with some example embodiments.

[0099] refer to Figure 8 and 9 In operation S810 , the filtering module 260 may receive the buffered optimized 3D position P′.

[0100] In operation S820, the filtering module 260 may average the buffered optimized 3D coupler positions P'. The filtering module 260 may average the buffered optimized 3D coupler positions by adding the buffered optimized 3D coupler positions in three axes and then dividing by the total number of optimized 3D coupler positions in the buffer.

[0101] In operation S830, the filtering module 260 may remove the optimized 3D positions outside the defined radius R from the average 3D coupler position. For example, the filtering module 260 may define the radius R as one quarter of the distance between the average 3D coupler position and the outermost optimized 3D position. However, example embodiments are not limited thereto.

[0102] Thereafter, in operation S840 , the filtering module 260 may average the remaining points within the defined radius R to determine a final 3D coupler position Pf.

[0103] Reference Figure 3 As discussed above, in operation S370, the attachment guidance or automatic attachment module 270 may utilize the final 3D coupler position Pf to perform one or more operations. For example, as shown below with reference to Fig.10As discussed, in some example embodiments, the hitch guidance or automatic hitch module 270 can guide the operator of the vehicle to approach the final 3D coupler position Pf. In other example embodiments, as described below with reference to Fig.12 As discussed, the hitch guidance or automatic hitch module 270 may automatically align the vehicle to the final 3D coupler position Pf.

[0104] Fig.10 Illustrated are experimental results of final coupler positions captured at different depths as a vehicle approaches a trailer, according to some example embodiments.

[0105] refer to Fig.10 As the vehicle approaches the trailer, the electronic device can continuously determine a final 3D coupler position Pf, which can be used to determine (or alternatively, communicate) a change between one or more of the depth, height, and angle between the trailer coupler and the vehicle's hitch ball.

[0106] Fig.11 Hook booting is illustrated according to some example embodiments.

[0107] refer to Fig.11 , the hitch guidance or automatic hitch module 270 can control the vehicle's infotainment module 183 to guide the operator based on the final 3D coupler position Pf.

[0108] For example, the hitch guidance or automatic hitch module 270 may determine the position of the hitch ball on the vehicle and may guide the operator of the vehicle using the infotainment module 183 to align the hitch ball position with the final 3D coupler position Pf.

[0109] In some example embodiments, the hitch guidance or automatic hitch module 270 may determine the location of the hitch ball on the vehicle by preset parameters input into the infotainment module 183 by, for example, an operator, a dealer, or a factory. In other example embodiments, the hitch guidance or automatic hitch module 270 may determine the location of the hitch ball on the vehicle by using a camera to identify the location of the hitch ball on the vehicle.

[0110] In some example embodiments, the hitch guidance or automatic hitch module 270 may guide the operator of the vehicle to align the position of the hitch ball with the final 3D coupler position Pf.

[0111] In some example embodiments, the hitch guidance or automatic hitch module 270 may display a visual element (such as a simple circle) at the final 3D coupler position Pf on the infotainment module 183 to enhance visibility. In some example embodiments, the hitch guidance or automatic hitch module 270 may also display the distance between the hitch ball and the final 3D coupler position Pf via a visual element (such as a measuring progress bar and / or graphics of the vehicle and / or trailer) to convey how far the customer still needs to guide the vehicle to connect the hitch ball and coupler at the final 3D coupler position Pf.

[0112] In some example embodiments, the hitch guidance or automatic hitch module 270 may display a virtual guide line from the position of the hitch ball to the final 3D coupler position Pf on the infotainment module 183. In some example embodiments, the virtual guide line from the position of the hitch ball to the final 3D coupler position Pf may change color and / or appearance as the user turns the steering wheel to the correct angle to align the position of the hitch ball and the final 3D coupler position Pf. In some example embodiments, the virtual guide line may change in length and angle so that when the steering wheel is turned to align the vehicle at the correct angle, the virtual guide line terminates at the intersection of the coupler rather than being drawn on an area of ​​the image containing the trailer.

[0113] In some example embodiments, the hitch guidance or automatic hitch module 270 may detect the height of the trailer's coupler from the ground based on the coupler's 3D position and determine whether the coupler's height sufficiently corresponds to the height of the vehicle's hitch ball. The hitch guidance or automatic hitch module 270 may display a visual element (such as a checkmark, plain text, or graphic of the trailer coupler and hitch ball) on the infotainment module 183 to convey that the ball is at a sufficiently low height so as not to collide with the trailer coupler or the trailer.

[0114] In some example embodiments, the hitch guidance or automatic hitch module 270 may output visual elements in the form of beeps, audio alerts, and / or tactile feedback to notify the operator that they are approaching the trailer, have achieved good alignment, and / or are about to collide with the trailer.

[0115] As discussed above, in other example embodiments, rather than guiding an operator, the hitch guidance or automatic hitch module 270 may automatically align the vehicle with the final 3D coupler position Pf.

[0116] Fig.12 Methods of automatically aligning a vehicle to a final 3D coupler position are illustrated according to some example embodiments.

[0117] refer to Fig.12In operation S1210, the hitch guidance or automatic hitch module 270 may receive a camera feed and various vehicle and camera parameters. For example, the hitch guidance or automatic hitch module 270 may receive camera intrinsic parameters and a relative position between the camera and a hitch ball on the vehicle.

[0118] In operation S1220 , the hitch guidance or automatic hitch module 270 may calculate a control input to a vehicle control unit for controlling the vehicle based on the parameters.

[0119] In operation S1230, the hitch guidance or automatic hitch module 270 may deliver the calculated control input to the vehicle controller to instruct the vehicle controller to move the vehicle. For example, the hitch guidance or automatic hitch module 270 may deliver the calculated control input to one or more of the electronic control module (ECM) 106, the power inverter module (PIM) 134, and the steering control module 140. In some example embodiments, when the operator manually operates the vehicle to slowly reverse the vehicle toward the coupling of the trailer, the hitch guidance or automatic hitch module 270 may control one or more of steering and braking. In other example embodiments, the hitch guidance or automatic hitch module 270 may also control the vehicle while controlling one or more of steering and braking to automatically reverse the vehicle.

[0120] In operation 1240, the hitch guidance or automatic hitch module 270 may continue to monitor the relative position of the vehicle trailer hitch (or alternatively, the hitch ball) and the trailer's coupler to determine whether the vehicle trailer hitch (or alternatively, the hitch ball) has sufficiently reached the final coupler position Pf.

[0121] If the hitch guidance or automatic hitch module 270 determines that the vehicle trailer hitch (or alternatively, the hitch ball) has not fully reached the final 3D coupler position Pf, the hitch guidance or automatic hitch module 270 may proceed back to operation S1230 and continue delivering control inputs.

[0122] In operation 1260 , in response to determining that the vehicle trailer hitch (or alternatively, hitch ball) has sufficiently reached the final coupler position Pf, the hitch guidance or automatic hitch module 270 may instruct one or more of the ECM 106 , the PIM 134 , and the steering control module 140 to stop moving the vehicle.

[0123] As discussed above, example embodiments may detect a final 3D coupler position Pf of a trailer's coupler and may guide a vehicle operator to drive their vehicle to approach the final 3D coupler position Pf and / or may automatically align the vehicle with the final 3D coupler position Pf. Such assistance may be provided, for example, upon request and / or automatically when the vehicle is set to reverse and the trailer's coupler is detected in an image frame captured by the vehicle's rear-mounted camera. Thus, example embodiments may assist an operator in coupling a trailer hitch of their vehicle to a trailer's coupler.

[0124] The foregoing description is merely illustrative in nature and is by no means intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in a variety of forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because other modifications will become apparent when studying the drawings, the specification and the following claims. It should be understood that, without changing the principles of the present disclosure, one or more steps in the method can be performed in different orders (or simultaneously). In addition, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in the features of any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and one or more embodiments and each other's permutation (permutation) are still within the scope of the present disclosure.

[0125] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "immediately adjacent," "on," "above," "below," and "disposed." Unless explicitly described as "direct," when describing a relationship between a first and a second element in the above disclosure, the relationship may be a direct relationship in which no other intervening elements exist between the first and second elements, but may also be an indirect relationship in which one or more intervening elements (either spatially or functionally) exist between the first and second elements. As used herein, the phrase "at least one of A, B, and C" should be interpreted to mean a logical (A or B or C) using a non-exclusive logical "or," and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0126] In the various figures, the direction of the arrow, as indicated by the arrow head, generally indicates the flow of information (such as data or instructions) of interest to the illustration. For example, when element A and element B exchange various information, but the information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. The unidirectional arrow does not imply that no other information is transmitted from element B to element A. In addition, for information sent from element A to element B, element B may send a request or a receipt confirmation of the information to element A.

[0127] In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or grouped) that executes code; a memory circuit (shared, dedicated, or grouped) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0128] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functions of any given module of the present disclosure may be distributed among multiple modules connected via the interface circuit. For example, multiple modules may allow load balancing. In a further example, a server (also referred to as a remote or cloud) module may implement some functions on behalf of a client module.

[0129] The term "code" as used above may include software, firmware, and / or microcode, and may refer to a program, a routine, a function, a class, a data structure, and / or an object. The term "shared processor circuit" covers a single processor circuit that executes some or all code from multiple modules. The term "group processor circuit" covers a processor circuit that executes some or all code from one or more modules in conjunction with an additional processor circuit. References to multiple processor circuits cover multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" covers a single memory circuit that stores some or all code from multiple modules. The term "group memory circuit" includes a memory circuit that stores some or all code from one or more modules in conjunction with additional memory.

[0130] The term "memory circuit" is a subset of the term "computer-readable medium". The term "computer-readable medium" as used herein does not encompass non-transitory electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term "computer-readable medium" may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0131] The apparatus and method described in the present application can be implemented in part or in whole by a special-purpose computer, which is created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flow chart components and other elements described above act as software specifications, which can be translated into computer programs by the routine work of skilled technicians or programmers.

[0132] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0133] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from languages ​​including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Version 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and .

Claims

1. A method for detecting and utilizing the three-dimensional (3D) position of a coupler of a trailer, comprising: calculating the motion of a camera mounted to the vehicle; determining a two-dimensional (2D) position of a connector within an input image captured by a camera; estimating a 3D estimated position of the connector based on the 2D position of the connector and the motion of the camera; optimizing the estimated 3D position of the coupler to generate an adjusted 3D coupler position; as well as The adjusted 3D coupler position is filtered to detect the 3D position of the coupler of the trailer.

2. The method according to claim 1, further comprising: receiving a plurality of input images captured by a camera; as well as detecting whether a joiner is present in one or more of the plurality of input images, wherein Determining the 2D position of the joiner is performed in response to detecting that the joiner is present in one or more of the plurality of input images.

3. The method of claim 2, wherein detecting whether a connector is present comprises: Object detection is performed on each of the plurality of input images using a trained machine learning or deep learning model.

4. The method according to claim 3, further comprising: Supervised machine learning is performed to generate the trained machine learning or deep learning model by iteratively applying a plurality of labeled test images having ground truth values ​​indicating the presence of a trailer coupler or the absence of a trailer coupler to the model to reduce an error between an output of the neural network and the ground truth values ​​of the plurality of labeled test images.

5. The method of claim 1 , wherein estimating the 3D estimated position of the coupling comprises: The 2D positions of the coupler in the plurality of input images captured by a camera mounted to the vehicle are triangulated to estimate a 3D estimated position of the coupler.

6. The method of claim 5, wherein the triangulation comprises: calculating a first depth value extending between one of the 3D estimated positions of the coupler and a camera focus with respect to a first of the plurality of input images, calculating a second depth value extending between the one of the 3D estimated positions of the coupler and the camera focus with respect to a second of the plurality of input images, and The one of the 3D estimated positions of the coupler is estimated by performing triangulation in 3D space using the first depth value and the second depth value.

7. The method of claim 1, wherein the optimizing comprises: reprojecting the 3D estimated positions of the connectors back onto corresponding ones of the input images to determine reprojected 2D positions; determining a reprojection error between two-dimensional (2D) positions of a coupler within the corresponding input image in the input image and corresponding 2D positions in the reprojected 2D positions; and The estimated 3D position of the coupler is adjusted to reduce the sum of the reprojection errors.

8. The method of claim 1, wherein the filtering comprises: averaging the adjusted 3D coupler positions to generate an average 3D coupler position; removing 3D coupler positions of the adjusted 3D coupler positions that are outside the defined radius from the average 3D coupler position to generate remaining adjusted 3D coupler positions; as well as The remaining adjusted 3D coupler positions are averaged to determine a final 3D coupler position.

9. The method according to claim 1, further comprising: The vehicle is guided based on at least the 3D position of the coupler such that the hitch ball of the vehicle approaches the coupler of the trailer.

10. The method of claim 9, wherein the directing comprises: A graphical user interface (GUI) is controlled to output a display to an operator of the vehicle that illustrates at least the relative position between the hitch ball of the vehicle and the coupler of the trailer based on the 3D position of the coupler.

Images