Trailer detection and autonomous bolting

Abstract

A method for autonomously maneuvering a towing vehicle (100). The method includes: receiving an image (143) from one or more cameras (142), the one or more cameras (142) being positioned on a rear portion of the towing vehicle (100); and receiving sensor data from an inertial measurement unit (144), the inertial measurement unit (144) being supported by the towing vehicle (100). The method also includes determining a pixel-wise intensity difference between a currently received image and a previously received image. The method includes determining a camera orientation and a trailer orientation relative to a world coordinate system. The camera orientation and the trailer orientation are based on the image (143), the sensor data (145), and the pixel-wise intensity difference. The method includes determining a towing vehicle path based on the camera orientation and the trailer orientation. The method also includes instructing a driving system (110) supported by the towing vehicle (100) to autonomously maneuver in an opposite direction along the towing vehicle path, resulting in the towing vehicle (100) being bolted to the trailer (200).

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for detecting a trailer positioned behind a vehicle and autonomously maneuvering the vehicle toward the trailer. Background Art

[0002] Trailers are usually unpowered vehicles pulled by powered traction vehicles. Trailers can be general trailers, pop-up campers, travel trailers, livestock trailers, flatbed trailers, enclosed dispatch winches, and boat trailers, etc. The traction vehicle can be a car, a cross-border vehicle, a truck, a van, a sports utility vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to be attached to the trailer and pull the trailer. The trailer can be attached to the powered vehicle using a trailer hitch. The receiver hitch is installed on the traction vehicle and is connected to the trailer hitch to form a connection. The trailer hitch can be a ball socket, a spare wheel and a gooseneck or a trailer jack. Other attachment mechanisms can also be used. In some examples, in addition to the mechanical connection between the trailer and the powered vehicle, the trailer is also electrically connected to the traction vehicle. In this way, the electrical connection allows the trailer to obtain the feed from the rear light circuit of the powered vehicle, allowing the trailer to have taillights, turn signals, and brake lights that are synchronized with the lights of the powered vehicle.

[0003] Recent advances in sensor technology have led to improved safety systems for vehicles. Thus, it is desirable to provide a system capable of determining the position of a trailer relative to a towing vehicle, which allows the towing vehicle to be autonomously maneuvered toward the trailer and autonomously bolted to the trailer. Summary of the invention

[0004] One aspect of the present disclosure provides a method for autonomously maneuvering a towing vehicle toward a trailer for autonomously bolting between the trailer and the towing vehicle. The method includes receiving images from one or more cameras at data processing hardware, the one or more cameras being positioned on a rear portion of the towing vehicle and communicating with the data processing hardware. The method also includes receiving sensor data from an inertial measurement unit at the data processing hardware, the inertial measurement unit communicating with the data processing hardware and supported by the towing vehicle. The method also includes determining the pixel-wise intensity difference between the current received image and the previous received image at the data processing hardware. The method includes determining the camera orientation and the trailer orientation relative to a world coordinate system at the data processing hardware. The camera orientation and the trailer orientation are based on the image, the sensor data, and the pixel-wise intensity difference. The method also includes determining the towing vehicle path based on the camera orientation and the trailer orientation at the data processing hardware. The method also includes sending instructions from the data processing hardware to a driving system supported by the towing vehicle. The instructions cause the towing vehicle to autonomously maneuver in the opposite direction along the towing vehicle path so that the towing vehicle is bolted to the trailer.

[0005] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, one or more cameras include a fisheye lens, wherein the fisheye lens includes an ultra-wide angle lens. The camera orientation may include a camera position and a camera angle in a world coordinate system. The trailer orientation may include a trailer position and a trailer angle in a world coordinate system.

[0006] In some examples, the method further includes identifying one or more feature points within the image. The feature points are associated with the trailer. The method may also include determining a plane based on the one or more feature points. The plane indicates a front face of the trailer. The method may also include determining a normal to the plane. In some examples, the method further includes determining an angle of the trailer based on the normal.

[0007] In some embodiments, the method may include determining a region of interest surrounding a representation of the trailer within the image. Determining a pixel-wise intensity difference between a currently received image and a previously received image includes determining a pixel-wise intensity difference between the region of interest of the currently received image and the region of interest of the previously received image. The method may also include identifying one or more feature points within the region of interest. The feature points are associated with the trailer. The method may also include tracking the one or more feature points as the vehicle autonomously maneuvers along the path.

[0008] In some examples, the method also includes determining a sequence of points associated with the path. The towing vehicle follows the sequence of points. The method may also include determining a steering wheel angle associated with each point within the sequence of points. The method may also include sending instructions to the driving system to adjust the current steering wheel angle based on the determined steering wheel angle. In some examples, the method also includes: as the vehicle autonomously maneuvers along the path, determining the difference between the steering angles of each front wheel of the towing vehicle. The method may also include sending instructions to the driving system to adjust the steering wheel angle of each wheel based on the difference between the steering wheel angles of each front wheel.

[0009] Another aspect of the present disclosure provides a system for autonomously maneuvering a towing vehicle toward a trailer for autonomously bolting between the trailer and the towing vehicle. The system includes: data processing hardware; and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, including the method described above.

[0010] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A is a schematic top view of an exemplary towing vehicle spaced apart from a trailer positioned behind the towing vehicle.

[0012] Figure 1B is a schematic illustration of an exemplary image captured by a camera positioned on a rear portion of a vehicle, wherein the image captures a trailer positioned behind the vehicle.

[0013] Figure 1C is a schematic illustration of an exemplary trailer orientation and an exemplary camera orientation relative to world coordinates.

[0014] Figure 2 is a schematic illustration of an exemplary towing vehicle with a user interface and a sensor system.

[0015] Figure 3 yes Figure 2 A schematic diagram of an exemplary position estimation system is shown in FIG.

[0016] Figure 4 yes Figure 2 A schematic view of an exemplary path planning system is shown in FIG.

[0017] Figure 5A – Figure 5C Is Figure 4Schematic view of an exemplary pathway generated by the Dubins pathway generation module shown in FIG.

[0018] Figure 6 is a schematic illustration of an exemplary arrangement for autonomously steering a towing vehicle toward a trailer.

[0019] In the various drawings, like reference numerals indicate like elements. DETAILED DESCRIPTION

[0020] A towing vehicle (e.g., but not limited to, a car, a crossover vehicle, a truck, a van, a sport utility vehicle (SUV), and a recreational vehicle (RV)) may be configured to tow a trailer. The towing vehicle is connected to the trailer by means of a trailer hitch. It is desirable to enable the towing vehicle to autonomously reverse toward the trailer, which is identified from one or more trailer representations of the trailer displayed on a user interface (e.g., a user display). In addition, it is also desirable to enable the towing vehicle to estimate the trailer position, such as the angle and position of the trailer relative to the towing vehicle, based on one or more images received from a camera positioned on the rear portion of the towing vehicle. The angle and position of the trailer relative to the towing vehicle may be determined, for example, by fusing visual information from the received images and IMU (inertial measurement unit) sensor data. In addition, it is desirable to enable the towing vehicle to autonomously maneuver toward the trailer along a planned path and align with the trailer hitch coupling of the trailer for autonomous bolting between the vehicle hitch ball of the towing vehicle and the trailer hitch coupling.

[0021] refer to Figure 1A – Figure 2In some embodiments, the driver of the towing vehicle 100 wishes to tow a trailer 200 positioned behind the towing vehicle 100. The towing vehicle 100 may be configured to receive an indication of a trailer selection 134 from the driver, which is associated with the selected trailer 200, 200a-c. In some examples, the driver steers the towing vehicle 100 toward the selected trailer 200, 200a-c, while in other examples, the towing vehicle 100 autonomously drives toward the selected trailer 200, 200a-c. The towing vehicle 100 may include a driving system 110, for example, the driving system 110 steers the towing vehicle 100 across the road surface based on a driving command having x, y, and z components. As shown, the driving system 110 includes a right front wheel 112, 112a, a left front wheel 112, 112b, a right rear wheel 112, 112c, and a left rear wheel 112, 112d. The driving system 110 may also include other wheel configurations. The driving system 110 may also include a braking system 114 including brakes associated with each wheel 112, 112a-d, and an acceleration system 116 configured to adjust the speed and direction of the towing vehicle 100. In addition, the driving system 110 may include a suspension system 118 including tires associated with each wheel 112, 112a-d, tire air, springs, shock absorbers, and linkages that connect the towing vehicle 100 to its wheels 112, 112a-d and allow relative movement between the towing vehicle 100 and the wheels 112, 112a-d. The suspension system 118 may be configured to adjust the height of the towing vehicle 100, allowing the towing vehicle hitch 120 (e.g., vehicle hitch ball 122) to align with the trailer hitch 210 (e.g., trailer hitch coupling or trailer hitch cup 212), which allows an autonomous connection between the towing vehicle 100 and the trailer 200.

[0022] The towing vehicle 100 can move across the road surface by various combinations of movement relative to three mutually perpendicular axes defined by the towing vehicle 100 (lateral axis X, front-to-back axis Y, and central vertical axis Z). The lateral axis X extends between the right and left sides of the towing vehicle 100. The forward driving direction along the front-to-back axis Y is represented as F, also referred to as forward movement. In addition, the rearward or rearward driving direction along the front-to-back direction Y is represented as R, also referred to as rearward movement. When the suspension system 118 adjusts the suspension of the towing vehicle 100, the towing vehicle 100 can tilt about the X-axis and / or the Y-axis, or move along the central vertical axis Z.

[0023] The towing vehicle 100 may include a user interface 130. In some examples, the user interface 130 is a touch screen display 132 that allows the driver to make selections via the display. In other examples, the user interface 130 is not a touch screen, and the driver may use an input device (such as, but not limited to, a knob or a mouse) to make selections. The user interface 130 receives one or more user commands from the driver via one or more input mechanisms or a touch screen display 132, and / or displays one or more notifications to the driver. The user interface 130 communicates with a vehicle controller 150, which in turn communicates with a sensor system 140. In some examples, the user interface 130 displays an image 143 of the environment of the towing vehicle 100, resulting in the user interface 130 receiving one or more commands (from the driver) that initiate the execution of one or more actions. In some examples, the user display 132 displays one or more trailer representations 136, 136a-c of a trailer 200 positioned behind the towing vehicle 100. In this case, the driver makes a trailer selection 134 of the trailer representations 136, 136a-c of the trailer 200. The controller 150 may then determine the position L of the trailer 200 associated with the trailer selection 134 relative to the towing vehicle 100. T In other examples, the controller 150 detects one or more trailers 200 and determines the position L of each of the trailers associated with each of the trailer representations 136 , 136a - c relative to the towing vehicle 100 . T The vehicle controller 150 includes a computing device (processor) 152 (e.g., a central processing unit having one or more computing processors) in communication with a non-transitory memory 154 (e.g., hard disk, flash memory, random access memory, memory hardware) capable of storing instructions executable on the computing processor(s) 152 .

[0024] The vehicle controller 150 executes a driving assistance system 180, which in turn includes a path following behavior 182. The driving assistance system 180 receives the planned path 500 from the path planning system 400 and executes behaviors 182a-182c, which send commands 184 to the driving system 110, resulting in the towing vehicle 100 autonomously driving around the planned path 500 in the rearward direction R and autonomously bolting to the trailer 200.

[0025] The path following behavior 182 includes a braking behavior 182a, a speed behavior 182b, and a steering behavior 182c. In some examples, the path following behavior 182 also includes a hitch connection behavior (allowing the vehicle hitch ball 122 to connect to the trailer hitch cup 212) and a suspension adjustment behavior (causing the vehicle suspension to be adjusted to allow the hitch connection of the vehicle hitch ball 122 and the trailer hitch coupling 212). Each behavior 182a-182c causes the towing vehicle 100 to take an action, such as driving backward, turning at a specific angle, braking, accelerating, decelerating, etc. The vehicle controller 150 can steer the towing vehicle 100 across the road surface in any direction by controlling the driving system 110 (more specifically, by issuing commands 184 to the driving system 110).

[0026] The towing vehicle 100 may include a sensor system 140 to provide reliable and robust driving. The sensor system 140 may include different types of sensors that may be used alone or in conjunction with each other to form a sense of the environment of the towing vehicle 100, which is used for the towing vehicle 100 to drive and help the driver make intelligent decisions based on objects and obstacles detected by the sensor system 140. The sensor system 140 may include one or more cameras 142, 142a-d. In some embodiments, the towing vehicle 100 includes a rear camera 142, 142a, which is installed to provide a view of the rear driving path of the towing vehicle 100. The rear camera 142a may include a fisheye lens, which includes an ultra-wide-angle lens that produces strong visual distortion, which is intended to form a wide panoramic or hemispherical image. The fisheye camera captures an image with an extremely wide-angle view. In addition, the image 143 captured by the fisheye camera 142a has a convex non-linear representation. Other types of cameras 142 may also be used to capture images 143 of the environment behind the towing vehicle 100 .

[0027] In some embodiments, the sensor system 140 may also include an IMU (Inertial Measurement Unit) 144, which is configured to measure the linear acceleration (using one or more accelerometers) and rotation rate (using one or more gyroscopes) of the vehicle. In some examples, the IMU 144 also determines the heading reference of the towing vehicle 100. Therefore, the IMU 144 determines the pitch, roll, and yaw of the towing vehicle 100.

[0028] The sensor system 140 may include other sensors such as, but not limited to, radar, sonar, LIDAR (light detection and ranging, which may require optical remote sensing, which measures the properties of scattered light to obtain the distance and / or other information of distant targets), LADAR (laser detection and ranging), ultrasonic sensors, etc. Additional sensors may also be used.

[0029] The vehicle controller 150 executes a position estimation and path planning system 400 that receives the image 143 from the camera 142 and the IMU data 145 from the IMU 144 and determines the position L of the trailer 200 relative to the towing vehicle 100 based on the received image 143. T .

[0030] The controller 150 receives image data 143 from the rear camera 142a and IMU sensor data 145 from the IMU 144, and based on the received data 143, 145, the controller 150 estimates the camera orientation 332 relative to the world coordinate origin (i.e., the camera position L C and angle α C ) and the trailer position 348 (ie, the trailer position L T and angle α T ). The world coordinate origin is defined at the starting position of the vehicle 100. The controller 150 includes a position estimation and path planning system 400, which includes a position estimation system. The position estimation system 300 fuses the camera data 143 (e.g., monocular camera data) and the IMU sensor data 145 to estimate the trailer position 348 (i.e., the trailer position L relative to the world coordinate origin). T and trailer angle α T ) and the camera position L relative to the world coordinates C and the camera angle α C Trailer angle α T It is the angle between the front-rear axis Y of the world coordinate system and the center line of the trailer 200. Camera angle α C is the angle between the front-rear axis Y of the world coordinate system and the optical axis of the camera. Once the position estimation system 300 determines the position L of the trailer 200a relative to the world coordinate system, T Angle α with trailer 200a T and the camera position L relative to the world coordinates C and the camera angle α C , the path planning system 400 determines a trajectory or path 500 to align the towing vehicle 100 with the trailer 200a.

[0031] The position estimation and path planning system 400 includes the position estimation system 300 and the path planning system 400. Figure 3 The position estimation system 300 includes an imaging module 310, which includes a trailer ROI detection module 312 and a feature detection and tracking module 314. In addition, the position estimation system 300 includes an iterative extended Kalman filter 320, a camera orientation module 330, and a trailer orientation estimator module 340.

[0032] The trailer ROI detection module 312 receives an image 143 including a trailer representation 136 from the rear camera 142a. As previously mentioned, in some examples, the image 143 includes one or more trailer representations 136, 136a-c (associated with one or more trailers 200, 200a-c positioned behind the towing vehicle 100). In some examples, when the image 143 includes more than one trailer representation 136, the position estimation and path planning system 400 instructs the display 132 to request a trailer selection 134 associated with a representation of one of the trailers 200. The driver selects the trailer 200a (i.e., the representation 136a of the trailer 200a) from the image 143. In some examples, once the trailer representation 136 is selected or identified, the trailer ROI detection module 312 delimits the identified trailer representation 136 (in the region of interest (ROI) 200) by a bounding box (also referred to as a region of interest (ROI) 200). Figure 1B 200a (i.e., the representation 136a of the trailer 200a) by defining the identified trailer representation 136a by a bounding box (i.e., the ROI 230). The driver can enter the bounding box 200 via the user interface 130 (e.g., via the touch screen display 132) or a knob or mouse to select the bounding box 200 surrounding the trailer representation 136 within the image 143.

[0033] Once the trailer ROI detection module 312 detects and identifies the ROI 230 that includes the trailer representation 136, the feature detection and tracking module 314 identifies visual feature points 316 within the ROI 230. The visual feature points 316 may be, for example, trailer wheels, hitches 212, trailer edges, or any other features associated with the trailer 200. Once the feature points 316 within the ROI 230 are detected, the feature detection and tracking module 314 tracks the detected features 316 as the towing vehicle 100 moves toward the trailer 200.

[0034] As previously mentioned, the position estimation system 300 includes an iterative extended Kalman filter 320 that receives the identified feature points 316 within the ROI 230 and the IMU sensor data 145 and analyzes the received data 145, 316. The iterative extended Kalman filter 320 fuses the identified feature points 316 and the IMU data 145. The iterative extended Kalman filter 320 uses a dynamic model of the system, control inputs known to the system, and multiple sequential measurements (from sensors) to estimate the amount of change in the system.

[0035] The extended Kalman filter is a nonlinear version of the Kalman filter that is linearized about the current mean and covariance estimates. The Kalman filter is an algorithm that uses a series of measurements observed over time and includes statistical noise and other inaccuracies, and outputs estimates of unknown variables that are more accurate than those based on a single measurement because the Kalman filter 320 estimates the joint probability distribution over the variables for each time frame. The Kalman filter 320 performs its calculations in a two-step process. During the first step (also called the prediction step), the Kalman filter 320 determines the current state 322c (states 324, 324a c –324j c ) together with the uncertainty associated with each current state variable 324a-j. When observing the results of the current measurement, in a second step (also called the update step), these current states 314c (states 324, 324a) are updated using a weighted average. c –324j c ), where more weight is given to the one with higher certainty (current state 314c (states 324, 324a c –324j c ) or current measurement). The algorithm is recursive and runs in real time, using the current input measurement and the current filter state 322c (states 324, 324a c –324j c ) to determine the updated filter state 322u (states 324, 324a u –324j u ), so no additional past information is needed. The iterative extended Kalman filter 320 improves the linearization of the extended Kalman filter by reducing the linearization error at the expense of increased computational requirements.

[0036] The iterative extended Kalman filter 320 receives the IMU sensor data 145 from the IMU 144 and the visual feature points 316 within the ROI 230. In some embodiments, the iterative extended Kalman filter 320 determines filter states 322, 322c, 322u that are continuously updated based on the received IMU sensor data 145 and the visual feature points 316. The filter states 322, 322c, 322u may include calibration values, such as the distance 324f between the IMU 144 and the rear camera 142a, since the positions of both within the towing vehicle 100 are known. In some examples, the filter states 322, 322c, 322u include an IMU position state 324a, an IMU velocity state 324b, and an IMU altitude state 324c, which are determined by the iterative extended Kalman filter 320 based on the IMU sensor data 145 (including acceleration and angular velocity data of the towing vehicle 100), the calibration distance 324f between the IMU 144 and the rear camera 142a, and the position of the camera 142a in a coordinate system (e.g., a world coordinate system). The world coordinate system defines a world origin (a point with coordinates [0,0,0]) and defines three unit axes that are orthogonal to each other. The coordinates of any point in the world space are defined relative to the world origin. Once the world coordinate system is defined, the position of the camera 142a can be defined by the position in the world space, and the orientation of the camera 142a can be defined by three unit vectors that are orthogonal to each other. In some examples, the world origin is defined by the initial position of the camera, and the three unit axes are defined by the initial camera orientation. The position of camera 142a is determined or known by camera orientation module 330, as will be described below. As previously mentioned, IMU 144 includes an accelerometer for determining the linear acceleration of the towing vehicle and a gyroscope for determining the rotation rate of the vehicle wheels. In addition, in some examples, filter states 322, 322c, 322u include accelerometer bias states 324d and gyroscope bias states 324e. Inertial sensors (e.g., accelerometers and gyroscopes) typically include small offsets in the average signal output (even when there is no movement). Accelerometer bias states 324d estimate small offsets of the accelerometer sensor in the average signal output, and gyroscope bias states 324e estimate small offsets of the gyroscope sensor in the average signal output.

[0037] In some examples, the filter states 322, 322c, 322u include a camera orientation state 324g to the ROI feature points, which is the orientation of the camera 142a to the one or more ROI feature points 316 identified by the feature detection and tracking module 314. The filter state 324 may also include a camera distance state 324h to the ROI feature points, which is the distance between the camera 142a and the one or more ROI feature points 316 identified by the feature detection and tracking module 314.

[0038] The iterative extended Kalman filter 320 generates a pixel-wise intensity difference 326 between the visual feature points 316 of the current image 143 and the previously tracked image 143. The extended iterative Kalman filter 320 determines the pixel-wise intensity difference 326 by running a routine on the pixel locations of the current image 143 and the previously tracked image 143, and returns the result, then moves to the next pixel location and repeats the same routine until all pixels of the current image 143 and the previously tracked image 143 are processed and the pixel-wise intensity difference 326 is determined.

[0039] In a first step, the iterative extended Kalman filter 320 predicts the current filter state 322c, 324a based on the received IMU sensor data 145 (ie, acceleration and angular velocity data) c –324j c In a second step, the iterative extended Kalman filter 320 updates the current state 324a based on the pixel-wise intensity difference 326 between the tracked feature point 316 of the current image 143 and the previous corresponding feature point 316 of the previous tracked image 143 c –324i c The value of to update and correct the update filter state 322u, 324a u –324i u Thus, each time the iterative extended Kalman filter 320 receives data from the IMU 144 and the rear camera 142a, the current state 324a of each of the mentioned states 322c is updated. c –324i c .

[0040] The position estimation system 300 further includes a camera position module 330 that calculates the position L of the camera 142a in the world coordinate system based on the updated filter state 322u. C and orientation or angle α C For example, the camera position module 330 calculates the camera position L C and the camera angle α C (like Figure 1C ), camera angle α CIt is the angle between the front-back axis Y of the world coordinates and the position of the camera at the world coordinate origin.

[0041] The trailer orientation estimator module 340 calculates the position 342 of the feature point 316 in the world coordinate system based on the updated filter state 322u. Then, the trailer orientation estimator module 340 (e.g., the plane extraction module 344) determines the plane P based on the feature point position 342. The plane P is the front of the trailer 200, such as Figure 1A Once plane P is identified, the trailer orientation estimator module 340 calculates a normal 346 to plane P to identify the trailer angle α relative to the world coordinate system. T .

[0042] As previously discussed, the position estimation system 300 utilizes an iterative extended Kalman filter 320 to fuse the camera image data 143 and the IMU sensor data 145 to estimate and determine the trailer position L in the world coordinate system. T and trailer angle α T and the camera position L in the world coordinate system C and the camera angle α C The position estimation system 300 provides high accuracy by using a tightly coupled fusion technique and increases robustness by utilizing a feature detection and tracking module 314 for updating the current filter state 322c in the iterative extended Kalman filter 320. As described above, the position estimation system 300 utilizes low computational resources and achieves real-time performance using low-cost hardware. In addition, because the feature detection and tracking module 314 for filter state update 322u relies on analysis of the image 143, the position estimation system 300 increases the robustness of the determination of the trailer position.

[0043] refer to Figure 4 The path planning system 400 includes a trajectory planning module 410 and a trajectory control module 420. The trajectory planning module 410 receives the position 332 (ie, the camera position L) from the camera position module 330. C and the camera angle α C ) and the trailer position 348 (ie, the trailer position L) from the trailer position estimator module 340 T and trailer angle α T). The trajectory planning module 410 includes a Dobbins path generation module 412 and a key point generation module 414. The Dobbins path generation module 412 receives the camera orientation 332 and the trailer orientation 348, and generates a Dobbins path 500. A Dobbins path refers to the shortest curve connecting two points in a two-dimensional Euclidean plane (i.e., an xy plane), which has a restriction on the curvature of the path, and has a prescribed initial and terminal tangent to the path, and it is assumed that a vehicle traveling on the path can travel in only one direction. The Dobbins path 500 is generated by connecting arcs and straight lines with maximum curvature. Figure 5A – Figure 5C The Dobbins path is shown. Figure 5A The CLC (circle-line-circle configuration) used when the distance between the towing vehicle 100 and the trailer 200 (rather than the orientation of the trailer 200 relative to the towing vehicle 100) is shown. Figure 5B The CL (circle line configuration) is shown for use when the distance between the towing vehicle 100 and the trailer 200 is at a minimum and only one turn is required to align the towing vehicle 100 and the trailer 200a. Figure 5C Another CLC configuration is shown when the towing vehicle 100 is aligned with the trailer 200 but at a lateral distance. Once the Dobbins path generation module 412 generates the path 500 , the key point generation module 414 identifies a sequence of points 415 along the path 500 for the towing vehicle 100 to follow.

[0044] Afterwards, the trajectory control module 420 determines the steering wheel angle of the towing vehicle 100 in the backward direction R to follow the path 500. In addition, the trajectory control module 420 performs an Ackerman angle calculation 422 based on the received point sequence 415 from the key point generation module 414. The Ackerman steering geometry is the geometric arrangement of the links in the steering of the towing vehicle 100, which solves the problem that the trajectory of the wheel 112 needs to be a circle of different radius on the inside and outside of the turn. Therefore, the Ackerman angle calculation 422 assumes that all wheels 112 have their axles arranged as the radius of a circle with a common center point. With the fixed rear wheels, this center point must be on a line extending from the rear wheel axle. Also, making the axles of the front wheels intersect on this line requires that when turning, the inner front wheel turns through a larger angle than the outer wheel. Therefore, the Ackerman angle calculation 422 determines the difference between the steering angles of each front wheel. The trajectory control module 420 updates the steering wheel angle based on the Ackerman angle calculation 422 and determines an updated steering wheel angle 424 for causing each of the front wheels 112a , 112b to follow the path 500 .

[0045] As the tow vehicle 100 autonomously maneuvers along the planned path 500, the path planning system 400 continuously updates the path based on the continuously received sensor data. In some examples, the object detection system identifies one or more objects along the planned path and sends data related to the location of the one or more objects to the path planning system 400. In this case, the path planning system 400 recalculates the planned path 500 to avoid the one or more objects while also performing the predetermined maneuver. In some examples, the path planning system determines a collision probability, and if the collision probability exceeds a predetermined threshold, the path planning system 400 adjusts the path 500 and sends it to the driver assistance system 180.

[0046] Once the path planning system 400 determines the planned path 500, the vehicle controller 150 executes the driving assistance system 180, which in turn includes a path following behavior 182. The path following behavior 182 receives the planned path and executes one or more behaviors 182a-b that send a command 184 to the driving system 110, resulting in the towing vehicle 100 autonomously driving along the planned path, which results in the towing vehicle 100 autonomously driving along the planned path.

[0047] The path following behaviors 182a-b may include one or more behaviors, such as, but not limited to, a braking behavior 182a, a speed behavior 182b, and a steering behavior 182c. Each behavior 182a-b causes the towing vehicle 100 to take an action, such as driving backward, turning at a specific angle, braking, accelerating, decelerating, etc. The vehicle controller 150 may steer the towing vehicle 100 across the road surface in any direction by controlling the driving system 110 (more specifically, by issuing commands 184 to the driving system 110).

[0048] Braking action 182a may be performed to stop or slow the towing vehicle 100 based on the planned path. Braking action 182a sends a signal or command 184 to the driving system 110 (eg, a braking system (not shown)) to stop or slow the towing vehicle 100.

[0049] Speed ​​behavior 182b may be executed to change the speed of towing vehicle 100 by accelerating or decelerating based on the planned path. Speed ​​behavior 182b sends a signal or command 184 to braking system 114 for deceleration or sends a signal or command 184 to acceleration system 116 for acceleration.

[0050] The steering action 182c may be executed to change the direction of the towing vehicle 100 based on the planned path. As such, the steering action 182c sends a signal or command 184 indicating a steering angle to the acceleration system 116, causing the driving system 110 to change direction.

[0051] As previously discussed, the position estimation and path planning system 400 provides the position of the selected trailer 200 in the world coordinate system without using one or more markers positioned on the trailer 200. In addition, the position estimation and path planning system 400 provides high accuracy by using a tightly coupled fusion technique, which requires low computational resources and can achieve real-time performance on low-cost hardware. Since no markers are required on the trailer 200, the complexity of operation and computation is reduced, and thus the production cost of the system is also reduced.

[0052] Figure 6 An exemplary arrangement of operations of a method 600 for autonomously maneuvering a towing vehicle 100 toward a trailer 200 is provided for autonomously bolting between a trailer 200 and a towing vehicle 100 using the systems described in FIGS. 1-5 . At box 602, the method 600 includes receiving an image at data processing hardware 152 from one or more cameras 142a positioned on a rear portion of the towing vehicle 100 and in communication with the data processing hardware 152. The camera may include a fisheye lens including an ultra-wide angle lens. At box 604, the method 600 includes receiving sensor data 145 at the data processing hardware 152 from an inertial measurement unit 144 in communication with the data processing hardware 152 and supported by the towing vehicle 100. At box 606, the method 600 includes determining, at the data processing hardware 152, a pixel-wise intensity difference 326 between a currently received image 143 and a previously received image 143. At box 608, the method 600 includes determining, at the data processing hardware 152, a camera orientation 332 and a trailer orientation 348 relative to a world coordinate system. The camera orientation 332 and the trailer orientation 348 are based on the image 143, the sensor data, and the pixel-wise intensity difference 326. At box 610, the method 600 includes determining, at the data processing hardware 152, a towing vehicle path 500 based on the camera orientation 332 and the trailer orientation 348. At box 612, the method 600 includes sending instructions 184 from the data processing hardware 152 to a steering system 110 supported by the towing vehicle 100. The instructions 184 cause the towing vehicle 100 to autonomously maneuver along the towing vehicle path 500 in a reverse direction R such that the towing vehicle 100 is bolted to the trailer 200.

[0053] In some examples, the camera position includes the camera position L in the world coordinate system Cand the camera angle α C The trailer orientation may include the trailer position L in the world coordinate system C and trailer angle α T The method 600 may include identifying one or more feature points within the image, the feature points 316 being associated with the trailer 200. The method 600 may also include determining a plane P based on the one or more feature points 316. The plane P indicates the front of the trailer 200. The method may include determining a normal to the plane P, and determining the trailer angle α based on the normal. T .

[0054] In some examples, the method includes determining an area of ​​interest 230 surrounding the representation 136 of the trailer 200 within the image 143. Determining a pixel-wise intensity difference 326 between the current received image 143 and the previous received image 143 includes determining a pixel-wise intensity difference 326 between the area of ​​interest 230 of the current received image 143 and the area of ​​interest 230 of the previous received image 143. The method 600 may also include identifying one or more feature points 316 within the area of ​​interest 230. The feature points 316 are associated with the trailer 200. The method 600 may also include tracking the one or more feature points 316 as the vehicle 100 autonomously maneuvers along the path 500.

[0055] In some embodiments, the method 600 includes determining a sequence of points associated with the path 500. The towing vehicle 100 follows the sequence of points along the path 500. The method 600 also includes determining a steering wheel angle associated with each point within the sequence of points, and sending instructions to the driving system 110 to adjust a current steering wheel angle based on the determined steering wheel angle.

[0056] The method 600 may also include determining a difference between the steering angles of each front wheel 112 of the towing vehicle 100 as the vehicle 100 autonomously maneuvers along the path 500. The method 600 may also include sending instructions 184 to the driving system 110 to adjust the steering wheel angle of each wheel 112 based on the difference between the steering wheel angles of each front wheel 112.

[0057] Various embodiments of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0058] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural language and / or an object-oriented programming language and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device and / or means (e.g., disks, optical disks, memories and programmable logic devices (PLDs)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0059] The subject matter and implementation scheme of functional operation described in this specification may be implemented in a digital electronic circuit system or in computer software, firmware or hardware, including the structure disclosed in this specification and its structural equivalent or one or more combinations thereof. In addition, the subject matter described in this specification may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device, or to control the operation of a data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition affecting a machine-readable propagation signal, or a combination of one or more thereof. The terms "data processing device", "computing device" and "computing processor" cover all devices, devices and machines for processing data, including programmable processors, computers or multiple processors or computers by way of example. In addition to hardware, the device may also include code, which forms an execution environment for the computer program involved, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system or one or more combinations thereof. A propagation signal is an artificially generated signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0060] Similarly, although operations are depicted in the accompanying drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems as a whole can be integrated together in a single software product, or packaged into multiple software products.

[0061] A number of embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method for autonomously maneuvering a towing vehicle towards a trailer, for autonomously bolting between the trailer and the towing vehicle, said method include: receiving images at data processing hardware from one or more cameras positioned on a rear portion of the towing vehicle and in communication with the data processing hardware; receiving sensor data at the data processing hardware from an inertial measurement unit in communication with the data processing hardware and supported by the towing vehicle; determining, at the data processing hardware, pixel-wise intensity differences between a currently received image and a previously received image; determining, at the data processing hardware, a camera orientation and a trailer orientation relative to a world coordinate system, the camera orientation and the trailer orientation being based on the image, the sensor data, and the pixel-wise intensity differences, wherein the camera orientation comprises a camera position and a camera angle in the world coordinate system, and the trailer orientation comprises a trailer position and a trailer angle in the world coordinate system; Determining the trailer angle includes: identifying one or more feature points within the image, the feature points being associated with the trailer; determining a plane based on the one or more feature points, the plane indicating a front face of the trailer; determining a normal to the plane; and determining the trailer angle based on the normal; determining, at the data processing hardware, a towing vehicle path based on the camera position and the trailer position; and Instructions are sent from the data processing hardware to a steering system supported by the towing vehicle to autonomously maneuver the towing vehicle in an opposite direction along the towing vehicle path, resulting in the towing vehicle being bolted to the trailer.

2. The method according to claim 1, in, The one or more cameras include a fisheye lens, and the fisheye lens includes an ultra-wide-angle lens.

3. The method according to claim 1, further comprising: include: determining a region of interest surrounding a representation of the trailer within the image, Wherein, determining the pixel-wise intensity difference between the current received image and the previous received image comprises: determining the pixel-wise intensity difference between the focus area of ​​the current received image and the focus area of ​​the previous received image.

4. The method according to claim 3, further comprising: include: identifying one or more feature points within the region of interest, the feature points being associated with the trailer; The one or more feature points are tracked as the vehicle autonomously maneuvers along the path.

5. The method according to claim 1, further comprising: include: determining a sequence of points associated with the path, wherein the towing vehicle follows the sequence of points; determining a steering wheel angle associated with each point in the sequence of points; and A command is sent to the driving system to adjust a current steering wheel angle based on the determined steering wheel angle.

6. The method according to claim 1, further comprising: include: determining a difference between the steering angles of each front wheel of the towing vehicle as the vehicle autonomously maneuvers along the path; as well as Instructions are sent to the steering system to adjust the steering wheel angle of each front wheel based on the difference between the steering wheel angles of each front wheel.

7. A system for autonomously maneuvering a towing vehicle towards a trailer, for autonomously bolting between said trailer and said towing vehicle, said system include: Data processing hardware; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: receiving images from one or more cameras positioned on a rear portion of the towing vehicle and in communication with the data processing hardware; receiving sensor data from an inertial measurement unit in communication with the data processing hardware and supported by the towing vehicle; determining pixel-wise intensity differences between a currently received image and a previously received image; determining a camera orientation and a trailer orientation relative to a world coordinate system, the camera orientation and the trailer orientation being based on the image, the sensor data, and the pixel-wise intensity differences, wherein the camera orientation comprises a camera position and a camera angle in the world coordinate system, and the trailer orientation comprises a trailer position and a trailer angle in the world coordinate system; Determining the trailer angle includes: identifying one or more feature points within the image, the feature points being associated with the trailer; determining a plane based on the one or more feature points, the plane indicating a front face of the trailer; determining a normal to the plane; and determining the trailer angle based on the normal; determining a towing vehicle path based on the camera position and the trailer position; and A command is sent to a steering system supported by the towing vehicle, the command causing the towing vehicle to autonomously maneuver in a reverse direction along the towing vehicle path, resulting in the towing vehicle being bolted to the trailer.

8. The system according to claim 7, in, The one or more cameras include a fisheye lens, and the fisheye lens includes an ultra-wide-angle lens.

9. The system according to claim 7, in, The operations also include: determining a region of interest surrounding a representation of the trailer within the image, Wherein, determining the pixel-wise intensity difference between the current received image and the previous received image comprises: determining the pixel-wise intensity difference between the focus area of ​​the current received image and the focus area of ​​the previous received image.

10. The system according to claim 9, in, The operations also include: identifying one or more feature points within the region of interest, the feature points being associated with the trailer; The one or more feature points are tracked as the vehicle autonomously maneuvers along the path.

11. The system according to claim 7, in, The operations also include: determining a sequence of points associated with the path, wherein the towing vehicle follows the sequence of points; determining a steering wheel angle associated with each point in the sequence of points; and A command is sent to the driving system to adjust a current steering wheel angle based on the determined steering wheel angle.

12. The system according to claim 7, in, The operations also include: determining a difference between the steering angles of each front wheel of the towing vehicle as the vehicle autonomously maneuvers along the path; and A command is sent to the steering system to adjust the steering wheel angle of each wheel based on the difference in the steering wheel angle of each front wheel.

Images