Visual Object Tracker

Abstract

A method for tracking a trailer coupler (212) positioned behind a towing vehicle (100) is provided. The method includes receiving one or more images (143) from a camera (142) positioned on the back of the towing vehicle. The method also includes determining a trailer coupler location within the one or more images. The trailer coupler location is associated with the trailer coupler (212). Additionally, the method includes receiving vehicle sensor data (141) from one or more vehicle sensors. The method includes determining vehicle motion data based on the received vehicle sensor data (141). The method also includes determining an updated coupler location based on the trailer coupler location and the vehicle motion data.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for a visual object tracking system supported by a towing vehicle. The visual object tracking system tracks a coupler of a trailer positioned behind the towing vehicle. Background Art

[0002] A trailer is typically an unpowered vehicle pulled by a powered towing vehicle. Trailers can be utility trailers, pop-up campers, travel trailers, livestock trailers, flatbed trailers, enclosed car haulers, and boat trailers, among others. The towing vehicle can be a car, crossover, truck, van, sport utility vehicle (SUV), recreational vehicle (RV), or any other vehicle configured to attach to and pull a trailer. A trailer can be attached to a motor vehicle using a trailer hitch. A receiver hitch is mounted on the towing vehicle and connects to the trailer hitch to form a connection. The trailer hitch can be a ball and socket, fifth wheel and gooseneck, or trailer jack. Other attachment mechanisms can also be used.

[0003] Recent advances in computing and sensor technology have led to improvements in vehicle autonomous driving. Therefore, it is desirable to have a towing vehicle that can autonomously drive in a rearward direction toward a trailer positioned behind the towing vehicle such that the towing vehicle is positioned attached to the trailer. Summary of the invention

[0004] One aspect of the present disclosure provides a method for tracking a trailer coupler positioned behind a towing vehicle. The method includes receiving one or more images at data processing hardware from a camera positioned on the back of the towing vehicle and in communication with the data processing hardware. The method also includes determining, at the data processing hardware, a trailer coupler location within the one or more images. The trailer coupler location is associated with the trailer coupler. The method also includes receiving, at the data processing hardware, vehicle sensor data from one or more vehicle sensors in communication with the data processing hardware. The method also includes determining, at the data processing hardware, vehicle motion data based on the received vehicle sensor data. The method also includes determining, at the data processing hardware, an updated coupler location based on the trailer coupler location and the vehicle motion data.

[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the vehicle motion data includes vehicle attitude, vehicle linear velocity, and vehicle angular velocity. The one or more vehicle sensors may include at least one of a steering wheel angle sensor, a wheel encoder, an inertial measurement unit, a radar, and a global positioning system receiver.

[0006] In some examples, the method also includes determining a pixel velocity of a trailer coupler within one or more images based on the vehicle motion data and a previous coupler location. The updated coupler location can be based on the pixel velocity and the previous coupler location. The pixel velocity can also be based on a coupler height in a real-world coordinate system.

[0007] In some embodiments, the method further includes determining a coupler height from a predetermined range. The method may also include determining a first pixel position associated with a trailer coupler based on a pixel velocity. The method may also include determining a second pixel position associated with a trailer coupler based on the one or more images. The method may also include comparing the first pixel position and the second pixel position. The method may also include adjusting the coupler height based on the compared first pixel position and the second pixel position. In some examples, the method includes storing an updated coupler position at a memory hardware that communicates with the data processing hardware. The method may also include retrieving the updated coupler position as a previous coupler position when one or more updated images are received.

[0008] In some embodiments, the method further includes determining a coupler position in world coordinates based on the updated coupler position. The method may also include transmitting an instruction to a driving system of the towing vehicle. The instruction causes the towing vehicle to autonomously maneuver in a rearward direction toward the coupler position.

[0009] Another aspect of the present disclosure provides a system for tracking a trailer coupler positioned behind a towing vehicle. The system includes: data processing hardware; and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including the above method.

[0010] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic top view of an exemplary towing vehicle;

[0012] Figure 2 Includes tracking system Figure 1 A schematic diagram of a towing vehicle is shown in;

[0013] Figure 3 yes Figure 2 A schematic diagram of an exemplary tracking system shown in;

[0014] Figure 4 is a schematic diagram of an exemplary towing vehicle;

[0015] Figure 5 Includes height adjustment module Figure 4 A schematic diagram of a towing vehicle is shown in;

[0016] Figure 6 is a schematic diagram of an exemplary method for tracking a coupler of a trailer positioned behind a towing vehicle;

[0017] The same reference numbers in different drawings identify the same elements. DETAILED DESCRIPTION

[0018] A towing vehicle, such as, but not limited to, a car, a crossover, a truck, a van, a sport utility vehicle (SUV), and a recreational vehicle (RV), may be configured to tow a trailer. The towing vehicle is connected to the trailer by way of a trailer hitch, which includes a coupler and a drawbar. It is desirable to have a towing vehicle that is capable of tracking a trailer coupler and autonomously maneuvering in a rearward direction toward the trailer. Object tracking computer vision algorithms typically rely on visual information to predict the location of an object in future images. While past motion of an object in an image is some indicator of future positioning, changes in camera orientation may confuse the object tracking computer vision algorithm and cause the object tracking computer vision algorithm to lose its position of the object. As such, a tracking system that relies on camera motion in addition to the location of the coupler in previously captured images reduces the chance of losing the object's position in future images.

[0019] refer to Figure 1-3In some embodiments, the driver of the towing vehicle 100 wants his towing vehicle 100 to autonomously maneuver toward the trailer 200 by tracking the selected or identified trailer 200. The towing vehicle 100 can autonomously maneuver toward the trailer 200 in the rearward direction R. When the towing vehicle 100 and the trailer 200 are aligned, the towing vehicle 100 can be coupled to the trailer 200 (e.g., by the driver's means or autonomously). In some examples, the towing vehicle 100 includes a towing hitch 120 having a towing ball 122. The trailer 200 includes a trailer hitch 210, which includes a trailer coupler 212 and a towing bar 214. Therefore, the towing ball 122 is coupled to the trailer coupler 212. The towing vehicle 100 can include a driving system 110 that, for example, maneuvers the towing vehicle 100 across a road surface based on a driving command having x, y, and z components. As shown, the driving system 110 includes a right front wheel 112, 112a, a left front wheel 112, 112b, a right rear wheel 112, 112c, and a left rear wheel 112, 112d. The driving system 110 may also include other wheel configurations. The driving system 110 may also include a braking system 114 and an acceleration system 116, wherein the braking system 114 includes a brake associated with each wheel 112, 112ad, and the acceleration system 116 is configured to adjust the speed and direction of the towing vehicle 100. In addition, the driving system 110 may include a suspension system 118, which includes tires associated with each wheel 112, 112ad, tire air, springs, shock absorbers, and a link that connects the towing vehicle 100 to its wheels 112, 112ad and allows relative movement between the towing vehicle 100 and the wheels 112, 112ad. The suspension system 118 may be configured to adjust the height of the towing vehicle 100 , thereby allowing the towing hitch 120 (eg, towing hitch ball 122 ) to align with the trailer hitch 210 (eg, trailer hitch coupler 212 ), which allows for an autonomous connection between the towing vehicle 100 and the trailer 200 .

[0020] The tractor vehicle 100 can move across a road surface by a combination of various movements relative to three mutually perpendicular axes defined by the tractor vehicle 100: a lateral axis X, a fore-aft axis Y, and a central vertical axis Z. The lateral axis X extends between the right side and the left side of the tractor vehicle 100. A forward driving direction along the fore-aft axis Y is designated as F, also referred to as forward movement. Additionally, a rearward or backward driving direction along the fore-aft direction Y is designated as R, also referred to as backward movement. When the suspension system 118 adjusts the suspension of the tractor vehicle 100, the tractor vehicle 100 can tilt about the X-axis and / or the Y-axis, or move along the central vertical axis Z.

[0021] The towing vehicle 100 may include a user interface 130. The user interface 130 may include a display 132, a knob, and a button used as an input mechanism. In some examples, the display 132 may show a knob and a button. In other examples, the knob and the button are a knob-button combination. In some examples, the user interface 130 receives one or more driver commands from the driver via one or more input mechanisms or a touch screen display 132, and / or displays one or more notifications to the driver. The user interface 130 communicates with the vehicle controller 170, which in turn communicates with the sensor system 140. In some examples, the display 132 displays an image of the environment of the towing vehicle 100, causing the user interface 130 to receive one or more commands (from the driver) that initiate the execution of one or more behaviors. In some examples, the user display 132 displays a representation image of the rear environment of the towing vehicle 100, which includes one or more representations 136, 136a-c of the trailers 200, 200ac positioned behind the towing vehicle 100. In this case, the driver makes a selection 134 of a representation 136 , 136a - c of a trailer 200 , 200 ac which indicates that the driver wants the vehicle to be driven autonomously towards the trailer 200 .

[0022] The towing vehicle 100 may include a sensor system 140 to provide reliable and robust driving. The sensor system 140 may include different types of sensors that may be used alone or together to create environmental awareness of the towing vehicle 100 that is used for autonomous driving of the towing vehicle 100 and / or to help the driver make intelligent decisions based on objects and obstacles detected by the sensor system 140. The sensor system 140 provides data 141 (such as measurements of the movement of the towing vehicle 100) by using known camera and vehicle parameters, which is used to calculate the expected change of a specific point (e.g., trailer coupler 212) in the camera image 143. If the object being tracked (e.g., trailer coupler 212) is stationary, the only movement in the image 143 is due to the movement of the towing vehicle 100, and the tracking system 300 can determine how the coupler position 312 will change in the next image 143.

[0023] The sensor system 140 may include one or more cameras 142 supported by the towing vehicle 100. In some embodiments, the towing vehicle 100 includes a rear camera 142 mounted to provide a view of the rear driving path of the towing vehicle 100. The rear camera 142 may include a fisheye lens including an ultra-wide angle lens that produces strong visual distortions intended to create a wide panoramic or hemispherical image. The fisheye camera captures images with an extremely wide view. In addition, the images captured by the fisheye camera have a characteristic convex non-linear appearance. Other types of cameras may also be used to capture images of the rear environment of the towing vehicle 100.

[0024] In some examples, the sensor system 140 also includes one or more wheel encoders 144 associated with one or more wheels 112, 112a-d of the tractor vehicle 100. The wheel encoder 144 is an electromechanical device that converts the angular position or movement of the wheel into an analog or digital output signal. Thus, the wheel encoder 144 determines the speed and distance that the wheel 112, 112a-d has traveled.

[0025] The sensor system 140 may also include an inertial measurement unit (IMU) 146 that is configured to measure the linear acceleration (using one or more accelerometers) and rotation rate (using one or more gyroscopes) of the vehicle. In some examples, the IMU 146 also determines a heading reference for the towing vehicle 100. Thus, the IMU 146 determines the pitch, roll, and yaw of the towing vehicle 100.

[0026] In some embodiments, the sensor system 140 includes a radar sensor 148 configured to determine the distance, angle, or speed of an object. The radar sensor 148 transmits a radio signal and measures the time it takes for a reflection of the transmitted signal to return. The radar sensor 148 determines the distance to the object as half the round trip time multiplied by the signal speed. The radar sensor 148 can also determine the speed of the towing vehicle 100. The speed can be based on the distance from the radar sensor 148 to the object at a first time, and the distance from the radar sensor 148 to the object at a second time. Therefore, the radar sensor 148 can determine the speed based on the difference between the distances at the first time and the second time divided by the difference between the first time and the second time.

[0027] The sensor system 140 also includes other sensors 150 that detect vehicle motion (i.e., speed, angular velocity, positioning, etc.). The other sensors 105 may include, but are not limited to, steering wheel sensors, sonar, LIDAR (light detection and ranging, which may require optical remote sensing, which measures the properties of scattered light to find the distance and / or other information of distant targets), LADAR (laser detection and ranging), ultrasonic, etc.

[0028] In some embodiments, the towing vehicle 100 includes a global positioning system (GPS) receiver 160 that receives information from GPS satellites and then calculates a geographic location 162 of the towing vehicle 100 .

[0029] In some embodiments, the vehicle controller 170 includes a computing device (or processor) 172 (e.g., a central processing unit having one or more computing processors) that communicates with a non-transitory memory 174 (e.g., a hard disk, flash memory, random access memory, memory hardware) capable of storing instructions executable on the computing processor(s) 172.

[0030] The vehicle controller 170 executes a tracking system 300 that analyzes the received image 143 and identifies the trailer coupler 212 within the image 143 and tracks the trailer coupler 212 as the towing vehicle 100 moves in the rearward direction R. The tracking system 300 includes a coupler point tracker 310, a low speed motion estimation module 320, an image point motion module 330, and a coupler point fusion module 340. The tracking system 300 provides an estimate of the coupler position L by fusing the data 312 of the coupler point tracker 310 within the image 143 and the camera motion determined by the low speed motion estimation module 320. C of the improvements.

[0031] refer to Figure 3 In some embodiments, the coupler point tracker module 310 receives the image 143 from the rear camera 142. The coupler point tracker 310 identifies the trailer coupler 212 within the image 143 and tracks the coupler 212 in subsequently received images 143. Thus, the coupler point tracker 310 determines the current coupler position 312 in the current image 143, (u + , v + ) The connector position 312 in the image is given by (u, v) Definition, where u is the pixel location along the x image axis of the image, v is the pixel location along the y image axis of the image.

[0032] The low speed motion estimation module 320 receives sensor data 141 from one or more sensors 142, 144, 146, 148, 150 of the sensor system 140. In some examples, the low speed motion estimation module 320 receives one or more of the image(s) 143 from the camera 142, wheel ticks 145 from the wheel encoder(s) 144, IMU data 147 from the IMU 146, radar data 149 from the radar 148, and other sensor data 151 from other sensors 150. In some examples, the low speed motion estimation module 320 receives GPS data 162 from the GPS receiver 160. In some examples, the low speed motion estimation module 320 determines the vehicle linear velocity and angular velocity 322 of the towing vehicle 100 based on the received sensor data 141, (v, ω,) ,(in v is the linear velocity, ω is the angular velocity) and the vehicle attitude 324, (x, y, θ) . The vehicle posture 324 is composed of (x, y, θ) Definition, where x is the location of the center of the tractor vehicle 100 along the horizontal axis X in the XY plane, y is the location of the center of the vehicle along the longitudinal axis Y in the XY plane, and θ is the heading of the towing vehicle 100. The low-speed motion estimation module 320 can also be based on the vehicle linear velocity and angular velocity 322, (v, ω) and vehicle posture 324, (x, y, θ) To determine camera motion information, such as camera speed and camera positioning (camera pose), because camera motion is directly related to vehicle motion because camera 142 is supported by towing vehicle 100. In other examples, low-speed motion estimation module 320 determines camera motion, and based on the camera motion, low-speed motion estimation module 320 determines vehicle linear velocity and angular velocity 322, (v, ω) and vehicle posture 324, (x, y, θ) .

[0033] In some embodiments, the image point motion module 330 estimates the pixel velocity 332 of pixels associated with the coupler 212 within one or more images 143. The image point motion module 330 provides the tracking system 300 with an estimate of where the coupler 212 will be (i.e., where it will move) in the next image 143. The image point motion module 330 receives the vehicle linear velocity and angular velocity 322 from the low speed motion estimation module 320, (v, ω,) and vehicle posture 324, (x, y, θ) In addition, the image point motion module 330 receives or determines the coupling height Z. The coupling height Z may be input by the driver via the user interface 130, or, for example, according to the Figure 5The image point motion module 330 also receives the previous connector positions 342p associated with the previous image 143. In some examples, each current connector position 342c, (u + *, v + *) is stored in the hardware memory 174 and is associated with a timestamp. Thus, when a new current coupling position 342c is determined, the previously stored current coupling position 342c becomes the previous coupling position 342p. Based on the received vehicle linear velocity and angular velocity 322, (v, ω) and vehicle posture 324, (x, y, θ) , coupler height Z, and previous coupler position 342p, the image point motion module 330 determines a pixel velocity 332 of pixels associated with the coupler 212 within the one or more images 143.

[0034] In some examples, the image point motion module 330 converts the real-world connector position L c Determined as ( X, Y, Z ). The image point motion module 330 can determine the position of the connector in the image 143 by the following formula: (u,v) :

[0035]

[0036] in (u,v) is the coupling position, and the vehicle attitude is given by x, y, θ Define, and the connector position L in the world c Depend on( X, Y, Z ) definition. Therefore, the connector position (u, v) is the vehicle posture 324, (x, y, θ) and the connector position L in real world coordinates c , ( X, Y, Z ) function.

[0037] The image point motion module 330 inverts equation (1) to obtain the position of the connector in the image. (u,v) and the real-world coupling height Z to determine the vehicle position in the xy plane (X, Y) In some examples, the coupling height Z is within a certain range (e.g., between 40 cm and 70 cm);

[0038] .

[0039] Next, the image point motion module 330 calculates the Jacobian matrix of the function according to the first equation (1). The low-speed motion estimation module 320 determines the speed of the image point or the pixel speed 332 based on the Jacobian matrix (3). , which is the speed of the connector pixels within the image;

[0040]

[0041] in is the change in pixel positioning relative to the change in time, is the change in pixel positioning with respect to the change in time. ; ; ;and .

[0042] In some embodiments, the connector point fusion module 340 is based on the connector pixel velocity 332, and the previously determined connector point position 342p (previously determined by the connector point fusion module 340 and stored in the memory hardware 174) to determine the updated connector point position 342c in the current image 143. In other words, the connector point fusion module 340 determines the updated connector point position 342c in the current image 143 based on the connector pixel velocity 332 and the previously determined connector point position 342p to update the connector point position 312 determined by the connector point tracker 310 in the current image 143, , to determine updated coupler position 342c in current image 143. As such, tracking system 300 improves coupler position 342c determination by combining image pixel velocity 332, coupler position 342p in previous image 143, and coupler position 312 based on coupler point tracker 310.

[0043] In some embodiments, controller 170 executes tracking system 300 for each received image 143 such that towing vehicle 100 autonomously maneuvers to move in rearward direction R based on updated coupler position 342 to align with tow bar 214 supporting coupler 212 for hitching.

[0044] The vehicle controller 170 executes a driving assistance system 190, which in turn includes a path following behavior 192. The path following behavior 192 receives the current coupler position 342c in the current image 143 and executes one or more behaviors 192 ab that send a command 194 to the driving system 110 to cause the towing vehicle 100 to autonomously drive in a rearward direction R toward the coupler position 342c in the current image 143, which causes the towing vehicle 100 to autonomously connect to the trailer 200. In some examples, the controller 170 determines the real-world location of the coupler position 342c in the current image 143 so that the driving assistance system 190 follows the coupler position L. C The real-world coordinates of .

[0045] The path following behavior 192 includes a braking behavior 192a, a speed behavior 192b, and a steering behavior 192c. In some examples, the path following behavior 192 also includes a hitch connection behavior and a suspension adjustment behavior. Each behavior 192a-192c causes the tractor vehicle 100 to take an action, such as driving backward, turning at a specific angle, braking, speeding, slowing down, among others. The vehicle controller 170 can manipulate the tractor vehicle 100 to move in any direction across the road surface by controlling the driving system 110, more specifically by issuing commands 194 to the driving system 110.

[0046] Based on the coupling position 342c in the current image 143, a braking action 192a may be performed to stop or slow the towing vehicle 100. The braking action 192a sends a signal or command 194 to the driving system 110, such as a braking system (not shown), to stop or slow the towing vehicle 100.

[0047] A speed behavior 192b may be executed to change the speed of the towing vehicle 100 by accelerating or decelerating based on the planned path. The speed behavior 192b sends a signal or command 194 to the braking system 114 for deceleration or to the acceleration system 116 for acceleration.

[0048] A steering action 192c may be executed to change the direction of the towing vehicle 100 based on the coupler position 342c in the current image 143. As such, the steering action 192c sends a signal or command 194 indicating a steering angle to the acceleration system 116, causing the steering system 110 to change direction.

[0049] Reference Figure 4 and Figure 5In some embodiments, the tracking system 300 includes an integration module 350, a summation module 360, and a height adjustment module 370, which are used together to determine the height Z of the trailer coupling 212 in the world coordinate system. The integration module 350, the summation module 360, and the height adjustment module 370 can be independent modules or part of the coupling point fusion module 340. The integration module 350 receives the image pixel velocity 332 from the image point motion module 330, (ie, image derivative), and outputs a pixel point or position 352 based on the pixel velocity determined from the pixel velocity 332. In some examples, the integration module 350 determines the pixel point 352 by integrating the received pixel velocity 332.

[0050] In some examples, under benign conditions (e.g., good lighting without sharp shadow lines or misleading image features, etc.), the connector point tracker 310 accurately follows one or more image pixels from one image frame to another over time. The connector point trackers 310 may (self-)report some metrics of the estimated quality of their tracking performance.

[0051] During periods when the coupler point tracker 310 performs well (either through its self-reported quality metric or some other means), its output 312 may be compared to the output of the image point motion module 330. Both the coupler point tracker 310 and the image point motion module 330 report the location of features (i.e., trailer coupler 212) in the world within the image 143. Therefore, the output of the coupler point tracker 310 and the output of the image point motion module 330 may be directly compared to each other by the summing module 360. If the two outputs 312, 352 are the same, then the height Z of the coupler 212 in the real world is accurate. However, if the outputs 312, 352 are not the same, then the height Z of the coupler 212 is adjusted by the height adjustment module 370. In an initial step, the height adjustment module 370 may select a coupler height Z between 40 centimeters and 70 centimeters from the road surface, and based on the output of the summing module 360, the height adjustment module 370 adjusts the height Z of the coupler 212. In some examples, the selection of the coupling height Z may be varied between limits until the outputs 312, 352 are the same. Once the height Z is adjusted, in the next captured image 143, the image point motion module 330 determines the pixel velocity 332 by taking the adjusted height Z into account.

[0052] The image point motion module 330 relies on the height Z of a known point in the world (i.e., the trailer coupler 212), while the coupler point tracker 310 does not rely on the height Z. Therefore, the tracking system 300 determines the coupler height Z, for example, by way of pre-input, user input, or calculation. In some examples, the height Z is estimated by comparing the output 312 of the coupler point tracker 310 and the output 332 of the image point motion module 330 under benign conditions of the coupler point tracker 310, and adjusting the height input Z of the image point motion module 330 until its output 332 matches the output 312 of the coupler point tracker 312. In this way, the system 300 can estimate the height Z of the feature used in the real world (i.e., the trailer coupler 212).

[0053] In some embodiments, when the condition of the coupler point tracker 310 is not benign, the system 300 can make better use of the image point motion module 330 to improve the tracking performance.

[0054] Tracking system 300 fuses image 143 with vehicle dynamics (i.e., sensor data 141) to predict the location of the connector points within image 143 and in world coordinates. By incorporating vehicle motion, the location of stationary objects within the image (relative to the world) can be determined in future image frames.

[0055] Figure 6 Provides the use of Figure 1-5 6 is an example operational arrangement of a method 600 for tracking a trailer coupler 212 positioned behind a towing vehicle 100 as described in the system described in FIG. 6. At block 602, the method 600 includes receiving one or more images 143 at the data processing hardware 172 from a camera 142 positioned on the back of the towing vehicle 100 and in communication with the data processing hardware 172. At block 604, the method 600 includes determining a towing coupler location 312 within the one or more images 143 at the data processing hardware 172. The trailer coupler location 312 is associated with the trailer coupler 212. Thus, the trailer coupler location 312 is a pixel location within the image 143. At block 606, the method 600 includes receiving vehicle sensor data 141 at the data processing hardware 172 from one or more vehicle sensors 142, 144, 146, 148, 150, 160 in communication with the data processing hardware 172. The one or more vehicle sensors may include, but are not limited to, a steering wheel angle sensor, a wheel encoder, an inertial measurement unit, a radar, and a global positioning system receiver.

[0056] Additionally, at block 608, the method 600 includes determining, at the data processing hardware 172, vehicle motion data 322, 324 based on the received vehicle sensor data 141. The vehicle motion data 322, 324 may include vehicle linear and angular velocities 322 and vehicle pose 324. At block 610, the method 600 also includes determining, at the data processing hardware 172, an updated coupler position 342c for the current image based on the trailer coupler position 312 and the vehicle motion data 322, 324.

[0057] In some embodiments, the method 600 includes determining a pixel velocity 332 of one or more points of the trailer coupler 212 within the one or more images 143 based on the vehicle motion data 322 , 324 and the previous coupler position 342 , The updated connector position 342c is based on the pixel velocity and the previous connector position. The pixel velocity 332, The method 600 may also include determining the coupling height Z from a predetermined range (eg, 40-70 cm). The method may also include determining the coupling height Z based on the pixel velocity 332, A first pixel location 352 associated with the trailer coupler 212 is determined, and a second pixel location 312 associated with the trailer coupler 212 is determined based on the one or more images 143. The method 600 also includes comparing the first pixel location 352 and the second pixel location 312, and adjusting the coupler height Z based on the compared first pixel location 352 and second pixel location 312.

[0058] In some embodiments, the method 600 includes storing the updated coupler location 342c at memory hardware 174 in communication with the data processing hardware 172, and retrieving the updated coupler location 342c as the previous coupler location 342p when one or more updated images 143 are received.

[0059] In some examples, the method 600 further includes determining a connector position L in world coordinates based on the updated connector position 342c. C The method 600 may include transmitting instructions 194 to the driving system 110 of the towing vehicle 100. The instructions 194 cause the towing vehicle 100 to autonomously maneuver in the rearward direction R toward the coupling position L. C move.

[0060] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0061] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0062] The subject matter and implementation of the functional operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. In addition, the subject matter described in this specification may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that implement a machine-readable propagation signal, or a combination of one or more of them. The terms "data processing device", "computing device" and "computing processor" cover all devices, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, a device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0063] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0064] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A method for tracking a trailer coupler positioned behind a towing vehicle, the method comprising: receiving, at the data processing hardware, one or more images from a camera positioned on the back of the towing vehicle and in communication with the data processing hardware; determining, at the data processing hardware, a trailer coupler location within the one or more images, the trailer coupler location being associated with the trailer coupler; receiving, at the data processing hardware, vehicle sensor data from one or more vehicle sensors in communication with the data processing hardware; determining, at the data processing hardware, vehicle motion data based on the received vehicle sensor data; determining, at the data processing hardware, an updated coupler position based on the trailer coupler position and the vehicle motion data; and determining a pixel velocity of a trailer coupler within the one or more images based on the vehicle motion data and a previous coupler location; The updated connector position is based on the pixel velocity and the previous connector position. 2 . The method according to claim 1 , wherein the vehicle motion data comprises vehicle attitude, vehicle linear velocity and vehicle angular velocity. 3 . The method of claim 1 , wherein the one or more vehicle sensors include at least one of a steering wheel angle sensor, a wheel encoder, an inertial measurement unit, a radar, and a global positioning system receiver.

4. The method of claim 1, wherein the pixel velocity is based on a connector height in a real-world coordinate system.

5. The method according to claim 4, further comprising: determining a coupling height from a predetermined range; determining a first pixel location associated with the trailer coupler based on the pixel velocity; determining a second pixel location associated with a trailer coupler based on the one or more images; Comparing the first pixel position and the second pixel position; and The connector height is adjusted based on the compared first pixel position and second pixel position.

6. The method according to claim 1, further comprising: storing the updated connector location at memory hardware in communication with the data processing hardware; and When one or more updated images are received, the updated coupler locations are retrieved as the previous coupler locations.

7. The method according to claim 1, further comprising: determining a connector position in world coordinates based on the updated connector positioning; and Commands are transmitted to the driving system of the towing vehicle, which cause the towing vehicle to autonomously maneuver in a rearward direction toward the coupler position.

8. A system for tracking a trailer coupler positioned behind a towing vehicle, the system comprising: Data processing hardware; and Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: receiving one or more images from a camera positioned on the back of a towing vehicle and in communication with the data processing hardware; determining a trailer coupler location within the one or more images, the trailer coupler location being associated with the trailer coupler; receiving vehicle sensor data from one or more vehicle sensors in communication with the data processing hardware; determining vehicle motion data based on the received vehicle sensor data; determining an updated coupler position based on the trailer coupler position and the vehicle motion data; and determining a pixel velocity of a trailer coupler within the one or more images based on the vehicle motion data and a previous coupler location; Wherein the updated connector position is based on the pixel velocity and the previous connector position. 9 . The system of claim 8 , wherein the vehicle motion data comprises vehicle attitude, vehicle linear velocity, and vehicle angular velocity.

10. The system of claim 8, wherein the one or more vehicle sensors include at least one of a steering wheel angle sensor, a wheel encoder, an inertial measurement unit, a radar, and a global positioning system receiver.

11. The system of claim 8, wherein the pixel velocity is based on a connector height in a real world coordinate system.

12. The system of claim 11, wherein the operations further comprise: determining a coupling height from a predetermined range; determining a first pixel location associated with the trailer coupler based on the pixel velocity; determining a second pixel location associated with a trailer coupler based on the one or more images; Comparing the first pixel position and the second pixel position; and The connector height is adjusted based on the compared first pixel position and second pixel position.

13. The system of claim 8, wherein the operations further comprise: storing the updated connector location at the memory hardware; and When one or more updated images are received, the updated coupler locations are retrieved as the previous coupler locations.

14. The system of claim 8, wherein the operations further comprise: determining a connector position in world coordinates based on the updated connector positioning; and Commands are transmitted to the driving system of the towing vehicle, which cause the towing vehicle to autonomously maneuver in a rearward direction toward the coupler position.

Images