Apparatus and method for determining the center of a trailer towing coupler

Abstract

A method for determining a position of a target (212) positioned behind a towing vehicle (100) is provided. The method includes receiving images from a camera (142) positioned on the back of the towing vehicle. The images include the target. The method includes applying one or more filter sets to the images. The method also includes determining a region of interest (400) within each image based on the applied filter sets. The region of interest includes the target. The method also includes identifying the target within the region of interest and determining a target position of the target, including a position in a real-world coordinate system. The method also includes transmitting instructions to a driving system (110) supported by the vehicle. The instructions cause the towing vehicle to autonomously maneuver toward the position in the real-world coordinate system.

Description

Technical Field

[0001] The present disclosure relates to apparatus and methods for determining the center of a trailer towing coupler. Background Art

[0002] A trailer is typically an unpowered vehicle pulled by a powered towing vehicle. A trailer may be a utility trailer, a pop-up camper, a travel trailer, a livestock trailer, a flatbed trailer, an enclosed car transporter, and a boat trailer, among others. The towing vehicle may be a car, a crossover, a truck, a van, a sport utility vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to attach to and pull the trailer. A trailer may be attached to a powered vehicle using a trailer hitch. A receiver hitch is mounted on the towing vehicle and is connected to the trailer hitch to form a connection.

[0003] In some examples, a trailer includes a coupler tongue (i.e., a trailer hitch), a drawbar connecting the coupler tongue to the trailer, and a jack support for supporting the trailer before connecting to the vehicle, the jack support pivoting around the axle center of the trailer wheel. The connections and features of trailers vary greatly. For example, trailer-drawbar-coupler features include different shapes, colors, drawbar types, coupler types, jack support types, tongue attachments to the trailer frame, the location of the jack support, additional objects of the trailer support, the size and load-bearing capacity of the trailer, and the number of axles. Therefore, different combinations of the above features result in many different trailer models, making the trailer types very diverse. In addition, in some examples, the trailer owner personalizes his / her trailer, which further makes the trailers more different. Therefore, a vehicle having a trailer detection system configured to identify a trailer positioned behind a vehicle may have difficulty identifying a trailer or trailer hitch due to the different types of trailers available. In addition, it may be difficult for a vehicle identification system to identify a trailer positioned on the back of a vehicle because the trailer may be positioned on various surfaces that are difficult to identify, such as grass, dirt roads, beaches, etc., in addition to well-maintained roads. In current trailer identification systems, if the identification system has not previously stored such an environment, the identification system cannot identify the trailer. In addition, current trailer identification systems that use vision-based methods to detect trailers tend to generalize the characteristics of trailer features, i.e., tow bars, couplers, which results in the inability to detect trailers with uncommon features, custom trailers, or personalized trailers in uncommon environments (i.e., environments other than well-maintained roads). In addition, due to the large amount of data processing required, the production cost of trailer identification system hardware for detecting different trailers is very high. Therefore, current trailer detection systems cannot distinguish specific trailer combinations in multiple trailer scenarios.

[0004] It would be desirable to have a system that could solve the above mentioned problems by quickly and easily identifying a trailer-drawbar-coupler combination, regardless of the shape and surface on which the trailer is positioned. Summary of the invention

[0005] One aspect of the present disclosure provides a method for determining the position of a target positioned behind a towing vehicle. The method includes receiving an image at data processing hardware from a camera positioned on the back of the towing vehicle and in communication with the data processing hardware. The image includes the target. The method also includes applying one or more filter groups to the image by the data processing hardware. The method includes determining, by the data processing hardware, a region of interest within each image based on the applied filter groups. The region of interest includes the target. The method also includes identifying, by the data processing hardware, the target within the region of interest. The method also includes determining, by the data processing hardware, a target position of the target, including a position in a real-world coordinate system. The method also includes transmitting instructions from the data processing hardware to a driving system supported by the vehicle and in communication with the data processing hardware. The instructions cause the towing vehicle to autonomously maneuver toward the position in the real-world coordinate system.

[0006] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the method further includes tracking the target by the data processing hardware as the towing vehicle autonomously maneuvers toward the identified target. The method may also include determining, by the data processing hardware, an updated target position. In some examples, the method includes transmitting updated instructions from the data processing hardware to the driving system. The updated instructions cause the towing vehicle to autonomously maneuver toward the updated target position.

[0007] In some implementations, the camera includes a fisheye camera that captures the fisheye image. In some examples, the method further includes correcting the fisheye image by the data processing hardware before applying the one or more filter banks.

[0008] In some embodiments, the method includes receiving, at data processing hardware, training images stored in a hardware memory in communication with the data processing hardware, and determining, by the data processing hardware, a training region of interest within each received image. The training region of interest includes a target. The method may include determining, by the data processing hardware, one or more filter banks within each training region of interest. In some examples, the method further includes: identifying, by the data processing hardware, a center of the target, wherein the target location includes a location of the center of the target.

[0009] The target may be a coupler in a drawbar-coupler supported by a trailer. The image may be a top-down view of the drawbar-coupler. In some examples, the target is a trailer positioned behind a towing vehicle, and the target position is a position at the center of the bottom of the trailer at the drawbar. The image is a perspective view of the trailer.

[0010] Another aspect of the present disclosure provides a system for determining the position of a target positioned behind a towing vehicle. The system includes: data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving images from a camera positioned on the back of a trailer and in communication with the data processing hardware. The images include the target. The operations include applying one or more filter groups to the images. The operations include determining a region of interest within each image based on the applied filter groups. The region of interest includes the target. The operations include identifying the target within the region of interest, and determining a target position of the target, including a position in a real-world coordinate system. The operations include transmitting instructions to a driving system supported by the vehicle and in communication with the data processing hardware. The instructions cause the towing vehicle to autonomously maneuver toward a position in a real-world coordinate system.

[0011] Implementations of this aspect of the disclosure may include one or more of the following optional features. In some implementations, the operations further include tracking the target as the towing vehicle autonomously maneuvers toward the identified target. The operations may include determining an updated target position and transmitting updated instructions to the driving system. The updated instructions cause the towing vehicle to autonomously maneuver toward the updated target position.

[0012] In some implementations, the camera includes a fisheye camera that captures the fisheye image. The operations may include correcting the fisheye image prior to applying the one or more filter banks.

[0013] In some examples, the operations further include: receiving training images stored in a hardware memory in communication with the data processing hardware; and determining a training region of interest within each received image. The training region of interest includes an object. The operations may also include determining one or more filter banks within each training region of interest. The operations further include identifying a center of the object, wherein the object location includes a location of the center of the object.

[0014] In some examples, the target is a coupler in a drawbar-coupler supported by a trailer. The image may be a top-down view of the drawbar-coupler. The target may be a trailer positioned behind a towing vehicle, and the target location is a location at the center of the bottom of the trailer at the drawbar. The image is a perspective view of the trailer.

[0015] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages should be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic top view of an exemplary towing vehicle at a distance from a trailer;

[0017] Figure 2 is a schematic diagram of an exemplary vehicle;

[0018] Figure 3 is a schematic diagram of an exemplary arrangement of operations performed during a training phase;

[0019] Figure 4A is a schematic diagram including an exemplary image of a drawbar and a coupler;

[0020] Figure 4B and 4C is a schematic diagram of exemplary images including a trailer at different distances from a towing vehicle;

[0021] Figure 5 is a schematic diagram of an exemplary operational arrangement for determining a region of interest proposal;

[0022] Figure 5A is a schematic diagram of an exemplary image of a trailer captured by a rear-view vehicle camera;

[0023] Figure 6 is a schematic diagram of an exemplary trailer coupler detector and an exemplary tracker;

[0024] Figures 7A-7C is a schematic side view of a trailer positioned at different distances behind a vehicle;

[0025] Fig.7D is a schematic top view of a trailer positioned behind a vehicle illustrating different positioning of the viewport;

[0026] Figure 8 is a schematic diagram of an exemplary operational arrangement for determining the location of a coupler and a center of a trailer bottom portion;

[0027] Fig. 9 is a schematic diagram of an exemplary arrangement of operations performed by an adaptive hitch ball height change detector;

[0028] Fig.10 is a schematic diagram of an exemplary operational arrangement for determining a target and causing a towing vehicle to steer toward the target;

[0029] The same reference numerals in different drawings denote the same elements. DETAILED DESCRIPTION

[0030] A towing vehicle, such as, but not limited to, a car, a crossover, a truck, a van, a sport utility vehicle (SUV), and a recreational vehicle (RV), may be configured to tow a trailer. The towing vehicle is connected to the trailer by means of a trailer hitch coupler and a trailer hitch ball supported by the vehicle. It is desirable to have a towing vehicle that is capable of detecting a tow bar of the trailer, and then detecting and locating the center of the trailer hitch coupler positioned on the tow bar of the trailer. Further, it is desirable that the towing vehicle detects the tow bar, and detects and locates the center of the trailer coupler when the vehicle is moving in reverse toward the trailer. As such, the towing vehicle having a detection module provides information to the driver and / or the vehicle that assists (either the driver or autonomously) in driving the towing vehicle in reverse toward the trailer. The detection module is configured to learn how to detect a trailer having a trailer hitch bar and a trailer coupler, and the detection module may receive images from a camera having a high distortion rate (e.g., a fisheye camera) based on the learned data, and determine the location of the tow bar and the center of the coupler when the towing vehicle approaches the trailer. In addition, a towing vehicle including a detection module easily identifies a trailer-drawbar-coupler combination regardless of the shape and surface on which the trailer is positioned. During the learning phase, the detection module learns to detect any shape of the drawbar and any shape of the coupler, and during the detection phase, the detection module can use the learned data to accurately locate the coupler center and / or (one or more) key points of the trailer. In some examples, the learning phase and the detection phase include using a cascade method on the rigidly connected components of the trailer-drawbar-coupler combination (e.g., a trailer body such as a box, V-nose, boat, etc., a trailer frame with two or more wheels for bolting the trailer body thereto, and a coupler tongue, which is sometimes part of the frame and sometimes belongs to a separate component bolted to the drawbar) so that the coupler center positioning confidence can be improved through the relationship between these components. For example, the cascade method implements an algorithm that first detects the trailer body, then detects the drawbar-coupler combination, then zooms in to observe the coupler center and its surrounding features at a close distance, and finally detects the coupler tongue. As long as there are enough features to identify the connector center L CCP , the zoom ratio can be anywhere between 1.5-2.5. The detection module is also able to differentiate between multiple classes of trailer-drawbar-coupler combinations parked side by side in a scene such as a tow yard.

[0031] Reference Figure 1A-4, in some embodiments, the driver of the tractor vehicle 100 desires to tow a trailer 200 positioned behind the tractor vehicle 100. The tractor vehicle 100 may be configured to receive an indication of a driver selection associated with a selected trailer 200. In some examples, the driver steers the tractor vehicle 100 toward the selected trailer 200; while in other examples, the tractor vehicle 100 autonomously steers toward the selected trailer 200. The tractor vehicle 100 may include a driving system 110 that steers the tractor vehicle 100 across a road surface, for example, based on a driving command having x, y, and z components. As shown, the driving system 110 includes a right front wheel 112, 112a, a left front wheel 112, 112b, a right rear wheel 112, 112c, and a left rear wheel 112, 112d. The driving system 110 may also include other wheel configurations. The driving system 110 may also include a braking system 114 including a brake associated with each wheel 112, 112 ad and an acceleration system 116 configured to adjust the speed and direction of the towing vehicle 100. In addition, the driving system 110 may include a suspension system 118 including tires associated with each wheel 112, 112 ad, tire air, springs, shock absorbers, and a link connecting the towing vehicle 100 to its wheels 112, 112 ad and allowing relative movement between the towing vehicle 100 and the wheels 112, 112 ad. The suspension system 118 may be configured to adjust the height of the towing vehicle 100, thereby allowing the towing vehicle hitch 120 (e.g., the towing vehicle hitch ball 122) to align with the trailer hitch 210 (e.g., the coupler-towbar combination 210) supported by the trailer drawbar 214, which allows connection between the towing vehicle 100 and the trailer 200.

[0032] The tractor vehicle 100 can move across a road surface by various combinations of movement relative to three mutually perpendicular axes defined by the tractor vehicle 100, the three mutually perpendicular axes being: a transverse axis X v , front and rear axis Y v and the central vertical axis Z v The transverse axis x extends between the right side and the left side of the towing vehicle 100. v The forward driving direction is designated as F v , also called forward motion. In addition, along the front-to-back direction Y v The backward or reverse driving direction is designated as R v , also known as backward or reverse motion. When the suspension system 118 adjusts the suspension of the towing vehicle 100, the towing vehicle 100 can move around X v Axis and / or Y v Axis tilt, or along the central vertical axis Z v move.

[0033] The towing vehicle 100 may include a user interface 130, such as a display. The user interface 130 receives one or more user commands from the driver via one or more input mechanisms or a touch screen display 132, and / or displays one or more notifications to the driver. The user interface 130 communicates with a vehicle controller 150, which in turn communicates with a sensor system 140. In some examples, the user interface 130 displays an image of the environment of the towing vehicle 100, resulting in the user interface 130 receiving (from the driver) one or more commands that initiate the performance of one or more actions. In some examples, the user display 132 displays one or more representations 136 of trailers 200 positioned behind the towing vehicle 100. In this case, the driver selects 134 which representation 136 of the trailer 200 the controller 150 should identify, detect, and locate the center of the coupler 212 associated with the selected trailer 200. In other examples, the controller 150 detects one or more trailers 200, and detects and locates the center of the trailer coupler 212 of the one or more trailers. The vehicle controller 150 includes a computing device (or processor) 152 (e.g., a central processing unit having one or more computing processors) in communication with a non-transitory memory 154 (e.g., a hard disk, flash memory, random access memory, memory hardware) capable of storing instructions executable on the computing processor(s) 152 .

[0034] The tractor vehicle 100 may include a sensor system 140 to provide reliable and robust driving. The sensor system 140 may include different types of sensors that may be used alone or together with each other to create a sense of the environment of the tractor vehicle 100, which is used to steer the tractor vehicle 100 and help the driver make intelligent decisions based on objects and obstacles detected by the sensor system 140, or help the driving system 110 autonomously maneuver the tractor vehicle 100. The sensor system 140 may include, but is not limited to, radar, sonar, LIDAR (light detection and ranging, which may require optical remote sensing to measure the properties of scattered light to find the range and / or other information of distant targets), LADAR (laser detection and ranging), (one or more) ultrasonic sensors, etc.

[0035] In some embodiments, the sensor system 140 includes one or more cameras 142, 142a-n supported by the vehicle. In some examples, the sensor system 140 includes a rear view camera 142a that is mounted to provide a view of the rear driving path of the towing vehicle 100. The rear view camera 410 may include a fisheye lens that includes an ultra-wide angle lens that produces strong visual distortions intended to create wide panoramic or hemispherical images. The fisheye camera 142a captures images with an extremely wide viewing angle. In addition, the images captured by the fisheye camera 142a have a characteristic convex non-linear appearance.

[0036] Detection module 160

[0037] The vehicle controller 150 includes a detection module 160 that receives the image 144 (i.e., the fisheye image) from the rear view camera 410a and determines the location of the tow bar 214 and the location of the coupler 212 within the image(s) 144 and in world coordinates, such as the location of the center of the coupler 212. In some examples, the detection module 160 determines the center of the tow bar 214 and the coupler 212 at long-range, mid-range, and short-range distances between the towing vehicle 100 and the trailer 200. In some embodiments, the detection module 310 includes a training / learning phase 170 followed by a detection phase 180. During the training / learning phase 170, the detection module 160 executes a trailer and coupler training module 172. Additionally, during the detection phase 180 , the detection module 160 executes the following: an ROI suggestion determiner 182 , a trailer and coupler detector 184 , a pose and scale estimator 186 , a Kalman multi-channel correlation filter (MCCF) tracker 188 , a confidence calculation module 190 , an adaptive hitch ball height change detector 192 , and a real-world positioning estimation module 194 .

[0038] Training / Learning Phase 170

[0039] Figure 3 and 4A -4C illustrates the training / learning phase 170. Figure 3 , the detection module 310 executes the trailer and coupler trainer module 172 to teach the Kalman MCCF tracker 188 how to detect the coupler 212 and the drawbar 214. Figure 3 , the trainer module 172 receives an input image 144 captured by a rear-view camera 410a (ie, a fisheye camera). The input image 144 includes a region of interest 400 of a trailer 200 or a coupler-drawbar combination 210 (including a coupler 212 and a drawbar 214). Figure 4A , the input image 144 includes the image 144 and the pixel position L of the center of the coupler 212 in the coupler-traction rod combination 210CCP , because in this case, the trailer and coupler trainer module 172 is learning how to detect the coupler 212 within the coupler-drawbar combination 210 ROI 400c. Figure 4B and 4C The trailer 200 is shown. In the training image 144, the pixel position L of the center of the coupling 212 is CCP ( Figure 4A ) or the pixel position L of the center of the trailer bottom at the tow bar 214 TCP is predetermined prior to the training / learning phase 170, e.g., pixel (64, 42) in the (100×100 pixel) training image 144. In some examples, the ROIs 400, 400c, 400t are rescaled to a fixed pixel size, e.g., (100×100 pixel) for the training and learning phase 170. The trainer module 172 maintains the center L of the connector 212 in the training image 144 for a particular pixel range. CCP or the center of the trailer bottom at the trailer drawbar 214 L TCP Remain in the same position within image 144 .

[0040] Figure 4A A top-down corrected image 146 based on the image 144 captured by the rear view camera 142a is illustrated. The controller 150 adjusts the received image 144 to remove distortion caused by the rear view camera 142a being a fisheye camera. Since the camera 142a is positioned above the coupler-towbar combination 210 relative to the road surface, the image 144 of the coupler-towbar combination 210 captured by the camera 142a is generally a top-down view of the coupler-towbar combination 210. In addition, the trainer module 172 determines a region of interest 400, 400c within the corrected image 146 that includes the coupler-towbar combination 210.

[0041] Figure 4B and 4C 2 are perspective views of the corrected image 146 including the trailer 200 at different distances. Figure 4B is the corrected image 146b associated with the image 144 captured at the first distance from the trailer 200, and Figure 4C146 is a corrected image 146 associated with another image 144 captured at a second distance from the trailer 200, wherein the second distance is greater than the first distance. Since the trailer 200 is positioned behind the towing vehicle 100, the camera 142a can capture the straight-ahead image 144 of the trailer 200. Therefore, the corrected images 146b, 146c based on the straight-ahead image 144 are also straight-ahead views. In some examples, the trainer module 172 determines a trailer ROI 400t within the corrected images 146b, 146c that includes the trailer 200. As shown, the corrected images 146b, 146c are also ROIs 400t identified by the training module 172.

[0042] As described above, trainer module 172 corrects captured image 144. In some examples, in the coupling-towbar combination 210 Figure 4A In the top-down view shown in FIG. 1 , the appearance of the drawbar-coupler (i.e., the aspect ratio of the shape) of the coupler-drawbar assembly 210 does not change. The change in the proportions of the coupler-drawbar assembly 210 from top to bottom indicates the change in the height of the coupler-drawbar assembly 210. Figure 4B and 4C In the rear view perspective shown in FIG. 1 , given a fixed pitch angle, the appearance of trailer 200 does not change; however, if the pitch angle changes, the appearance of trailer 200 changes. In some examples, given a reference image path with a fixed pitch angle, the above-mentioned appearance change due to the change in the trailer pitch angle can be used to estimate the trailer pitch angle. The image path is around the trailer.

[0043] Furthermore, given a reference patch with a fixed yaw angle, the above appearance changes can be used to estimate the trailer yaw angle (i.e., orientation). The correlation energy is maximized when the test image matches the training image. Back to Reference Figure 3 At step 302, the trainer module 172 performs a normalization function on the ROI 400, 400c, 400t to reduce illumination variations. Then, at step 304, the trainer module 172 performs a gradient function, for example, by determining the difference in image pixel values ​​within the ROI 400, 400c, 400t in both the x and y directions. The trainer module 172 determines the magnitude and orientation according to the following formula:

[0044] Amplitude: (sqrt(Gx 2 +Gy 2 )); (1)

[0045] Orientation: (arctan2(Gy / Gx)). (2)

[0046] The trainer module 172 may determine equations (1) and (2) based on the gradient in the x-direction (Gx) and the gradient in the y-direction (Gy) to determine the directional change in intensity or color in the received image ROI 400, 400c, 400t and determine a histogram of the gradients.

[0047] The Histogram of Gradients (HOG) is a feature descriptor used in computer vision and image processing to characterize one or more objects within an image. HOG determines multiple bins (e.g., 5 bins) of gradient orientation in a local portion of an image. Each bin represents a specific range of orientations.

[0048] At step 304, based on the HOG, the trainer module 172 determines a plurality of bins associated with the HOG. The trainer module 172 performs a cell normalization function and a block normalization function. During the cell normalization function, the average values ​​of the gradient magnitude and gradient orientation on each cell are determined, the cell size being, for example, (5×5 pixels). The trainer module 172 determines the number of cells, for example (20×20 pixels), based on the size of the ROI 400, 400c, 400t (e.g., 100×100 pixels) divided by the cell size (e.g., (5x5)). If the trainer module 172 determines that the average gradient magnitude in a particular cell is below zero, the trainer module 172 sets the value of the cell to 0. If the trainer module 172 determines that the average gradient orientation in a particular cell is below zero, the trainer module 172 sets the value of the cell to 0. The trainer module 172 determines an average gradient orientation bin based on the average gradient orientation, and then multiplies the average gradient orientation bin by the inverse of the average gradient magnitude plus 0.1. During the block normalization function, all per-cell normalized gradient orientation channels are squared and summed by the trainer module 172 and averaged over the cell size (e.g., (5×5)). When the gradient squared interval sum (GSsum) is less than zero, the trainer module 172 sets GSsum to zero. The trainer module 172 obtains the final gradient sum (Gsum) by taking the square root of GSsum. The trainer module 172 normalizes the final HOG interval by Gsum, where the cell normalized gradient interval is again divided by Gsum plus 0.1. The steps of box 304 adapt to lighting and environmental changes.

[0049] At step 306, the trainer module 172 applies a Gaussian filter around the target location, which is the center of the connector L CCP (as "pos" in the equation below) or the pixel position L of the center of the trailer bottom at the tow bar 214 TCP :

[0050] rsp(i,j) = exp( -((i-pos(1))2 +(j-pos(2)) 2 ) / (2*S 2 ) ) (3)

[0051] S is sigma, an adjustable parameter. Since this is a learning step, the trainer module 172 knows the location of the connector 212 in the image and therefore applies the Gaussian filter around the known location. The Gaussian response, the size of the image patch (as "rsp") is also transformed to the frequency domain for fast computation time, thereby obtaining the Gaussian response in the frequency domain (as "rsp_f") (which is the size of the image 144 in the frequency domain).

[0052] At step 308, the trainer module 172 applies a mask to the ROI 400, 400, 400, which improves performance. The trainer module 172 applies a mask to mask the area around the trailer 200 or the coupler-drawbar combination 210. At step 310, the trainer module 172 applies a cosine window function on the unit / block normalized HoG channel to reduce the high frequency of the image boundary of the corrected image 146 (i.e., ROI 400), and transforms the HoG channel to the frequency domain for fast calculation time, thereby obtaining HoG_f (HOG frequency). At step 312, the trainer module 172 calculates the autocorrelation energy xxF and the cross-correlation energy xyF, respectively. The autocorrelation energy xxF is obtained by multiplying HoG_f by the transpose of HoG_f. The cross-correlation energy xyF is obtained by multiplying rsp_f by the transpose of HoG_f. At step 314, the trainer module 172 sums the autocorrelation energy xxF and the cross-correlation energy xyF in the same range across multiple ROIs 400. At step 316, the trainer module 172 solves the equation:

[0053] MCCF = lsqr(xxF+lambda,xyF) (4)

[0054] Equation (4) solves for a filter bank 322 (i.e., MCCF) and is transformed from the frequency domain back to the image domain at step 318. The filter bank 322 is a multi-channel correlation filter bank 322 that provides features of the trailer or coupler-drawbar combination 210 (i.e., trailer hitch 201). As such, the filter bank 322 is subsequently used during the detection phase 180 to determine the location of the trailer 200 or coupler 212 within the captured image 144, where the positioning of the trailer 200 or coupler 212 is unknown within the image 144. At step 320, the trainer module 172 stores the filter bank 322 (i.e., MCCF) associated with the trailer 200 and coupler-drawbar combination 210 determined by equation (4) in the memory hardware 154.

[0055] In some embodiments, the trainer module 172 performs the training of the images 144 captured at the far range, the medium range, and the near range. Figure 3 The steps in are used to respectively determine three filter banks 322 to adapt to the changes in appearance, resolution, and scale of trailer 200 and coupler-drawbar combination 210 within each image 144.

[0056] In some embodiments, the training / training and learning phase 170 may be performed on the raw fisheye images 144 along the vehicle centerline Y having a zero orientation angle at a specific distance from the trailer 200 in a supervised manner, such as, for example, in a dynamic driving scenario. In some examples, the training / training and learning phase 170 may be further simplified by using a single image frame or a few image frames. In some examples, the captured fisheye images 144 may be Figures 4A-4C 146. The trainer 172 uses a top-down view for filter learning associated with the coupler-drawbar combination 210, while the trailer 172 uses a front-view perspective for filter learning of the trailer 200. In some examples, the learning images 144 are stored in the memory 154. The learning images 144 may include the trailer 200 and the towing vehicle in a hitched position, where the position of the truck hitch ball 122 is known. The filter bank 322 learned based on the corrected images 146 during the training and learning phase 170 can be applied to the corrected images 146 for determining the center position L of the trailer 200 during real-time operation. CCP and the pixel position L of the center of the trailer bottom at the tow bar 214 TCP In some examples, the center position L of the trailer 200 CCP By capturing image 144 in the front view perspective ( Figure 4B and 4C The center of the coupling can be determined by capturing image 144 in a close range top-down view ( Figure 4A ). In some embodiments, the training and learning phase 170 may also include images of the trailer 200 and the towing vehicle 100 in an unhitched position. In some examples, the image 144 of the trailer 200 may be captured three to four meters from the towing vehicle 100. The image 144 may include the side of the trailer 200 facing the towing vehicle 100.

[0057] In some examples, during the training and learning phase 170, trailer 200 and coupler-drawbar combination 210 in corrected images 146 are not randomly oriented, but have known orientation angles, such as, for example, 0° or 90°. In addition, trailer 200 and coupler-drawbar combination 210 are in a front perspective ( Figure 4B and 4C ) and a top-down view ( Figure 4A) are orthogonally connected to each other within the captured image 144, similar to how they are connected in real life.

[0058] In some embodiments, the trainer module 172 corrects the captured top-down view image 146a of the drawbar-coupler ROI 400c so that the drawbar-coupler ROI 400c is at a zero orientation relative to the towing vehicle 100, i.e., the longitudinal axis Y of the towing vehicle 100, when the towing vehicle 100 is hitched to the trailer 200. In addition, the trainer corrects the perspective images 146b, 146c of the trailer 200 so that the trailer ROI 400b is at a zero orientation relative to the towing vehicle 100, i.e., the longitudinal axis Y of the towing vehicle 100, when the towing vehicle 100 is hitched to the trailer 200.

[0059] In some examples, the filter bank 322 learned during the learning process 170 is used to estimate the trailer pitch of the trailer 200. The trainer module 172 can mask out areas within the image 144 that are not the coupler-drawbar combination 210 or the trailer 200. Thus, the trainer module 172 uses only the areas of the coupler-drawbar combination 210 or the trailer 200 within the ROI 400 of the image 144, and the surrounding areas are masked out during training, so that the trained ROI 400 can be used in different environmental conditions with consistent results. This approach reduces the amount of training data stored in the memory 154.

[0060] In some embodiments, the trainer module 172 analyzes the center L of the connector 212. CCP or the center L of the trailer bottom at the drawbar 214 TCP 400. Thus, during the detection phase 180, the controller 150 can use the associated energy from the additional keypoints during real-time operation to determine a confidence value. Thus, the keypoints within the training ROI 400 can be matched with the keypoints identified in the image captured in real time to increase the confidence of the detection phase 180.

[0061] In some implementations, the trainer module 172 may generate a three-dimensional (3D) view of the trailer 200, the drawbar 214, and the coupler 212 during the training and learning phase 170. Furthermore, the trainer module 172 may generate the 3D view at scale so the trainer module 172 can determine the ratio between the physical trailer components and the normalized ROI 400. In some examples, the trainer module 172 determines and learns shape characteristics of the trailer 200 and the coupler-drawbar combination 210.

[0062] In some examples, the trainer module 172 is executed when a driver of the towing vehicle 100 first uses the towing vehicle 100. Thus, the trainer module 172 determines the filter set 322 based on one or more images 144 received during the driver's first use of the towing vehicle. In some examples, the filter set 322 can also be used in an online adaptive learning process, or the towing vehicle 100 can receive additional filter sets 322 from an online system.

[0063] In some examples, the training and learning phase 170 is performed on a cloud computing hardware device that is separate from the towing vehicle 100. For example, the rear view camera 142a of the towing vehicle 100 captures an image 144 of the environment behind the vehicle and transmits the captured image 144 to the cloud computing hardware device via a wireless Internet connection. The cloud computing hardware device executes the trainer module 172, and once the filter bank 322 is determined, the cloud computing hardware device transmits the filter bank 322 back to the towing vehicle 100 by way of a wireless Internet connection. The towing vehicle 100 receives the filter bank 322 and stores the filter bank 322 in the memory hardware 154.

[0064] Detection stage 180

[0065] Once the training and learning phase 170 is performed and completed, the detection phase 180 may be performed. The detection module 160 performs the following operations: ROI suggestion determiner 182, trailer and coupler detector 184, pose and scale estimator 186, Kalman multi-channel correlation filter (MCCF) tracker 188, confidence calculation module 190, adaptive hitch ball height change detector 192, and real world position estimation module 194. The ROI suggestion determiner 182 (which may be optionally implemented) analyzes the captured images 144 and outputs ROI suggestions 400p that include the coupler-towbar combination 210 or trailer 200. The trailer and coupler detector 184 then analyzes the ROI suggestion 400p and uses the information from the real world position estimation module 194 to determine the ROI suggestion 400p. Figure 4A–4C ROI 400c, 400t to locate the trailer (if far) and the drawbar-coupler (if close). The pose and scale estimator 186 analyzes the image as a whole and determines the pose of the trailer 200. When the towing vehicle 100 is moving towards the trailer 200, behind the Kalman MCCF tracker 188 is the located trailer or drawbar coupler (located by the trailer and coupler detector 184). The confidence calculator 190 calculates the confidence to determine the location of the coupler. In some examples, the adaptive vehicle hitch ball height change detector 192 checks the vehicle hitch ball height change based on the current trailer hitch 210 positioning. The adaptive vehicle hitch ball height change detector 192 can be executed when the trailer 200 and the towing vehicle 100 are in the hitch positioning, so the adaptive vehicle hitch ball height change detector checks the hitch ball height change and verifies that the training / learning information stored in the memory hardware 152 matches the features learned from the current hitch ball height, so that a potential collision can be avoided in the next hitch attempt. Finally, the real-world position estimation module 194 determines the position of the coupler 212 based on the above modules 182 - 192 in real-world coordinates.

[0066] ROI Proposal Determiner 182

[0067] Figure 5 A method 500 is shown, which is performed by the ROI suggestion determiner 182 to determine one or more ROI suggestions 400p based on the received image 144, and automatically detect the coupler-drawbar combination 210 or trailer 200 within the received image 144 by applying a close-fitting bounding box (ROI) around the coupler-drawbar combination 210 or trailer 200. At box 502, the ROI suggestion determiner 182 receives one or more images 144. At box 504, the ROI suggestion determiner 182 analyzes the received image 144 and determines a valid region for analysis. For example, referring to Figure 5A For the detection of the coupler 212, the ROI suggestion determiner 182 determines the valid zone 530 based on the camera calibration to exclude the area above the horizon and the area including the body of the towing vehicle 100. Similarly, for the detection of the trailer, the ROI suggestion determiner 182 determines the valid zone 530 based on the camera calibration to exclude the area above the trailer 200 and the area below the trailer 200.

[0068] Next, the ROI suggestion determiner 182 may perform one of three methods to determine the ROI suggestion 400p (i.e., the ROI suggestion 400p may include the coupler ROI 400c or the trailer ROI 400t). Method 1: At box 506, the ROI suggestion determiner 182 iteratively applies the learned filter bank 322 (i.e., MCCF) on a series of scan windows covering the image area 530 until the maximum peak is found. This first method is called a brute force search method and may be used when runtime is not an issue).

[0069] Method 2: The ROI suggestion determiner 182 may perform Method 2 for determining the ROI suggestion 520. At box 508, the ROI suggestion determiner 182 first segments the received region 530 of the image 144 using SLIC (Simple Linear Iterative Clustering) to find superpixels, and then constructs a RAG (Region Adjacency Graph) to merge regions with similarity. Pixel similarity may be based on intensity, distance, color, edge, and texture. At box 510, the ROI suggestion determiner 182 constructs a region adjacency graph based on the segmentation of box 508. RAG is a data structure used for segmentation algorithms and provides vertices representing regions and edges representing connections between adjacent regions.

[0070] At box 512, the ROI suggestion determiner 182 merges the regions within the determined region 530 of the image 144. For example, if two adjacent pixels are similar, the ROI suggestion determiner 182 merges them into a single region. If two adjacent regions are collectively similar enough, the ROI suggestion determiner 182 similarly merges them. The merged region is called a superpixel. This collective similarity is typically based on comparing the statistics of each region.

[0071] At block 514, the ROI suggestion determiner 182 identifies possible trailer superpixels. If the superpixel has minimum and maximum size constraints and excludes irregular shapes based on predetermined trailer types, the superpixel qualifies as a trailer region of interest. At block 516, the detection module 310 identifies and merges possible trailer superpixels based on the estimated trailer shape and size at a particular image location to obtain ROI suggestions 400p. The ROI suggestion determiner 182 associates these ROI suggestions 400p with the trained MCCF filter 322 to find the center L for the connector 212. CCP or pixel position L at the center of the trailer bottom at the tow bar 214 TCP Automatic pixel positioning of peak energy.

[0072] Method 3: The ROI suggestion determiner 182 may perform method 3 for determining ROI suggestions 520. At box 518, the ROI suggestion determiner 182 may use any other object detection and classification method that can generalize the detection of multiple coupler-drawbar combinations 210 or trailers 200, but cannot identify a specific preferred trailer in a trailer park. In some examples, a deep neural network (DNN) or other generalization method may be used, but cannot identify a specific trailer type. The ROI suggestions 520 may be provided by other vision-based object detection and classification methods. The ROI suggestions 400p may also be determined by other sensors, such as radar or lidar, if available. Multiple ROI suggestion methods may be combined to reduce false positive ROI suggestions 400p.

[0073] Trailer and coupler detector 184

[0074] Figure 6 1 shows a flow chart of the trailer and coupler detector 184 as the towing vehicle 100 approaches the trailer 200. At box 602, the starting position of the towing vehicle 100 is generally within a distance D (in meters) from the trailer 200. At box 604, the trailer and coupler detector 184 receives parameters and the trained MCCF filter 322 (stored by the training phase) from the memory hardware 152. The parameters may include, but are not limited to, camera intrinsic and extrinsic calibration information, towing vehicle configuration, such as the distance between the towing ball 122 and the towing vehicle front axle, the wheel base of the towing vehicle and the initial height of the towing ball, and trailer configuration, such as the width of the trailer 200, the distance between the coupler 212 and the center of the trailer wheel. The camera intrinsic parameters may include focal length, image sensor format, and principal point, while the camera extrinsic parameters may include a coordinate system transformation from 3D world coordinates to 3D camera coordinates, in other words, the extrinsic parameters define the location of the camera center and the heading of the camera in world coordinates. At box 606, the trailer and coupler detector 184 may receive a pose estimate of the trailer 200 or coupler-drawbar combination 210 from other modules, or the pose and scale estimator 186 may iteratively estimate the pose and scale of the trailer 200 or coupler-drawbar combination 210 based on the MCCF 322 to find the maximum correlation energy between the trained filter and the trailer and coupler region of interest at a specific viewport pose and viewport center.

[0075] Then at block 608, the trailer and coupler detector 184 determines a dynamic viewport by adjusting the center of the viewport and the rotation angles (pitch, yaw, roll) of the viewport so that the corrected image 146 looks similar to a trained pattern with a known orientation (i.e., Figure 4A –ROI 400c, 400t shown in C). Fig. 7A–7D, by changing the distance of the viewport center from the camera and the viewing angle (such as pitch and yaw), the trailer’s appearance will change until it matches the trained pattern (i.e., Figure 4A –C) where the maximum correlation energy is achieved. In the real world, the orientation angle between the trailer 200 and the towing vehicle 100 is not perfectly aligned with zero, and the distance between the trailer 200 and the towing vehicle 100 also varies. The viewport 700 for the top-down view and the front view is defined with the center at (x, y, z) in the world coordinate system in millimeters. The size of the viewport 700 is defined in (width_mm × height_mm) and (width_px × height_px) with pitch, yaw and roll rotation angles relative to the AUTOSAR (Automatic Open System Architecture) vehicle coordinate system. Millimeters per pixel and pixels per millimeter of the image can be calculated. If the center of the correction viewport 700 is fixed, a scale estimation is required. The correction viewport center can be adapted in such a way that the size of the trailer in the corrected image is the same as the size of a single channel of the learned filter. Estimating the scale can be avoided by changing the viewport center. If the truck and trailer are not perfectly aligned in the real world, the correction viewport yaw angle can be adapted so that the appearance of the trailer in the front view maintains the same orientation and appearance as the training patch. The viewport yaw angle can be iteratively determined by finding the peak correlation energy with the trained filter. During trailer keypoint localization, in order to maintain the trailer size in the corrected image the same as the single-pass learned filter, a series of virtual cameras are placed at the center of the viewport in each frame.

[0076] The dynamic viewport 700 is configured to adjust the viewing distance, which is the longitudinal distance of the viewport center from the camera 142a, and the viewing angle, which is the viewport rotation angle (also referred to as the viewport attitude), such as pitch, yaw, and roll. Therefore, the appearance of the trailer 200 changes based on the viewport 700 with different center positions and attitude configurations. Therefore, at block 608, the detector 184 adjusts the viewport of the current image 144 so that it matches the trained ROI 400. Fig. 7A The viewport is shown at an intermediate distance between the towing vehicle 100 and the trailer 200 . Figure 7B The viewport is shown at a closer distance relative to the intermediate distance. Figure 7C A viewport is shown capturing a towing vehicle 100 and an attached trailer 200 . Fig.7D A top view illustration of different view centers and perspectives of viewport 700 is shown (the "eye" position is the viewport center, the "eye" direction is the viewport pose. The "eye" position can be indicative, for example, indicating a person standing between camera 142a and trailer 200 and looking at trailer 200 or looking around).

[0077] At block 610, the trailer and coupler detector 184 determines whether it is detecting a coupler-drawbar combination 210 or a trailer 200. Thus, when the trailer and coupler detector 184 determines that it is detecting a coupler-drawbar combination 210, then at block 612, the captured image 144 includes the image 144 as previously described in Figure 4A However, if the trailer and coupler detector 184 determines that it is detecting a trailer 200, then at block 614, the captured image 144 includes the image 144 as previously described in Figure 4B and 4C A perspective view of a trailer 200 is shown in FIG.

[0078] Next, at block 616, the trailer and coupler detector 184 instructs the ROI suggestion determiner 182 to determine a trailer ROI suggestion 400p or a towbar-coupler ROI suggestion 400p based on the decision of block 610. The trailer and coupler detector 184 also generates a lookup table (LUT) for each of the coupler-drawbar combination 210 and the trailer 200, respectively, wherein the lookup table LUT includes entries corresponding to pixel positions in the fisheye image 144 from the dynamic viewport (determined at block 608).

[0079] At box 618, the trailer and coupler detector 184 determines the coupler center L CCP or pixel position L at the center of the trailer bottom at the tow bar 214 TCP The location of the peak correlation energy within the ROI proposal 400p of the dynamic viewport image determined at block 608. The correlation energy refers to the correlation energy between features of two images (such as between the ROI proposal 400p image patch and the pre-trained trailer patch 400 - the key point of the trailer patch 400 is at the center of the bottom of the trailer (L TCP ) and also at the peak of the Gaussian curve - or a measure of similarity between the drawbar-coupler patch Gaussian-weighted at the coupler center) as a function of the pixel position relative to the other. The peak correlation energy in the ROI suggestion 400p corresponds to the Gaussian-weighted keypoint in the training image patch 400. In some examples, the detector 184 performs Figure 8 The method 800 shown in FIG. 8 is used to determine the coupling center L at the coupling-drawbar combination 210. CCP or the pixel position L of the center of the trailer bottom TCPAt block 802, the detector 184 retrieves the ROI proposal 400p determined by the determiner 182 from the memory hardware 154 and rescales it to the same size as the image patch used during the training phase. At block 802, the detector 184 performs a normalization function on the ROI proposal 400p to reduce illumination variations. At block 804, the detector 184 computes multi-channel normalized HOG features, similar to Figure 3 At block 806, the detector 184 applies a cosine window to the HOG channels and transforms them to the frequency domain, similar to Figure 3 At block 808, the detector 184 determines a correlation energy map as a function of pixel position between the ROI suggestion patch 400p and the trained filter 322 in the frequency domain and transforms back to the image domain. At block 810, the detector 184 determines the coupling center L at the coupling-traction rod combination 210 by finding the pixel position of the peak energy in the correlation energy map. CCP or the pixel position L of the center of the trailer bottom TCP .

[0080] Return to reference Figure 6 At block 620, the trailer and coupler detector 184 relies on a lookup table to locate the coupler center L determined within the ROI suggestion 400p of the dynamic viewport. CCP or pixel position L at the center of the trailer bottom at the tow bar 214 TCP Mapped to the original fisheye image 144 .

[0081] Kalman MCCF Tracker 188

[0082] At block 622, in some examples, the tracker 188 maps the connector center L at block 620 to the CCP or pixel position L at the center of the trailer bottom at the tow bar 214 TCP , tracking the coupling center L at the coupling-traction rod combination 210 in the fisheye image 144 CCP or the pixel position L of the center of the trailer bottom TCP The tracker 188 may include a Kalman MCCF tracker configured to track the viewport 700 and key points. At block 622, the tracker 188 tracks the viewport center and viewport pose in the original image and the trailer bottom center L TCP and connector center L CCP At block 624, the tracker 188 predicts the viewport center and viewport pose, and predicts an updated coupler center L at the coupler-tugbar combination 210 in the new viewport based on the predicted keypoints in the original image. CCP Or the center pixel position of the trailer bottom L TCPThe predicted viewport 700 may be used in block 186 as a reference to determine the next viewport pose and viewport center.

[0083] Confidence Calculation 190

[0084] In some embodiments, the confidence calculation module 190 determines the tracked coupler center L at the coupler-drawbar combination 210 CCP or the pixel position L of the center of the bottom of the tracked trailer TCP The confidence calculation module 190 may use the coupling center L at the coupling-drawbar combination 210 CCP Or the position L of the center of the trailer bottom TCP To determine the orientation of the trailer 200 and coupler-drawbar combination 210 in a top view or 3D view to check trailer pose confidence.

[0085] In addition, the confidence calculation module 190 applies a cascade method to determine the confidence of the key point location. First, the confidence calculation module 190 detects the trailer body, then the confidence calculation module 190 detects the coupler-drawbar combination 210, and the confidence calculation module 190 checks the constraints between the two key points. Then, the confidence calculation module 190 zooms in to take a closer look at the coupler center L CCP and its surrounding features. As long as there are enough features to identify the connector center L CCP , the zoom ratio can be 1.5-2.5. The confidence calculation module 190 checks the connector center L according to the cascade method CCP Finally, the confidence calculation module 190 determines a high resolution patch of the connector itself, and the confidence calculation module 190 analyzes its edges to find out the localization accuracy of the center of the connector. The confidence calculation module 190 uses several methods to estimate a metric to increase the confidence in the localization accuracy. One metric that the confidence calculation module 190 attempts to determine is the location of the center of the connector, L CCP Therefore, the confidence calculation module 190 analyzes the physical constraints and relationships within the trailer-drawbar-coupler combination, as well as key points with textures of different sizes, and the confidence calculation module 190 also analyzes the high-resolution edge features of the coupler. Once the confidence calculation module 190 completes its analysis, the confidence calculation module 190 determines the coupler center position L CCP If the confidence is below a threshold, the hitch operation may be stopped, or the algorithm may be reset to perform a more extensive search. The confidence calculation module 190 may analyze the history of previously determined confidences to determine a mismatch between the trained filter 322 and the actual trailer and drawbar-coupler.

[0086] Adaptive tow ball height change detector 192

[0087] Reference Fig. 9 In some embodiments, after the towing vehicle 100 is hitched to the trailer 200 , the adaptive hitch ball height change detector 192 determines whether the proportions of the features of the attached coupler-drawbar combination 210 (based on one or more captured images 144 ) are the same as the proportions associated with a previously learned coupler-drawbar combination 210 .

[0088] At box 902, the adaptive hitch ball height change detector 192 receives an image 144 from a rear view camera 142a of a towing vehicle 100 attached to a trailer 200. Since the towing vehicle 100 is attached to the trailer 200, the coupler center L CCP Overlaps with the center of vehicle hitch ball 122. Since image 144 is associated with coupler-tow bar combination 210, the image is Figure 4A . At block 904, the adaptive hitch ball height change detector 192 loads the learned filter bank 322 and the parameters of the hitch bar from the memory hardware 154. Next, at block 906, the adaptive hitch ball height change detector 192 generates a corrected top-down view 146 of the image 144. At block 908, the adaptive hitch ball height change detector 192 generates a corrected top-down view 146 of the image 144 by performing the following steps: Figure 8 The method 800 shown in finds the coupling center. At box 910, the adaptive traction ball height change detector 192 crops multiple image patches with different scale factors. The scale factor can be searched in an iterative manner in the direction where the larger correlation energy is found. These image patches are adjusted to the size of the patch (ROI 400) used during the training and learning phase 170, and then the correlation with the trained filter 322 is applied. The scale factor of the image patch with the largest peak correlation energy will be selected. At box 912, the adaptive traction ball height change detector 192 determines whether the scale of the feature within the captured image 144 with the largest peak energy is different from the scale of the feature associated with the trained ROI 400c, and whether the maximum peak energy is greater than a predetermined threshold. If one or both conditions are not met, the adaptive traction ball height change detector 192 determines that the image 144 does not include the coupling-traction bar combination 210 or the scale of the coupling-traction bar combination 210 has not changed. However, if both conditions are met, then at box 914, the adaptive traction ball height change detector 192 sets an indication flag to notify the driver that the traction ball height has changed, and therefore, the training data can be updated. At box 916, the adaptive traction ball height change detector 192 updates the information obtained during the learning phase ( Figure 3) during the training and learning phase 170 (i.e., MCCF). At box 918, the adaptive hitch ball height change detector 192 updates the scale associated with the image patch used during the training and learning phase 170, where the training patch has a fixed size. At box 920, the adaptive hitch ball height change detector 192 stores the updated filter bank 322 in the memory hardware 154. Therefore, the adaptive hitch ball height change detector 192 is a procedure for synchronizing the scale of the hitch bar-coupler patch with the current hitch ball height for the sake of a safety feature to prevent the hitch ball 122 and the coupler 212 from colliding with each other during the hitch. Once synchronized, when the coupler 212 and the hitch ball 122 are separated from each other, for example within one meter, a similar procedure with a safety feature can be performed to determine whether the coupler 212 is above or below the hitch ball 122, thereby preventing the coupler 212 and the hitch ball 122 from colliding with each other during the hitch process. This process is also referred to as relative height determination.

[0089] Real World Position Estimation Module 194

[0090] The real world position estimation module 194 determines the position of the trailer coupler 212 determined by the tracker 188 in the world coordinate system. The viewport center is a three-dimensional coordinate. If the same scale is maintained in each image frame during operation, and the 3D physical width of the trailer 200 is known, the relationship between the 3D coordinates of the trailer 200 and the viewport center can be determined. This method of trailer 3D distance estimation that relies on texture width and real world trailer width is robust to situations where there are uneven surfaces (such as beaches, dirt roads, grass). Further, in the real world, L TCP A fixed distance between the coupler center and the center of the trailer bottom at the drawbar is another useful constraint to optimize the 3D distance estimate of the coupler center and coupler height.

[0091] Therefore, once the real-world position estimation module 194 determines the real-world position of the trailer coupler 212, the real-world position estimation module 194 sends it to the driving assistance system 196. Based on the received real-world position, the driving assistance system 196 determines the path between the towing vehicle 100 and the trailer 200, and guides the towing vehicle 100 to align with the trailer 200 for hitching. In addition, the driving assistance system 196 sends one or more commands 198 to the driving system 110, so that the driving system 110 autonomously manipulates the towing vehicle 100 to move in the rearward direction R v Move toward trailer 200. In some examples, driver assistance system 196 instructs driver system 110 to position towing vehicle 100 so that the front and rear axles Y of towing vehicle 100 are aligned with the towing vehicle 100. V The front and rear axles Y of the trailer 200 T coincide.

[0092] Fig.10 Provides for use Figure 1-9 The system described in the foregoing claims is a method 1000 for determining the position of an object 200, 210, 212, 214 (e.g., trailer 200, coupler-drawbar combination 210, coupler 212, and drawbar 214) positioned behind a towing vehicle 100. At block 1002, the method 1000 includes receiving an image 144 at the data processing hardware 152 from a camera 142a positioned on the back of the towing vehicle 100. The camera 142a may include a fisheye camera. The image 144 includes the objects 200, 210, 212, 214. At block 1004, the method 1000 includes applying, by the data processing hardware 152, one or more filter banks 322 to the image 144. The filter banks 322 stored in the memory hardware 154 communicate with the data processing hardware 152. At block 1006, the method 1000 includes determining, by the data processing hardware 152, a region of interest (ROI) 400, 400c, 400t within each image 144 based on the applied filter bank 322. The ROI 400, 400c, 400t includes the object 200, 210, 212, 214. At block 1008, the method 1000 includes identifying, by the data processing hardware 152, the object 200, 210, 212, 214 within the ROI 400, 400c, 400t. At block 1010, the method 1000 includes determining, by the data processing hardware 152, the object location L of the object 200, 210, 212, 214. CCP , L TCP , including a position in a real-world coordinate system. At block 1012, the method 1000 includes transmitting instructions 195, 198 from the data processing hardware 152 to a driving system 110 supported by the towing vehicle 100 and in communication with the data processing hardware 152. The instructions 195, 198 cause the towing vehicle 100 to autonomously maneuver toward the position of the target 200, 212, 214 in the real-world coordinate system.

[0093] In some embodiments, the method 1000 includes: tracking the target 200, 210, 212, 214 by the data processing hardware 152 as the towing vehicle 100 autonomously maneuvers toward the identified target 200, 210, 212, 214; and determining, by the data processing hardware 152, an updated target position L TCP , L CCP The method 1000 may also include transmitting updated instructions 195, 198 from the data processing hardware 152 to the driving system 110. The updated instructions 195, 198 cause the towing vehicle 100 to autonomously maneuver toward the updated target position L TCP , L CCPIn some examples, where camera 142 a is a fisheye camera, method 1000 further includes correcting, by data processing hardware 152 , fisheye image 144 prior to applying one or more filter banks 322 .

[0094] In some embodiments, method 1000 includes receiving, at data processing hardware 152, training images 144 stored in a hardware memory 154 in communication with the data processing hardware 152. Method 1000 may also include determining, by the data processing hardware 152, a training ROI 400, 400c, 400t within each received image, the training ROI 400, 400c, 400t including the target 200, 210, 212, 214. Method 1000 may include determining, by the data processing hardware 152, one or more filter banks 322 within each training ROI 400, 400c, 400t. In some examples, method 1000 further includes identifying, by the data processing hardware 152, a center of the target, wherein the target location L TCP , L CCP Includes the location of the target center.

[0095] In some embodiments, the target 200, 210, 212, 214 is the coupler 212 of the coupler-drawbar combination 210 supported by the trailer 200. Thus, the image 144 is as shown in FIG. Figure 4A A top-down view of the coupler-drawbar combination 210 is shown in FIG.

[0096] In some embodiments, the target 200, 210, 212, 214 is a trailer 200 positioned behind the towing vehicle 100, and the target position L TCP is the location of the center of the bottom of the trailer at the drawbar 214. Thus, the image 144 is a perspective view of the trailer 200.

[0097] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0098] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0099] The subject matter and implementation of the functional operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. In addition, the subject matter described in this specification may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that implements a machine-readable propagation signal, or a combination of one or more of them. The terms "data processing device", "computing device" and "computing processor" cover all devices, devices and machines for processing data, including, by way of example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated in order to encode information for transmission to a suitable receiver device.

[0100] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0101] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A method for determining the position of an object positioned behind a towing vehicle, the method comprising: receiving, at data processing hardware, an image from a camera positioned on a back of the towing vehicle and in communication with the data processing hardware, the image including the target; applying, by the data processing hardware, one or more filter banks to the image; determining, by the data processing hardware, a region of interest within each image based on the applied filter bank, the region of interest including the object; identifying, by the data processing hardware, the target within the region of interest; determining, by the data processing hardware, a target position of the target including a position in a real-world coordinate system; as well as sending instructions from the data processing hardware to a driving system supported by the vehicle and in communication with the data processing hardware, the instructions causing the towing vehicle to autonomously maneuver toward a position in the real-world coordinate system, receiving, at the data processing hardware, a training image stored in a hardware memory in communication with the data processing hardware; determining, by the data processing hardware, a training region of interest within each received image, the training region of interest including a target; and One or more filter banks within each training region of interest are determined by the data processing hardware.

2. The method according to claim 1, further comprising: tracking the target by the data processing hardware as the towing vehicle autonomously maneuvers toward the identified target; determining, by the data processing hardware, an updated target location; as well as Updated instructions are sent from the data processing hardware to the driving system, the updated instructions causing the towing vehicle to autonomously maneuver toward the updated target position. The method of claim 1 , wherein the camera comprises a fisheye camera that captures fisheye images. 4 . The method of claim 3 , further comprising correcting the fisheye image by the data processing hardware before applying the one or more filter banks.

5. The method according to claim 1, further comprising: A center of the target is identified by the data processing hardware, wherein the target location includes a location of the center of the target.

6. The method of claim 1, wherein the target is a coupler in a drawbar-coupler supported by a trailer.

7. The method of claim 6, wherein the image is a top-down view of the drawbar-coupler.

8. The method of claim 1, wherein the target is a trailer positioned behind the towing vehicle and the target position is a position at the center of the bottom of the trailer at the drawbar.

9. The method of claim 8, wherein the image is a perspective view of the trailer.

10. A system for determining the position of an object positioned behind a towing vehicle, the system comprising: Data processing hardware; and Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: receiving an image from a camera positioned on a back of a towing vehicle and in communication with the data processing hardware, the image including a target; applying one or more filter banks to the image; determining a region of interest within each image based on the applied filter bank, the region of interest including the object; identifying the target within the region of interest; determining a target position of the target including a position in a real-world coordinate system; sending instructions to a driving system supported by the vehicle and in communication with the data processing hardware, the instructions causing the towing vehicle to autonomously maneuver toward a position in the real-world coordinate system; receiving a training image stored in a hardware memory in communication with the data processing hardware; determining a training region of interest within each received image, the training region of interest including the object; and One or more filter banks are determined within each training region of interest.

11. The system of claim 10, wherein the operations further comprise: tracking the target as the towing vehicle autonomously maneuvers toward the identified target; Determine the target location for the update; as well as Updated instructions are sent to the driving system, the updated instructions causing the towing vehicle to autonomously maneuver toward the updated target position.

12. The system of claim 10, wherein the camera comprises a fisheye camera that captures fisheye images.

13. The system of claim 12, wherein the operations further comprise correcting the fisheye image prior to applying the one or more filter banks.

14. The system of claim 10, wherein the operations further comprise identifying a center of the target, wherein the target location comprises a location of the center of the target.

15. The system of claim 10, wherein the target is a coupler in a drawbar-coupler supported by a trailer.

16. The system of claim 15, wherein the image is a top-down view of the drawbar-coupler.

17. The system of claim 10, wherein the target is a trailer positioned behind the towing vehicle and the target location is a location of the center of the bottom of the trailer at the drawbar.

18. The system of claim 17, wherein the image is a perspective view of the trailer.

Images