Autonomous agricultural equipment attached to a tractor

Abstract

A method for autonomously maneuvering a tractor vehicle to attach to a trailer using a hitch assist system, the method comprising: entering a hitch assist mode; displaying a camera image on a user interface; and receiving an indication of a selected trailer. The method also includes determining, using a computing device, a planned path, i.e., a tractor vehicle path from an initial position to a final position adjacent to the trailer, the tractor vehicle path including maneuvers configured to move the tractor vehicle along the tractor vehicle path from the initial position to the final position. The method also includes autonomously following the tractor vehicle path from the initial position to the final position at a drive system in communication with the computing device.

Description

Technical Field

[0001] The present disclosure relates to a tractor configured to identify one or more pieces of agricultural equipment positioned proximate to the tractor and to travel to one of the one or more pieces of agricultural equipment in order to attach the equipment to a trailer. Background Art

[0002] A trailer is typically an unpowered vehicle, implement, or equipment pulled by a powered tractor vehicle. When referring to agricultural equipment, the powered tractor vehicle typically refers to a tractor (rear-loading or front-loading) or any other vehicle configured to attach to and pull a trailer. The trailer may be any type of agricultural implement, such as a bailer, flatbed trailer, plow, seeder, harvester, baler, etc. In the case of a front-loading tractor, the trailer (i.e., the unpowered vehicle) may actually be located in front of the tractor vehicle (i.e., the powered vehicle).

[0003] A trailer can be attached to a tractor using a trailer hitch. A receiver hitch is mounted on the tractor and connects to the trailer hitch to form a connection. When handling farm equipment, the connection point between the tractor and trailer, or hitch, may vary depending on the type of implement being towed.

[0004] The trailer hitch may be a ball and socket type, spare wheel gooseneck type, trailer jack type, eyebolt type, lanyard drive type, drawbar type, or three-point hitch. Other attachment mechanisms may also be used. In addition to the mechanical connection between the trailer and the tractor, in some examples, the trailer is electrically connected to the tractor. Thus, the electrical connection allows the trailer to draw a feed from the tail light circuit of the powered tractor, thereby allowing the trailer to have lights that are synchronized with the lights of the powered tractor.

[0005] Some challenges faced by tractor operators are in connecting a tractor to a trailer because more than one person is required. For example, one person (e.g., the operator) drives the tractor, while another person or persons need to observe the tractor and trailer and provide guidance to the operator regarding the path the tractor must take to align with the hitch. If the person providing guidance to the operator is not comfortable with hooking a tractor to a trailer, then they may have difficulty providing effective instructions to direct the path of the tractor.

[0006] Recent advances in sensor technology have led to improved vehicle safety systems. Arrangements and methods for detecting and avoiding collisions have become available. Such operator assistance systems use sensors located on the vehicle to detect a collision that is occurring. In some examples, the system can alert the operator to one or more driving situations to prevent or minimize the collision. In addition, sensors and cameras can also be used to alert the operator of possible obstacles when the vehicle is traveling in a forward direction. Therefore, it is desirable to provide a system including sensors to overcome the challenges faced by operators of tractor-trailers. Summary of the invention

[0007] One general aspect includes a method of reverse maneuvering a tractor vehicle to attach to a trailer using a hitch assist system, the method comprising: entering a hitch assist mode. The method also includes displaying a camera image on a user interface, wherein the displayed image displays at least one camera. The method also includes receiving an indication of a selected trailer at the user interface in communication. The method also includes determining, using a computing device, a planned path, i.e., a tractor vehicle path from an initial position to a final position adjacent to the trailer, the tractor vehicle path including maneuvers configured to move the tractor vehicle along the tractor vehicle path from the initial position to the final position. The method also includes autonomously following the tractor vehicle path from the initial position to the final position at a drive system in communication with the computing device.

[0008] Implementations may include one or more of the following features. The method also includes: continuously detecting one or more objects within the tractor path at the neural network as the tractor moves along the tractor path. The method may also include changing the tractor path at the computing device when the object is detected. The method of detecting one or more trailers includes: capturing one or more images at one or more imaging devices in communication with the neural network, at least one of the one or more imaging devices being positioned on a rear side of the trailer facing a rearward direction. The method may also include determining one or more trailers in the one or more images at the neural network. The method of displaying on the user interface also includes: receiving one or more images from one or more cameras positioned on the rear of the tractor and in communication with the controller at the controller; and superimposing at the controller a path representation indicating an expected path for the tractor to travel along it, the expected path starting at the tractor hitch. The method of selecting a trailer also includes: receiving a first command at the controller through the user interface, the first command indicating a change in the path representation so that the path representation ends at a point of interest; and adjusting the path representation at the controller based on the first command, and wherein the path planning is performed by the controller of the hitch assist system. The method also includes: stopping or pausing the tractor at an intermediate position at the drive system before reaching the final position, the intermediate position being closer to the final position than the initial position. The method may also include modifying one or more tractor suspensions associated with the tractor at the drive system to align the tractor hitch with the trailer hitch. The method may also include autonomously following the tractor path from the intermediate position to the final position at the drive system. The method may also include connecting the tractor hitch with the trailer hitch at the drive system. The method of connecting the tractor hitch with the trailer hitch includes modifying one or more tractor suspensions associated with the tractor to align the tractor hitch with the trailer hitch.

[0009] One aspect of the present disclosure provides a method for autonomously driving a vehicle in a rearward direction toward a point of interest. The method includes receiving one or more images from a camera positioned on the rear of the vehicle and communicating with the data processing hardware at data processing hardware. The method also includes receiving an operator-planned path from a user interface in communication with the data processing hardware at the data processing hardware. The operator-planned path includes a plurality of waypoints. The method includes transmitting one or more commands from the data processing hardware to a drive system in communication with the data processing hardware, the commands causing the vehicle to autonomously maneuver along the operator-planned path. The method includes determining a current vehicle position at the data processing hardware. In addition, the method includes determining an estimated subsequent vehicle position based on the operator-planned path at the data processing hardware. The estimated subsequent vehicle position is at a subsequent waypoint along the operator-planned path from the current vehicle position. The method also includes determining a path adjustment from the current vehicle position to the estimated subsequent vehicle position at the data processing hardware. In addition, the method includes transmitting instructions from the data processing hardware to the drive system, the instructions causing the vehicle to autonomously maneuver toward the estimated subsequent vehicle position based on the path adjustment.

[0010] Another aspect of the present disclosure provides a system for autonomously maneuvering a vehicle in a rearward direction toward a point of interest. The system includes: data processing hardware; and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including the above method.

[0011] Implementations of aspects of the present disclosure may include one or more of the following optional features. In some implementations, the method includes: overlaying a path on one or more images; and receiving a command through a user interface in communication with data processing hardware. The command includes an instruction to adjust the path to an operator-planned path. The command may include an instruction to adjust the distance of the path. In some examples, the command includes an instruction to adjust the angle of the path. The command may include an instruction to adjust the angle of an end portion of the path.

[0012] In some examples, determining the current vehicle position includes receiving wheel encoder sensor data associated with one or more wheels and receiving steering angle sensor data. The current vehicle position is based on the wheel encoder sensor data and the steering angle sensor data.

[0013] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1is a schematic diagram of an exemplary tractor with a user interface displaying an indication of a trailer behind the tractor.

[0015] Figure 2 is a schematic diagram of an exemplary tractor.

[0016] Figure 3A and Figure 3B is a perspective view of an exemplary tractor autonomously maneuvering along a planned path.

[0017] Figure 4A is a perspective view of an exemplary tractor in an initial position.

[0018] Figure 4B is a perspective view of an exemplary tractor in a neutral position.

[0019] Figure 4C is a perspective view of an exemplary tractor in a final position.

[0020] FIG. 5A to FIG. 5D is a perspective view of an exemplary hitch coupled to a trailer hitch of an exemplary tractor system.

[0021] Fig. 6A and Figure 6B is a schematic diagram of a second exemplary tractor system.

[0022] Fig. 7A and Figure 7B is a schematic diagram of an exemplary state diagram of a system.

[0023] Fig. 8A and Figure 8B is a schematic diagram of an exemplary angle mode of a second exemplary tractor system.

[0024] Fig.9A and Fig. 9B is a schematic diagram of an exemplary distance mode of a second exemplary tractor system.

[0025] Fig. 10A and Fig. 10B is a schematic diagram of an exemplary double arc mode of a second exemplary tractor system.

[0026] FIG. 11A to FIG. 11B It is a schematic diagram of trajectory generation calculation.

[0027] FIG. 12A to FIG. 1 2D is a schematic diagram of motion estimation calculations for a second exemplary tractor system.

[0028] Fig.13 is a schematic diagram of an exemplary method for autonomously driving a vehicle in a rearward direction toward a point of interest.

[0029] Fig.14 is a flow chart of an exemplary arrangement for reversing the operation of a tractor to attach to a trailer of first and second exemplary tractor systems.

[0030] 15 is a schematic diagram of an exemplary method for autonomously driving a vehicle in a rearward direction toward a point of interest.

[0031] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0032] A tractor (such as but not limited to a rear-loading tractor or a front-loading tractor, hereinafter referred to as a tractor) may be configured to tow a trailer. A tractor may be other types of vehicles and agricultural equipment that are powered and configured to tow other agricultural implements. A trailer may be any type of agricultural implement, such as a watering machine, a flatbed trailer, a plow, a seed drill, a harvester, a baler, etc. In the case of a front-loading tractor, the trailer (i.e., a non-powered vehicle) may actually be located in front of the tractor (i.e., a powered vehicle). All such equipment will be referred to as trailers hereinafter.

[0033] A tractor is connected to a trailer via a trailer hitch. It would be desirable to have a tractor that can autonomously maneuver toward and attach to a trailer, thereby eliminating the need for an operator to steer the tractor (e.g., in a rearward direction) while another person or persons provide guidance to the operator regarding the path that must be taken to align with the trailer and ultimately the trailer's hitch. In this way, a tractor with autonomous driving and hitching features provides the operator with a safer and faster experience when hitching the tractor to a trailer.

[0034] refer to Figure 1 to Figure 2B. In some embodiments, the operator of the tractor 100 wants to tow a trailer 200, 200a-200c, for example, a specific trailer 200a-200c from a group of trailers 200, 200a-200c. The tractor 100 can be configured to receive an indication of an operator selection 144 associated with the selected trailer 200, 200a-200c and autonomously drive toward the selected trailer 200, 200a-200c. The tractor 100 can include a drive system 110 that steers the tractor 100 across a surface based on a drive command having, for example, x, y, and z components. As shown, the drive system 110 includes a right front wheel 112, 112a, a left front wheel 112, 112b, a right rear wheel 112, 112c, and a left rear wheel 112, 112d. The drive system 110 can also include other wheel configurations. The drive system 110 may also include a braking system 120 including brakes associated with each wheel 112, 112a-112d and an acceleration system 130 configured to adjust the speed and direction of the tractor 100. In addition, the drive system 110 may include a suspension system 132 including tires associated with each wheel 112, 112a-112d, tire air, springs, shock absorbers, and linkages that connect the tractor 100 to its wheels 112, 112a-112d and allow relative movement between the tractor 100 and the wheels 112, 112a-112d. The suspension system 132 improves the handling of the tractor 100 and provides a better ride quality by isolating noise, bumps, and vibrations. Additionally, the suspension system 132 is configured to adjust the height of the tractor 100 , allowing the tractor hitch 160 to align with the trailer hitch 210 , which allows for an autonomous connection between the tractor 100 and the trailer 200 .

[0035] The tractor 100 can move across a surface by various combinations of motions relative to three mutually perpendicular axes defined by the tractor 100: a lateral axis X, a fore-aft axis Y, and a central vertical axis Z. The lateral axis X extends between the right side R and the left side of the tractor 100. The forward drive direction along the fore-aft axis Y is designated as F, also referred to as forward motion. Additionally, the rearward or reverse drive direction along the fore-aft direction Y is designated as R, also referred to as reverse motion. When the suspension system 132 adjusts the suspension of the tractor 100, the tractor 100 can tilt about the X axis and / or the Y axis, or move along the central vertical axis Z. The example shown here is a trailer 200 located in a rearward position of the tractor 100. However, this is shown merely as exemplary in nature, and a person skilled in the art will be able to apply the embodiment to a trailer 200 also located in a front or generally rearward position of the tractor 100. Alternatively, a person skilled in the art will be able to apply the embodiment to a tractor 100 having a front loading tractor hitch 160. Furthermore, to simplify the drawing, the tractor 100 and trailer 200 are shown using a hitch with a ball hitch and receiver. Other types of hitches may be used with the exemplary embodiments.

[0036] The tractor 100 may include a user interface 140, such as a display. The user interface 140 receives one or more user commands from an operator via one or more input mechanisms or a touch screen display 142, and / or displays one or more notifications to the operator. The user interface 140 communicates with a controller 300, which in turn communicates with a sensor system 400. In some examples, the user interface 140 displays an image of the environment of the tractor 100, resulting in one or more commands being received by the user interface 140 (from the operator), which initiates the performance of one or more actions. The controller 300 includes a computing device (or processor) 302 (e.g., a central processing unit having one or more computing processors) in communication with a non-transitory memory 304 (e.g., a hard disk, flash memory, random access memory) capable of storing instructions executable on the computing processor.

[0037] The controller 300 executes an operator assistance system 310, which in turn includes a path following subsystem 320. The path following subsystem 320 receives a planned path 552 ( Figure 3A and Figure 3B ), and executes actions 322-330, which send command 301 to drive system 110, causing tractor 100 to autonomously travel in a rearward direction R around planned path 552.

[0038] The path following subsystem 320 includes a braking behavior 322, a speed change behavior 324, a steering behavior 326, a hitch connection behavior 328, and a suspension adjustment behavior 330. Each behavior 330, 330a-330c causes the tractor 100 to take an action, such as driving backward, turning at a specific angle, braking, accelerating, decelerating, etc. By controlling the drive system 110, and more specifically, by issuing commands 301 to the drive system 110, the controller 300 can steer the tractor 100 across the surface in any direction. For example, the controller 300 can move the tractor 100 from an initial position (such as Figure 5A as shown) to the final position (as shown Figure 5C In the final position, the hitch coupler 162 of the tractor 100 is aligned with the hitch coupler 212 of the trailer 200, and the hitch coupler 212 connects the tractor 100 and the selected trailer 200, 200a-200c.

[0039] The tractor 100 may include a sensor system 400 to provide reliable and robust autonomous driving. The sensor system 400 may include different types of sensors that may be used alone or together with each other to generate a perception of the tractor's environment that is used for the tractor 100 to autonomously drive and make intelligent decisions based on objects and obstacles detected by the sensor system 400. The sensors may include, but are not limited to, one or more imaging devices (such as cameras) 410 and sensors 420, such as, but not limited to, radar, sonar, LIDAR (light detection and ranging, which may require measuring the properties of scattered light to find the distance and / or other information of distant targets), LADAR (laser detection and ranging), etc. In addition, when the tractor 100 is traveling in the forward direction F or in the rearward direction R, the camera 410 and the sensor 420 can be used to warn the operator of possible obstacles through an audible alarm and / or a visual alarm via the user interface 140. Therefore, the sensor system 400 is particularly useful for improving the safety of the tractor 100 operating under semi-autonomous or autonomous conditions.

[0040] In some embodiments, the tractor 100 includes a rear camera 410, 410a mounted to provide a view of a rear travel path of the tractor 100. Additionally, in some examples, the tractor 100 includes a front camera 410, 410b that provides a view of a front travel path for the tractor 100, a right camera 410, 410c positioned on the right side of the tractor 100, and a left camera 410, 410d positioned on the left side of the tractor 100. The left camera 410, 410d and the right camera 410, 410c provide additional side views of the tractor 100. In this case, in addition to objects and obstacles detected along the front and rear travel paths, the tractor 100 can also detect objects and obstacles positioned on either side of the tractor 100. The cameras 410, 410a-410d can be monocular cameras, binocular cameras, or another type of sensing device capable of providing a view of a rear travel path of the tractor 100.

[0041] In some embodiments, the tractor 100 includes one or more neural networks (NN) 500, such as deep neural networks (DNNs), to improve the autonomous driving of the tractor 100. DNNs 500 are computational methods used in computer science and other disciplines, and are based on a collection of a large number of neural units, roughly imitating the way biological brains solve problems using large clusters of biological neurons connected by axons. DNNs 500 are self-learning and trained, rather than programmed, and excel in areas where solution feature detection is difficult to express in traditional computer programs. In other words, DNNs 500 are a set of algorithms designed to recognize patterns. DNNs 500 interpret sensor system data 402 (e.g., from sensor systems 400) by machine sensing, labeling, or clustering raw inputs. The recognized patterns are numerical vectors, and all real-world data such as images, text, sound, or time series are converted into vectors. DNNs 500 include multiple layers of nonlinear processing units 502 that communicate with DNN non-transitory memory 504. The DNN non-transitory memory 504 stores instructions that, when executed on the non-linear processing units 502, cause the DNN 500 to provide outputs 506, 508. Each non-linear processing unit 502 is configured to transform an input or signal (e.g., sensor system data 402) using parameters learned through training. A series of transformations from input (e.g., sensor system data 402) to output 506, 508 occur at multiple layers of non-linear processing units 502. Thus, the DNN 500 is able to determine a location based on the image 412 or sensor data 422, thereby eliminating the need to have a DGPS or GPS.

[0042] The DNN 500 receives the sensor system data 402 (including the image 412 and / or the sensor data 422) and provides an image output 506 to the user interface 140 and / or a data output 508 to the controller 300 based on the received data 402. In some examples, the DNN 500 receives an image 412 of a rear view of the tractor 100 from a camera 410 in communication with the DNN 500. The DNN 500 analyzes the image 412 and identifies one or more trailers 200 in the received image 412. The DNN 500 may also receive sensor data 420 from a sensor 420 in communication with the DNN 500 and analyze the received sensor data 420. Based on the analyzed image 412 (or the analyzed image 412 and the sensor data 422), the DNN 500 identifies the position of each identified trailer 200 relative to the tractor 100, for example, by a coordinate system. As such, the DNN 500 displays the received image 412 on the user interface 140, which displays the trailer representation 146, 146a-146c of the identified trailer 200, 200a-200c located a distance behind the tractor 100. Figure 1 As shown, the first trailer 200a, the second trailer 200b, and the third trailer 200c are positioned behind the tractor 100. As such, the user interface 140 displays a first trailer representation 146a, a second trailer representation 146b, and a third trailer representation 146c associated with the first trailer 200a, the second trailer 200b, and the third trailer 200c, respectively.

[0043] The operator may select one of the trailer representations 146, 146a-146c, which indicates that the operator wishes the tractor 100 to automatically drive and connect to the trailer 200, 200a-200c associated with the selected trailer representation 146, i.e., the operator selection 144. In some examples, the user interface is a touch screen display 142; thus, the operator may point with his finger and select the trailer representation 146. In other examples, the user interface 140 is not a touch screen, and the operator may use an input device (such as, but not limited to, a knob or a mouse) to select one of the trailer representations 146, 146a-146c.

[0044] When the operator selects a trailer 200, 200a-200c to which he / she wishes the tractor 100 to be connected, the path planning system 550 plans a path 552 ( FIG. 3 ) between the tractor 100 and the trailer 200 based on the location (e.g., orientation and distance) of the selected trailer 200, 200a-200c relative to the location (e.g., orientation and distance) of the tractor 100 as determined by the DNN 500 from the received sensor system data 402. Figure 1As the first trailer 200a) is shown autonomously backing up, the planned path 552 allows the tractor 100 to autonomously travel and connect to the trailer 200. The path planning system 550 plans the path 552 for the tractor 100 to autonomously maneuver so that the tractor 100 is in an orientation aligned generally parallel to and away from the trailer 200 at an intermediate position within a predetermined distance D from the trailer 200, wherein the hitch 160 of the tractor 100 is substantially aligned with the hitch 210 of the trailer 200.

[0045] In some examples, the path planning system 550 is part of the controller 300, such as Figure 2 A; in other examples, the path planning system 550 is part of the DNN 500, such as Figure 2 B. Reference Figure 2 A, when the operator selects the trailer representation 146a associated with the trailer 200a to which the operator wishes the tractor 100 to autonomously drive and connect, the DNN 500a sends a data output 508 to the controller 300, the data output including the selected trailer 200a and the position of the selected trailer 200a relative to the tractor 100. In this case, the path planning system 550a plans a path 552 between the tractor 100 and the selected trailer 200a. The path planning system 550a can use several methods to determine the path 552. Figure 3A and Figure 3BA method for path planning is provided. In some examples, the path planning system 550a extends its fore-and-aft axis Y in the rearward direction R, while the trailer 200a extends the fore-and-aft axis in the forward direction around the length of the trailer 200. The path planning system 550a draws a first circle 554 and a second circle 556, the first circle 554 being tangent to the tractor fore-and-aft axis Y facing the trailer fore-and-aft axis at a first tangent point 555, and the second circle 556 being tangent to the trailer fore-and-aft axis Y facing the tractor fore-and-aft axis at a second tangent point 557. The first circle 554 and the second circle 556 intersect at an intersection point 558. The size of the first circle 554 and the second circle 556 can be adjusted and manipulated based on the distance between the tractor 100 and the trailer 200, obstacles and objects located between the tractor 100 and the trailer 200, and any other considerations. The path planning system 550a determines a path 552 that follows the tractor fore-aft axis Y until a first tangent point 555, then moves along the arc of a first circle 554 until an intersection point 558, then moves along the arc of a second circle 556 until a second tangent point 557, and then follows the trailer fore-aft axis. As such, the planned path 552 positions the tractor 100 in an orientation that is aligned generally parallel to the trailer 200 and facing away from the trailer 200a, with the hitch 160 of the tractor 100 substantially aligned with the hitch receiver 212 of the trailer hitch 210 of the trailer 200. In other words, the fore-aft axis Y of the tractor 100 is substantially aligned with the fore-aft axis T of the trailer 200a. Figure 3A An example of a path 552 is shown in which the fore-aft axis Y of the tractor 100 is substantially parallel to the fore-aft axis T of the trailer 200a. Figure 3B An example of a path 552 is shown in which the fore-aft axis Y of the tractor 100 is not substantially parallel to the fore-aft axis T of the trailer 200a.

[0046] Continue to refer to Figure 2 A. Figure 3A and Figure 3BIn some examples, when the tractor 100 autonomously drives along the planned path 552, the DNN 500 continuously sends the position of the selected trailer 200a relative to the tractor 100 as the tractor 100 moves along the planned path 552 to the controller 300 based on the received sensor system data 402 (i.e., the image 412). Since the image 412 is updated as the tractor 100 approaches the selected trailer 200a, the position of the selected trailer 200a relative to the tractor 100 also changes as the tractor 100 moves closer to the selected trailer 200a. In some examples, the DNN 500 identifies one or more objects along the planned path 552 and sends data related to the position of the one or more objects to the path planning system 550a. In this case, the path planning system 550a can recalculate the planned path 552 to avoid the one or more objects. In some examples, the path planning system 550a determines the probability of a collision, and if the probability of a collision exceeds a predetermined threshold, the path planning system 550a adjusts the path and sends it to the path following subsystem 320.

[0047] Re-reference Figure 2 B. In some embodiments, DNN 500 includes a path planning system 550b. Thus, path planning system 550b can determine path 552 based on learned behavior. For example, DNN 500 determines the location of selected trailer 200a and determines path 552 of tractor 100 based on the location. In some examples, the determined path 552 can be similar to the path 552 described in relation to Figure 3A and Figure 3B Path described. However, other path determination methods are possible. Similarly, DNN 500 (i.e., path planning system 550b of DNN 500) adjusts planned path 552 based on one or more objects (moving or stationary) that may be identified within planned path 552 as tractor 100 moves around path 552. In some examples, path planning system 550b determines the probability of a collision, and if the probability of a collision exceeds a predetermined threshold, path planning system 550b adjusts the path and sends it to path following subsystem 320.

[0048] Re-reference Figure 2 A and Figure 2B. Once the path planning system 550 has planned the path 552, the path following subsystem 320 is configured to execute behaviors that cause the drive system 110 to autonomously follow the planned path 552. Therefore, the path following subsystem 320 includes one or more behaviors 322-330 that, once executed, allow the tractor 100 to autonomously travel along the planned path 552. The behaviors 322-330 may include, but are not limited to, a braking behavior 322, a speed change behavior 324, a steering behavior 326, a hitch connection behavior 328, and a suspension adjustment behavior 330.

[0049] Braking action 322 may be performed based on planned path 552 to stop or slow down tractor 100. Braking action 322 sends a signal or command 301 to drive system 110 (eg, brake system 120) to stop or slow down tractor 100.

[0050] The speed change behavior 324 can be executed based on the planned path 552 to change the speed of the tractor 100 by accelerating or decelerating. The speed change behavior 324 sends a signal or command 301 to the braking system 120 for deceleration or to the acceleration system 130 for acceleration.

[0051] A steering action 326 may be executed based on the planned path to change the direction of the tractor 100. As such, the steering action 326 sends a signal or command 301 indicating a steering angle to the acceleration system 130, causing the drive system 110 to change direction.

[0052] FIG. 4A to FIG. 4C The selected trailer 200a is shown in the initial position P I ( Figure 4A ), middle position P M ( Figure 4B ) and the final position P F ( Figure 4C ) or the tractor 100 in the connection position. Figure 4A , before initiating the autonomous maneuver towards the selected trailer 200a, the tractor 100 is in an initial position P relative to the trailer 200a. I In some examples, the initial distance D between the tractor 100 and the selected trailer 200a is I is about 15 meters. Figure 4B As shown, the tractor 100 autonomously maneuvers along the planned path 552 until the tractor 100 reaches an intermediate distance D from the selected trailer 200a. M The middle position P M In the middle position P M, the tractor hitch 160 is oriented in a generally parallel alignment with the selected trailer 200a, and the tractor hitch 160 is substantially aligned with the trailer hitch receiver / coupler 212 of the trailer hitch 210. In other words, the tractor fore-aft axis Y defines a plane that extends along the tractor vertical axis Z and along the trailer fore-aft axis T along the trailer vertical axis. In some examples, the intermediate distance D M About 1 meter.

[0053] refer to Figures 5A-5D In some examples, when the tractor 100 is in the middle position P M , a hitch connection behavior 328 is performed to connect the tractor hitch 160 to the trailer hitch 210. The example shown uses a ball-type hitch and receiver. However, FIG. 5A to FIG. 5D The exemplary embodiments of the present invention may be implemented with any type of hitch that utilizes a vertical arrangement between the tractor hitch and the hitch receiver. For hitch types that involve a horizontal arrangement between the tractor hitch and the hitch receiver, vertical manipulation may be required only to obtain the desired horizontal alignment between the components, or not at all.

[0054] The controller 300 determines the relative height H between the top portion of the tractor hitch coupler 162 and the bottom portion of the trailer hitch coupler 212. R To connect the tractor 100 and the selected trailer 200a, the trailer hitch coupler 212 releasably receives the tractor hitch coupler 162. Thus, to connect the tractor hitch coupler 162 to the trailer hitch coupler 212, the relative height H R must be equal to zero to allow the tractor hitch coupler 162 to move under and insert into the trailer hitch coupler 212. Therefore, when the hitch connection behavior 328 receives a relative height H between the tractor hitch coupler 162 and the trailer hitch coupler 212 greater than zero from the controller 300, R , the hitch connect behavior 328 sends a command to the suspension adjust behavior 330 to execute and issues a command 301 to the suspension system 132 to adjust the height of the tractor 100 to reduce the relative height H based on the measurement from the controller 300. R When the hitch connection behavior 328 receives a relative height H equal to zero R , the hitch connection behavior 328 issues a command 301 to the drive system 110 to follow the remainder of the path 552 (i.e., from the intermediate position P M To the final position P F ( Figure 4C)) is manipulated to connect the tractor 100 to the selected trailer 200a.

[0055] refer to Figure 1 and 6A to 7B , a second exemplary tractor 100 is shown. Again, the tractor 100 may include a camera 410 configured to capture an image 412 of an environment of the tractor 100. The image 412 is displayed on the display 142, which shows a trailer representation of the trailer 200 positioned behind the tractor 100. The operator identifies a track or path 554 in the received image 412 that positions the tractor hitch coupler 552 under or near the trailer hitch coupler 212. Based on the operator-identified path 554 in the image 412, the tractor 100 autonomously drives along the identified path 554.

[0056] The tractor 100 may include a user interface 400. The user interface 400 may include a display 142, a knob 143, and a button 145 used as an input mechanism. In some examples, the display 142 may display the knob 143 and the button 145. In other examples, the knob 143 and the button 145 are a knob-button combination. In some examples, the user interface 400 receives one or more operator commands from an operator via one or more input mechanisms or a touch screen display 142, and / or displays one or more notifications to the operator. The user interface 400 communicates with the tractor controller 300, which in turn communicates with the sensor system 400. In some examples, the display 142 displays an image of the environment of the tractor 100, which causes one or more commands to be received by the user interface 400 (from the operator), which initiates the execution of one or more actions. In some examples, the user display 142 displays a representation image of the environment behind the tractor 100. In this case, the operator can select a position within the representation of the image, which indicates the environment that the operator wants the vehicle to autonomously maneuver toward. In some examples, the user display 142 displays one or more representations of a trailer 200 positioned behind the tractor 100. In this case, the operator selects a representation of the trailer 200 that the tractor 100 autonomously maneuvers toward.

[0057] The display 142 displays an expected path 554 for the tractor 100, which is superimposed on the camera image 412 of the environment behind the tractor 100. The operator can change the expected path 554 using the user interface 400. For example, the operator can turn a knob 143 that simulates a virtual steering wheel. As the operator turns the knob 143, the expected path displayed on the display 142 updates. The operator adjusts the displayed path 554 until the expected path displayed on the display 142 intersects with the trailer representation or other object that the user wants the tractor 100 to drive to. Once the user is satisfied with the displayed expected path 554, the operator performs an action indicating the finalized path 554, which allows the tractor 100 to autonomously follow the planned path 554. The planned path 554 is the path from the tractor hitch coupler 552 to the trailer hitch receiver / coupler 212. The wheel path 552 is the estimated path that the wheels will move along the planned path 554.

[0058] In some embodiments, the user may adjust three path modes 312, including: Fig. 8A and Figure 8B The angle mode 314 shown is used to adjust the bending angle of the path, as shown in FIG. Fig.9A and Fig. 9B The distance mode 316 shown is used to adjust the length of the expected path 554, and optionally Fig. 10A and Fig. 10B The double arc mode 318 is shown for adjusting the "approach angle," which is the angle for how the vehicle will be oriented relative to the trailer (or other object) at the end of the maneuver. Since the double arc mode is optional, if the user is satisfied with the currently constructed path (arc), the user can finalize the maneuver selection by pressing the button again. Otherwise, the user adjusts the knob a third time to change the shape of the double arc or other suitable path. This allows the final approach angle to the trailer or other object to be adjusted. Once the user is satisfied with the choice of approach angle, he / she presses the button to finalize the path selection.

[0059] The tractor controller 300 includes a computing device (or processor) 302 (e.g., a central processing unit having one or more computing processors) in communication with a non-transitory memory 304 (e.g., a hard disk, flash memory, random access memory) capable of storing instructions executable on the computing processor 302.

[0060] The tractor controller 300 executes an operator assistance system 310 which receives the planned path 552 from the path system 550 and executes actions 330, 330a-330c which send commands 301 to the drive system 110 causing the tractor 100 to autonomously travel around the planned path 554 in a rearward direction R (in the example shown).

[0061] The path following behaviors 330 include braking behaviors 330a, speed change behaviors 330b, and steering behaviors 330c. In some examples, the path following behaviors 330 also include hitch connection behaviors and suspension adjustment behaviors. Each behavior 330, 330a-330c causes the tractor 100 to take an action, such as driving backward, turning at a specific angle, braking, accelerating, decelerating, etc. By controlling the drive system 110, and more specifically, by issuing commands 301 to the drive system 110, the tractor controller 300 can steer the tractor 100 across the surface in any direction.

[0062] The tractor 100 may include a sensor system 400 to provide reliable and robust driving. The sensor system 400 may include different types of sensors that may be used alone or together with each other to generate a perception of the environment of the tractor 100, which is used for the tractor 100 to navigate and assist the operator in making intelligent decisions based on objects and obstacles detected by the sensor system 400. The sensor system 400 may include one or more cameras 410. In some embodiments, the tractor 100 includes a rear camera 410, 410a, which is mounted to provide a view of the rear travel path of the tractor 100. The rear camera 410 may include a fisheye lens, which includes an ultra-wide-angle lens that produces strong visual distortion, intended to create a wide panoramic or hemispherical image. The fisheye camera captures images with a very wide viewing angle. In addition, the images captured by the fisheye camera have a characteristic convex non-linear appearance. Other types of cameras may also be used to capture images of the rear of the tractor 100.

[0063] The sensor system 400 may also include an IMU (Inertial Measurement Unit) 420, which is configured to measure the linear acceleration (using one or more accelerometers) and rotation rate (using one or more gyroscopes) of the tractor. In some examples, the IMU 420 also determines the heading reference of the tractor 100. Thus, the IMU 420 determines the pitch, roll, and yaw of the tractor 100.

[0064] The sensor system 400 may include other sensors such as, but not limited to, radar, sonar, LIDAR (light detection and ranging, optical remote sensing which may require measuring properties of scattered light to find the distance and / or other information of distant targets), LADAR (laser detection and ranging), etc.

[0065] The tractor controller 300 executes the hitch assist system 310 which receives the image 412 from the camera 410 and superimposes the tractor path 552 on the received image 412 .

[0066] Re-reference FIG. 6A to FIG. 6B, the tractor controller 300 executes the hitch assist system 310, which assists the operator in selecting a path 552 to autonomously steer the tractor 100 toward the trailer 200. The operator may initiate execution of the hitch assist system 160 through the user interface 140 (e.g., by making a selection on the display 142). Once initiated, the hitch assist system 160 instructs the display 142 to display the path 552 of the tractor 100, which is superimposed on the camera image 143 of the environment behind the tractor 100. The operator may change the planned path 552 using the user interface 140. For example, the operator may turn a knob 143 that simulates a virtual steering wheel. As the operator turns the knob 143, the planned path 552 displayed on the display 142 is updated. The operator adjusts the displayed path 552 until the updated planned path 552 displayed on the display 142 intersects the trailer representation 138 or other object that the operator wants the tractor 100 to drive toward. Once the operator is satisfied with the displayed planned path 552, the operator performs an action indicative of finalizing the path 552, which allows the tractor 100 to autonomously follow the planned path.

[0067] In some embodiments, the hitch assist system 160 includes a trajectory generator 560, a motion estimator 570, and a path tracker 580. The trajectory generator 560 determines an estimated position of the tractor 100 based on the operator-selected path 552. The motion estimator 570 determines the actual position of the tractor 100, and the path tracker 580 determines an error 562 based on the estimated position Pe and the actual position Pa, and adjusts the planned path 552 of the tractor 100 to eliminate the error 562 between the actual position Pa and the estimated position Pe.

[0068] In some embodiments, the trajectory generator 560 receives the image 143 from the camera 410 and overlays the vehicle path 552 on the received image 143. The operator can adjust the selection of the path 552 based on one or more path modes 312. In some examples, the path mode 312 includes an arc mode 312 having an angle submode 314 and a distance submode 316. In some examples, the path mode 312 can include a dual arc mode 318. Thus, the operator can select between the angle submode 314, the distance submode, and / or the dual arc mode 318 to determine and adjust the path 552 to the trailer 200 or the object.

[0069] refer to Fig. 7A and Figure 7B In some examples, the angle sub-mode 314 and the distance sub-mode 316 are arc modes 312 ( Fig. 7A), so the operator first selects a mode 312, 318 and then selects a sub-mode within the selected mode 312, 318. Thus, for example, the display 142 may display an arc mode button 145 and a dual arc mode button 145 from which the operator may select. Figure 7B An example is shown in which each sub-mode / mode 314, 316, 318 is independent. Thus, pressing or pushing the button 145 rotates between the three modes 314, 316, 318.

[0070] Angle sub-mode 314 is configured to adjust the bending angle of path 552, such as Fig. 8A and Figure 8B Thus, the operator can turn knob 143 to the right, causing displayed path 552 to have a rightward curve, as shown. Fig. 8A In addition, the operator can turn knob 143 to the left, causing the displayed path 552 to have a leftward bend, as shown. Figure 8B The distance sub-mode 316 is configured to adjust the length of the expected path 552, such as Fig. 8A and Figure 8B For example, refer to Fig. 8A , the operator may rotate knob 143 to position the destination of path 552 near trailer representation 138 in image 143. Figure 8B , image 143 shows a ratio Fig. 8A Thus, in this case, the operator may wish to autonomously move the tractor 100 a few meters in the rearward direction R. The dual arc mode 318 is configured to adjust the approach angle, which indicates how the tractor 100 will be oriented relative to the trailer 200 (or other object) at the end of the path 552, such as Fig. 10A and Fig. 10B For example, the double arc mode 318 helps the operator align the tractor 100 with the trailer 200 so that the front-rear axis Y of the tractor 100 is aligned with the front-rear axis Y of the trailer 200, which helps the operator during the coupling process between the tractor 100 and the trailer 200. Fig.11A and Fig. 11B , both the arc mode 312 and the double arc mode 318 have the same endpoints; however, the double arc mode 318 allows for adjustment of the approach angle toward the trailer 200. In some examples, when the operator switches to the double arc mode 318, the trajectory generator 560 keeps the starting position and the ending position the same. In the double arc mode 318, the operator can only adjust the approach angle to the trailer 200. In the double arc mode 318, the operator does not adjust the distance. In some examples, the radius and length of the path 552 selected by the operator determines the final position of the tractor 100. The vehicle controller 150 uses the Dobbins path to determine the path 552 as the best calculation.

[0071] In some embodiments, where the dual arc mode 318 is optional, if the operator is satisfied with the path 552 based on the selection of the arc mode 312, the operator can finalize the path 552 by pressing the button 145. Otherwise, the operator adjusts the knob 143 a third time to change the shape of the dual arc or other suitable path 552. This allows the final approach angle to the trailer 200 or other object to be adjusted. Once the operator is satisfied with the selection of the approach angle, he / she presses the button 145 to finalize the path selection.

[0072] In some embodiments, the operator parks the tractor 100 in a position where the trailer 200 or other object or point of interest is within the field of view of the rear camera 410 of the tractor 100. The engine of the tractor 100 may be idling and the transmission may be in the "park" position. The operator may initiate the trajectory generator 560 by pressing a button 145 and / or making a selection on the display 142. In some examples, the display 142 displays a selectable option or button 145 that allows the operator to initiate the arc mode 312. The trajectory generator 560 begins by executing the angle sub-mode 314 of the arc mode 312, such as Fig. 7A and Figure 7B The operator switches to the distance sub-mode 316 to adjust the distance of the path 552 by, for example, pressing the button 145. In some examples, the operator can adjust the path 552 by switching between the angle sub-mode 314 and the distance sub-mode 316 and adjusting the path 552 until the desired path 552 is displayed on the display. The operator adjusts the path 552 so that the outer boundary 554 of the path 552 intersects the trailer 200 (i.e., the trailer representation 138 within the image 143) or other points of interest.

[0073] In some embodiments, the final approach angle to the trailer 200 or point of interest is important, for example, to align the vehicle front-to-rear axis Y with the trailer front-to-rear axis Y. In this case, the operator can select or press the "Arc / Dual Arc Mode" button 145 (displayed on the display 142) and switch to the dual arc mode 318. In the dual arc mode 318, the previously set endpoints of the path 552 remain unchanged, and the operator adjusts the final approach angle with the knob 143. When the operator is satisfied with the final approach angle and the complete trajectory or path 552, the operator can confirm the selected path 552 by performing an action. In some examples, the operator shifts the transmission to reverse gear, which indicates that the operator is satisfied with the displayed path 552. In some examples, the operator shifts the transmission to reverse gear with the brakes depressed, then releases the brakes, and the tractor 100 follows the selected path 552. In some examples, the operator can stop the tractor 100 by, for example, pressing the brakes as the vehicle autonomously maneuvers along the path 552 in the rearward direction R. This causes the vehicle controller 150 to exit the hitch assist system 160.

[0074] In some embodiments, trajectory generator 560 sets the path distance to a default value, which allows the operator to simply adjust the steering angle until it intersects trailer 200 or other point of interest.

[0075] In some embodiments, the final approach angle is not adjusted. Instead, the final approach angle is always the same as the initial vehicle departure angle. Therefore, the final vehicle front-to-back axis Y is parallel to the initial vehicle front-to-back axis Y. In this case, the operator adjusts the final position of the path 552 to intersect the trailer.

[0076] In some examples, as the tractor 100 maneuvers in the rearward direction R along the path 552, the display 142 may display the progress of the tractor 100 along the path 552. For example, the display 142 may display the original track projected on the ground, but updated by the changing position of the vehicle. The display 142 may also display an indication of the degree to which the vehicle is following the track.

[0077] In some embodiments, the trajectory generator 560 receives data from other vehicle systems to generate the path 552. In some examples, the trajectory generator 560 receives vehicle pose data defined by (x, y, θ), where x is the position of the center of the tractor 100 along the lateral axis X in the XY plane, y is the position of the center of the vehicle along the fore-aft axis Y in the XY plane, and θ is the heading of the tractor 100. In addition, the trajectory generator 560 can receive the position of the knob 143 from the knob 143, such as the knob angle. The trajectory generator 560 can also receive the mode button state (i.e., arc mode 312 or dual arc mode 318) and the sub-mode button state (i.e., angle sub-mode 314 or distance sub-mode 316). Based on the received data, the trajectory generator 560 adjusts the path 552 and instructs the display 142 to display the path 552. In some examples, the path 552 includes an outer boundary 554 and a hitch ball path 166, which is an estimated path of the hitch ball 122. Trajectory generator 560 may also instruct display 142 to display a current mode or sub-mode state, which indicates the mode / sub-mode that the driver has selected to adjust path 552 .

[0078] Re-reference Fig. 6A and Figure 6B Once the operator indicates via the user interface 140 that his path selection is complete, the vehicle controller 150 executes the operator assistance system 320 to follow the planned path 552. The operator assistance system 320 includes a path following behavior 330. The path following behavior 330 receives the selected path 552 and executes one or more behaviors 330a-330b that send commands 194 to the drive system 110, causing the tractor 100 to autonomously travel along the planned path 552. As the tractor 100 autonomously maneuvers along the planned path 552, the hitch assist system 160 continuously updates the path 552 based on the motion estimator 570 and the path tracker, as discussed below.

[0079] Re-reference Fig. 6A and Figure 6B, when the tractor 100 autonomously maneuvers in the backward direction R along the path 552, the motion estimator 570 determines the current position Pa of the tractor 100. As previously discussed, the trajectory generator 560 determines where the tractor 100 should be based on the planned path 552, that is, the estimated position Pe; therefore, the motion estimator determines the actual position Pa of the tractor 100. In some examples, the motion estimator includes a motion estimation algorithm that outputs relative vehicle position and speed. For example, the motion estimation algorithm may include an extended Kalman filter (EKF). The EKF uses measurements such as, but not limited to, encoders from 4 wheels (e.g., 96 ticks per revolution) and steering angles. The motion estimator 570 fuses the measurements to determine the actual position of the tractor 100. Since the speed of the tractor 100 is slow when the tractor 100 autonomously moves in the backward direction, the motion estimator 570 may use a bicycle model. The bicycle model uses a single steered wheel in the front to represent the two front wheels, and a single non-steered wheel in the back to represent the two rear wheels. The wheels are connected by a single rigid link. Motion is constrained to a two-dimensional horizontal ground plane. The inputs to the bicycle model are speed and steering angle, while its state is position and heading. Motion estimator 570 estimates linear and rotational speeds as well as position (e.g., orientation of the vehicle). In some examples, motion estimator 570 takes into account sensor data from sensor system 400, such as camera, radar, GPS measurements, to improve any drift.

[0080] In some embodiments, the motion estimator 570 uses an extended Kalman filter (EKF). The EKF equations are also provided below as equations (3)-(7). The state vector has nine elements as shown in equation 1:

[0081]

[0082] The first three are the "attitude" of the vehicle (x, y, θ). The next two are the linear and angular velocities (v, ω). The last four are the distance traveled by the four tires (d lr , d rr , d lf , d rf ), respectively left rear, right rear, left front, and right front.

[0083] The complete measurement vector has the five elements shown in Equation 2:

[0084]

[0085] Distance traveled by the first four or four tires (d lr , d rr , d lf , d rf). The last element φ is the average front wheel angle (not the steering wheel angle, but the average angle of the front tires relative to the longitudinal axis).

[0086] The motion estimator 570 provides a vehicle speed estimate. The approximate speed is usually calculated by dividing the change in distance by the change in time (Δd / Δt), but this method will be very noisy for the case here, where the wheel encoder counts are relatively small and the vehicle moves relatively slowly. Therefore, to avoid the direct calculation involving dividing the change in distance by the change in time (Δd / Δt), the motion estimator 570 estimates the vehicle linear speed v based on the measured wheel cumulative distance by using an EKF, and there is no explicit rate calculation involving division.

[0087] The extended Kalman filter can be written as two prediction equations and three measurement equations. The prediction equation is:

[0088]

[0089] Equation (3) provides an update of the state μ. Equation (4) provides an update of the covariance Σ. The covariance provides an estimate of the current uncertainty of the state. The matrix R is the noise covariance of the state μ.

[0090] The measurement update equation is:

[0091]

[0092] (7)

[0093] Equation (5) sets the value of the optimal Kalman gain K.

[0094] Equation (6) provides the update of the state μ. Equation (7) provides the update of the covariance Σ. The matrix Q is the noise covariance of the measurement z.

[0095] For prediction, a nonlinear vector function g needs to be defined. The matrix G is the derivative of this vector function. For convenience, it is also provided. The vector function g is given by:

[0096]

[0097] Here, w is the "track width" of the vehicle. More specifically, it is the lateral distance from the center of the left tire to the center of the right tire. Assume that the front and rear distances between the tires are the same. The wheelbase is represented by l. Note that in the last two elements, the distance from d lf and d rfThere is a minus sign where the expression is subtracted from . This minus sign assumes backward motion. Therefore, this prediction equation cannot be used for forward motion. However, if there were some measurements of the vehicle's direction (forward or backward), it would be a simple matter to change the signs of the last two elements (positive for forward and negative for backward) to make the equation valid for both the forward and backward directions.

[0098] The matrix G has nine rows and nine columns. Let

[0099]

[0100] I is a 9×9 identity matrix. Then,

[0101]

[0102] All other elements are zero.

[0103] The complete update assumes that both wheel scale and wheel angle measurements are available. If only wheel scale is available, then wheel scale can be combined separately, if only wheel angle is available, then wheel angle can be combined separately. For the complete update, a vector h is defined. The matrix H is the derivative of this vector function. For convenience, it is also provided. The vector function h is given by:

[0104]

[0105] The matrix H is the derivative of h and is a 5×9 matrix. The non-zero elements are

[0106]

[0107] Note that some quantities (h5, H 54 , H 55 ) contains divisors that could easily be zero. Therefore, implementations need to test whether these divisors are zero before performing the division.

[0108] It is possible to consider measurements consisting only of wheel scales, as well as measurements consisting only of steering angles. However, these variants are not included, since they are straightforward given the information already provided.

[0109] During the autonomous maneuvering of the vehicle in the rearward direction R along the planned path 552, the trajectory generator 560 determines where the tractor 100 should be based on the planned path 552 (i.e., the estimated position Pe); while the motion estimator 570 determines the actual position Pa of the tractor 100; therefore, the path tracker 580 determines the error 562 based on the estimated position Pe and the actual position Pa. The path tracker 580 adjusts the current position Pa of the vehicle based on the error 562 so that the tractor 100 continues to follow the planned path 552.

[0110] refer to FIG. 12A to FIG. 1 2D, in some embodiments, the path tracker 580 performs a pure tracking method to keep the tractor 100 on the planned path 552. The operator-selected path 552 is sampled at predetermined time intervals, such as every minute, to generate a plurality of waypoints 168 located along the hitch ball path 166 of the planned path 552. The path tracker 580 (e.g., an algorithm) compares the current hitch ball position and heading Pa received from the motion estimator 570 with the next waypoint position Pb. The vehicle hitch assist system 160 continuously adjusts the vehicle steering toward the current waypoint Pb (i.e., the waypoint the vehicle is heading toward). The path tracker 580 allows the vehicle hitch ball 122 to track each waypoint 168. In other words, the path tracker 580 allows the hitch ball 122 to reach each waypoint along the hitch ball path 166. In some examples, the waypoints 168 are converted from world coordinates to vehicle coordinates. For example, the path tracker 580 calculates the center Cc of the turning circle based on the hitch ball position Pa and the waypoint position Pb. The path tracker 580 then calculates the vehicle turning radius Rr based on the center of the turning circle. Finally, the path tracker 580 calculates the steering angle based on the center of the turning circle using the Ackerman angle. In other words, the path tracker 580 compares the estimated position Pe with the current position Pa to ensure that the vehicle is following the path, and determines the next waypoint Pb, and determines or adjusts the path from the current vehicle position and heading to the next or subsequent waypoint Pb. Thus, the path tracker 580 keeps the tractor 100 autonomously maneuvering along the planned path, and adjusts the behavior or travel of the vehicle when the tractor 100 deviates from the planned path.

[0111] In some examples, the controller includes an object detection system (not shown) that identifies one or more objects along the planned path 552. In this case, the hitch assist system 160 adjusts the path 552 to avoid the detected one or more objects. In some examples, the hitch assist system 160 determines the probability of a collision, and if the probability of a collision exceeds a predetermined threshold, the hitch assist system 160 adjusts the path 552 and sends it to the operator assistance system 320.

[0112] Once the operator instructs entry into the selected path 552, the vehicle controller 150 executes the operator assistance system 320, which in turn includes a path following behavior 330. The path following behavior 330 receives the selected path 552 and executes one or more behaviors 330a-330b that send commands 194 to the drive system 110, causing the tractor 100 to autonomously travel in the rearward direction R along the planned path.

[0113] The path following behaviors 330a-b may include one or more behaviors, such as, but not limited to, a braking behavior 330a, a speed change behavior 330b, and a steering behavior 330c. Each behavior 330a-330b causes the tractor 100 to take an action, such as driving backward, turning at a specific angle, braking, accelerating, decelerating, etc. By controlling the drive system 110, and more specifically, by issuing commands 194 to the drive system 110, the vehicle controller 150 may steer the tractor 100 across the road surface in any direction.

[0114] Braking action 330a may be performed based on the planned path to stop or slow down the tractor 100. Braking action 330a sends a signal or command 194 to the drive system 110 (eg, a brake system (not shown)) to stop or slow down the tractor 100.

[0115] The speed change behavior 330b may be executed based on the planned path 552 to change the speed of the tractor 100 by accelerating or decelerating. The speed change behavior 330b sends a signal or command 194 to the brake system 120 for deceleration or to the acceleration system 1320 for acceleration.

[0116] The steering action 330c may be executed based on the planned path 552 to change the direction of the tractor 100. In this manner, the steering action 330c sends a signal or command 194 indicating a steering angle to the acceleration system 140, causing the drive system 110 to change direction.

[0117] Fig.13 Autonomously maneuvering the tractor 100 toward a selected trailer 200, 200a is shown. Figure 1 to Figure 12D). In box 802, the method 800 includes initiating a hitch assist mode by receiving an indication that an operator wants to autonomously hitch the tractor 100 to the trailer 200, 200a. The indication can be through a selection on the user interface 140 of the tractor 100, placing the tractor in reverse (without reversing), or any other indication. In box 804, the DNN 500 detects and locates one or more trailers 200, 200a-200c behind the tractor 100, and displays trailer representations 146, 146a-146c associated with the one or more identified trailers 200, 200a-200c, respectively, through the user interface 140. Alternatively, the camera image is displayed on the display 142. In decision box 806, the method 800 waits for the operator selection 144. The operator may select from the displayed image by selecting one of the one or more trailer representations 146, 146a-146c on the display, or by manipulating the user input knob 143 until the path representation (i.e., the path tracker 480) is aligned with the point of interest on the displayed image (i.e., the trailer representation 146 including the hitch image location). When the operator selects the trailer representation 146, 146a-146c associated with the selected trailer 200a, at block 808, the method 800 includes planning a path from the initial position P of the tractor 100. I To the final position P relative to the selected trailer 200 F tractor path 552. In some examples, the path planning system 550, 550a, 550b plans the path 552. The path planning system 550, 550a, 550b can be part of the controller 300 or part of the DNN 500. At block 810, the method 800 includes executing the path following subsystem 320. At decision block 812, the method 800 determines whether the tractor 100 is within a predetermined distance from the selected trailer 200a, that is, the method 800 determines whether the tractor 100 has reached the intermediate position P M When the tractor 100 reaches the intermediate position P M When the method 800 determines at decision block 812 the relative height H between the top portion of the tractor hitch coupler 162 of the tractor 100 and the bottom portion of the hitch coupler 212 of the selected trailer 200a. R , and based on the relative height H R Determine whether the hitch coupler 212 is capable of releasably receiving the tractor hitch coupler 162. In other words, the method 800 determines the relative height H R Is it equal to zero? If the relative height H R is not equal to zero, then at block 816, method 800 adjusts the suspension of tractor 100 and then determines the relative height H R And check the relative height H in box 814R Is it equal to zero? Once the relative height H R is equal to zero, the method 800 continues at block 818 around the path 552 from the intermediate position P M Manipulate to the final position P F , the final position P F The tractor hitch coupler 162 of the tractor 100 is connected to the hitch coupler 212 of the selected trailer 200a.

[0118] Fig.14 Provides the use of Figures 1 to 8B An example arrangement of operations of a method 900 for autonomously maneuvering a tractor 100 (e.g., a towing vehicle) in a rearward direction R toward a point of interest (such as a trailer 200) is provided for the system described in FIG. 1 . At block 902, the method 900 includes receiving, at data processing hardware 152, one or more images 143 from a camera 142 positioned on a rear portion of the tractor 100 and in communication with the data processing hardware 152. At block 904, the method 900 includes receiving, at the data processing hardware 152, an operator planned path 162 from a user interface 130 in communication with the data processing hardware 152. The operator planned path 162 includes a plurality of waypoints 168. At block 906, the method 900 includes transmitting, from the data processing hardware 152 to the drive system 110 in communication with the data processing hardware 152, one or more commands 161, 194 that cause the tractor 100 to autonomously maneuver along the operator planned path 162. At block 908, the method 900 includes determining, at the data processing hardware 152, a current vehicle position Pa. At block 910, the method 900 includes determining, at the data processing hardware 152, an estimated subsequent vehicle position based on the operator planned path, the estimated subsequent vehicle position being at a subsequent waypoint Pb along the operator planned path 162 from the current vehicle position Pa. At block 912, the method 900 includes determining, at the data processing hardware 152, a path adjustment from the current vehicle position Pa to the estimated subsequent vehicle position Pb. At block 914, the method 900 includes transmitting instructions from the data processing hardware 152 to the drive system 110 that cause the tractor 100 to autonomously maneuver toward the estimated subsequent vehicle position Pb based on the path adjustment.

[0119] Figure 10 provides the use of Figures 1 to 8BAn example arrangement of operations of another method 1000 of autonomously maneuvering a tractor 100 (e.g., a towing vehicle) in a rearward direction R toward a point of interest (such as a trailer 200) by the system described in FIG. 1002 . At block 1004 , the method 1000 includes receiving, at the data processing hardware 152 , one or more images 143 from one or more cameras 142 positioned on a rear portion of the tractor 100 and in communication with the data processing hardware 152 . At block 1006 , the method 1000 includes transmitting, from the data processing hardware 152 to the drive system 110 in communication with the data processing hardware 152 , one or more commands that cause the tractor 100 to autonomously maneuver along the operator planned path 162 . At block 1008 , the method 1000 includes determining, at the data processing hardware 152 , an estimated vehicle position Pe based on the operator planned path 162 . At block 1010, the method 1000 includes determining a current vehicle position Pa at the data processing hardware 152. At block 1012, the method 1000 includes determining an error 184 at the data processing hardware 152 based on the estimated vehicle position Pe and the current vehicle position Pa. At block 1014, the method 1000 includes determining one or more path adjustment commands at the data processing hardware 152 that cause the tractor 100 to autonomously maneuver from the current vehicle position Pa to the estimated vehicle position Pe, thereby eliminating the error 184. At block 1016, the method 1000 transmits the one or more path adjustment commands from the data processing hardware 152 to the drive system 110.

[0120] In some examples, the method 900, 1000 includes overlaying the path on the one or more images 143 and receiving a command through the user interface 130 in communication with the data processing hardware 152. The command includes an instruction to adjust the path to the operator planned path 162. In some examples, the command includes an instruction to adjust the distance of the path. The command may include an instruction to adjust the angle of the path and / or an instruction to adjust the angle of the end portion of the path.

[0121] In some embodiments, determining the current vehicle position includes receiving wheel encoder sensor data 145 associated with one or more wheels 112 and receiving steering angle sensor data 145. The current vehicle position Pa is based on the wheel encoder sensor data 145 and the steering angle sensor data 145.

[0122] As discussed previously, the proposed algorithm is designed to work in real-time in a standard CPU with or without the use of a GPU, graphics accelerator, training, or FPGA. In addition, the proposed method provides an automated method that requires only initial input from the operator. In addition, the described system provides a compromise between providing guidance to the operator and automating all backward functions.

[0123] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0124] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0125] The subject matter and the implementation of the functional operation described in this specification can be implemented in digital electronic circuits, or in computer software, firmware or hardware including the structure disclosed in this specification and its structural equivalent, or in a combination of one or more of them. In addition, the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for being executed by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of materials that implement a machine-readable propagation signal, or a combination of one or more of them. The terms "data processing device", "computing device" and "computing processor" cover all devices, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0126] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0127] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A method of reverse maneuvering a tractor vehicle to attach to a trailer using a hitch assist system, the method comprising: Enter the hitch assist mode; displaying a camera image on a user interface, wherein the displayed image shows at least one trailer; Select a trailer; receiving, at a user interface in communication, an indication of a selected trailer; determining, using a computing device, a tractor path from an initial position to a final position adjacent to a selected trailer, the tractor path including maneuvers configured to move the tractor along the tractor path from the initial position to the final position; and autonomously following the tractor path from the initial position to the final position at a drive system in communication with the computing device, wherein displaying on the user interface further comprises: receiving at a controller one or more images from one or more cameras positioned on a rear portion of the tractor and in communication with the controller; and superimposing at the controller a path representation indicating an expected path along which the tractor is to travel, the expected path beginning at a tractor hitch, Wherein, selecting a trailer further comprises: receiving a first command at the controller through a user interface, the first command indicating a change in the path representation so that the path representation ends at a point of interest; and adjusting the path representation at the controller based on the first command, wherein the path planning is performed by a controller of the hitch assist system.

2. The method according to claim 1, wherein: Displaying on the user interface includes displaying more than one trailer, and wherein selecting the trailer includes selecting a graphical representation of a desired trailer, and wherein the path planning is performed by a deep neural network.

3. The method according to claim 2, further comprising: continuously detecting, at the neural network, one or more objects within the tractor path as the tractor moves along the tractor path; and When an object is detected, the tractor path is altered at the computing device.

4. The method according to claim 2, wherein: Detecting one or more trailers includes: capturing one or more images at one or more imaging devices in communication with the neural network, at least one of the one or more imaging devices being positioned on a rear side of the trailer facing in a rearward direction; and The one or more trailers within the one or more images are determined at the neural network.

5. The method according to claim 1, further comprising: Before reaching the final position, stopping or pausing the tractor at the drive system at an intermediate position, the intermediate position being closer to the final position than the initial position; modifying one or more tractor suspensions associated with the tractor at the drive system to align the tractor hitch with the trailer hitch; autonomously following at the drive system the tractor path from the intermediate position to the final position; and The tractor hitch is connected to the trailer hitch at the drive system.

6. The method according to claim 1, wherein: Connecting the tractor hitch to the trailer hitch includes modifying one or more tractor suspensions associated with the tractor to align the tractor hitch with the trailer hitch.

Images