Method and device for determining a trailer's path

Abstract

Procedures (600), comprising: (610) Recognizing an obstacle position in response to an image; (620) Predictions of a tow vehicle path in response to a tow vehicle steering angle and a tow vehicle position; (630) Predictions, by a processor, of a trailer wheel path in response to the towing vehicle path and a trailer dimension, wherein an odometry of the towing vehicle and the trailer is predicted as a function of the telemetry; (640) Generating a warning signal in response to an intersection between the obstacle position and the trailer wheel path; (650) Displaying an obstacle position indicator and a trailer wheel path indicator on a vehicle cabin display; and (660) Modifying map data in response to the obstacle position and the towing vehicle position, whereby the trailer wheel path is determined in response to the map data.

Description

[0001] The present disclosure relates generally to a system for providing trailer guidance and support in a motor vehicle. More specifically, aspects of the present disclosure relate to systems, methods, and devices for determining a trailer's path during trailer operation, for detecting objects within the determined trailer path, and for providing a warning to a vehicle system algorithm or a vehicle operator.

[0002] Passenger cars have long been equipped with towing devices for trailers. These trailers can be caravans, boat trailers, or utility trailers. They are typically connected to the towing vehicle via a mechanically attached to the towing vehicle's frame. The trailers are connected to the towing vehicle by a coupling assembly that is attached to the trailer's frame. Electrical connections between the towing vehicle and the trailer provide power to the trailer's lighting and braking systems, and transmit electronic signals that control the application of the trailer brakes and the illumination of various trailer lights, such as brake lights, turn signals, and reversing lights.

[0003] The difficulty lies in the fact that the trailer's path during towing can lie outside the towing vehicle's path. When cornering, the trailer's wheels often follow a path that lies within the towing vehicle's wheel path, and the trailer's wheels can come into contact with curbs if the towing vehicle does not turn with sufficient clearance. This contact can lead to damage to the trailer, its wheels, and / or the curb. Furthermore, other parts of the trailer can touch objects that the towing vehicle has successfully avoided. It would be desirable to provide a means of giving feedback to the towing vehicle and / or its driver, while simultaneously overcoming the aforementioned problems.

[0004] EP 3 334 639 B1 describes a method for preventing damage to a vehicle and a trailer during forward travel.

[0005] DE 10 2020 123 087 A1 concerns the towing of trailers and in particular a lateral detection and avoidance system for vehicles towing trailers.

[0006] The objective can be considered to be to provide a method and a device with which the path of a towing vehicle trailer can be predicted and collisions avoided. The aforementioned problem is solved by a method with the features of claim 1 and by a device with the features of claim 8. Advantageous embodiments are described in the dependent claims.

[0007] This document discloses vehicle sensor methods and systems and the associated control logic for providing vehicle systems, methods for manufacturing and operating such systems, and motor vehicles equipped with on-board control systems. Various embodiments of systems for object detection and the prediction of contact events for a towing vehicle and a trailer are presented by way of example and without limitation.

[0008] The method according to the invention comprises detecting an obstacle position in response to an image, predicting a towing vehicle path in response to a towing vehicle steering angle and a towing vehicle position, predicting a trailer wheel path in response to the towing vehicle path and a trailer dimension by a processor, wherein an odometry of the towing vehicle and the trailer is predicted as a function of the telemetry, generating a warning signal in response to an intersection of the obstacle position and the trailer wheel path, displaying an indicator for the obstacle position and an indicator for the trailer wheel path on a vehicle cabin display, and changing map data in response to the obstacle position and the towing vehicle position, wherein the trailer wheel path is determined in response to the map data.

[0009] According to one embodiment, the method includes determining the obstacle position in response to a distance measurement received from a distance sensor.

[0010] According to one embodiment, the method includes predicting a trailer wheel path in response to the towing vehicle path, a vehicle dimension, and a trailer dimension using a processor.

[0011] According to one embodiment, the image is captured by a camera mounted on the towing vehicle, the camera having a forward-facing field of view from the towing vehicle.

[0012] According to one embodiment, the obstacle position is confirmed as a reaction to a depth measurement by a side-view sensor attached to the towing vehicle.

[0013] According to one embodiment, the vehicle dimension is a distance from the rear axle of the vehicle to a trailer coupling point, and the trailer dimension is at least a distance from a trailer wheel axle to a trailer coupling point and a trailer track width.

[0014] According to one embodiment, the obstacle position is confirmed in response to a depth measurement by an ultrasonic sensor attached to the towing vehicle.

[0015] According to one embodiment, the trailer wheel path is predicted in response to a trailer coupling deflection angle.

[0016] According to one embodiment, the towing vehicle's path is predicted in response to at least one acceleration detected by an inertial measurement unit attached to a towing vehicle and a displacement detected by a wheel encoder attached to the towing vehicle.

[0017] According to one embodiment, the image is captured in response to activation of the trailer system and detection of movement by a towing vehicle.

[0018] According to one embodiment, the warning signal is generated in response to at least one of the time until the collision and / or a distance between the obstacle position and the trailer wheel path.

[0019] The device according to the invention comprises a camera configured to capture an image of a field of view containing an object; a distance sensor configured to determine a distance to the object; a positioning sensor configured to determine a towing vehicle position; a processor configured to determine an object position relative to a towing vehicle position in response to the image; predict a towing vehicle path in response to a towing vehicle steering angle and the towing vehicle position; predict a trailer wheel path in response to the towing vehicle path and a trailer dimension, predicting an odometry of the towing vehicle and the trailer as a function of the telemetry; and generates a warning signal in response to an intersection of the object position and the trailer wheel path.a user interface configured to display a warning in response to the warning signal; and a memory for storing map data, wherein the processor further serves to modify the map data in response to the object position and the towing vehicle position, and wherein the trailer wheel path is determined in response to the map data.

[0020] According to one embodiment, the obstacle is a curb.

[0021] According to one embodiment, the location sensor is a global positioning sensor.

[0022] According to one embodiment, the camera is a stereo camera for capturing a multitude of stereo images, and the object position is determined in response to the multitude of stereo images.

[0023] According to one embodiment, the distance sensor is a side-view ultrasonic sensor.

[0024] According to one embodiment, the processor further comprises a memory for storing map data and wherein the processor is further able to modify the map data in response to the object position and the towing vehicle position, with the trailer wheel path being determined in response to the map data.

[0025] According to one embodiment, the image is captured in response to activation of the trailer system and detection of movement of the towing vehicle.

[0026] According to one embodiment, the vehicle dimension is a distance from the rear axle of the vehicle to a trailer coupling point, and the trailer dimension is at least one of a distance from the trailer wheel axle to a trailer coupling point and a trailer track width.

[0027] According to one embodiment, a trailer path hazard detection system comprises a trailer interface for detecting a connection to a trailer, a forward-facing camera for capturing a first image containing a curb hazard, a side-facing camera for capturing a second image containing the curb hazard, a distance sensor for determining a distance to the curb hazard, a processor for estimating a position of the curb hazard in response to the first image, the second image relative to a towing vehicle position, confirming the position of the curb hazard in response to the distance to the curb hazard, predicting a towing vehicle path in response to a towing vehicle deflection angle and the towing vehicle position, predicting a trailer wheel path in response to the towing vehicle path, a vehicle dimension, and a trailer dimension.and generating a warning signal in response to an intersection of the obstacle position and the trailer cycle path, and a display to show the second image and to overlay a first indicator of the curb position and a second indicator of the trailer cycle path onto the second image.

[0028] According to one embodiment, a loudspeaker is also provided to generate an acoustic alarm in response to the warning signal.

[0029] The exemplary embodiments are described below in conjunction with the following drawings, where identical numbers denote identical elements and where: Fig. 1 shows an exemplary environment for the use of the trailer path hazard detection system, according to an embodiment of the present disclosure; Fig. 2 a block diagram showing a system for implementing the trailer path hazard detection system, according to an exemplary embodiment of the present disclosure; Fig. 3 shows a flowchart illustrating an exemplary method for controlling the trailer path hazard detection system, according to an exemplary embodiment of the present disclosure; Fig. 4 a user interface display which provides the driver with feedback on the trailer path hazard detection system, according to an exemplary embodiment of the present disclosure; Fig. 5 shows a block diagram illustrating a system for implementing the trailer path hazard detection system, according to an exemplary embodiment of the present disclosure; and Fig. Figure 6 shows a further flowchart illustrating an exemplary method for controlling the trailer path hazard detection system, according to an exemplary embodiment of the present disclosure.

[0030] As used herein, the term module refers to an application-specific integrated circuit (ASIC), an electronic circuit, a processor (common, dedicated or as a group) and memory executing one or more software or firmware programs, a combinational logic circuit and / or other suitable components providing the described functionality.

[0031] In Fig. Figure 1 shows an environment 100 for the use of the trailer path hazard detection system according to an exemplary embodiment of the present disclosure. In the exemplary environment 100, a towing vehicle 105 and a trailer 110 are shown.The towing vehicle 105 can be equipped with optical imaging sensors, for example a front camera with a camera field of view (FOV) 130 and a right side camera with a right side field of view (FOV) 131 and a left side camera with a left side field of view (FOV) 132, as well as with distance measuring sensors, for example a front left ultrasonic sensor with a front left ultrasonic detection range 135, a front right ultrasonic sensor with a front right ultrasonic detection range 137, a rear left ultrasonic sensor with a rear left ultrasonic detection range 139 and a rear right ultrasonic sensor with a rear right ultrasonic detection range 138.

[0032] For vehicle operators, it is desirable that vehicles and trailers are not scratched or damaged. One source of damage to vehicle and trailer wheels is damage from curbs and parking blocks, primarily caused by the wheel rim contacting the curb when turning. The exemplary system is configured to detect curb hazards and estimate the vehicle's wheel path (120) in relation to curb hazards and other objects that may occur, providing guidance to the driver to avoid collisions. The exemplary system is also configured to detect curb hazards and estimate a trailer's wheel path (125) in relation to curb hazards and other objects that may occur, providing guidance to the driver to avoid collisions while towing a trailer.The exemplary system can use a fusion of vehicle perception sensors, including optical image sensors such as cameras and distance image sensors such as ultrasonic sensors, lidar or short-range radar, and other kinematic and dynamic vehicle data together with algorithms to detect, locate, track and report curb hazards and other dangerous objects within the turning radius of the trailer.

[0033] In some exemplary embodiments, a towing vehicle 105 is configured with a system that detects curbs and other hazards near the towing vehicle 105 and the trailer 110 when trailer operation is being carried out. The exemplary system determines whether the wheels of the vehicle or the trailer, or other parts of the trailer, could touch a curb 115 or other obstacle. If so, the exemplary system gives instructions to a driver or a vehicle control system to prevent such contact and subsequent damage to the vehicle 105 or trailer 110.

[0034] The exemplary system can include a number of perception sensors, including optical image sensors, e.g., front and side cameras, and distance image sensors, e.g., ultrasonic sensors or short-range radars, together with a processor, an algorithm, and a customer interface. In some exemplary embodiments, the tow vehicle 105 can be equipped with ultrasonic sensors at its front and rear corners for detecting and locating a curb and / or other objects adjacent to the tow vehicle 105. The tow vehicle 105 is further equipped with one or more forward-facing cameras with a forward field of view 130 and left and right side-facing cameras with a left side field of view 132 and a right side field of view 131, respectively.

[0035] In some exemplary embodiments, the system can be configured to perform sensor fusion of the individual sensor readings to create a local area map referenced to the towing vehicle 105. As the towing vehicle 105 moves, the local area map is updated with newly acquired information, and existing information is confirmed or updated in response to subsequent sensor data. Combining data from different sources using multi-sensor data fusion algorithms leverages data redundancy to reduce uncertainty in object position. In some exemplary embodiments, the sensor data can be fused using Bayesian fusion algorithms with Kalman filtering.

[0036] The system can further perform an estimation model for detecting and locating a curb or objects within the curb by combining two complementary in-vehicle sensors, including optical image sensors such as surround-view cameras and distance image sensors such as ultrasonic sensors. The estimation model can use a camera-based algorithm to acquire and map front and side views of the towing vehicle 105 and then fuse lateral ultrasonic sensor data with the front and side-view cameras using model-based or non-model-based Bayesian filtering. In some exemplary embodiments, the curb hazard or obstacles detected by the camera and ultrasonic sensor fusion can be combined to create a map of the area around the towing vehicle 105.

[0037] The system can also be configured to monitor the telemetry of the towing vehicle and trailer, such as the steering angle and speed of the towing vehicle. In response to this telemetry, the system is then configured to predict the odometry of the towing vehicle and trailer based on the telemetry data. This towing vehicle and trailer odometry maps the predicted towing vehicle path 120 and / or the predicted trailer path 125. This predicted trailer path 125 can be compared with detected obstacles and curbs on the local map. If a contact event is predicted in response to the predicted trailer path 125, a warning can be provided to the driver via a user interface, for example, in the form of an audible signal, a warning light, a seat vibration, or a graphic on an in-vehicle display.

[0038] A landmark estimation model can be further used to enhance the estimation model through the local odometry algorithm, which integrates several onboard sensors such as GPS, IMU, wheel impulses, and steering angle. The example algorithm can use local odometry to spatially and temporally stamp landmarks and, as a result, create a local map of recently seen landmarks. The algorithm can then use a kinematic / dynamic model of the towing vehicle 105 with / without trailer 110 to estimate the projected path of the towing vehicle 105 or the trailer wheels and assess whether this path would collide with the detected landmarks in the local map. The path guidance line is annotated on the vision system so that the feature can interact with the driver. The path guidance line can adapt and change in response to changes in steering direction and coupling angle.

[0039] The exemplary system can further include a user interface for providing feedback to the driver, such as audible signals, displays, and haptic feedback when a potential hazard is detected. For example, guide lines can be displayed to the driver, with the color of the guide lines representing the probability of a possible collision between the trailer wheels and the curb, based on the distance of the wheels to the curb and the time until the wheels collide with the curb.

[0040] In Fig. Figure 2 shows a block diagram of a system 200 for implementing a trailer path hazard detection system according to an exemplary embodiment of the present disclosure. The exemplary system 200 can be located in a towing vehicle and comprise a processor 240, a left distance imaging sensor 221, a right distance imaging sensor 223, a left optical sensor 224, a right optical sensor 225, a camera 222, a GPS sensor 245, an IMU sensor 235, a vehicle controller 270, a user interface 260, a memory 265, and a trailer interface 255.

[0041] The camera 222 can be a forward-facing camera installed on a front part of the towing vehicle, for example, on the front fairing or behind a rearview mirror. The camera 222 can include one or more image sensors to capture a forward-facing field of view from the front of the towing vehicle. In some exemplary embodiments, the images captured by the camera 222, the left optical sensor 224, and / or the right optical sensor 225 can be fused to produce a combined image of an area near the towing vehicle.

[0042] If more than one image sensor is used, a stereo image can be generated. Image processing algorithms can be performed on the stereo image to estimate depth information in the front field of view. One or more images captured by the camera 222 can then be connected to the processor 240. In addition, the exemplary system can include one or more lateral optical sensors, e.g., an optical sensor 224 for the left side and an optical sensor 225 for the right side, with which images are captured and / or depth information for the left and right sides of the vehicle and / or trailer is determined.

[0043] The left-side distance imaging sensor 221 and the right-side distance imaging sensor 223 can be distance imaging sensors and / or depth sensors configured to emit a pulse, receive a reflection of the pulse, and, in response to the travel time of the emitted pulse, estimate a distance to a vertical surface of an object, such as a curb or other obstacle. The left-side distance imaging sensor 221 and the right-side distance imaging sensor 223 can have fields of view orthogonal to the centerline of the towing vehicle and be oriented such that the field of view covers an expected position of a curb or other object near the side of the towing vehicle.In some exemplary embodiments, the left-side distance imaging sensor 221 and the right-side distance imaging sensor 223 can be ultrasonic sensors configured to emit ultrasonic audio pulses. The distance to a curb or object can be estimated based on the travel time of the ultrasonic audio pulse. Alternatively, the left-side distance imaging sensor 221 and the right-side distance imaging sensor 223 can be a lidar sensor, radar, or other rangefinder.

[0044] The GPS sensor 245 is configured to receive a multitude of timestamped satellite signals containing the location data of a transmitting satellite. The GPS sensor 245 then uses this information to determine its precise location. The processor 240 can receive the location data from the GPS sensor 245 and / or store this location data in the memory 265. The memory 265 can be used to store map data for use by the processor 240.

[0045] The IMU 235 is a device used to measure a specific force acting on a body, such as the angular or linear acceleration of the body to which it is attached. The IMU 235 can measure angular or linear acceleration and can be used to determine the lateral acceleration, longitudinal acceleration, yaw rate, and pitch of a vehicle. The IMU 235 is mounted inside the towing vehicle and is used to generate a control signal indicating the measured specific forces and to connect this control signal to the Processor 240.

[0046] The user interface 260 can serve to create a human-machine interface between the vehicle control system and the driver. The user interface 260 can be a touchscreen, a display, one or more buttons, dials, switches, etc., and / or one or more LEDs, sound-generating devices such as a speaker, or other indicators. The user interface 260 is used to receive user requests, such as a request to activate a vehicle system, like settings for ADAS operation. In an exemplary embodiment, the user interface 260 can be used to receive an activation of a trailer mode in a towing vehicle in response to user input. The user interface 260 can also be configured to display user warnings when a vehicle or trailer contact event is predicted.The User Interface 260 can display a predicted towing vehicle path and / or a predicted trailer path on a display in the towing vehicle cab. The User Interface 260 can be configured to receive information about the dimensions and / or configuration of the trailer in response to user input, or it can initiate an algorithm to estimate the dimensions of the trailer in response to user input.

[0047] In some exemplary embodiments, the user interface 260 can issue audible and / or visual warnings to the driver, which intensify depending on the time to collision and / or the distance to the collision. For example, if a projected trailer wheel path passes within a threshold distance of a curb or other object, the color of a displayed trailer wheel path indicator can change from green to yellow or from yellow to red. An audible warning signal can be generated by the user interface 260 in response to the projected trailer wheel path passing within a threshold distance of a curb or other object. The amplitude or frequency of an audible warning signal can change as the distance between the projected wheel path and an obstacle, such as a curb, changes.

[0048] The vehicle control unit 270 can be operated to control the operation of the vehicle depending on a control algorithm or similar. In an exemplary embodiment, the vehicle control unit 270 can be operated to generate vehicle control signals for coupling with a throttle control, a brake control, and a steering control, or the like, to control the vehicle in response to an algorithm of an advanced driver assistance system (ADAS) or in response to input from the vehicle operator via a steering wheel, vehicle pedals, and the like. The vehicle control unit 270 can also be used to generate system control signals for coupling with the processor 240, which display information such as the vehicle's speed, acceleration, vehicle motion cycles, the input state of the vehicle operator, the trailer, etc.

[0049] The trailer interface 255 can be a module within the towing vehicle configured to receive and transmit control signals and / or data to the trailer. The trailer interface 255 can be electrically connected to a trailer socket. The socket can be a 7-pin electrical socket, with each pin separately electrically connected to the trailer interface 255. The socket is configured to accept a plug that is electrically connected to the trailer's wiring system by an electrical cable long enough to remain attached to the towing vehicle's socket while recording the trailer's movement during towing operations. In some exemplary embodiments, the trailer interface 255 can be configured to detect when the plug is inserted into the socket.This detection can occur in response to a change in capacitance or voltage at one or more of the electrical terminals on the socket. For example, the Trailer Interface 255 can detect a change in voltage at the ground pin of the socket. Alternatively, plugging in can be detected in response to a change in the state of a physical switch, such as a momentary switch that is pressed by the plug when it is inserted into the socket.

[0050] The processor 240 is configured to execute the hazard detection algorithm for the trailer path. In response to image data received from the camera 222 and depth map information received from the left distance imaging sensor 221 and the right distance imaging sensor 223, the processor 240 is initially configured to create a local area map of objects and road features, such as curbs, near the towing vehicle. The processor 240 can supplement this local area map with map data stored in memory 265 or with map data received from other sources. The local area map is continuously updated as image data and depth information are received.The position of the towing vehicle and the trailer are located within the local area map, so that distances between the detected obstacles, the position of the towing vehicle and the position of the trailer can be determined.

[0051] The Processor 240 is also configured to process vehicle telemetry data, such as...

[0052] The Processor 240 receives vehicle speed, steering angle, braking information, and similar data from the vehicle control unit 270. The Processor 240 can also receive acceleration data from the IMU 235. Based on the telemetry and acceleration data, the Processor 240 is configured to estimate the expected towing vehicle path. Based on the expected towing vehicle path, the hitch deflection angle, the geometry of the towing vehicle and trailer, and measurements, the Processor 240 then predicts the expected trailer path. The expected towing vehicle path and the expected trailer path are compared to the map data, and potential contact events involving the towing vehicle, trailer, or trailer wheels are predicted.When a potential contact event is predicted, a warning control signal is generated and connected to the user interface 260 and / or the vehicle control unit 270. In some exemplary embodiments, the potential contact event can be based on the distance of the wheels to the curb as well as the time until the wheels collide with the curb. The vehicle control unit 270 can forward the warning control signals as information to an ADAS algorithm for assisted or autonomous vehicle operation.

[0053] In some exemplary embodiments, the processor 240 can first receive an image and / or image data from the camera 222, which maps the forward field of view of the towing vehicle. The processor 240 can then perform edge detection techniques or other image processing techniques to estimate the position of curbs near the towing vehicle. The processor 240 can subsequently receive depth data from the left distance imaging sensor 221 and the right distance imaging sensor 223, generated in response to a pulse reflection from one or more vertical surfaces, and confirm the position of the curbs. This confirmed information is stored as map data in memory 265 or other accessible memory. The processor 240 can then predict a trailer path in response to the trailer measurements and the towing vehicle telemetry.In response to the predicted trailer path, the processor 240 can next predict possible contact events between the towing vehicle, the towing vehicle's wheels, and / or the trailer's wheels and the curb. If a contact event is predicted, a warning signal is transmitted from the processor 240 to the user interface 260 to alert the driver to the potential contact event.

[0054] In some exemplary embodiments, a top-down view of the towing vehicle and trailer can be generated from various cameras positioned around the towing vehicle and / or trailer. An image of the towing vehicle and an image of the trailer can be superimposed on the top-down view, as can indicators of the predicted path of the towing vehicle and the predicted path of the trailer. Curbs, lane markings, and other objects can be highlighted in the top-down view or displayed through additional overlays. Anticipated contact points can be displayed in the top-down view. Potential contact points can be highlighted by color (e.g., red), indicated by a symbol (e.g., an exploded view), or otherwise displayed to the operator.

[0055] Fig. Figure 3 shows a flowchart illustrating an exemplary method 300 for controlling a trailer path hazard detection system according to an exemplary embodiment of the present disclosure.

[0056] The method is initially configured to receive 310 images from a camera mounted on the towing vehicle. The camera can be a wide-angle camera with a front field of view from the towing vehicle, or a wide-angle camera with a side field of view from the towing vehicle. In some exemplary embodiments, the camera can include a plurality of image sensors capable of capturing a stereo image from which depth information for objects within the stereo image can be determined. The image, or a series of images, captured at regular intervals, can then be connected to an image processor. In some exemplary embodiments, the processor can be configured to detect horizontal edges or lines within the image that might indicate a curb.

[0057] Next, the method detects 315 objects in the captured image or the multitude of images. The method can use successive images taken at different times to estimate distances to objects using the parallax within the images caused by the movement of the towing vehicle. In some exemplary embodiments, the images can be converted to grayscale, an inverse perspective mapping can be performed, and then image smoothing can be applied. Canny edge detection can be used to detect edges within the image, such as curbs, lampposts, etc. A Hough transform can also be used to extract features from the images and detect edges or other shapes.

[0058] Next, the method receives depth information from the side-view sensors, which have a lateral field of view near the front of the towing vehicle. These side-view sensors can include ultrasonic sensors, cameras, lidar, radar, or similar devices. In some exemplary embodiments, the side-view sensors are configured to generate depth maps of their respective fields of view and couple these depth maps to the processor. The depths can indicate the distance between the side-view sensor and a vertical surface. In some exemplary embodiments, depth information is coupled to the processor, for example, two-dimensional depth information acquired at regular intervals as the towing vehicle moves. The field of view can, for example, be in the form of a vertical fan representing a variety of depths at different heights with a single azimuth.The movement of the towing vehicle allows for a variation in the azimuth measurements, which are used to create the depth map.

[0059] Next, procedure 325 confirms the position of objects determined from the image using depth information received from the side-view sensors. For example, a horizontal edge detected in the image can be correlated with a vertical edge detected in response to the depth information to estimate the position of a curb. If the object's position is not confirmed, the procedure returns to receiving the next image 310. If the position is confirmed, procedure 330 updates map information stored in memory.

[0060] The procedure is configured to update the map with object positions in response to a landmark estimation model that combines the front view captured by the camera and the depth information captured by the side-view sensors using Bayesian filtering. The stored map section can be cropped depending on the tow vehicle's position. For example, the procedure can store map information for a predetermined radius around the tow vehicle, discarding the object information as soon as the object falls outside the predetermined radius.

[0061] Next, the procedure is configured to receive telemetry data from the tow vehicle's control system or from other sensors on the tow vehicle, 335. The telemetry data may include the vehicle's speed, location, velocity, steering angle, degree of brake application, various accelerations detected by the vehicle's IMU, or similar information. The landmark estimation model can be further improved by the local odometry algorithm, which combines several onboard sensors, such as GPS, IMU, wheel impulses, and steering angle, using Bayesian filtering.

[0062] The method next predicts the towing vehicle path in response to the received telemetry data. The towing vehicle path is predicted using the local map and a kinematic / dynamic model of the towing vehicle to estimate the projected vehicle path. Next, the method uses the predicted towing vehicle path, the dimensions of the trailer, and, in some exemplary embodiments, the coupling deflection angle to predict the trailer path. In particular, the method can predict the trailer wheel paths.

[0063] Next, the procedure 350 predicts possible contact events in response to the map data and the predicted trailer path. In some exemplary embodiments, the trailer path hazard detection algorithm can assess whether the trailer path might intersect with the detected landmarks in the local map. If no intersection is predicted, meaning no contact event will occur, the procedure is configured to return to receiving the next image 310.

[0064] When a contact event is predicted, the procedure is next configured to generate a warning that is communicated to the driver or a vehicle control system executing an ADAS algorithm. For driver warning, a projected trailer path line can be displayed on a graphical user interface of a vehicle cabin display. In some exemplary embodiments, the path line can be adaptive and change its position relative to a displayed obstacle as the steering wheel and coupling angle change. For example, a curb can be displayed, and the projected trailer path can also be projected.As the projected trailer wheel path approaches the curb, the displayed projected trailer wheel path approaches the curb indicator, and the color of the displayed projected trailer wheel path may change to highlight the increased likelihood of contact. An audible, haptic, or other warning to the driver may be triggered when the distance between the projected trailer wheel path and the obstacle reaches a threshold. After the warning is generated, the procedure returns to receiving a subsequent image.

[0065] In some exemplary embodiments, the method uses the vehicle's odometry model to create a spatiotemporal map of recently seen landmarks by combining sensor data such as cameras, ultrasound (and / or short-range radar), vehicle dynamics and kinematics sensors, and models such as steering angle, GPS, IMU, and wheel encoders. The method fuses the spatiotemporal map of the landmarks with the projected / predictive trajectory of the vehicle / trailer wheels, which is designed to assess whether the vehicle wheels are about to collide with these landmarks, based on the wheels' distance to the landmarks, the time until the wheels collide with the landmarks, or a combination of both.

[0066] The method can determine the projected or predictive trajectory of the vehicle / trailer wheels by using the vehicle / trailer model, including but not limited to the dynamics or kinematics model, as well as the vehicle's own sensors, including but not limited to the steering angle and the IMU, together with the coupling angle between the vehicle and trailer and other vehicle and trailer dimensions. The vehicle's user interface and / or feedback system can utilize the assessment and provide feedback to the driver, for example, in the form of audible beeps, displays, and haptic feedback when a potential hazard is detected. Examples include a guide color indicating a possible collision between the rear wheels and the curb based on the distance of the wheels to the curb and the time until the wheels collide with the curb.

[0067] Fig. Figure 4 shows an exemplary user interface display that provides the driver with feedback about the trailer hazard detection system according to an exemplary embodiment of the present disclosure. The first image 400 shows a side view of the towing vehicle as it begins to execute a right turn during towing operations. The detected curb is highlighted by the system by displaying a first indicator 405 above the detected curb. A trailer guidance line 410 is also displayed on the user interface image to indicate the expected trailer wheel path to the driver. The first indicator 405 and the trailer guidance line 410 can be displayed in different colors on the user interface image.In some exemplary embodiments, the color of the trailer guidance line can change as the turning process progresses and as the projected trailer wheel path approaches the curb, in order to indicate to the driver the increased risk of a collision with the curb.

[0068] The second figure, 430, shows a curve where the projected trailer wheel path is about to make contact with the curb. In some exemplary embodiments, the curb indicator 415 and the trailer guidance line 420 are displayed almost overlapping on the user interface. In some embodiments, the color of the trailer guidance line can change to alert the vehicle operator to the impending collision with the curb. Furthermore, additional warnings can be provided to the vehicle operator, such as a haptic warning signal, an audible warning signal, flashing LEDs, or similar indicators of the impending collision with the curb.

[0069] In Fig. Figure 5 shows a block diagram of a trailer path hazard detection system 500 according to an exemplary embodiment of the present disclosure. The exemplary system may comprise a camera 510, a distance sensor 515, a processor 520, a position sensor 525, a memory 535, and a user interface 530.

[0070] The camera 510 can be configured to capture an image of a field of view containing an object. In some exemplary embodiments, the object is a curb. The obstacle can be another obstruction, such as a traffic sign, a safety bollard, a light pole, a utility pole, or similar, located near the edge of the roadway. In some exemplary embodiments, the camera can be a stereo camera capable of capturing multiple stereo images, with the object's position determined based on the multiple stereo images. The image can be captured in response to activation of the trailer system and detection of movement by the towing vehicle.

[0071] The distance sensor 515 is configured to determine the distance to the object. The distance sensor 515 can have a laterally oriented field of view and determine the distance to the object at a different time than when the image of the object was captured by the camera. These different time-of-view measurements can be correlated with the positions of the towing vehicle at those different times to confirm the object's location and generate map data based on the object's location. In some exemplary embodiments, the distance sensor 515 can be a side-view ultrasonic sensor.

[0072] The 525 tracking sensor is configured to determine the tow vehicle's position. The 525 can operate as a global positioning sensor, determining the vehicle's position in response to data transmitted by a variety of satellites. Alternatively, the tracking sensor can also use vehicle-to-infrastructure (V2I) signals from the local infrastructure to determine the tow vehicle's position.

[0073] The Processor 520 is configured to detect an object's position in response to the image and the position of the towing vehicle. The Processor 520 can use image processing techniques, such as edge detection, to identify the object in the image. Furthermore, the Processor 520 can estimate the distance to the object based on one or more images and the position of the towing vehicle.

[0074] The processor 520 can further predict a tow vehicle path in response to the tow vehicle steering angle and the tow vehicle position. The tow vehicle path can be located in the map data stored in memory 535 and added to the map data. The processor 520 can then predict a trailer wheel path in response to the tow vehicle path and a trailer dimension. In some embodiments, the trailer dimension is a distance from a trailer wheel to a trailer coupling point. Alternatively, the trailer dimension can include a variety of dimensions, such as the width of the trailer track, the distance from the coupling point to the trailer axle, the trailer width, the trailer height, the trailer length, the length of the coupling mechanism, and the coupling deflection angle.Furthermore, the processor can be configured to generate a warning signal in response to an intersection between the object position and the trailer wheel path.

[0075] The 530 user interface can be configured to display a warning in response to the warning signal. The 530 user interface can be a warning light, a speaker for playing an audible alarm, or a haptic device. The user interface can also be an in-cabin display showing one or more images containing the curb, overlaying a curb marker and indicating the trailer wheel path.

[0076] The exemplary system can further include a memory 535 for storing map data, wherein the processor further serves to modify the map data in response to the object position and the towing vehicle position, and wherein the trailer wheel path is determined in response to the map data.

[0077] In some exemplary embodiments, the trailer path hazard detection system 500 may include a trailer interface for detecting a connection to a trailer, a forward-facing camera for capturing a first image containing a curb, a side-facing camera for capturing a second image containing the curb, a processor for determining a curb position in response to the first image, the second image and a towing vehicle position, predictions of a towing vehicle path depending on a towing vehicle steering angle and the towing vehicle position, predictions of a trailer wheel path depending on the towing vehicle path and a trailer dimension, and generating a warning signal depending on an intersection of the obstacle position and the trailer wheel path.and a display for showing the second image and for overlaying a first indicator of the curb position and a second indicator of the trailer wheel path onto the second image. The Trailer Path Hazard Detection System 500 may further include a loudspeaker to generate an audible alarm in response to the warning signal.

[0078] In Fig.Figure 6 is a flowchart illustrating an exemplary method 600 for implementing a trailer roll hazard detection system according to an exemplary embodiment of the present disclosure. The exemplary method can be performed by a processor in a towing vehicle. The processor can be communicatively connected to sensors, controllers, and interfaces within the towing vehicle. The processor can also be communicatively coupled to a user interface, such as a vehicle cabin display, a vehicle infotainment system, or an application running on a smartphone or the like.

[0079] The exemplary method can detect an obstacle position in response to an image, wherein the image is captured by a camera mounted on the towing vehicle. The camera can have a front field of view from the towing vehicle. In some exemplary embodiments, the obstacle position can be confirmed in response to a depth measurement from a side-view sensor mounted on the towing vehicle. The obstacle position can be confirmed in response to a depth measurement from an ultrasonic sensor mounted on the towing vehicle. The image can be captured in response to activation of the trailer system and detection of movement by the towing vehicle.

[0080] The method can predict a tow vehicle path in response to a tow vehicle steering angle and tow vehicle position. Furthermore, the tow vehicle path can be predicted in response to vehicle telemetry data, map data, and data from positioning sensors. For example, the tow vehicle path can be predicted based on an acceleration detected by an inertial measurement unit mounted on the tow vehicle. In some embodiments, the tow vehicle path can be predicted in response to a displacement detected by a wheel encoder mounted on the tow vehicle.

[0081] The procedure can then predict a trailer wheel path 630, which is derived from the towing vehicle path and the dimensions of the vehicle and trailer. The vehicle dimension can be the distance between the vehicle's rear axle and the trailer coupling point. The trailer dimension can be the distance from the trailer's wheel axle to the trailer coupling point and the trailer's track width. The trailer wheel path can be predicted depending on the coupling deflection angle.

[0082] The method can then compare the predicted trailer wheel path and the obstacle position to determine whether a contact event can occur. A contact event can occur if there is a predicted intersection between the obstacle position and the trailer wheel path. The method can generate a warning signal in response to the predicted contact event. 640 In some exemplary embodiments, the method can be operational to display an indicator for the obstacle position and an indicator for the trailer wheel path on a vehicle cab display. 650 The color of the trailer wheel path indicator can change depending on the current distance between the trailer wheel and the obstacle. The method can further be used to modify map data depending on the obstacle position and the towing vehicle position, 660 whereby the trailer wheel path is determined depending on the map data.

Claims

[1] Procedure (600), comprising: (610) Recognizing an obstacle position in response to an image; (620) Predictions of a tow vehicle path in response to a tow vehicle steering angle and a tow vehicle position; (630) Predictions, by a processor, of a trailer wheel path in response to the towing vehicle path and a trailer dimension, wherein an odometry of the towing vehicle and the trailer is predicted as a function of the telemetry; (640) Generating a warning signal in response to an intersection between the obstacle position and the trailer wheel path; (650) Displaying an obstacle position indicator and a trailer wheel path indicator on a vehicle cabin display; and (660) Modifying map data in response to the obstacle position and the towing vehicle position, whereby the trailer wheel path is determined in response to the map data. [2] Method (600) according to claim 1, wherein the image is captured by a camera (222) mounted on the towing vehicle (105) which has a forward-facing field of view (130) from the towing vehicle (105). [3] Method (600) according to claim 1, wherein the obstacle position is confirmed as a response to a depth measurement of a side view sensor attached to the towing vehicle (105). [4] Method (600) according to claim 1, wherein the obstacle position is confirmed as a response to a depth measurement of an ultrasonic sensor attached to the towing vehicle (105). [5] Method (600) according to claim 1, wherein the trailer wheel path (125) is predicted in response to a trailer coupling deflection angle. [6] Method (600) according to claim 1, wherein the towing vehicle path (120) is predicted in response to an acceleration detected by an inertial measurement unit attached to a towing vehicle (105). [7] Method (600) according to claim 1, wherein the image is captured in response to a trailer system activation and a detection of movement of a towing vehicle (105). [8] Device (500), comprising: a camera (510) configured to take an image of a field of view containing an object; a distance sensor (515) configured to determine a distance to the object; a locating sensor (525) configured to determine a towing vehicle position; a processor (520) configured to In response to the image and the towing vehicle position, an object position is determined. predicts a towing vehicle path in response to a towing vehicle steering angle and the towing vehicle position. In response to the towing vehicle path and a trailer dimension, a trailer wheel path is predicted, whereby an odometry of the towing vehicle and the trailer is predicted depending on the telemetry, and a warning signal is generated in response to an intersection between the object's position and the trailer's wheel path; a user interface (530) configured to display a warning in response to the warning signal; and a memory (535) for storing map data, wherein the processor further serves to modify the map data in response to the object position and the towing vehicle position, and wherein the trailer wheel path is determined in response to the map data.

Images