Trailer Backup Track Overlay Using a Trailer Camera Display System

Abstract

The controller is connected to a plurality of cameras on the vehicle. At least one side camera is configured to define a rearward lateral view, and at least one rear camera is configured to generate a rearward-facing view. The memory-stored instructions cause the processor to determine a trailer angle of the trailer relative to the tractor based on images provided by the at least one side camera, estimate a trailer angle rate, and determine trailer end positions at a plurality of stages based at least in part on the vehicle speed, the estimated trailer angle rate, and the determined trailer angle. A predicted trailer path is determined using the determined trailer end positions, and an overlay showing the predicted trailer path is generated on the display.

Description

[Technical Field]

[0001] The present disclosure relates to a camera monitoring system (CMS) for use in a vehicle towing a trailer, and more particularly to a system for displaying a prediction of the expected trailer path during a reversing maneuver.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 426,391, filed November 18, 2022. [Background technology]

[0003] Mirror replacement systems, and camera systems that complement mirror views, are utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. Camera monitoring systems (CMS) utilize one or more cameras positioned around the vehicle to provide the vehicle operator with an expanded field of view. In some instances, the mirror replacement system within a CMS can cover a wider field of view than a traditional mirror or can include views not fully obtainable through a traditional mirror.

[0004] The area behind the trailer is a typical blind spot in conventional mirror systems, making reversing maneuvers difficult while a trailer is attached. Further contributing to the difficulty of vehicle maneuvering is the fact that trailer behavior during reversing maneuvers differs from trailer behavior during forward maneuvers, and driver assistance systems and estimation techniques available for forward maneuvers are typically unavailable during reversing maneuvers. Summary of the Invention

[0005] In one exemplary embodiment, a camera monitoring system (CMS) for a vehicle includes a CMS controller including a memory and a processor. The CMS controller is connected to a plurality of cameras arranged around the vehicle and configured to receive a video feed from each camera in the plurality of cameras. The CMS controller includes at least one side camera configured to define a rearward-side view and at least one rear camera configured to generate a rearward-facing view. Memory-stored instructions cause the processor to determine a trailer angle of a trailer relative to a tractor based on images provided by the at least one side camera, estimate a trailer angle rate, and determine a trailer end position at a plurality of stages based at least in part on vehicle speed, the estimated trailer angle rate, and the determined trailer angle. A predicted trailer path is determined using the determined trailer end positions, and the processor generates an overlay showing the predicted trailer path and applies the overlay to a rearview display.

[0006] In a further embodiment of any of the above, determining the end position of the trailer in the process includes one of determining the end position of the trailer at a plurality of time intervals and determining the end position of the trailer at a plurality of distance intervals.

[0007] In a further embodiment of any of the above, the rearward looking view includes at least one of a Class VIII view and a rearview mirror replacement view.

[0008] In a further embodiment of any of the above, the rearward facing view includes at least a portion of the trailer.

[0009] In a further embodiment of any of the above, the processor is configured to estimate trailer angle rate using Kalman filtering.

[0010] In a further embodiment of any of the above, determining a predicted trailer path using the determined trailer end positions includes calculating a 3D trajectory using a least squares approximation to calculate a trailer trajectory in 3D space and transforming the 3D trajectory.

[0011] In a further embodiment of any of the above, generating an overlay indicative of the predicted trailer path includes converting the 3D trajectory into a 2D image.

[0012] In a further embodiment of any of the above, determining a trailer angle of a trailer relative to a tractor based on images provided by the at least one side camera includes determining the trailer angle without using a dedicated angle detection sensor.

[0013] In a further embodiment of any of the above, the memory further stores instructions configured to cause the processor to identify at least one object in a rearview image including the rear side view and the rearward view, and instructions configured to modify the overlay in response to at least one object intersecting the overlay in the rearview image.

[0014] In a further embodiment of any of the above, the overlay is changed by changing the color of the overlay. [Brief explanation of the drawings]

[0015] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0016] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitoring system (CMS) used to provide at least Class II and Class IV views.

[0017] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera mirror system providing Class II, Class IV, Class V, Class VI, and Class VIII views.

[0018] [Figure 2] 1 is a schematic view of the interior of a vehicle cab.

[0019] [Figure 3] 10A and 10B show schematic illustrations of rear view displaced display scenes including predicted trailer trajectories;

[0020] [Figure 4] 4 illustrates a method for creating a rear view trajectory overlay for the rear view replacement display scene of FIG. 3.

[0021] [Figure 5] We show how to generate warnings based on predicted trajectories.

[0022] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0023] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. FIG. 2 is a schematic top perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While commercial trucks are contemplated in this disclosure, the present invention is applicable to other types of vehicles. The vehicle 10 incorporates a camera monitor system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b (generally "16") mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, each arm housing one or more cameras and / or mirrors.

[0024] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b (collectively, "20") is disposed within each camera arm. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 16a, 16b. If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an illustration of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing enclosing electronics configured to provide various features of the CMS 15.

[0025] First and second video displays 18a, 18b (collectively, "18") are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing side views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0026] If Class V and / or Class VI view footage is also desired, a camera housing 16c and camera 20c may be positioned at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located within the cab 12 near the top center of the windshield can be used to display Class V and Class VI views forward of the vehicle 10 to the driver. Displays 18a, 18b, and 18c face a driver area 24 within the cab 22, where the driver is seated in a driver's seat 26. The location, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0027] If a Class VIII view is required, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the vehicle's Class VIII zone. As shown, the Class VIII view includes a view that surrounds the immediate vicinity of the trailer and a rearward close-up view of the vehicle that includes the area behind the trailer. In one example, the rearward close-up view of the vehicle is generated by a rear-facing camera positioned at the rear of the vehicle and may include both an immediate rearward close-up view and a traditional rearward view (e.g., a field of view extending rearward to the horizon provided by a rearview mirror on a vehicle without a trailer). In such an example, the third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near the first, second, and third displays 18a, 18b, and 18c to provide dedicated displays that provide the Class VIII view.

[0028] In some cases, the Class VIII view is generated using a trailer-mounted camera 30. The trailer-mounted camera 30 is a rear-facing camera that provides a field of view 32 that encompasses a portion of the trailer, a rear-facing Class VIII view, and a conventional rearview mirror. This rearview mirror portion is identified by the CMS 15 and provided on either displays 18a, 18b and / or another display 18c within the vehicle cab 22, either as a replacement for the rearview mirror or as a supplement to the rearview mirror. This view is particularly beneficial because the trailer 14 may obstruct some or all of the view provided by a conventional rearview mirror.

[0029] CMS 15 is also configured to utilize imagery from cameras 20a, 20b, 30, as well as imagery from other cameras that may be positioned around the vehicle to determine vehicle characteristics, identify objects, and facilitate driver assistance features such as display overlays and semi-autonomous driver assistance systems.

[0030] These features and functions of the CMS 15 are used to implement multiple CMS 15 systems that assist in the operation of a vehicle. Note that a controller 28 (FIG. 2) for the CMS 15 can be used to implement the various functions disclosed herein. The controller 28, which communicates with the displays 18 and the cameras 20, may include one or more separate units. For example, a centralized architecture may have a common controller located on the vehicle 10, while a distributed architecture may use a controller located on each of the displays 18, for example. Furthermore, part of the controller 28 may be located on the vehicle 10, while another part of the controller 28 may be located elsewhere, such as on the camera arm 16. In another example, a master-slave display configuration may be used, where one display includes the controller 28 and the other display receives commands from the controller 28.

[0031] In terms of hardware architecture, such a controller may include a processor, memory (e.g., memory), and one or more input and / or output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but is not limited to, one or more buses and / or other wired or wireless connections. The local interface may also include additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers that enable communication, which are omitted for simplicity. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0032] Controller 28 may be a hardware device for executing software, particularly software stored in a memory (e.g., a memory). Controller 28 may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device for general-purposely executing software instructions.

[0033] The memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). Furthermore, the memory may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory may have a distributed architecture where various components are located remotely from each other but are accessible by the processor.

[0034] Software in memory may include one or more separate programs, each containing an ordered list of executable instructions for implementing a logical function. A system component embodied as software may be constructed as a source program, an executable program (physical code), a script, or any other entity containing a set of instructions to be executed. If constructed as a source program, the program is translated via a compiler, assembler, interpreter, etc., which may or may not be contained in memory.

[0035] Input / output devices of the present disclosure that may be coupled to the system I / O interface(s) may include, but are not limited to, input devices such as a keyboard, mouse, scanner, microphone, camera, mobile device, proximity device, etc. They may also include, but are not limited to, output devices such as a printer, display, etc. Finally, input / output devices may further include devices that communicate as both input and output, such as, but are not limited to, a modulator / demodulator (i.e., for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc.

[0036] When controller 28 is in operation, the processor may be configured to execute software stored in the memory, communicate data to and from the memory, and generally control the operation of the computing device in accordance with the software. The software in the memory is read, in whole or in part, by the processor and often buffered within the processor before being executed.

[0037] In various examples, controller 28 includes one or more modules having algorithm(s), equation(s), and / or decision manager(s) that receive input(s) from sensors and / or stored values. During vehicle operation, controller 28 may use outputs (e.g., display 18, speaker, etc.) to communicate information to the driver, fleet operator, or others.

[0038] One such CMS system is a reversing assist system that generates trailer trajectory predictions for a reversing maneuver of the vehicle 10. An example output of the reversing assist system is shown in rearview replacement scene 100 in FIG. 3. The illustrated replacement scene 100 includes a single person 120 and a single tree 130 for ease of explanation, but it will be understood that in a practical example, the replacement scene 100 may include more objects, a greater variety of objects, roads, multiple types of objects, etc. In the illustrated example, the scene 100 includes at least a portion of the rear end of the trailer 14. The scene 100 is displayed on one or more of the monitors 18a, 18b, 18c and / or another monitor within the vehicle.

[0039] During a backing maneuver, CMS 15 uses the backing assist system to determine a predicted rear trajectory (i.e., the expected path of the rear end of trailer 14) and provides the predicted trajectory as an overlay 110 on top of scene 100. Overlay 110 extends from the rear end of trailer 14 into scene 100 and tracks the expected position of the rear end of trailer 14 over time and / or distance. If the predicted trajectory intersects with an object (e.g., person 120), CMS 15 can generate a warning indicating a potential collision may occur. The warning can take the form of an audio output to the operator, a shaded identifier 122 in overlay 110, a color change, or any combination thereof. In other examples, any other method of drawing the operator's attention to object 120 can be utilized.

[0040] Continuing with reference to the scene 100 of FIG. 3, FIG. 4 schematically illustrates a process 300 for generating the overlay 110. Initially, the CMS 15 receives images from the rear-facing camera(s) 30, the Class II / IV cameras, and other cameras within the CMS 15. Next, the CMS 15 uses image analysis techniques to determine the end position of the trailer 14 in three-dimensional (real-world) space and the trailer angle relative to the tractor 12 in a "Determine Trailer End Position and Angle" step 310. In one example, the trailer angle and end position are determined using image analysis alone, without the use of any other sensors other than the angle sensors or image sensors (cameras) of the CMS 15. Additionally, during this step, the CMS 15 receives multiple parameters from the vehicle controller, including truck speed, yaw rate, steering angle, gear, and other camera extrinsic parameters.

[0041] As the vehicle 10 operates, the trailer end position and angle are calculated multiple times from the images, and the rate of change of the trailer angle and trailer position is determined in step 320 "Estimate Trailer Angle Rate of Change." The rate of change can be over time, over distance, or a combination of both. In one example, the rate of change is determined by applying a Kalman filter to the determined trailer end position and trailer angle and additional parameters received from the vehicle controller, with the output of the Kalman filter being the rate of change. The rate of change tracks the change in the trailer end position in 3D space and is redetermined at each iteration of process 300. In one example, the trailer angle rate of change and truck speed are converted to trailer end motion in two perpendicular (x and y) directions. An integral formula calculates the change in trailer end position over a predetermined period (e.g., 1 second, 2 seconds, etc.). With a prediction of the trailer end position over the calculated period, a trajectory is obtained by connecting the points.

[0042] Once the rate of change of the trailer end is determined, CMS 15 calculates what the estimated location of the trailer end will be in three-dimensional space at a given time and / or distance interval in "Calculate Trailer End Position" step 330. Process 300 loops through step 330 multiple times, with each loop determining the estimated end position at a separate time and / or distance interval. The time and / or distance intervals, in some examples, are fixed intervals stored in memory of CMS 15. In alternative examples, the time and / or instantaneous intervals may depend on speed, yaw rate, or any other parameter.

[0043] After determining the trailer end position at each interval, process 300 combines the trailer end positions to generate a predicted trajectory for the trailer end in "Determine Trailer Trajectory in 3D Space" step 340. The trailer trajectory is the route that the trailer end is expected to travel in three-dimensional space as it moves from each determined interval to the next determined interval.

[0044] In one example, a complete trajectory connecting the trailer's end positions at each determined interval is determined using least-squares filtering of the trailer's end points at each interval, and the resulting curve is the predicted trajectory.

[0045] After determining the 3D trajectory of the trailer end, the 3D trajectory is converted to a 2D graphical overlay in a "Convert 3D Trajectory to 2D Overlay" step 350. This conversion converts the 3D trailer end route into a 2D trajectory through the scene 100 and creates a transparent overlay 110 of the trajectory.

[0046] Once the transparent overlay 110 is created, in an "apply 2D overlay to rearview display" step 360, the overlay 110 is applied to the image and displayed to the operator.

[0047] In some examples, after determining the trajectory and before applying the overlay to the scene 100, the CMS 15 identifies any objects 120, 130 in the scene 100 that intersect with the trajectory and outputs a warning to the vehicle operator. The warning can take the form of an audio output, a visual indicator (as in the example scene 110), a color change, or any similar warning. Figure 5 shows a method 400 for achieving this warning.

[0048] First, CMS 15 uses image-based object identification techniques to identify objects 120, 130 in scene 100 and identify the two-dimensional positions of the objects in scene 100 in "Identify Objects in View" step 410. Next, the two-dimensional positions of objects 120, 130 in scene 110 are converted to three-dimensional positions of objects 120, 130 in real space. After determining the three-dimensional trajectories of the ends of trailer 14, CMS 15 compares the three-dimensional position of each object to this trajectory in "Compare Object Positions to Trajectories" step 420 and displays a warning if the end of trailer 14 passes through the same three-dimensional space as objects 120, 130 in "Generate Display Alert" step 430.

[0049] In more complex systems, the trajectory of a moving object (e.g., person 120) can be estimated using a similar trajectory estimation process, and the predicted trajectory of the moving object is compared to the predicted trajectory of the end of trailer 14. In such an example, an alert is generated if the object's trajectory intersects with the trajectory of trailer 14 simultaneously or within a predetermined time window (e.g., + / - 10 seconds).

[0050] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A camera monitoring system (CMS) for a vehicle, comprising: a CMS controller including a memory and a processor; the CMS controller is connected to a plurality of cameras arranged around the vehicle and configured to receive a video feed from each camera in the plurality of cameras, the CMS controller including at least one side camera configured to define a rearward lateral view and at least one rear camera configured to generate a rearward facing view; the memory storing instructions for causing the processor to determine a trailer angle of the trailer relative to the tractor based on images provided by the at least one side camera; estimating a trailer angle rate and determining trailer end positions at multiple stages based at least in part on vehicle speed, the estimated trailer angle rate, and the determined trailer angle; determining a predicted trailer path using the determined trailer end positions; and generating an overlay showing the predicted trailer path and applying the overlay to a rearview display.

2. 2. The camera monitoring system of claim 1, wherein determining the trailer end position in the process includes one of determining the trailer end position at a plurality of time intervals and determining the trailer end position at a plurality of distance intervals.

3. The camera monitoring system of claim 1 , wherein the rearward-facing view comprises at least one of a Class VIII view and a rearview mirror-replacement view.

4. The camera monitoring system of claim 3 , wherein the rearward-facing view includes at least a portion of the trailer.

5. The camera monitoring system of claim 1 , wherein the processor is configured to estimate trailer angle rate using Kalman filtering.

6. 2. The camera monitoring system of claim 1, wherein determining a predicted trailer path using the determined trailer end positions includes calculating a 3D trajectory using a least squares approximation to calculate a trailer trajectory in 3D space and transforming the 3D trajectory.

7. The camera monitoring system of claim 6 , wherein generating an overlay showing the predicted trailer path includes converting the 3D trajectory into a 2D image.

8. 2. The camera monitoring system of claim 1, wherein determining a trailer angle of a trailer relative to a tractor based on images provided by the at least one side camera includes determining the trailer angle without using a dedicated angle detection sensor.

9. 2. The camera monitoring system of claim 1, wherein the memory further stores instructions configured to cause the processor to identify at least one object in a rear-view image including the rear side view and the rearward view, and instructions configured to modify the overlay in response to at least one object intersecting the overlay in the rear-view image.

10. The camera monitoring system of claim 9 , wherein the overlay is changed by changing the color of the overlay.

Images