Trailer backup trajectory overlay using trailer camera display system

Abstract

The controller is connected to a plurality of cameras on the vehicle. The at least one side camera is configured to define a rear side view, and the at least one rear camera is configured to generate a rear view. A memory storing instructions causes the processor to determine a trailer angle of the trailer relative to the tractor based on images provided by the at least one side camera, cause the processor to estimate a trailer angular rate, a processor is caused to determine a trailer end position in a plurality of instances based at least in part on the vehicle speed, the estimated trailer angular rate, and the determined trailer angle. A projected trailer path is determined using the determined trailer end position, and an overlap depicting the projected trailer path is generated on a display.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority of U.S. Provisional Application No. 63 / 426,391, filed on November 18, 2022. Technical field

[0003] The present disclosure relates to a camera monitoring system (CMS) for a vehicle towing a trailer, and more particularly to a system for displaying a projection of an expected trailer path during a reverse maneuver. Background art

[0004] In commercial vehicles, mirror replacement systems and camera systems for supplementing the mirror view are utilized to enhance the ability of a vehicle operator to see the surrounding environment. A camera monitoring system (CMS) uses one or more cameras disposed around the vehicle to provide an enhanced field of view to the vehicle operator. In some examples, the mirror replacement system within the CMS can cover a larger field of view than a conventional mirror, or can include views that cannot be fully obtained via a conventional mirror.

[0005] The area behind a trailer is a typical blind spot in a conventional mirror system, resulting in difficulty in performing reverse maneuvers when attaching a trailer. Further affecting the difficulty of vehicle operation is the fact that the movement of the trailer during a reverse maneuver is different from the movement of the trailer during a forward maneuver, and driver assistance systems and estimation techniques available for forward maneuvers are generally not available during a reverse maneuver. Summary of the invention

[0006] In one exemplary embodiment, a camera monitoring system (CMS) for a vehicle includes a CMS controller that includes a memory and a processor. The CMS controller is connected to a plurality of cameras disposed around the vehicle and is configured to receive video feeds from each of the plurality of cameras. The CMS controller includes at least one side camera configured to define a rear - side view and at least one rear camera configured to generate a rear - view. The memory storing instructions causes the processor to determine a trailer angle of the trailer relative to the towing vehicle based on an image provided by the at least one side camera, causes the processor to estimate a trailer angular rate, causes the processor to determine trailer end positions at a plurality of instances at least in part based on the vehicle speed, the estimated trailer angular rate, and the determined trailer angle. The determined trailer end positions are used to determine a projected trailer path, and causes the processor to generate an overlay depicting the projected trailer path and apply the overlay to a rear - view display.

[0007] In another embodiment of any of the above embodiments, determining the trailer end positions at instances includes determining the trailer end positions at a plurality of time intervals and determining the trailer end positions at a plurality of distance intervals.

[0008] In another embodiment of any of the above, the rear view includes at least one of a Class VIII view and a rearview mirror replacement view.

[0009] In another embodiment of any of the above, the rear view includes at least a portion of the trailer.

[0010] In another embodiment of any of the above embodiments, the processor is configured to use Kalman filtering to estimate the trailer angular rate.

[0011] In another embodiment of any of the above embodiments, determining the projected trailer path using the determined trailer end position includes using least squares fitting to calculate a 3D trajectory to calculate the trailer trajectory in 3D space and transforming the 3D trajectory.

[0012] In another embodiment of any of the above embodiments, generating an overlay depicting the projected trailer path includes transforming the 3D trajectory into a 2D image.

[0013] In another embodiment of any of the above embodiments, determining the trailer angle relative to the tractor based on images provided by at least one side camera includes determining the trailer angle without using a dedicated angle detection sensor.

[0014] In another embodiment of any of the above embodiments, the memory further stores instructions that are configured to cause the processor to identify at least one object within a rear view image that includes a rear side view and a rear view, and that are configured to change the overlay in response to at least one object intersecting the overlay in the rear view image.

[0015] In a further embodiment of any of the above embodiments, the overlay is changed by changing the color of the overlay. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure can be further understood when considered in conjunction with the following detailed description, in which:

[0017] Figure 1A is a schematic front view of a commercial truck having a camera monitoring system (CMS) for providing at least Class II and Class IV views.

[0018] Figure 1B is a schematic top view of a commercial truck having a camera mirror system providing Class II, Class IV, Class V, Class VI, and Class VIII views.

[0019] Figure 2 is a schematic view of the interior of a vehicle cab.

[0020] Figure 3 Schematically shows a rear view alternative display scenario including a projected trailer trajectory.

[0021] Figure 4 shows a method for creating a rear - view trajectory overlap for an alternative display scene of a Figure 3 rear - view.

[0022] Figure 5 shows a method for generating an alert based on a projected trajectory.

[0023] Any of the foregoing paragraphs, claims, or examples, embodiments, and alternatives of the following description and drawings, including any of their aspects or respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment apply to all embodiments unless those features are incompatible. Detailed Description

[0024] A schematic diagram of a commercial vehicle 10 is shown in Figure 1A and Figure 1B is shown. Figure 2 is a schematic top - perspective view of a vehicle 10 cabin including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or the trailer 14 can be of any configuration. Although a commercial truck is envisioned in the present disclosure, the present invention can also be applied to other types of vehicles. The vehicle 10 includes a camera monitoring system (CMS) 15 ( Figure 2 ), which has a driver - side camera arm 16a and a passenger - side camera arm 16b (collectively referred to as "16") mounted to the outside of the vehicle cab 12. If desired, the camera arms 16a, 16b can also include conventional side mirrors integrated therewith, but the CMS 15 can be used to fully replace the side mirrors. In additional examples, each side can include multiple camera arms, each arm accommodating one or more cameras and / or side mirrors.

[0025] Each of the camera arms 16a, 16b includes a base fixed to, for example, the cab 12. A pivot arm is supported by the base and can be hinged relative to the base. At least one rear - facing camera 20a, 20b (collectively referred to as "20") is respectively disposed within the camera arm. The external cameras 20a, 20b respectively provide external fields of view FOV EX1 , FOV EX2 , each field of view including at least one of a Class II view and a Class IV view ( Figure 1B), which are legally required views in the commercial trucking industry. If desired, multiple cameras can also be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in European R46 legislation, and similar driving visibility requirements for commercial trucks exist in the United States and other countries. Any reference to "Class" views is not intended to be restrictive but rather to serve as an example of the type of views provided by a particular camera to the display. Each arm 16a, 16b can also provide a housing that encloses the electronics configured to provide CMS 15.

[0026] The first video display 18a and the second video display 18b (collectively referred to as "18") are disposed on each of the driver side and the passenger side within the vehicle cab 12 on or near the A-pillars 19a, 19b to display Class II and Class IV views on their respective sides of the vehicle 10, which provide rearward-facing side views of the vehicle 10 captured by the external cameras 20a, 20b.

[0027] If video of Class V and / or Class VI views is also desired, the camera housing 16c and the camera 20c can be disposed at or near the front of the vehicle 10 to provide those views ( Figure 1B ). A third display 18c disposed within the cab 12 near the center of the top of the windshield can be used to display Class V and Class VI views toward the front of the vehicle 10 to the driver. The displays 18a, 18b, 18c face the driver area 24 within the cab 22, where the operator sits in the driver's seat 26. The position, size, and field of the views streamed to any particular display can be different from the configuration described in this disclosure and still incorporate the disclosed invention.

[0028] If video of Class VIII views is desired, camera housings can be provided on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII regions of the vehicle 10. As shown, the Class VIII views include views directly surrounding the trailer and are near the rear of the vehicle including the rear of the trailer. In one example, a view of the vehicle's rear proximity is generated by a rear-facing camera disposed at the rear of the vehicle and can include an immediate rear proximity and a traditional rear view (e.g., a view extending rearward to the horizon, such as a view that could be generated by a rearview mirror in a vehicle without a trailer). In such an example, the third display 18c can include one or more frames that display the Class VIII views. Alternatively, additional displays can be added near the first display 18a, the second display 18b, and the third display 18c, and a display dedicated to providing the Class VIII views can be provided.

[0029] In some cases, a trailer-mounted camera 30 is used to generate a Class VIII view. The camera 30 mounted on the trailer is a rear-facing camera that provides a field of view 32 that encompasses a portion of the trailer, a rearward Class VIII view, and a conventional rearview mirror. This rearview mirror portion can be recognized by the CMS15 and provided to one and / or the other of the displays 18a, 18b in the passenger compartment 22 and / or the other display 18c as an alternative to or in addition to the rearview mirror. This view is particularly beneficial because the trailer 14 may block some or all of the views provided by the conventional rearview mirror.

[0030] The CMS15 is also configured to utilize images from the cameras 20a, 20b, 30 and images from other cameras that may be disposed around the vehicle to determine vehicle characteristics, identify objects, and implement driver assistance features such as display overlays and semi-automatic driver assistance systems.

[0031] These features and functions of the CMS15 are used to implement multiple CMS15 systems that assist in vehicle operation. It should be noted that the controller 28 for the CMS15 ( Figure 2 ) can be used to implement the various functions disclosed in this application. The controller 28 that communicates with the displays 18 and the cameras 20 can include one or more discrete units. For example, a centralized architecture can have a common controller disposed in the vehicle 10, while a decentralized architecture can use, for example, controllers disposed in each of the displays 18. Additionally, a portion of the controller 28 can be disposed in the vehicle 10, while another portion of the controller 28 can be located elsewhere, such as the camera arm 16. In another example, a master-slave display configuration can be used, where one display includes the controller 28 and the other display receives commands from the controller 28.

[0032] In terms of hardware architecture, such a controller can include a processor, a memory (e.g., a memory), and one or more input and / or output (I / O) device interfaces communicatively coupled via a local interface. The local interface can include, for example, but not limited to, one or more buses and / or other wired or wireless connections. The local interface can have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, which are omitted for simplicity to enable communication. Additionally, the local interface can include address, control, and / or data connections to enable proper communication between the above components.

[0033] The controller 28 can be a hardware device for executing software, particularly software stored in a memory (e.g., a memory). The controller 28 can be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device commonly used to execute software instructions.

[0034] The memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM), such as DRAM, SRAM, SDRAM, VRAM, etc.) and / or non-volatile memory elements (e.g., ROM, hard disk drive, magnetic tape, CD-ROM, etc.). In addition, the memory may comprise electronic, magnetic, optical, and / or other types of storage media. The memory may also have a distributed architecture where various components are remote from each other but accessible by a processor.

[0035] The software in the memory may include one or more separate programs, each program including an ordered list of executable instructions for implementing a logical function. System components embodied as software may also be interpreted as a source program, executable program (object code), script, or any other entity that includes a set of instructions to be executed. When constructed as a source program, the program is compiled via a compiler, assembler, interpreter, etc., which may or may not be included in the memory.

[0036] The disclosed input and output devices that may be coupled to the system I / O interface may include input devices such as, but not limited to, a keyboard, mouse, scanner, microphone, camera, mobile device, proximity device, etc. In addition, output devices such as, but not limited to, a printer, display, etc. Finally, the input and output devices may also include devices that communicate as both input and output, such as, but not limited to, a modem (modulator / demodulator; for accessing another device, system, or network), radio frequency (RF) or other transceiver, telephone interface, bridge, router, etc.

[0037] When the controller 28 is in operation, the processor may be configured to execute software stored in the memory, transfer data to and from the memory, and generally control the operation of the computing device according to the software. The software in the memory is read, possibly buffered within the processor, and then executed, in whole or in part, by the processor.

[0038] In various examples, the controller 28 includes one or more modules having algorithms, equations, and / or decision managers that receive inputs from sensors and / or store values. During vehicle operation, the controller 28 may use outputs (e.g., the display 18, speaker, etc.) to communicate information to a driver, fleet operator, or other person.

[0039] One such CMS system is a reverse assist system that generates a trailer trajectory projection for a reverse maneuver of the vehicle 10. In Figure 3The rear view alternative scenario 100 shows an example output of the reverse assist system. Although for ease of description, the shown alternative scenario 100 includes a single person 120 and a single tree 130, it should be understood that in actual examples, the alternative scenario 100 may include more objects, more diverse objects, roads, multiple categories of objects, etc. In the shown example, scenario 100 includes at least a portion of the rear end of the trailer 14. Scenario 100 is displayed on one or more of the monitors 18a, 18b, 18c and / or another monitor within the vehicle.

[0040] During a reverse maneuver, the CMS 15 uses the reverse assist system to determine the projected rearward trajectory (i.e., the expected path of the rear end of the trailer 14) and provides the projected trajectory as an overlay 110 on top of scenario 100. The overlay 110 extends from the rear end of the trailer 14 into scenario 100 and tracks the expected position of the rear end of the trailer 14 over time and / or distance. When the predicted trajectory intersects an object (e.g., person 120), the CMS 15 may generate an alert indicating that a potential collision may occur. The alert takes the form of an audio output to the operator, a shaded identifier 122 in the overlay 110, a color change, or any combination thereof. In other examples, any other method that directs the operator's attention to the object 120 may be utilized.

[0041] Continuing to refer to Figure 3 scenario 100 of Figure 4 FIG. 3 schematically shows a process 300 for generating the overlay 110. Initially, the CMS 15 receives images from the rear camera 30, the class II / IV camera, and other cameras in the CMS 15. Then, in the "Determine trailer end position and angle" step 310, the CMS 15 uses image analysis techniques to determine the end position of the trailer 14 in three-dimensional (real-world) space and the trailer angle relative to the towing vehicle 12. In one example, only image analysis is used to determine the trailer angle and end position without using an angle sensor or other sensors outside of the image sensors (cameras) of the CMS 15. Additionally, during this step, the CMS 15 receives a plurality of parameters from the vehicle controller, including truck speed, yaw rate, steering angle, gear position, and other camera external parameters.

[0042] When the vehicle 10 is operating, the trailer end position and angle are calculated multiple times from the images, and the rates of change of the trailer position and the trailer angle are determined in the "Estimate Trailer Angle Rate of Change" step 320. The rate of change can be with respect to time, with respect to distance, or a combination of both. In one example, the rate of change is determined by applying a Kalman filter to the determined trailer end position and trailer angle and additional parameters received from the vehicle controller, where the output of the Kalman filter is the rate of change. The rate of change tracks the change in the position of the trailer end in 3D space and is re-determined at each iteration of process 300. In one example, the trailer angular rate and the truck speed are converted into the motion of the trailer end in two perpendicular (x and y) directions. An integration formula calculates the change in the position of the trailer end over a period of time (e.g., 1 second, 2 seconds, etc.). By predicting the trailer end position over the calculated time period, a trajectory is obtained by connecting the points.

[0043] Once the rate of change of the trailer end is determined, the CMS 15 calculates the estimated position of the trailer end in three-dimensional space at a given time and / or distance interval in the "Calculate Trailer End Position" step 330. Process 300 loops through step 330 multiple times, where each loop determines the estimated end position at a different time and / or distance interval. In some examples, the time and / or distance interval is a fixed interval stored in the memory of the CMS 15. In alternative examples, the time and / or instantaneous interval can depend on speed, yaw rate, or any other parameter.

[0044] After determining the trailer end position at each interval, process 300 combines the trailer end positions to create a projected trajectory of the trailer end in the "Determine Trailer Trajectory in 3D Space" step 340. The trailer trajectory is the route that the trailer end is expected to travel through in three-dimensional space as the trailer end travels from each determined interval to the next.

[0045] In one example, a least squares filter is used on the trailer end points at each interval to determine the complete trajectory connecting the trailer end positions at each determined interval, and the resulting curve is the predicted trajectory.

[0046] After determining the 3D trajectory of the trailer end, in the "Convert 3D Trajectory to 2D Overlay" step 350, the 3D trajectory is converted into a two-dimensional graphic overlay. This conversion transforms the three-dimensional trailer end route into a two-dimensional trajectory through scene 100 and creates a transparent overlay 110 of the trajectory.

[0047] Once the transparent overlay 110 is created, the overlay 110 is applied to the image and displayed to the operator in the "Apply 2D Overlay to Rear View Display" step 360.

[0048] In some examples, after determining the trajectory and before applying the overlay to scene 100, the CMS 15 identifies any objects 120, 130 in scene 100 that will intersect the trajectory and outputs a warning to the vehicle operator. The warning can take the form of an audio output, a visual indicator (such as in example scene 110), a color change, or any similar alert. Figure 5 A method 400 for implementing such an alert is shown.

[0049] Initially, the CMS 15 uses image-based object recognition techniques to identify objects 120, 130 within scene 100 and identifies the two-dimensional positions of the objects in scene 100 in the "Identify Objects in View" step 410. The two-dimensional positions of the objects 120, 130 within scene 110 are then transformed into the three-dimensional positions of the objects 120, 130 in real space. After determining the three-dimensional trajectory of the end of the trailer 14, the CMS 15 compares the three-dimensional position of each object with the trajectory in the "Compare Object Positions with Trajectory" step 420 and indicates an alert in the "Generate Display Alert" step 430 when the end of the trailer 14 passes through the same three-dimensional space as the objects 120, 130.

[0050] In a more complex system, a similar trajectory estimation process can be used to estimate the trajectory of a moving object (e.g., person 120) and the projected trajectory of the moving object is compared with the projected trajectory of the end of the trailer 14. In such an example, an alert is generated when the trajectory of the object intersects the trajectory of the trailer 14 either simultaneously or within a predefined time span (e.g., + / - 10 seconds).

[0051] Although example embodiments have been disclosed, those of ordinary skill in the art will recognize that certain modifications will fall within the scope of the claims. For this reason, the following claims should be studied to determine their true scope and content.

Claims

1. A camera monitoring system (CMS) for a vehicle, comprising: A CMS controller, the CMS controller including a memory and a processor; The CMS controller is connected to a plurality of cameras disposed around the vehicle and is configured to receive video feeds from each of the plurality of cameras, the CMS controller including at least one side camera configured to define a rear side view and at least one rear camera configured to generate a rearward view; And The memory stores instructions for: causing the processor to determine a trailer angle of a trailer relative to a towing vehicle based on an image provided by the at least one side camera; causing the processor to estimate a trailer angular rate; Causing the processor to determine trailer end positions in a plurality of instances at least partially based on vehicle speed, the estimated trailer angular rate, and the determined trailer angle; Using the determined trailer end positions to determine a projected trailer path, and causing the processor to generate an overlay depicting the projected trailer path and apply the overlay to a rear view display.

2. The CMS according to claim 1, wherein, Determining the trailer end positions in the plurality of instances includes: determining the trailer end positions at a plurality of time intervals and determining the trailer end positions at a plurality of distance intervals.

3. The CMS according to claim 1, wherein, The rearward view includes at least one of a Class VIII view and a rearview mirror replacement view.

4. The CMS according to claim 3, wherein, The rearward view includes at least a portion of the trailer.

5. The CMS according to claim 1, wherein The processor is configured to use a Kalman filter to estimate the trailer angular rate.

6. The CMS according to claim 1, using the determined trailer end position to determine the projected trailer path includes: Using least squares fitting to calculate a 3D trajectory to calculate a trailer trajectory in 3D space and transform the 3D trajectory.

7. The CMS according to claim 6, wherein, Generating an overlay depicting the projected trailer path includes: transforming the 3D trajectory into a 2D image.

8. The CMS according to claim 1, wherein, Determining a trailer angle of a trailer relative to a towing vehicle based on an image provided by the at least one side camera includes: determining the trailer angle without using a dedicated angle detection sensor.

9. The CMS according to claim 1, wherein, The memory further stores instructions configured to cause the processor to identify at least one object within a rear view image including the rear side view and the rearward view, and configured to change the overlay in response to the at least one object intersecting the overlay in the rear view image.

10. The CMS according to claim 9, wherein the overlay is changed by changing the color of the overlay.

Images