Camera monitor system with curve-cutting warning function

Abstract

A vehicle camera monitor system (CMS) includes a camera configured to provide a captured image of a field of view, a display in communication with the camera and configured to portray a displayed image including at least a portion of the captured image, and a controller in communication with the camera and the display. The controller includes a lane detection module configured to determine lane boundaries for the vehicle, a trailer detection module configured to determine trailer boundaries, and a collision warning module configured to determine an impending crossing between the trailer boundary and the lane boundary and provide a warning in response.

Description

[Technical Field]

[0001] The present disclosure relates to a camera monitor system (CMS) for use on a tractor towing a trailer, and more particularly to a system for increasing driver awareness during turning maneuvers.

[0002] (Priority Claim) This application claims priority to U.S. Patent Application No. 18 / 116,627, filed March 2, 2023, entitled "TRAILER STRIKING PREDICTION USING CAMERA MONITORING SYSTEM." [Background technology]

[0003] Mirror replacement systems, and camera systems that complement mirror views, are utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. Camera monitor systems (CMS) utilize one or more cameras positioned around the vehicle to provide the vehicle operator with an expanded field of view on one or more displays located within the vehicle cab. In some instances, the mirror replacement system within a CMS can cover a wider field of view than a traditional mirror or can include views not fully obtainable through a traditional mirror.

[0004] Forward turning maneuvers of commercial tractor-trailer configurations require a wider turn than other vehicles to prevent the side of the trailer from inadvertently striking an object inside the turning arc. Even when the inside portion of the turn is visible through mirrors and / or camera monitoring systems, it can be difficult for an inexperienced operator to grasp the movement of the side of the trailer using only a conventional view.

[0005] Typically, vehicle operators compensate for the difficulty by using unnecessarily wide turns to ensure that the trailer does not hit objects on the inside of the turn. Summary of the Invention

[0006] In one exemplary embodiment, a vehicle camera monitor system (CMS) includes a camera configured to provide a captured image of a field of view, a display in communication with the camera and configured to portray a displayed image including at least a portion of the captured image, and a controller in communication with the camera and the display, the controller including a lane detection module configured to determine lane boundaries for the vehicle, a trailer detection module configured to determine trailer boundaries, and a collision warning module configured to determine an impending crossing between the trailer boundary and the lane boundary and to provide a warning in response.

[0007] In a further embodiment of any of the above, the alert is provided by an overlay depicted on the display.

[0008] In a further embodiment of any of the above, the overlay is configured to increase in intensity based on severity level.

[0009] In a further embodiment of any of the above, the overlay is an indicia and the severity level defines a blink rate of the indicia.

[0010] In a further embodiment of any of the above, the overlay is an indicia and the severity level defines a color of the indicia.

[0011] In a further embodiment of any of the above, the severity level is based on at least one of a time to the impending intersection and a time to a collision of the vehicle with a vehicle in an adjacent lane.

[0012] In a further embodiment of any of the above, the camera is a rear-facing camera mounted on a tractor, and the field of view captures at least a portion of a trailer.

[0013] In a further embodiment of any of the above, the lane detection module includes an algorithm configured to identify lane markings on a road, and the captured image includes the lane markings.

[0014] In a further embodiment of any of the above, the algorithm is configured to determine the lane markings by filtering the color of the lane markings from surrounding portions of the captured image.

[0015] In a further embodiment of any of the above, the controller includes a kinematics module including a bicycle model indicative of a trailer predicted path, and the trailer detection module determines the trailer boundary based on the trailer predicted path.

[0016] In a further embodiment of any of the above, the bicycle model is a second bicycle model, and the controller includes a first bicycle model that indicates a predicted tractor path, the second bicycle model being connected to the first bicycle model by a hitch point, and the first bicycle model including Ackermann steering.

[0017] In a further embodiment of any of the above, the controller is configured to receive a steering angle and a vehicle speed, and the trailer predicted path is based on the steering angle and the vehicle speed.

[0018] In a further embodiment of any of the above, the trailer detection module includes an object detection algorithm based on trailers in the captured image, and the trailer boundary is based on trailer features detected by the object detection algorithm.

[0019] In a further embodiment of any of the above, the trailer feature is at least one trailer wheel.

[0020] In a further embodiment of any of the above, the trailer feature is at least one trailer edge line.

[0021] In another exemplary embodiment, a method for monitoring a trailer of a vehicle includes capturing an image of a field of view including at least a portion of the trailer, displaying at least a portion of the captured image, detecting lane boundaries for the vehicle, determining a trailer boundary for the trailer, and outputting an alert regarding an impending crossing between the trailer boundary and the lane boundary.

[0022] In a further embodiment of any of the above, the warning is provided by an overlay depicted on the display, the overlay configured to increase in intensity based on a severity level, the severity level based on at least one of a time until the impending intersection and a time until a collision of the vehicle with a vehicle in an adjacent lane.

[0023] In a further embodiment of any of the above, detecting lane boundaries includes identifying lane markings on a road, the captured image including the lane markings, and identifying is performed by at least one of filtering the color of the lane markings and performing deep learning from surrounding portions of the captured image.

[0024] In a further embodiment of any of the above, detecting the trailer boundary includes using a kinematic model with a first bicycle model that represents a tractor predicted path, a second bicycle model connected to the first bicycle model by a hitch point, the first bicycle model including Ackermann steering, and the second bicycle model representing the trailer predicted path, the kinematic model configured to receive a steering angle and a vehicle speed, and the trailer predicted path is based on the steering angle and the vehicle speed.

[0025] In a further embodiment of any of the above, the step of detecting a trailer boundary includes detecting trailer features including at least one of at least one trailer wheel and at least one trailer edge line. [Brief explanation of the drawings]

[0026] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0027] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitor system (CMS) used to provide at least Class II and Class IV views.

[0028] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera mirror system providing Class II, Class IV, Class V, Class VI, and Class VIII views.

[0029] [Figure 2] FIG. 1 is a schematic diagram of the interior of a vehicle cab and CMS system.

[0030] [Figure 3] The kinematic models of the tractor and trailer are shown.

[0031] [Figure 4] A bird's-eye view of the collision area of ​​the tractor and trailer based on the kinematic model of Figure 3 is shown.

[0032] [Figure 5] 5A-5C show schematic diagrams of a commercial truck turning operation at the beginning (FIG. 3A), middle (FIG. 3B), and end (FIG. 3C) of a forward turning operation.

[0033] [Figure 6] It shows how CMS can be used to provide collision warning for trailer collision areas.

[0034] [Figure 7] 1 shows a schematic diagram of a predicted collision area of ​​a trailer.

[0035] [Figure 8] We present a detailed method for generating the collision area for the trailer in Figure 6 and using it to generate the overlay.

[0036] [Figure 9] 1 is a flowchart of various functions performed in a CMS.

[0037] [Figure 10] A bird's-eye view of a vehicle traveling along a curved highway in traffic.

[0038] [Figure 11] 11A-11B are the driver's and passenger's respective views provided on the CMS display for the vehicle position shown in FIG.

[0039] [Figure 12] 12A-12B are the driver's and passenger's respective views provided on the CMS display for the vehicle position shown in FIG. 3B.

[0040] [Figure 13] 13A-13C are alternative overlays to the overlay shown in FIG. 12B.

[0041] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0042] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. FIG. 2 is a schematic top perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While commercial trucks are contemplated in this disclosure, the present invention is applicable to other types of vehicles. The vehicle 10 incorporates a camera monitor system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b (generally "camera arms 16" or "wings") mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, each housing one or more cameras and / or mirrors.

[0043] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. Fixed wings may also be used. At least one rear-facing camera 20a, 20b is disposed within each camera arm. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1B), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2The camera arms 16a, 16b each have an image capture unit that captures at least a portion of the trailer 14 within their field of view, such as the side and / or end of the trailer, throughout vehicle operation. If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an illustration of the type of view provided on the display by a particular camera. Each arm 16a, 16b may also provide a housing enclosing electronics configured to provide various features of the CMS 15.

[0044] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II (narrow angle view) and Class IV (wide angle view) views on each side of the vehicle 10 (e.g., Class II depicted above Class IV in a portrait configuration), which provide rear-facing views along the vehicle 10 (e.g., a portion of a trailer) captured by exterior cameras 20a, 20b.

[0045] If video of Class V and / or Class VI views is also required, camera housing 16c and camera 20c may be positioned at or near the front of vehicle 10 to provide these views (FIG. 1B). A third display 18c located within cab 12 near the top center of the windshield can be used to display Class V and Class VI views forward of vehicle 10 to the driver. Displays 18a, 18b, 18c face a driver area 24 within cab 22, where the driver is seated in driver's seat 26. The position, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0046] If a Class VIII view is required, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the vehicle's Class VIII zone. As shown, the Class VIII view includes a view that surrounds the immediate vicinity of the trailer and a rearward close-up view of the vehicle that includes the area behind the trailer. In one example, the rearward close-up view of the vehicle is generated by a rear-facing camera positioned at the rear of the vehicle and may include both an immediate rearward close-up view and a traditional rearward view (e.g., a field of view extending rearward to the horizon provided by a rearview mirror on a vehicle without a trailer). In such an example, the third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near the first, second, and third displays 18a, 18b, and 18c (generally, "displays 18") to provide dedicated displays that provide the Class VIII view.

[0047] In some cases, the Class VIII view is generated using trailer-mounted camera 30. Trailer-mounted camera 20d is a rear-facing camera that provides a field of view behind the trailer. This rear view may be provided on one of displays 18a, 18b and / or another display 18c in vehicle cab 22 as an alternative to or as a supplement to a rearview mirror. This view is particularly beneficial because trailer 14 may obstruct some or all of the view provided by a conventional rearview mirror.

[0048] CMS 15 is also configured to utilize imagery from cameras 20a, 20b, 20c, and 20d (collectively "cameras 20"), as well as imagery from other cameras that may be positioned around the vehicle or in communication with the vehicle, to determine vehicle characteristics, identify objects, and facilitate driver assistance features such as display overlays and semi-autonomous driver assistance systems.

[0049] These features and functions of the CMS 15 may be used to implement multiple CMS 15 systems to assist in the operation of a vehicle. Note that a controller 30 (FIG. 2) for the CMS 15 may be used to implement the various functions disclosed herein. The controller 30, which communicates with the displays 18 and the cameras 20, may include one or more separate units. For example, a centralized architecture may have a common controller located on the vehicle 10, while a distributed architecture may use a controller located on each of the displays 18, for example. Furthermore, part of the controller 30 may be located on the vehicle 10, while another part of the controller 30 may be located elsewhere, such as on the camera arm 16. In another example, a master-slave display configuration may be used, where one display includes the controller 30 and the other display receives commands from the controller 30.

[0050] In terms of hardware architecture, such a controller may include a processor, memory (e.g., memory 31, FIG. 2), and one or more input and / or output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but is not limited to, one or more buses and / or other wired or wireless connections. The local interface may also include additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers that enable communication, which are omitted for simplicity. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0051] Controller 30 may be a hardware device for executing software, particularly software stored in a memory (e.g., memory 31, FIG. 2). Controller 30 may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device for general-purposely executing software instructions.

[0052] The memory 31 may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). Additionally, the memory 31 may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory 31 may have a distributed architecture in which various components are located remotely from one another but are accessible by the processor.

[0053] The software in memory 31 may include one or more separate programs, each containing an ordered list of executable instructions for implementing a logical function. A system component embodied as software may be constructed as a source program, an executable program (object code), a script, or any other entity containing a set of instructions to be executed. If constructed as a source program, the program is translated via a compiler, assembler, interpreter, etc., which may or may not be contained within memory 31.

[0054] Input / output devices of the present disclosure that may be coupled to the system I / O interface(s) may include, but are not limited to, input devices such as a keyboard, mouse, scanner, microphone, camera, mobile device, proximity device, etc. They may also include, but are not limited to, output devices such as a printer, display, etc. Finally, input / output devices may further include devices that communicate as both input and output, such as, but are not limited to, a modulator / demodulator (i.e., for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc.

[0055] When controller 30 is in operation, the processor may be configured to execute software stored in memory 31, to communicate data to and from memory 31, and to generally control the operation of the computing device in accordance with the software. The software in memory 31 is read, in whole or in part, by the processor and often buffered within the processor before being executed.

[0056] In various examples, the controller 30 includes one or more modules having algorithm(s), equation(s), and / or decision manager(s) that receive input(s) from sensors and / or stored values. Exemplary modules include a lane detection module 100, an object detection module 101, a trailer edge detection module 102, a kinematics module 104, a trailer crash area prediction module 106, a tractor crash area prediction module 108, and a collision warning module 110. Example inputs include a steering angle sensor 32, a vehicle speed sensor 34, and other sensor data. Vehicle configuration information 36 relates to vehicle characteristics (e.g., trailer length, axle position, trailer type / wheelbase, tractor configuration / wheelbase, hitch point location, etc.) and may be provided by the manufacturer, operator, and / or determined by one or more modules. During vehicle operation, the controller 30 may communicate information to the driver, fleet operator, or others using an output 39 (e.g., a display 18, a speaker, etc.).

[0057] The object detection module 101 includes one or more image processing algorithms configured to identify objects in captured images. The algorithms may be used to identify VRUs (e.g., pedestrians or bicyclists), attributes of the tractor 12 and / or trailer 14, other vehicles, signs, curbs, trees, buildings, and / or other inanimate objects.

[0058] The lane detection module 100 also uses image processing of the captured image to identify markings on the road, such as lane markers that visually separate adjacent lanes. An example algorithm is described in U.S. Publication No. 2023 / 117,719, entitled "CAMERA MIRROR SYSTEM DISPLAY FOR COMMERCIAL VEHICLES INCLUDING SYSTEM FOR IDENTIFYING ROAD MARKINGS," which is incorporated by reference in its entirety. This publication describes a lane detection module in which an object detection algorithm identifies lane markings on a road by filtering the lane marking colors from surrounding portions of the captured image. Other techniques based on deep learning or other computer vision methods may also be used, if desired.

[0059] The trailer end detection module 102 is another image processing module that extracts one or more trailer features from the captured image to determine the location of the trailer end in 3D space. These extracted attributes can be used to detect objects such as tractor wheels, trailer ends, and other features. An exemplary wheel detection algorithm technique is disclosed in U.S. Publication No. US 2023 / 202,394, entitled "CAMERA MONITOR SYSTEM FOR COMMERCIAL VEHICLES INCLUDING WHEEL POSITION ESTIMATION," which is incorporated herein by reference in its entirety. An example of a trailer end detection algorithm technique is disclosed in U.S. Publication No. US 2023 / 125,045, entitled "TRAILER END TRACKING IN CAMERA MONITORING SYSTEM," which is incorporated herein by reference in its entirety. Other techniques can also be used if desired.

[0060] Many of the described functions utilize a kinematics model (provided by the kinematics module 104) to determine where one or more features on the tractor 12 and / or trailer 14 are currently located or predicted to be located. The kinematics model, shown schematically in FIG. 3, models the tractor turning radius R1 and the trailer turning radius R2, as well as the separate and distinct radii. The tractor 12 has front wheels 42 and rear wheels 43. The kinematics model models the front wheels 42 to account for their Ackermann steering characteristics. This is a common steering geometry approach that allows the outer and inner wheels to travel in different radial paths to reduce tire wear. The trailer 14, which connects to the tractor 12 at a hitch point (i.e., a fifth wheel), has rear wheels 44. V T is the tractor speed (corresponding to the vehicle speed N of 34 in Figure 2), and V nis the trailer velocity in its longitudinal direction. Vy is the trailer end velocity in the same direction as the tractor's direction of travel, and Vx is the trailer end velocity component in the direction intersecting Vy. The trailer angle θ, which is the angle between the trailer 14 and the tractor 12, can be determined by a variety of suitable approaches. The trailer angle is calculated using a kinematic model during forward driving and estimated using image processing methods during reverse driving. Another method may use a LiDAR point cloud to calculate the trailer angle and dimensions. Yet another method may be a relative / absolute angular position sensor mounted at the hitch location.

[0061] The kinematic model of the vehicle 10 is simplified by using a two-bicycle or half-track model. That is, not all wheels need to be represented in the model. In other words, the bicycle model used as the kinematic model in the kinematics module 104 is a simplified representation of a four-wheeled vehicle. Only the inside wheel of a turning motion can be modeled because it is the side of the vehicle 10 that is most at risk of collision. This is used to predict the attitude of the vehicle 10 using the instantaneous positions, angles, velocities, and accelerations acting on and within the system. Furthermore, in this kinematic model, all slip angles are assumed to be zero. As a result, the velocity and future displacement of vehicle components (e.g., wheel positions and / or trailer ends) can be propagated very quickly through mathematical algorithms. Furthermore, only trailer angle, vehicle speed, and steering angle are required as inputs, providing a simple, accurate, and fast approach to path prediction.

[0062] The disclosed kinematic model is provided by a first bicycle model with Ackermann steering that represents the predicted tractor path (112 in FIG. 4). A second bicycle model is connected to the first bicycle model by hitch point 48, and the second bicycle model represents the predicted trailer path (114 in FIG. 4). The paths of the inside tractor and trailer wheels are indicated at 112' and 114' at the inner boundaries of the respective tractor and trailer paths 112, 114, respectively.

[0063] The kinematics module 104 receives the current trailer angle, steering angle, and vehicle speed to calculate a predicted tractor and trailer path 112, 114. If desired, one or both of the predicted tractor and trailer paths 112, 114 can be shown as an overlay on one or more displays 18 (e.g., in at least one of a Class II view and a Class IV view) to assist the driver in maneuvering the vehicle 10 (e.g., in a bird's-eye view such as in FIGS. 12A-13C and / or FIG. 4). In one example, the kinematics algorithm runs (i.e., calculates) continuously during driving. In one example, a collision area is indicated when the trailer angle is greater than a certain threshold (e.g., 5° or 10°). This threshold may be adjusted by the driver as needed.

[0064] In one example of operation, CMS 15 utilizes kinematics module 104 to predict the impact zone of trailer 14 during a turning maneuver and generate a two-dimensional overlay that is digitally superimposed on at least one of the displayed Class II / IV images, thereby showing the vehicle operator the expected impact zone of trailer 14 and allowing the vehicle operator to adjust vehicle operation accordingly. CMS 15 uses captured images received from cameras 20 a, 20 b, as well as other camera and vehicle operation data received from a common vehicle controller via a data connection, such as a CAN bus or LIN bus, to estimate the expected positions of the tractor and / or trailer sides at each of multiple lateral positions and multiple time points. These positions are converted into a geometric region that encompasses all positions. In this way, the shape and size of the geometric region are not fixed but rather reflect the actual expected impact zone of the trailer.

[0065] 1A-2, FIGS. 5A, 5B, and 5C provide a scene 200 illustrating an example of a forward turning operation of a tractor 12 towing a trailer 14, with FIG. 5A showing the beginning of the turn, FIG. 5B showing the middle of the turn, and FIG. 5C showing the end of the turn. While the schematic tractor 12 and trailer 14 are schematic representations of the tractor 12 and trailer 14 shown in FIGS. 1A and 1B, the systems and processes for estimating a trailer crash area described herein can be incorporated into any similar tractor-trailer configuration including a CMS 15, and the present disclosure is not limited to the particular example environment. While the disclosed systems and methods determine and display not only the trailer crash area but also the tractor crash area, for simplicity, and because the trailer 14 is typically the portion of the vehicle 10 most at risk of colliding with an object, only the tractor crash area will be described below.

[0066] In the illustrated turning sequence, the tractor 12 pulls the trailer 14 around a right turn on a roadway 202. The turn proceeds along a turning path 210, which is directly controlled by the steering angle of the tractor 12. As the vehicle 10 (including the tractor 12 and trailer 14) proceeds along the turning path 210, the trailer 14, and in particular the inside side 14' of the trailer 14, cuts toward the inside of the turn, causing the trailer 14 to cross a portion of the roadway and adjacent ground 220 not traversed by the tractor 12. The portion of the area on the inside of the turn through which the trailer 14 passes is referred to as the "crash zone" because objects located within the crash zone will either collide with the side 14' of the trailer 14 if they are tall enough, or be susceptible to striking the tires or passing under the trailer 14 if they are not tall enough.

[0067] To avoid accidental collisions, the collision area prediction system uses vehicle data (e.g., steering angle, steering speed, trailer angle, vehicle speed, trailer wheelbase, tractor wheelbase, hitch point location, yaw rate, etc.) to generate a predicted collision area over time using process 300 shown in Figure 6. The predicted collision area is a prediction of the path the trailer will take during a turn and is continuously recalculated as the turn progresses.

[0068] The process first identifies that a turn operation is occurring in "Identify Turn Operation" step 310. A turn operation may be identified automatically by detecting a change in steering angle, a change in geospatially detected vehicle path direction, a combination of these, or a manual turn initiation input from the vehicle operator.

[0069] Once a turn is identified, the trailer crash area prediction system 40 determines a predicted crash area in "Predict Trailer Crash Area" step 320. The crash area prediction is a zone extending away from the side of the trailer 14 that the trailer 14 will pass through as the vehicle 10 completes the turn. In some examples, the prediction is based on a snapshot of the current steering angular velocity and other vehicle parameters. In other examples, the change in steering angle over time is used rather than an instantaneous steering angle. Similarly, in other examples, one or more additional vehicle parameters may be values ​​over time rather than an instantaneous snapshot. In another example, known, known, or detectable external factors (e.g., road conditions, road gradient, weather conditions, etc.) are further incorporated into the prediction process.

[0070] In addition to predicting the collision area, the process 300 identifies objects (e.g., signs 222, trees 224, curbs 226) in the received image, or another system in the CMS 15 in communication with the trailer collision area prediction system 40 provides the object identification to the trailer collision area prediction system 40. An overlay may be displayed on the display 18 over the curb for improved visibility to the driver (FIG. 12B). After receiving the object identification, the process 300 compares the location of the identified objects 222, 224, 226 to the estimated collision area in a "Compare Object Detection to Collision Area" step 330. If the objects 222, 224, 226 intersect with the estimated collision area, the process 300 outputs a warning to the vehicle operator in a "Output Warning If Detected Object is in Collision Area" step 340. The warning can be either an audio and / or visual output that alerts the vehicle operator to a potential collision. Because the process is predictive in nature, the alert allows the vehicle operator to adjust turning maneuvers to avoid the anticipated collision, and the system automatically updates the predicted collision area to compensate for the correction.

[0071] Continuing with reference to the overall process of Figure 6, Figures 7 and 8 show a detailed process for generating estimated collision area 410. As described above, prediction system 40 first receives trailer angle and trailer edge detection results in "Receive Trailer Angle and Trailer Edge Detection Results" step 510. Trailer angle and edge detection can be performed using any conventional detection, including image-based detection, sensor-based detection, and / or any other existing detection system(s).

[0072] The CMS controller then uses the trailer angle and edge information to identify multiple detection points or locations along the inside edge 14' of the trailer 14 in step 520, "Define Detection Points Along Trailer in 3D." As used herein, "3D" refers to location in three-dimensional real space relative to the trailer, and "2D" refers to location in an image frame along two-dimensional (e.g., XY) axes. Existing systems, particularly those in vehicle camera monitoring technology, can utilize any number of established processes or methodologies to convert 3D locations for a given camera view to 2D locations.

[0073] A plurality of detection points 430-438 are distributed along the side 14' of the trailer 14 within the turn. In the illustrated example, the detection points 430-438 are evenly distributed along the side 14'. In an alternative example, even distribution is not required, and the detection points 430-438 can be concentrated near the end (trailer end 430), with detection points near the tractor end spread out farther. In one example, a fixed number of detection points (e.g., five) are used, and the points are distributed across the side 14'. In another example, the number of detection points 430-438 is determined based on the length of the trailer and the desired distribution of the detection points.

[0074] After defining the initial detection points 430-438 along the side of the trailer 14, the collision area prediction system 40 applies the steering angle, trailer angle, rate of change of trailer angle, vehicle speed, yaw rate, and / or any similar parameters known by the CMS 15 to the kinematic model to predict the three-dimensional position of each detection point at a future time (t), a predetermined duration in the future (e.g., 1 second), in step 530, "Predict Future Positions of Detection Points in 3D." Each predicted point at t is stored, and step 520 is repeated (522), using the t positions of the detection points 430'-438' as the starting point and generating new predicted positions of the detection points 430"-438" at t. In the example of FIGS. 7 and 8, the prediction is repeated twice, generating two predicted future positions (t, t). It will be appreciated that in alternative examples, the number of iterations can be increased when predictions of a longer duration are required. Similarly, the number of iterations can be increased with shorter time durations between prediction points (e.g., t0 to t1, t1 to t2, etc.), resulting in a larger number of prediction points that produce the predicted collision area geometry.

[0075] After iteration 522, the process aggregates the detected points 430-438 in "Aggregate Detected Points and Convert 3D Locations to 2D Image Points" step 540 and converts the 3D locations of the predicted points to two-dimensional locations in the Class II / IV image plane to which the overlay is applied. After converting the image points to two-dimensional image points, the aggregated image points are converted to a collision region 410 in "Convert 2D Detected Points to Collision Region in Class II / IV Image" step 550 by defining a boundary in 2D space that contains all predicted locations 430-438 from t0 to tn. The boundary is defined as the minimum space required to contain all predicted locations 430-438. The collision region 410 is then aligned with the sides of the truck in the Class II / IV image and shaded as an overlay.

[0076] The trailer crash area is also useful in potential "curvecut" scenarios when the vehicle 10 is traveling on a curved roadway in traffic with other vehicles 700, 702 (FIG. 10). On a curved road, the trailer edge is more likely to cross the lane markers 706 that indicate the boundary to the adjacent lane 704, creating a potentially dangerous situation. To address this, the CMS 15 utilizes the lane detection module 100 to determine lane boundaries for the vehicle 10, the trailer edge detection module 102 to determine trailer boundaries, and the collision warning module 110 to determine impending intersections between the trailer and lane boundaries. The trailer crash area prediction module 106 can be used to predict future crash areas for the trailer and to predict impending intersections. When the crash area is used to determine trailer curve cuts, the kinematic model used may be different from the one used to calculate the trailer angle. The trailer crash kinematic model uses the trailer angle as an input, along with the other signals described above. However, if only the final wheel position / trailer end needs to be checked instantaneously against lane markings, the kinematic model used to calculate the trailer angle may be sufficient. Alerts may be based on time to intersection and / or time to collision. Alerts are output at different levels based on severity level.

[0077] Examples of driver and passenger displays 18a, 18b are shown in Figures 11A and 11B. A vehicle 10 and its trailer 14 (and trailer end) are moving toward a vehicle 702 in an inside lane 704 where a lane boundary (marker 706) is being cut into. A warning 708 is overlaid on the display 18b on the side where the cut is about to occur. The warning 708 may be accompanied by an audible warning, provided in a conspicuous color (perhaps flashing), and does not obstruct the lane 704. The warning 708 encourages the driver to pay attention to the trailer's position and momentum.

[0078] The overlay is continuously updated as the process is repeated, thereby allowing the collision area overlay to be accurate throughout the vehicle's turn. In one example, the overlay includes a first overlay on the display image corresponding to a first area encompassing a predicted trailer path for a current trailer position, the first overlay including a first boundary provided by a first curve indicating the inside trailer path and a trailer collision area from the first boundary to the trailer. In another example, the overlay includes a second overlay on the display image corresponding to a second area encompassing a predicted tractor path for a current tractor position, the first overlay including a second boundary provided by a second curve indicating the inside tractor path and a tractor collision area from the second boundary to the tractor.

[0079] Examples of overlays are shown in Figures 12A-13C. Figures 12A (outside turning radius) and 12B (inside turning radius) show the driver and passenger side displays 18a, 18b, respectively, during a turning maneuver similar to that shown in Figure 5C. The overlay representing the trailer crash area includes a first boundary 804 provided by a curved line indicating the inside trailer path. The trailer crash area 806 extends from the first boundary 804 to the trailer 14, and in particular to a second boundary 808, e.g., a straight line projected from the side of the trailer 14 onto the ground. In the example shown in Figure 12B, the trailer crash area 806 is cross-hatched to provide improved visibility of the crash area to the driver and to allow the captured image to remain visible. The overlay 802 may be provided on the curb 800 in a contrasting color to the trailer crash overlay, if desired.

[0080] Additional examples of crash area overlays are shown in FIGS. 13A-13C and may be available to the driver through a display customization menu. In the example of FIG. 13A, a first boundary 804′ is defined by a colored line, and a trailer crash area 806′ has only a captured image without an overlay. In the example of FIG. 13B, a first boundary 804″ is defined by a different colored line and a trailer crash area 806″, which is shaded with a prominent shaded area and surrounded by a first boundary 806″ and a second boundary 808″. In the example of FIG. 13C, a first boundary 804′″ is defined by a dark colored line and a trailer crash area 806′″, which is lightly shaded and surrounded by a first boundary 806′″ and a second boundary 808′″. Display schemes other than those shown may also be used.

[0081] The CMS 15 includes a decision manager or collision warning module 110, shown in FIG. 9, that communicates with modules 101-108 for assessing the proximity between a predicted tractor and / or trailer path 112, 114 (i.e., tractor and trailer collision area) and one or more objects (e.g., predicting an impending curb collision, curve cut, object collision, etc.). The decision manager considers the estimated time to event, the severity (what the object is), the closing speed between the objects, etc. The decision manager outputs a soft warning 602 or a sharp warning 604 via one or more output devices 39. In the example of a displayed warning (e.g., an overlaid sign), the severity may be conveyed to the driver based on the flash rate of the sign or a change in color of the sign. In the example of an audible warning, the sound level or pattern may be altered based on the severity.

[0082] Although described above in connection with a commercial tractor pulling a trailer, it is understood that wide turning requirements exist for any similar vehicle, and therefore the inventive features, systems, and apparatus described herein are applicable to any similar vehicle configuration and are not limited to commercial tractor-trailer configurations.

[0083] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. A camera monitor system (CMS) for a vehicle, comprising: a camera configured to provide a captured image of the field of view; a display in communication with the camera and configured to render a displayed image including at least a portion of the captured image; a controller in communication with the camera and the display; Equipped with The controller a lane detection module configured to determine lane boundaries for the vehicle; a trailer detection module configured to determine a trailer boundary; a collision warning module configured to determine an imminent crossing between the trailer boundary and the lane boundary and to provide a warning in response thereto; A camera monitor system comprising:

2. The camera monitor system of claim 1 , wherein the alert is provided by an overlay depicted on the display.

3. The camera monitor system of claim 2 , wherein the overlays are configured to increase in intensity based on severity level.

4. 4. The camera monitor system of claim 3, wherein the overlay is an indicia and the severity level defines a blink rate of the indicia.

5. 4. The camera monitor system of claim 3, wherein the overlay is an indicia and the severity level defines a color of the indicia.

6. 4. The camera monitor system of claim 3, wherein the severity level is based on at least one of a time to the impending intersection and a time to a collision of the vehicle with a vehicle in an adjacent lane.

7. 3. The camera monitor system of claim 2, wherein the camera is a rear-facing camera mounted on a tractor, and the field of view captures at least a portion of a trailer.

8. The camera monitor system of claim 1 , wherein the lane detection module includes an algorithm configured to identify lane markings in a road, and the captured image includes the lane markings.

9. 9. The camera monitor system of claim 8, wherein the algorithm is configured to determine the lane markings by filtering the color of the lane markings from surrounding portions of the captured image.

10. the controller includes a kinematics module including a bicycle model representing a trailer predicted path; The camera monitor system of claim 1 , wherein the trailer detection module determines the trailer boundary based on the predicted trailer path.

11. 11. The camera monitor system of claim 10, wherein the bicycle model is a second bicycle model, the controller includes a first bicycle model that indicates a tractor predicted path, the second bicycle model is connected to the first bicycle model by a hitch point, and the first bicycle model includes Ackermann steering.

12. 12. The camera monitor system of claim 11, wherein the controller is configured to receive a steering angle and a vehicle speed, and the predicted trailer path is based on the steering angle and the vehicle speed.

13. The camera monitor system of claim 1 , wherein the trailer detection module includes an object detection algorithm based on trailers in the captured image, and the trailer boundary is based on trailer features detected by the object detection algorithm.

14. 14. The camera monitor system of claim 13, wherein the trailer feature is at least one trailer wheel.

15. 14. The camera monitor system of claim 13, wherein the trailer feature is at least one trailer edge line.

16. 1. A method for monitoring a trailer of a vehicle, comprising: capturing an image of a field of view that includes at least a portion of the trailer; displaying at least a portion of the captured image; detecting lane boundaries for the vehicle; determining a trailer boundary for the trailer; outputting an alert regarding an impending crossing between the trailer boundary and the lane boundary; Including, a method.

17. the alert is provided by an overlay depicted on the display; the overlay is configured to increase in intensity based on severity level; 17. The method of claim 16, wherein the severity level is based on at least one of a time to the impending intersection and a time to a collision of the vehicle with a vehicle in an adjacent lane.

18. the step of detecting lane boundaries includes identifying lane markings on a road; the captured image includes the lane markings; 17. The method of claim 16, wherein identifying is performed by at least one of filtering the color of the lane markings and performing deep learning from surrounding portions of the captured image.

19. 17. The method of claim 16, wherein the step of detecting the trailer boundary includes using a kinematic model with a first bicycle model that indicates a tractor predicted path, a second bicycle model connected to the first bicycle model by a hitch point, the first bicycle model including Ackermann steering, the second bicycle model that indicates a trailer predicted path, the kinematic model configured to receive a steering angle and a vehicle speed, and the trailer predicted path is based on the steering angle and the vehicle speed.

20. The method of claim 16 , wherein the step of detecting a trailer boundary includes detecting trailer features including at least one of at least one trailer wheel and at least one trailer edge line.

Images