Trailer change detection system for commercial vehicles.

Abstract

A method for detecting a change in trailer load status includes responding to a first trigger event by storing a first snapshot of the trailer generated by at least one camera local to the vehicle including the trailer, responding to a subsequent second trigger event by storing a second snapshot of the trailer generated by at least one camera local to the vehicle including the trailer, comparing the first snapshot with the second snapshot to determine a difference between the image of the first snapshot and the image of the second snapshot, and identifying that a change in trailer load status has occurred in response to the difference exceeding a predefined magnitude.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 405,919, filed September 13, 2022.

[0002] FIELD OF THE DISCLOSURE This disclosure relates to monitoring commercial vehicles, and more particularly to monitoring commercial vehicles for changes in trailer conditions between engine cycles. [Background technology]

[0003] Camera surveillance systems (CMS) utilize one or more cameras to provide vehicle operators with an enhanced field of view. In one example, mirror replacement systems and camera systems that supplement the mirror view are utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. In some examples, camera surveillance systems cover a wider field of view than traditional mirror-based systems or include views that are not fully available via traditional mirror-based systems.

[0004] Some vehicle operations depend on or otherwise utilize the load in the trailer. As an example, a kinematic model of trailer movement may require knowledge of whether the trailer is fully loaded, partially loaded, or empty to provide a more accurate model of trailer behavior. Additionally, a particular tilt or inclination of the trailer affects trailer operation. While the load is not expected to change during vehicle operation, it is important to identify when the load has changed during operation to ensure that operations that use the load condition operate correctly. Relying on the driver or vehicle operator to identify that these changes have occurred is not always possible because the driver may be away from the vehicle during loading and unloading and / or other extended periods while the vehicle is off. Summary of the Invention

[0005] In one exemplary embodiment, a method for detecting a change in trailer load status includes responding to a first trigger event by storing a first snapshot of a trailer generated by at least one camera local to a vehicle including the trailer, responding to a subsequent second trigger event by storing a second snapshot of the trailer generated by the at least one camera local to the vehicle including the trailer, comparing the first snapshot with the second snapshot to determine a difference between an image of the first snapshot and an image of the second snapshot, and identifying a change in trailer load status as having occurred in response to the difference exceeding a predefined magnitude.

[0006] In a further embodiment of any of the above, the first trigger event is the end of an ignition cycle.

[0007] In a further embodiment of any of the above, the second trigger event is the start of an ignition cycle.

[0008] In a further embodiment of any of the above, the first trigger event is an ignition cycle end and the second trigger event is an ignition cycle start, with no intervening ignition cycle between the ignition cycle end and the subsequent ignition cycle start.

[0009] In a further embodiment of any of the above, the method further includes, before comparing the first snapshot and the second snapshot, detecting the trailer in each of the first snapshot and the second snapshot using an object detection algorithm.

[0010] In a further embodiment of any of the above, the difference comprises a change in position within the image of the detected trailer object between the first snapshot and the second snapshot.

[0011] In a further embodiment of any of the above, identifying the position changes includes detecting a set of position changes and normalizing the position changes to a single quantified value.

[0012] In a further embodiment of any of the above, the single quantified value is compared to the predefined magnitude.

[0013] In a further embodiment of any of the above, the method further comprises responding to changes in load condition by automatically detecting a level state of the trailer.

[0014] In a further embodiment of any of the above, automatically detecting the horizontal state of the trailer includes receiving a plurality of inputs at a controller including at least one of the first snapshot and the second snapshot of the trailer, and the controller outputting the horizontal state of the trailer.

[0015] In a further embodiment of any of the above, the at least one camera is the same camera of the same configuration for the first snapshot and the second snapshot.

[0016] In a further embodiment of any of the above, the at least one camera is a rear-facing camera mounted on a tractor.

[0017] In another exemplary embodiment, a system for detecting changes in trailer status includes at least one camera defining a field of view that includes a trailer, and a controller including a memory and in communication with the at least one camera and the memory. The controller is configured to store instructions that cause the CMS to respond to a first trigger event by storing a first snapshot of the trailer generated by the at least one camera and to respond to a subsequent second trigger event by storing a second snapshot of the trailer generated by the at least one camera. Using the first snapshot as the second snapshot, determine a difference between an image of the first snapshot and an image of the second snapshot, and identify a change in load status as occurring in response to the difference exceeding a predefined magnitude.

[0018] In a further embodiment of any of the above, the memory further stores instructions for causing the controller to respond to changes in load condition by automatically detecting a level state of the trailer.

[0019] In a further embodiment of any of the above, the at least one camera includes a rear-facing mirror replacement camera.

[0020] In a further embodiment of any of the above, the rear-facing mirror replacement camera is positioned in a camera arm that extends from one of a driver's side of the cab and a passenger's side of the cab.

[0021] In a further embodiment of any of the above, the first trigger event is the end of an ignition cycle.

[0022] In a further embodiment of any of the above, the second trigger event is the start of an ignition cycle.

[0023] In a further embodiment of any of the above, the first trigger event is an ignition cycle end and the second trigger event is an ignition cycle start, with no intervening ignition cycle between the ignition cycle end and the subsequent ignition cycle start. [Brief explanation of the drawings]

[0024] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0025] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views.

[0026] [Figure 1B] 1 is a schematic top view of a commercial truck equipped with a camera surveillance system providing Class II, Class IV, Class V, and Class VI views.

[0027] [Figure 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an interior camera.

[0028] [Figure 3] 1 illustrates a process for identifying trailer load changes.

[0029] [Figure 4] An example of a CMS image including a trailer after the first engine cycle is shown.

[0030] [Figure 5] 6 shows an example CMS image of the trailer of FIG. 5 at the start of the second engine cycle.

[0031] [Figure 6]4 illustrates an optional system for identifying the severity of trailer out-of-level conditions due to load changes detected by the process of FIG. 3.

[0032] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0033] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. FIG. 2 is a schematic top perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While commercial trucks are contemplated in this disclosure, the present invention is applicable to other types of vehicles. The vehicle 10 incorporates a camera monitoring system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, each arm housing one or more cameras and / or mirrors.

[0034] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within each camera arm. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOVEX2 16a, 16b. If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an illustration of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing enclosing electronics configured to provide various features of the CMS 15.

[0035] 1B , in addition to cameras 20a, 20b in camera arms 16a, 16b, CMS 15 includes at least a rear-facing camera 60 and an interior trailer camera 62. Rear-facing camera 60 is mounted to the rear of tractor 12 and captures a view of the exterior of trailer 14, while interior camera 62 is positioned within the trailer cargo area and captures a view of the interior of trailer 14, including objects loaded on trailer 14. In an alternative example, either camera 60, 62 may be a camera incorporated into a secondary system connected to CMS 15 and configured to provide a video feed to controller 30, and the following description would function similarly.

[0036] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0037] If Class V and / or Class VI view footage is also desired, a camera housing 16c and camera 20c may be positioned at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located within the cab 12 near the top center of the windshield can be used to display Class V and Class VI views forward of the vehicle 10 to the driver. Displays 18a, 18b, and 18c face a driver area 24 within the cab 22, where the driver is seated in a driver's seat 26. The location, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0038] If Class VIII view video is desired, camera housings may be positioned on the sides and rear of vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of vehicle 10. In such an example, third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near first, second, and third displays 18a, 18b, 18c to provide dedicated displays that provide the Class VIII view.

[0039] In some examples, the controller 30 is in communication with a vehicle controller and / or is connected to a vehicle data bus (e.g., a CANN bus). This communication enables the controller 30 to receive and utilize sensor and operational information from the vehicle.

[0040] 1A-2, FIG. 3 illustrates a process 300 for identifying load changes in a trailer using the camera surveillance system 15. FIG. 4 illustrates an example snapshot A of step 320 of the process 300. FIG. 5 illustrates an example snapshot B of step 340 of the process 300. Initially, the controller 30 of the CMS 15 detects a first trigger event in step 310, "identify a first trigger event." In one example, the first trigger event is the end of an ignition cycle (i.e., the vehicle operator turns off the engine). As used herein, "ignition cycle" refers to a driving cycle that begins with an engine start, satisfies engine start conditions for at least a predetermined period of time, and ends with the engine shut off. In some examples, the predetermined period of time may be at least two seconds.

[0041] After identifying the end of the ignition cycle, controller 30 stores a first snapshot (snapshot A shown in FIG. 4 ) including at least a portion of trailer 110 in memory (e.g., local memory) in a “Store Trailer Snapshot A” step 320. In some examples, controller 30 may include an object detection system to identify and flag the portion of the snapshot that corresponds to trailer 110 for later analysis. In other examples, the stored snapshot may be stored in memory (e.g., remote memory), and controller 30 is configured to access the memory (e.g., remote memory) as needed during operation of process 300. In examples including additional cameras 60, 62, the snapshot may further include exterior and interior views generated by the additional cameras 60, 62, either as separate images or as a stitched single image.

[0042] Once the engine is restarted, controller 30 identifies a second trigger event in "identify second trigger event" step 330. In one example, the second trigger event is the start of an engine cycle (i.e., the engine is started by the vehicle operator). Once a subsequent engine cycle is started, controller 30 stores a second snapshot (snapshot B shown in FIG. 5) in "store trailer snapshot B" step 340. The second trailer snapshot (B) is generated using the same camera(s) with the same configuration as snapshot (A). If there is no change in the load of trailer 110, the portions of snapshots (A) and (B) that include trailer 110 will be substantially identical.

[0043] After storing the second snapshot (B), the controller 30 compares snapshots (A) and (B) to determine how much difference there is between the image of the trailer 110 in snapshot (A) and the image of the trailer in snapshot (B). The differences between the images generally refer to differences between the trailer 110 portions of the images. These differences are generally the result of trailer state changes, such as load changes, lift gate opening, etc. In one example, the differences are determined by applying an edge detection algorithm to the snapshots to determine the pixel difference of identified edges between the first and second snapshots. The pixel difference represents how much the edge has shifted within the snapshot image.

[0044] In some cases, environmental changes (e.g., fog, rain, snow) can cause small differences, which may require the use of thresholds to filter out false positives. If the controller 30 includes software-based object recognition, the relative position of the trailer 110 itself can be compared between snapshots. For example, the distance 302 from the image edge 306 to the trailer corner 304, the length 312 of the top edge of the trailer 110 visible in the image, the angle 332 of the rear trailer edge 334 relative to the ground surface, and / or any similar measurable aspect can be compared between two snapshots (A) and (B), and the determined difference is the difference between snapshots (A) and (B). In examples involving an internal camera 62, object detection can utilize feature-based object detection and / or region-of-interest-based methods to determine the number and type of objects removed and / or added to the trailer 14, thereby determining a loss or increase in load. In other examples, entire images are compared, and any quantifiable measure of the difference between the two images can be used. In either case, process 300 compares the snapshots in a "Compare Snapshots and Determine Amount of Difference" step 350.

[0045] This difference is quantified to determine a single "difference" value with a numerical magnitude, and the magnitude of the difference value is compared to a threshold value in step 360, "Compare Difference to Threshold and Alert If Difference Exceeds Threshold." This alert is provided to the vehicle operator via any means, including an audio indicator, a visual alert, a message sent to a messaging system, or any other notification means. This notification notifies the operator that the trailer 110 is not in the same position as when the previous engine cycle ended. Upon such notification, the operator can manually check the trailer 110 to identify whether the load has changed, whether the lift gate has opened, or any similar function. The particular threshold value depends on the difference quantification used and can be set large enough so that environmental variations between engine cycles (e.g., shifts from day to night, shifts from rain to cloudy, etc.) do not generate false positives.

[0046] In some instances, such as when trailer 14 is a box trailer, the inclusion of rear-facing camera 60 can provide additional benefits. By way of example, rear-facing camera 60 has a square / rectangular field of view of trailer 14 that is particularly suited to assessing trailer level based on the edges of the trailer using edge detection.

[0047] In some examples, identifying that the trailer 110 has shifted can further be used to trigger an automatic level check. Parking and operating a semi-truck on an uneven surface / road can cause interference with normal operation. Additionally, parking when the trailer is uneven, either due to being on an uneven surface or having an uneven load being loaded / unloaded, can alter the vehicle's behavior and how the trailer responds to standard maneuvers.

[0048] The automatic level check is performed using depth perception technology that combines trailer imagery, steering angle, road imagery or wheel imagery, shift stick position, trailer load status (e.g., loaded, unloaded, partially loaded), and trailer dimensions or trailer height. The automatic level check uses depth perception combined with data from sensors and cameras received from controller 30 and the vehicle controller via a communications connection to determine first, whether the trailer is out of level, and second, how out of level the trailer is. The magnitude of the out-of-level condition is reported as the angle of the trailer bed relative to the angle of the ground.

[0049] In certain examples, one or more cameras from CMS 15 can provide additional views that can be analyzed to identify possible causes of the unlevelness. For example, a rear-facing camera that includes a wheel view can identify whether one or more wheels have gone over a curb, entered a grass area, or similar occurrence. This information is then correlated and provided to the vehicle operator, allowing the operator to distinguish between a trailer bed that is unlevel due to an uneven load and / or mechanical features of trailer 110 and a trailer bed that is unlevel due to environmental features. The operator can then use this information to adapt vehicle operation in the correct manner.

[0050] 6 schematically illustrates a control system 600 including an algorithm 610 configured to receive multiple inputs 620 from the controller 30 and / or a vehicle communication bus (e.g., a CANN bus), and an output 630 configured to output a value indicative of a level or non-level status of the trailer. In some examples, the inputs 620 may include a complete trailer image 622 from the CMS, a steering angle 624 from the vehicle controller, a road image 626 from the CMS, an image of one or more wheels in the trailer 628, a current gear 642 from the vehicle controller, a trailer load status 644 from either the CMS, the vehicle controller, or manual input from the vehicle operator, and a set of trailer dimensions 646 from the CMS or manual input from the vehicle operator.

[0051] In another example, any combination of the above values ​​can be provided to the algorithm 610 and a corresponding non-horizontal value 630 can be output from the algorithm 610.

[0052] In some examples, algorithm 610 outputs a binary value of "level" or "not level," providing a simple warning to the vehicle operator so they can correct accordingly. In other examples, output 630 can be a vector value indicating the magnitude of tilt of the trailer bed and the direction of the tilt. In the latter case, the vector value can be used by driver assistance systems, automated driving systems, and any similar systems that can automatically compensate for the tilt or provide instructions to the vehicle operator to compensate for the tilt.

[0053] As described above, the controller 30 can be used to implement the various functions disclosed herein. The controller 30 may include one or more individual units. In terms of hardware architecture, such a computing device may include a processor, memory, and one or more input / output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but is not limited to, one or more buses and / or other wired or wireless connections. The local interface may also include additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers that enable communication, which are omitted for simplicity. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0054] The controller 30 may be a hardware device for executing software, particularly software stored in a memory, and may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device for general-purposely executing software instructions.

[0055] The memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). Furthermore, the memory may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory may have a distributed architecture where various components are located remotely from each other but are accessible by the processor.

[0056] Software in memory may include one or more separate programs, each containing an ordered list of executable instructions for implementing a logical function. A system component embodied as software may be constructed as a source program, an executable program (object code), a script, or any other entity containing a set of instructions to be executed. If constructed as a source program, the program is translated via a compiler, assembler, interpreter, etc., which may or may not be contained in memory.

[0057] Input / output devices of the present disclosure that may be coupled to the system I / O interface(s) may include, but are not limited to, input devices such as a keyboard, mouse, scanner, microphone, camera, mobile device, proximity device, etc. They may also include, but are not limited to, output devices such as a printer, display, etc. Finally, input / output devices may further include devices that communicate as both input and output, such as, but are not limited to, a modulator / demodulator (i.e., for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc.

[0058] When controller 30 is in operation, the processor may be configured to execute software stored in the memory, communicate data to and from the memory, and generally control the operation of the computing device in accordance with the software. The software in the memory is read, in whole or in part, by the processor and often buffered within the processor before being executed.

[0059] Although described above with reference to a particular mirror replacement system, it should be understood that this process can be applied to any camera surveillance system that includes a camera that captures all or part of a trailer within its field of view.

[0060] Also, while particular component arrangements are disclosed in the illustrated embodiments, it should be understood that other arrangements would benefit from the present disclosure. Although a particular sequence of steps is shown, described, and claimed, it should be understood that, unless otherwise indicated, the steps may be performed in any order, separated, or combined, and still benefit from the present invention.

[0061] Although the different examples have specific components shown, embodiments of the present invention are not limited to those specific combinations, and some components or features of one example may be used in combination with features or components of another example.

[0062] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for detecting a change in trailer load status, comprising: Responding to a first trigger event by storing a first snapshot of the trailer generated by at least one camera local to a vehicle including the trailer; Responding to a subsequent second trigger event by storing a second snapshot of the trailer generated by the at least one camera local to the vehicle including the trailer; comparing the first snapshot to the second snapshot to determine differences between the image of the first snapshot and the image of the second snapshot; identifying a change in trailer load status as occurring in response to the difference exceeding a predefined magnitude; and A method comprising:

2. The method of claim 1 , wherein the first trigger event is an ignition cycle end.

3. The method of claim 1 , wherein the second trigger event is an ignition cycle start.

4. 2. The method of claim 1, wherein the first triggering event is an ignition cycle end and the second triggering event is an ignition cycle start, with no intervening ignition cycles between the ignition cycle end and a subsequent ignition cycle start.

5. detecting the trailer in each of the first snapshot and the second snapshot using an object detection algorithm before comparing the first snapshot and the second snapshot; The method of claim 1 further comprising:

6. The method of claim 5 , wherein the difference comprises a change in position within the image of a detected trailer object between the first snapshot and the second snapshot.

7. The method of claim 5 , wherein identifying the position changes comprises detecting a set of position changes and normalizing the position changes to a single quantified value.

8. The method of claim 6 , wherein the single quantified value is compared to the predefined magnitude.

9. Responding to changes in load conditions by automatically detecting the level of the trailer. The method of claim 1 further comprising:

10. 10. The method of claim 9, wherein automatically detecting the horizontal state of the trailer includes receiving, by a controller, a plurality of inputs including at least one of the first snapshot and the second snapshot of the trailer, and the controller outputting the horizontal state of the trailer.

11. The method of claim 1 , wherein the at least one camera is the same camera with the same configuration for the first snapshot and the second snapshot.

12. The method of claim 11 , wherein the at least one camera is a rear-facing camera mounted on a tractor.

13. 1. A system for detecting a change in a trailer condition, comprising: at least one camera defining a field of view that includes the trailer; a controller including a memory and in communication with the at least one camera and the memory; The controller is configured to: Responding to a first trigger event by storing a first snapshot of the trailer generated by the at least one camera; responding to a subsequent second trigger event by storing a second snapshot of the trailer generated by the at least one camera; comparing the first snapshot with the second snapshot to determine a difference between the image of the first snapshot and the image of the second snapshot; and identifying a change in load status as occurring in response to the difference exceeding a predefined magnitude. a system configured to store instructions for causing the

14. The memory responds to changes in loading conditions by automatically detecting the level state of the trailer. The system of claim 13 , further storing instructions for causing the controller to:

15. The system of claim 13 , wherein the at least one camera comprises a rear-facing mirror replacement camera.

16. 16. The system of claim 15, wherein the rear-facing mirror replacement camera is disposed in a camera arm extending from one of a driver's side of the cab and a passenger's side of the cab.

17. 14. The system of claim 13, wherein the first trigger event is an ignition cycle end.

18. The system of claim 13 , wherein the second trigger event is an ignition cycle start.

19. 14. The system of claim 13, wherein the first triggering event is an ignition cycle end and the second triggering event is an ignition cycle start, with no intervening ignition cycles between the ignition cycle end and a subsequent ignition cycle start.

Images