Trailer change detection system for commercial vehicle

Abstract

A method for detecting a change in a load state of a trailer includes: responding to a first trigger event by storing a first snapshot of the trailer generated by at least one camera local to a vehicle including the trailer; responding to a subsequent second trigger event by storing a second snapshot of the trailer generated by at least one camera local to the vehicle including the trailer; comparing the first snapshot with the second snapshot and determining a deviation between an image of the first snapshot and an image of the second snapshot; and identifying the trailer load state change as having occurred in response to the deviation exceeding a predefined magnitude.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 405,919, filed on September 13, 2022. Technical Field

[0003] The present disclosure relates to commercial vehicle monitoring, and in particular to monitoring trailer status changes of commercial vehicles between engine cycles. Background Art

[0004] A camera monitoring system (CMS) utilizes one or more cameras to provide an enhanced field of view to a vehicle operator. In one example, a mirror replacement system and a camera system to supplement the mirror view are utilized in a commercial vehicle to enhance the vehicle operator's ability to see the surrounding environment. In some examples, the camera monitoring system covers a larger field of view than a conventional mirror-based system, or includes views that are not fully available via a conventional mirror-based system.

[0005] Some vehicle operations depend on or otherwise utilize the load within the trailer. For example, a kinematic model of the trailer's motion may need to know whether the trailer is fully loaded, partially loaded, or empty in order to provide a more accurate model of the trailer's behavior. In addition, the specific roll or tilt of the trailer will affect the operation of the trailer. Although it is expected that the load will not change during operation of the vehicle, it is important to identify when the load changes between operations in order to ensure that operations using the load state operate correctly. It is not always possible to rely on the driver or vehicle operator to identify that these changes have occurred because the driver may be away from the vehicle during loading and unloading and / or during any other extended period of time when the vehicle is shut down. Summary of the invention

[0006] In one exemplary embodiment, a method for detecting a change in a load state of a trailer includes: responding to a first triggering event by storing a first snapshot of the trailer generated by at least one camera local to a vehicle including the trailer; responding to a subsequent second triggering event by storing a second snapshot of the trailer generated by at least one camera local to the vehicle including the trailer; comparing the first snapshot to the second snapshot and determining a deviation between an image of the first snapshot and an image of the second snapshot; and identifying a change in the trailer load state as having occurred in response to the deviation exceeding a predefined size.

[0007] In a further embodiment of any of the above embodiments, the first triggering event is the end of an ignition cycle.

[0008] In a further embodiment of any of the above embodiments, the second triggering event is the start of an ignition cycle.

[0009] In a further embodiment of any of the above embodiments, the first triggering event is an end of an ignition cycle, the second triggering event is a start of an ignition cycle, and there are no intervening ignition cycles between the end of an ignition cycle and the start of a subsequent ignition cycle.

[0010] In a further embodiment of any of the above embodiments, the method further includes detecting a trailer in each of the first snapshot and the second snapshot using an object detection algorithm before comparing the first snapshot to the second snapshot.

[0011] In a further embodiment of any of the above embodiments, the deviation comprises a change in a position within the image of the detected trailer object between the first snapshot and the second snapshot.

[0012] In another embodiment of any of the above embodiments, identifying the position changes includes detecting a set of position changes and normalizing the position changes into a single quantized value.

[0013] In a further embodiment of any of the above embodiments, the single quantized value is compared to a predefined size.

[0014] In another embodiment of any of the above embodiments, the method further comprises responding to a change in load status by automatically detecting a level status of the trailer.

[0015] In another embodiment of any of the above embodiments, automatically detecting the levelness of the trailer includes receiving a plurality of inputs at a controller including at least one of a first snapshot and a second snapshot at the trailer, and the controller outputs the levelness of the trailer.

[0016] In another embodiment of any of the above embodiments, the at least one camera is the same camera of the same configuration used for the first snapshot and the second snapshot.

[0017] In a further embodiment of any of the above embodiments, the at least one camera is a rear-facing camera mounted to the towing vehicle.

[0018] In another exemplary embodiment, a system for detecting a change in trailer state includes: at least one camera defining a field of view that includes a trailer; and a controller including a memory and in communication with the at least one camera and the memory. The controller is configured to store instructions for causing a CMS to: respond to a first triggering event by storing a first snapshot of the trailer generated by the at least one camera, respond to a subsequent second triggering event by storing a second snapshot of the trailer generated by the at least one camera, compare the first snapshot to the second snapshot, and determine a deviation between an image of the first snapshot and an image of the second snapshot, and identify a load state change as having occurred in response to the deviation exceeding a predefined magnitude.

[0019] In another embodiment of any of the above embodiments, the memory further stores instructions for causing the controller to respond to changes in load conditions by automatically detecting a level state of the trailer.

[0020] In a further embodiment of any of the above embodiments, the at least one camera includes a rear view mirror replacement camera.

[0021] In a further embodiment of any of the above embodiments, the rear view mirror replacement camera is disposed in a camera arm extending from one of a driver's side of the cab and a passenger's side of the cab.

[0022] In a further embodiment of any of the above embodiments, the first triggering event is the end of an ignition cycle.

[0023] In a further embodiment of any of the above embodiments, the second triggering event is the start of an ignition cycle.

[0024] In a further embodiment of any of the above embodiments, the first triggering event is an end of an ignition cycle, the second triggering event is a start of an ignition cycle, and there are no intervening ignition cycles between the end of an ignition cycle and the start of a subsequent ignition cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present disclosure may be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0026] Figure 1A is a schematic front view of a commercial truck having a camera monitoring system (CMS) for providing at least Class II and Class IV views.

[0027] Figure 1B is a schematic top view of a commercial truck with a camera monitoring system providing Class II, Class IV, Class V, and Class VI views.

[0028] Figure 2 is a schematic top perspective view of a vehicle cab including a display and interior cameras.

[0029] Figure 3 A process for identifying trailer load changes is shown.

[0030] Figure 4 An example CMS image including a trailer after the first engine cycle is shown.

[0031] Figure 5 At the beginning of the second engine cycle Figure 5 An example CMS image of a trailer.

[0032] Figure 6 shows the use of Figure 3An optional system that detects the severity of load changes that result in trailer misleveling.

[0033] The embodiments, examples and alternatives of the preceding paragraphs, claims or the following description and drawings, including any of their various aspects or corresponding individual features, may be taken independently or in any combination. Features described in conjunction with one embodiment are applicable to all embodiments, unless these features are incompatible. DETAILED DESCRIPTION

[0034] A schematic diagram of a commercial vehicle 10 is shown in Figure 1A and Figure 1B Shown in. Figure 2 1 is a schematic top perspective view of a cab of a vehicle 10 including a display and an internal camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. Although commercial trucks are contemplated in the present disclosure, the present invention may also be applied to other types of vehicles. The vehicle 10 includes a camera monitoring system (CMS) 15 ( Figure 2 ), which has a driver-side camera arm 16a and a passenger-side camera arm 16b mounted to the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, but the CMS 15 may be used to completely replace the mirrors. In another example, each side may include multiple camera arms, each arm housing one or more cameras and / or mirrors.

[0035] Each of the camera arms 16a, 16b includes a base fixed to, for example, the cab 12. The pivot arm may be supported by the base and may be articulated relative to the base. At least one rear-facing camera 20a, 20b is disposed within the camera arm, respectively. The external cameras 20a, 20b provide an external field of view FOV, respectively. EX1 FOV EX2 , each field of view includes at least one of a Class II and a Class IV view ( Figure 1B ), which are legally required views in the commercial trucking industry. If desired, multiple cameras may also be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. Any reference to "class" views is not intended to be limiting, but rather is intended as an example of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing that encloses electronic equipment configured to provide the various features of the CMS 15.

[0036] exist Figure 1BIn one example shown, in addition to the cameras 20a, 20b in the camera arms 16a, 16b, the CMS 15 includes at least a rear-facing camera 60 and an interior trailer camera 62. The rear-facing camera 60 is mounted to the rear of the tractor 12 and captures a view of the exterior of the trailer 14, while the interior camera 62 is disposed within the trailer cargo area and captures a view of the interior of the trailer 14, including objects loaded into the trailer 14. In an alternative example, either camera 60, 62 may be a camera incorporated into an auxiliary system connected to the CMS 15 and configured to provide a video feed to the controller 30, and the following description may function similarly.

[0037] A first video display 18a and a second video display 18b are arranged on each of the driver's side and the passenger side within the vehicle cab 12 on or near the A-pillars 19a, 19b to display Class II and Class IV views on their respective sides of the vehicle 10, which provide rearward side views along the vehicle 10 captured by the external cameras 20a, 20b.

[0038] If video of Class V and / or Class VI views is also desired, camera housing 16c and camera 20c may be disposed at or near the front of vehicle 10 to provide those views ( Figure 1B ). A third display 18c disposed within the cab 12 near the top center of the windshield may be used to display to the driver Class V and Class VI views toward the front of the vehicle 10. The displays 18a, 18b, 18c face a driver area 24 within the cab 22 where an operator sits in a driver's seat 26. The location, size, and field of view streamed to any particular display may vary from the configuration described in this disclosure and still incorporate the disclosed invention.

[0039] If video of a Class VIII view is desired, camera housings may be provided on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII area of ​​the vehicle 10. In such an example, the third display 18c may include one or more frames that display the Class VIII view. Alternatively, an additional display may be added adjacent to the first display 18a, the second display 18b, and the third display 18c and provide a display dedicated to providing a Class VIII view.

[0040] In some examples, controller 30 is in communication with a vehicle controller and / or connected to a vehicle data bus (eg, a CANN bus). This communication allows controller 30 to receive and utilize sensor and operational information from the vehicle.

[0041] Continue to refer Figures 1A to 2 , Figure 3 A process 300 for identifying load changes in a trailer using the camera monitoring system 15 is shown. Figure 4An exemplary snapshot A of step 320 of process 300 is shown. Figure 5 An exemplary snapshot B of step 340 of process 300 is shown. Initially, the controller 30 of CMS 15 detects a first triggering event in an "identify first triggering event" step 310. In one example, the first triggering event is the end of an ignition cycle (i.e., the vehicle operator turns off the engine). As used herein, an "ignition cycle" refers to a driving cycle that begins with engine start, satisfies engine start conditions for at least a predetermined period of time, and ends with an engine shutoff. In some examples, the predetermined time may be at least two seconds.

[0042] After identifying the end of the ignition cycle, the controller 30 stores a first snapshot (snapshot A, e.g., a snapshot containing at least a portion of the trailer 110) in a "store trailer snapshot A" step 320. Figure 4 The controller 30 may store the snapshots in memory (e.g., local memory) as shown. In some examples, the controller 30 includes an object detection system and may identify and mark the portion of the snapshot that corresponds to the trailer 110 for later analysis. In alternative examples, the stored snapshots may be stored in memory (e.g., remote memory), where the controller 30 is configured to access the memory (e.g., remote memory) as needed during operation of the process 300. In examples that include additional cameras 60, 62, the snapshots may also include exterior views and interior views generated by the additional cameras 60, 62, as different images, or as a single image stitched together.

[0043] When the engine is restarted, the controller 30 identifies a second triggering event in an “identify second triggering event” step 330. In one example, the second triggering event is the start of an engine cycle (i.e., the engine is started by the vehicle operator). When a subsequent engine cycle begins, the controller 30 stores the second snapshot (snapshot B, e.g., Figure 5 110). A second trailer snapshot (B) is generated using the same camera in the same configuration as snapshot (A). If the loading of trailer 110 has not changed, the portions of snapshots (A) and (B) containing trailer 110 will be substantially the same.

[0044] After storing the second snapshot (B), the controller 30 compares the snapshots (A), (B) to determine how much deviation there is between the image of the trailer 110 in the snapshot (A) and the image of the trailer in the snapshot (B). Deviations between images generally refer to differences between the trailer 110 portions of the images. These differences are generally the result of changes in the state of the trailer, such as changes in loading, lift gate opening, etc. In one example, the deviation is determined by applying an edge detection algorithm to the snapshots and determining the pixel difference of the identified edge between the first snapshot and the second snapshot. The pixel difference indicates how much the edge has shifted within the image of the snapshot.

[0045] In some cases, environmental changes (e.g., fog, rain, or snow) may also result in smaller deviations that require the use of thresholds to filter out false positives. When the controller 30 includes software-based object recognition, the relative position of the trailer 110 itself can be compared between snapshots. As an example, the distance 302 of the trailer corner 304 to the edge 306 of the image, the length 312 of the top edge of the trailer 110 visible in the image, the angle 332 of the rear trailer edge 334 relative to the ground plane, and / or any similar measurable aspects can be compared between two snapshots (A), (B), and the determined difference is the deviation between the snapshots (A), (B). In examples including an internal camera 62, object detection can utilize feature-based object detection and / or region-of-interest-based methods to determine the number and type of objects that have been removed and / or added to the trailer 14 to determine a load reduction or increase. In other examples, the entire image is compared, and any quantifiable measure of the difference between the two images can be utilized. In either case, the process 300 compares the snapshots in the "compare snapshots and determine the magnitude of the deviation" step 350.

[0046] The deviation is quantified to determine a single "deviation" value having a numerical magnitude, and the magnitude of the deviation value is compared to the threshold value in a "compare deviation to threshold value and issue an alarm when deviation exceeds threshold value" step 360. An alarm is provided to the vehicle operator via any means including an audio indicator, a visual alarm, a message sent to a messaging system, or any other notification means. The notification informs the operator that the trailer 110 is not in the same position as at the end of the previous engine cycle. Once so informed, the operator can manually inspect the trailer 110 and identify whether the load has changed, the lift gate is open, or any similar function. The specific threshold value depends on the deviation quantification used, and can be set to a size large enough so that environmental changes between engine cycles (e.g., changes from day to night, changes from rain to cloudy, etc.) do not produce false alarms.

[0047] In some examples, such as those where the trailer 14 is a box trailer, further advantages may be provided by including a rearward-facing camera 60. As an example, the rearward-facing camera 60 has a square / rectangular field of view of the trailer 14, and this field of view is particularly suitable for making trailer edge-based assessments of trailer level using edge detection.

[0048] In some examples, identifying that the trailer 110 has shifted can further be used to trigger an automatic leveling check. Parking and driving a semi-trailer truck on an uneven surface / road can interfere with normal operation. Additionally, when a trailer is not level, either due to being on an uneven surface or due to uneven loads being loaded / unloaded, it can change the operation of the vehicle and alter how the trailer responds to standard maneuvers.

[0049] The automatic leveling check is performed using depth sensing technology that combines trailer images, steering wheel angles, road images or wheel images, stickshift position, trailer load status (e.g., loaded, unloaded, partially loaded), and trailer size or trailer height. The automatic level check uses depth sensing in conjunction with data from sensors and cameras received from the controller 30 and the vehicle controller via a communication connection to determine firstly whether the trailer is out of level and secondly the degree to which the trailer is out of level. The magnitude of the out of level is reported as the angle of the trailer bed relative to the ground plane.

[0050] In some examples, one or more cameras from CMS 15 can provide additional views that are analyzed and potential causes of non-levelness can be identified. For example, a rear-facing camera that includes a view of the wheels may be able to identify whether one or more wheels have passed over a curb, passed through a grassy area, or any similar instance. This information is then correlated and can be provided to the vehicle operator, allowing the operator to distinguish between an unlevel trailer chassis due to uneven loading and / or mechanical characteristics of the trailer 110 and an unlevel trailer chassis due to environmental characteristics. This in turn allows the operator to use this information to adjust vehicle operation in the correct manner.

[0051] Continue to refer Figure 3 process, Figure 6 A control system 600 is schematically shown including an algorithm 610 configured to receive a plurality of inputs 620 from the controller 30 and / or a vehicle communication bus (e.g., a CANN bus) and an output 630 configured to output a value indicative of a level or unlevel state of the trailer. In some examples, the inputs 620 may include a full trailer image 622 from the CMS, a steering wheel angle 624 from the vehicle controller, a road image 626 from the CMS, an image of one or more wheels in the trailer 628, a current gear 642 from the vehicle controller, a trailer load state 644 from the CMS, the vehicle controller, or manually input from a vehicle operator, and a set of trailer dimensions 646 from the CMS or manually input from a vehicle operator.

[0052] In alternative examples, any combination of the above values ​​may be provided to the algorithm 610 , and a corresponding out-of-levelness value 630 may be output from the algorithm 610 .

[0053] In some examples, the algorithm 610 outputs a binary "level" or "not level" value that provides a simple alert to the vehicle operator who can compensate accordingly. In other examples, the output 630 can be a vector value indicating the magnitude of the tilt of the trailer chassis and the direction of the tilt. In the latter case, the vector value can be used by driver assistance systems, automatic driver systems, and any similar system that can automatically compensate for the tilt or provide instructions to the vehicle operator to compensate for the tilt.

[0054] As described above, the controller 30 can be used to implement the various functions disclosed in the present application. The controller 30 may include one or more discrete units. In terms of hardware architecture, such a computing device may include a processor, a memory, and one or more input and / or output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but not limited to, one or more buses and / or other wired or wireless connections. The local interface may have additional elements omitted for simplicity, such as a controller, a buffer (cache), a driver, a repeater, and a receiver to enable communication. In addition, the local interface may include addresses, controls, and / or data connections to enable appropriate communication between the above components.

[0055] The controller 30 may be a hardware device for executing software, particularly software stored in a memory. The controller 30 may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device generally used to execute software instructions.

[0056] The memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). In addition, the memory may contain electronic, magnetic, optical and / or other types of storage media. The memory may also have a distributed architecture in which various components are remote from each other but can be accessed by the processor.

[0057] The software in the memory may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. System components embodied as software may also be constructed as source programs, executable programs (object code), scripts, or any other entity including a set of instructions to be executed. When constructed as a source program, the program is translated via a compiler, assembler, interpreter, etc., which may or may not be included in the memory.

[0058] The disclosed input and output devices that can be coupled to the system I / O interface can include input devices such as, but not limited to, keyboards, mice, scanners, microphones, cameras, mobile devices, proximity devices, etc. In addition, output devices such as, but not limited to, printers, displays, etc. Finally, input and output devices can also include devices that communicate as input and output, such as, but not limited to, modulators / demodulators (modems; used to access another device, system, or network), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc.

[0059] When the controller 30 is in operation, the processor may be configured to execute software stored in the memory, transfer data to and from the memory, and generally control the operation of the computing device according to the software. The software in the memory is read in whole or in part by the processor, possibly buffered within the processor, and then executed.

[0060] Although described above in the context of a particular mirror replacement system, it should be understood that the process may be applied to any camera monitoring system that includes a camera that captures all or a portion of a trailer within its field of view.

[0061] It should also be understood that although a specific component arrangement is disclosed in the illustrated embodiment, other arrangements will benefit from this. Although a specific order of steps is shown, described, and claimed, it should be understood that unless otherwise indicated, the steps may be performed in any order, separated, or combined, and will still benefit from the present invention.

[0062] Although the different examples have specific components shown in the drawings, embodiments of the present invention are not limited to those specific combinations. Some components or features from one example may be used in combination with features or components from another example.

[0063] Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

1. A method for detecting a change in a load state of a trailer, comprising: responding to a first triggering event by storing a first snapshot of the trailer generated by at least one camera local to the vehicle including the trailer; responding to a subsequent second triggering event by storing a second snapshot of the trailer generated by the at least one camera local to the vehicle including the trailer; comparing the first snapshot to the second snapshot and determining a deviation between an image of the first snapshot and an image of the second snapshot; and In response to the deviation exceeding a predefined magnitude, a trailer load state change is identified as having occurred.

2. The method according to claim 1, wherein: The first triggering event is the end of an ignition cycle.

3. The method according to claim 1, wherein: The second triggering event is the start of an ignition cycle.

4. The method according to claim 1, wherein: The first triggering event is an end of an ignition cycle, the second triggering event is a start of an ignition cycle, and there is no intervening ignition cycle between the end of the ignition cycle and the start of a subsequent ignition cycle.

5. The method according to claim 1, further comprising: Prior to comparing the first snapshot to the second snapshot, an object detection algorithm is used to detect the trailer in each of the first snapshot and the second snapshot.

6. The method according to claim 5, wherein: The deviation comprises a change in a position within the image of a detected trailer object between the first snapshot and the second snapshot.

7. The method according to claim 5, wherein: Identifying the position changes includes detecting a set of position changes and normalizing the position changes into a single quantized value.

8. The method according to claim 6, wherein: The single quantized value is compared to the predefined size.

9. The method according to claim 1, further comprising: The trailer is automatically detected to respond to changes in load conditions.

10. The method according to claim 9, wherein: Automatically detecting the level status of the trailer includes receiving, by a controller, a plurality of inputs including at least one of the first snapshot and the second snapshot at the trailer, and the controller outputting the level status of the trailer.

11. The method according to claim 1, wherein: The at least one camera is the same camera of the same configuration used for the first snapshot and the second snapshot.

12. The method according to claim 11, wherein: The at least one camera is a rear-facing camera mounted to the tractor.

13. A system for detecting a change in trailer status, comprising: at least one camera, the at least one camera defining a field of view that includes the trailer; as well as a controller including a memory and in communication with the at least one camera and the memory, the controller configured to store instructions for causing the CMS to: respond to a first triggering event by storing a first snapshot of the trailer generated by the at least one camera, respond to a subsequent second triggering event by storing a second snapshot of the trailer generated by the at least one camera; comparing the first snapshot to the second snapshot and determining a deviation between an image of the first snapshot and an image of the second snapshot; And in response to the deviation exceeding a predefined magnitude, identifying a load state change as having occurred.

14. The system according to claim 13, wherein: The memory also stores instructions for causing the controller to respond to changes in load conditions by automatically detecting a level state of the trailer.

15. The system of claim 13, wherein: The at least one camera includes a rear view mirror replacement camera.

16. The system of claim 15, wherein: The rearview mirror replacement camera is disposed in a camera arm extending from one of a driver's side of a cab and a passenger's side of the cab.

17. The system of claim 13, wherein: The first triggering event is the end of an ignition cycle.

18. The system of claim 13, wherein: The second triggering event is the start of an ignition cycle.

19. The system of claim 13, wherein: The first triggering event is an ignition cycle end, the second triggering event is an ignition cycle start, and there is no intermediate ignition cycle between the ignition cycle end and a subsequent ignition cycle start.

Images