Automatic Panning Camera Mirror System with Weighted Trailer Angle Estimation

Abstract

To provide a method for automatically panning a view for a commercial vehicle that includes determining a plurality of estimated trailer angles and to provide a camera mirror system (CMS).SOLUTION: Each estimated trailer angle is determined using a distinct estimation method, and the method assigns a confidence value to each estimated trailer angle in the plurality of estimated trailer angles. The method determines a weighted sum of the plurality of estimate trailer angles, and automatically pans the view based on the weighted sum and a current vehicle operation at least partially.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure relates to a camera mirror system (CMS) for use in commercial trucks, and more particularly to a CMS with auto-panning functionality that includes fused trailer angle estimation. [Background technology]

[0002] Mirror replacement systems and camera systems to supplement mirror views are utilized in commercial vehicles to enhance vehicle operators' ability to view their surrounding environment. Camera mirror systems (CMS) utilize one or more cameras to provide the vehicle operator with an expanded field of view. In some instances, mirror replacement systems cover a wider field of view than traditional mirrors or include views not fully accessible in traditional mirrors.

[0003] In some maneuvers, such as backing up a trailer, a fixed view, such as that provided by a fixed mirror or fixed field of view camera, may not provide a complete view of the maneuver and may not present desirable information to the driver. Manual panning systems, in which an operator manually adjusts a physical camera or mirror angle, require frequent stops of the operation to adjust the view provided and may provide insufficient precision in the adjustments.

[0004] Some exemplary systems attempt to minimize the problems of manual panning by implementing automatic or semi-automatic panning. Such systems rely on potentially inaccurate trailer angle estimates, and kinematic models of vehicle operation, especially in reversing maneuvers, can have difficulty accounting for potential variability in trailer angle estimates. Summary of the Invention [Means for solving the problem]

[0005] An exemplary method for automatically panning a view for a commercial vehicle includes identifying a plurality of estimated trailer angles, where each estimated trailer angle is identified using a different estimation method; assigning a confidence value to each estimated trailer angle in the plurality of estimated trailer angles; identifying a weighted sum of the plurality of estimated trailer angles; and automatically panning the view based at least in part on the weighted sum and a current vehicle operation.

[0006] Another example of the above method for automatically panning a view for a commercial vehicle further includes adding the identified weighted sum to an ordered list of past weighted sums and low-pass filtering the ordered list.

[0007] In another example of any of the above methods for automatically panning a view for a commercial vehicle, automatically panning the view is based on a filtered ordered list.

[0008] Another example of any of the above methods for automatically panning a view for a commercial vehicle further includes, after assigning a confidence value to each estimated trailer angle, and before determining the weighted sum, discarding estimated trailer angles having a confidence value below a predetermined threshold.

[0009] In another example of any of the above methods for automatically panning a view for a commercial vehicle, the predetermined threshold is at least 85%.

[0010] In another example of any of the above methods for automatically panning a view for a commercial vehicle, the weighted sum is determined by multiplying each trailer angle estimate by a corresponding confidence value to determine a weighted estimate, summing the weighted estimates, and dividing the summed weighted estimate by the sum of the confidence values.

[0011] In another example of any of the above methods for automatically panning a view for a commercial vehicle, the different estimation methods include at least two of image-based wheel detection, image-based trailer trailing edge angle detection, image-based trailer marking angle detection, image-based wheel angle detection, road edge departure detection, lane marker departure detection, a hitch angle sensor, and a wheel angle sensor.

[0012] In another example of any of the above methods for automatically panning a view for a commercial vehicle, automatically panning the view includes adjusting a Class II view within a Class IV view.

[0013] In another example of any of the above methods for automatically panning a view for a commercial vehicle, automatically panning the view includes maintaining the rear end of the trailer within a Class II view.

[0014] Another example of any of the above methods for automatically panning a view for a commercial vehicle further includes continuously repeating the method, thereby producing real-time trailer angle monitoring.

[0015] In another example of any of the above methods for automatically panning a view for a commercial vehicle, determining a plurality of estimated trailer angles, assigning a confidence value to each estimated trailer angle in the plurality of estimated trailer angles, and determining a weighted average of the plurality of estimated trailer angles occurs within a camera mirror system controller of the vehicle.

[0016] In another example of any of the above methods for automatically panning a view for a commercial vehicle, at least one of determining a plurality of estimated trailer angles, assigning a confidence value to each estimated trailer angle in the plurality of estimated trailer angles, and determining a weighted average of the plurality of estimated trailer angles is performed remotely from and transmitted to the camera mirror system controller.

[0017] In one exemplary embodiment, a camera mirror system for a vehicle includes a first camera having a first field of view; and a controller configured to receive the first field of view and output a portion of the first field of view to a first display, the controller including a trailer angle detection module configured to identify a plurality of trailer angle estimates; a confidence value module configured to identify a confidence value for each trailer angle estimate; and a fusion module configured to fuse the plurality of trailer angle estimates and the confidence values ​​into a single trailer angle estimate, and automatically pan at least one view of the camera mirror system based at least in part on the single trailer angle estimate so that a feature of the trailer is maintained within the at least one view.

[0018] In another example of the camera mirror system for a vehicle described above, the fusion module is configured to determine a weighted sum trailer angle based on a plurality of trailer angle estimates and corresponding confidence values.

[0019] In another example of any of the above camera mirror systems for vehicles, the fusion module is further configured to add the weighted sum trailer angle to a historical weighted sum trailer angle data set and low-pass filter the historical weighted sum trailer angle data set.

[0020] In another example of any of the above camera mirror systems for vehicles, each trailer angle estimate in the plurality of trailer angle estimates is determined using a different angle estimation method.

[0021] In another example of any of the above camera mirror systems for vehicles, the controller is further configured to add the single trailer estimate to an ordered list of past trailer angle estimates and low-pass filter the ordered list.

[0022] In another example of any of the above camera mirror systems for vehicles, the automatic panning is based at least in part on a low-pass filtered ordered list. [Brief explanation of the drawings]

[0023] The present disclosure may be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings. [Figure 1A] FIG. 1 is a schematic front view of a commercial truck having a camera mirror system (CMS) used to provide at least a Class II view and a Class IV view. [Figure 1B] 1 is a schematic top view of a commercial truck equipped with a camera mirror system providing Class II, Class IV, Class V, and Class VI views. [Figure 2] FIG. 1 is a schematic top perspective view of a vehicle cabin including a display and an interior camera. [Figure 3] Figure 3A shows the vehicle at the beginning of a reverse maneuver with no trailer angle, and Figure 3B shows the vehicle midway through a reverse maneuver with a large trailer angle. [Figure 4] A method for obtaining an estimated weighted trailer angle is shown. [Figure 5] A system is presented for determining the exact trailer angle from the estimated weighted angle and automatically panning the camera mirror system.

[0024] The embodiments, examples and alternatives of the preceding paragraphs, claims or the following description and drawings, including any of their various aspects or their respective individual features, can be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments unless such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0025] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. While this disclosure contemplates a commercial truck, the invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera mirror system (CMS) 15 (FIG. 2) having driver-side and passenger-side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may include conventional mirrors integrated therewith, although the CMS 15 may be used to replace mirrors entirely. In a further example, each side may include multiple camera arms, each housing one or more cameras and / or mirrors.

[0026] Each of the camera arms 16a, 16b includes a base that is fixed to, for example, the cab 12. A pivoting arm is supported by and can articulate relative to the base. At least one rear-facing camera 20a, 20b is disposed within each of the camera arms. Each of the exterior cameras 20a, 20b has an exterior field of view (FOV), each of which includes at least one of a Class II view and a Class IV view (FIG. 1B), which are legally defined views in the commercial truck industry. EX1 , FOV EX2 A Class II view of a particular side of the vehicle 10 is a subset of a Class IV view of the same side of the vehicle 10. If desired, multiple cameras can be used on each camera arm 16 a, 16 b to provide these views. Each arm 16 a, 16 b can also provide a housing enclosing electronics configured to provide various functions of the CMS 15.

[0027] First and second video displays 18a, 18b are positioned at or near the A-pillars 19a, 19b on the driver's and passenger's sides, respectively, within the vehicle cab 12 and display Class II and Class IV views of each side of the vehicle 10, which provide rearward-facing side views along the vehicle 10 captured by external cameras 20a, 20b.

[0028] If video of Class V and Class VI views is also desired, camera housing 16c and camera 20c can be positioned at or near the front of vehicle 10 to provide these views (FIG. 1B). A third display 18c located within vehicle cab 12 near the top center of the windshield can be used to display Class V and Class VI views facing forward of vehicle 10 to the driver.

[0029] If video of a Class VIII view is desired, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of the vehicle 10. In such an example, the third display 18c can include one or more frames that display the Class VIII view. Alternatively, additional displays can be added near the first, second, and third displays 18a, 18b, 18c to provide displays dedicated to providing the Class VIII view.

[0030] 1A, 1B, and 2, FIGS. 3A and 3B show the vehicle 100 performing a reverse maneuver. In the initial position (FIG. 3A), the trailer 110 is initially at approximately 0 degrees relative to the vehicle cab 120. trailer angle (i.e., the angle between the central longitudinal axis A1 of the cab 120 and the central longitudinal axis A2 of the trailer 110) which has Trailer 110 This means that the trailer 110 is aligned with the orientation of the vehicle cab 120. Alternatively, this angle can be expressed as 180 degrees relative to the cab 120. During the process of backing up, especially when backing up during a turn, the trailer 110 will be at an angle relative to the cab 120 (FIG. 3B), which will affect the backing maneuver. Between axes A1 and A2 Trailer angle θ The particular skew in FIG. 3B is exaggerated relative to most expected angles for illustrative purposes.

[0031] To assist the driver in performing a reversing maneuver, it is beneficial to ensure that the rear 112 of the trailer 110 is visible to the driver on at least one display throughout the reversing maneuver. In some particular instances, it is desirable not only to include the rear 112 of the trailer 110 but also to center the rear 112 of the trailer 110 in the Class II view. However, as shown in FIG. 3B , a static Class II view can result in the rear 112 of the trailer 110 extending beyond the boundaries of the Class II view, even when the rear 112 remains within the Class IV field of view. To prevent the view of the rear 112 of the trailer 110 from being lost in the Class II view or to maintain the center of the Class II view on the rear 112 of the trailer 110, the vehicles 10, 100 illustrated herein include an automatic panning feature within the camera mirror system.

[0032] The auto-panning feature uses a combination of different trailer angle estimation and detection systems to estimate the trailer angle relative to the tractor at any particular time. The estimated trailer angle is assigned a "weight" corresponding to how likely it is to be accurate given the current driving conditions and provided to a fusion system within the vehicle controller. For example, a wheel detection-based system may have a high probability of accuracy (over 90%) in daytime conditions where black wheels stand out against the surrounding environment, but a lower probability of accuracy (50-70%) in nighttime conditions where black wheels blend into the dark environment. Similarly, a lane marker-based detection system, such as a line detection system using a Hough transform, may have a low probability of accuracy in low-light weather conditions (e.g., rain, snow, fog) but a high probability of accuracy in clear weather conditions. A bottom-edge detection system may have a high probability of accuracy with container trailers but a low probability of accuracy with tanker trailers.

[0033] The vehicle controller fuses multiple trailer angle estimates using a weighted sum based on the confidence values ​​of each detection method. The weighted sum is referred to as the raw estimate. The raw estimate is then combined with past estimates and filtered using a low-pass filter to provide a trailer angle that transitions smoothly to the CMS's auto-panning function and other vehicle systems that can benefit from utilizing trailer angle estimates. In some examples, the camera mirror system can generate meta-information corresponding to the side of the trailer, which can be utilized to prevent false positives and / or provide additional confidence in an otherwise determined trailer angle. To reduce the phase lag in determining the trailer angle estimate, and therefore the reporting delay, to a negligible level, the low-pass filter starts from the first measurement. This reduces the phase lag by providing an initial allocation value. While some delay still exists in such a system, the delay is minimized so as not to affect the reversing system.

[0034] Continuing with reference to FIGS. 1-3B, FIG. 4 illustrates a process for determining a more accurate trailer angle. Initially, process 300 determines angle estimates in an "Identify Angle Estimates" step 310. The number of angle estimates determined will vary depending on the particular system involved. In some examples, at least some of the angle estimates are purely visually based, utilizing feature tracking of objects identified in the CMS video feed (e.g., wheels, rear end, trailer markings, etc.) to determine the estimated trailer angle. Similarly, some angle estimates may be determined based on road edge and / or lane departure detection, comparison of positioning satellites to stored maps, hitch angle sensors, trailer edge detection, lane detectors, radar sensors, lidar sensors, and other similar trailer angle detection systems.

[0035] Once the trailer angle estimates have been identified, the controller that identifies the trailer angle estimates assigns a confidence value to each trailer angle estimate in an "identify confidence value" step 320. The method for identifying the confidence of each estimate depends on how the particular estimate is made and may be identified by one of ordinary skill in the art using any suitable technique. In some examples, the confidence value may depend on weather conditions, lighting conditions, trailer type, historical accuracy data, and other characteristics that may be related to the likelihood of accuracy. The confidence value is expressed as a percentage of accuracy (e.g., a trailer angle based on a 15-degree wheel is accurate 94% of the time).

[0036] After determining the confidence value, the controller discards all estimates below a minimum confidence threshold in step 330 "Discard angles below confidence threshold." In one example, estimates below 85% confidence are considered erroneous or inaccurate in some systems and are removed from consideration. In another example, estimates below 90% confidence are considered erroneous. Discarding estimates below the minimum confidence threshold eliminates outliers that may be caused by inaccurate sensors, particularly adverse conditions for a particular estimation technique, and other similar conditions that result in inaccurate estimates. Eliminating extreme outliers improves the accuracy of the estimate. In some examples, discarding angles below the threshold can be omitted if a significant number of estimates are provided to the controller and / or if a low confidence value does not indicate an error.

[0037] After all estimates below the threshold are discarded or step 330 is ignored, a fusion algorithm within the controller determines a weighted sum of the angle estimates in "Determine Weighted Sum" step 340. In one example of determining a weighted average, a first trailer angle estimate of 14 degrees has 98% confidence, a second trailer angle estimate of 10 degrees has 86% confidence, and a third trailer angle estimate of 15 degrees has 94% confidence. A predetermined confidence threshold is set at 85%, and all three values ​​are considered acceptable. The fusion algorithm multiplies each angle by its corresponding confidence, sums the results, and divides the sum by the sum of the confidences. In the exemplary case, the fusion algorithm results in ((14 * 98) + (10 * 86) + (15 * 94)) / (98 + 86 + 94) = 13.10 degrees. Thus, in the exemplary case, the determined angle (also called the raw measurement) is 13.10 degrees, and this angle is output to the low-pass filter. The low-pass filtered angle is output to the automatic panning system. It is understood that actual implementations may utilize substantially more than three angle estimates, and the more angle estimates used, the more accurate the resulting value will be.

[0038] Once the weighted average of the estimated angle is identified, the controller adds the weighted average to a historical data set that includes the previously identified weighted average for the current maneuver. In one example, trailer angle estimates are made approximately every 200 ms, and the historical data set includes each subsequent entry in turn. A low-pass filter is applied to the historical data set that includes the newly identified weighted average. The low-pass filter smooths the transitions, eliminating "jerky" or "abrupt" trailer angle transitions, providing a more accurate representation of trailer angle changes over time and allowing automatic panning systems and / or other vehicle systems to take the exact changes into account.

[0039] Continuing with reference to FIG. 4, FIG. 5 schematically illustrates an exemplary automatic panning system for a vehicle 410. A controller 420 receives images and other sensor information from the vehicle 410, and the controller 420 determines a raw angle estimate from the received images and sensor information using an angle detection module 424. In parallel with the angle detection module 424, a confidence value determination module 422 utilizes received data 421 indicating conditions and other aspects that affect the confidence of each detected angle. The confidence value determination module 422 determines the confidence of each trailer angle detection, and the detection and confidence values ​​are provided to a fusion module 426.

[0040] The fusion module 426 determines a weighted average of the estimated trailer angle and fuses the weighted average with previous trailer angles stored in the trailer angle history 428. The fusion module 426 also applies a low-pass filter to the combined trailer angle and past trailer angle data to determine a two-dimensional trailer angle. The two-dimensional trailer angle is an accurate estimate of the current trailer angle on a two-dimensional plane. The two-dimensional trailer angle is converted to a three-dimensional trailer position based on the trailer angle and geographic features (e.g., the slope of a hill). The three-dimensional trailer angle is then provided to an auto-panning function, and a camera mirror system automatically pans at least one camera view within the image. In one example, the auto-panning is configured to maintain the rear end of the trailer within a Class II view throughout the vehicle's operation. In other implementations, the auto-panning can maintain other objects or portions of objects within the view.

[0041] The fusion of multiple estimated angles from various sources into a single, more reliable trailer angle estimate can be used across multiple systems. Fusion of multiple trailer angle estimates into a single value also allows the system to reliably identify and pan the correct trailer angle while taking into account the unreliability of certain trailer angle estimation techniques for certain conditions and / or certain types of trailers.

[0042] While exemplary embodiments are disclosed, one of ordinary skill in the art would recognize that certain modifications would come within the scope of the following claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for automatically panning a view for a commercial vehicle, comprising: determining a plurality of estimated trailer angles between a central longitudinal axis of the tractor and a central longitudinal axis of the trailer, each estimated trailer angle being determined using a different estimation method; assigning a confidence value to each estimated trailer angle in the plurality of estimated trailer angles; determining a weighted sum of the plurality of estimated trailer angles based on the confidence value; automatically panning a view of the camera mirror system on an electronic display based on said weighted sum and current vehicle operation; A method comprising:

2. The method of claim 1 , further comprising adding the determined weighted sum to an ordered list of past weighted sums and low-pass filtering the ordered list.

3. The method of claim 2 , wherein automatically panning the view is based on a filtered ordered list.

4. 2. The method of claim 1, further comprising, after assigning the confidence value to each estimated trailer angle and before determining the weighted sum, discarding estimated trailer angles having a confidence value below a predetermined threshold.

5. The method of claim 4 , wherein the predetermined threshold is at least 85%.

6. 2. The method of claim 1, wherein the weighted sum is determined by multiplying each trailer angle estimate by a corresponding confidence value to determine a weighted estimate, summing the weighted estimates, and dividing the summed weighted estimates by the sum of the confidence values.

7. 2. The method of claim 1, wherein the different estimation methods include at least two of image-based wheel detection, image-based trailer trailing edge angle detection, image-based trailer marking angle detection, image-based wheel angle detection, road edge departure detection, lane marker departure detection, and a hitch angle sensor.

8. The method of claim 1 , wherein automatically panning the views includes adjusting a Class II view within a Class IV view.

9. The method of claim 8 , wherein automatically panning the view includes maintaining the trailing edge of the trailer within the Class II view.

10. The method of claim 1 , further comprising continuously repeating the method, thereby producing real-time trailer angle monitoring.

11. 2. The method of claim 1, wherein determining the plurality of estimated trailer angles, assigning a confidence value to each estimated trailer angle in the plurality of estimated trailer angles, and determining a weighted average of the plurality of estimated trailer angles occurs within a camera mirror system controller of the vehicle.

12. 2. The method of claim 1 , wherein at least one of determining the plurality of estimated trailer angles, assigning a confidence value to each estimated trailer angle in the plurality of estimated trailer angles, and determining a weighted average of the plurality of estimated trailer angles is performed remotely from and transmitted to a camera mirror system controller.

13. 1. A camera mirror system for a vehicle, comprising: a first camera having a first field of view; a controller configured to receive the first field of view and output a portion of the first field of view to a first display; Equipped with the controller includes a trailer angle detection module configured to determine a plurality of trailer angle estimates for a trailer angle between a central longitudinal axis of a tractor and a central longitudinal axis of a trailer; a confidence value module configured to determine a confidence value for each trailer angle estimate; and a fusion module configured to determine a weighted sum trailer angle based on the plurality of trailer angle estimates and corresponding confidence values ​​and to fuse the plurality of trailer angle estimates into a single trailer angle estimate using the weighted sum trailer angle; A camera mirror system that automatically pans at least one view of the camera mirror system based on the single trailer angle estimate so that features of the trailer are maintained within the at least one view.

14. The camera mirror system of claim 13, wherein the controller is further configured to, after determining a confidence value for each trailer angle estimate, discard trailer angle estimates having a confidence value below a predetermined threshold before determining the weighted sum trailer angle.

15. 14. The camera mirror system of claim 13, wherein the fusion module is further configured to add the weighted sum trailer angle to a historical weighted sum trailer angle data set and low-pass filter the historical weighted sum trailer angle data set.

16. The camera mirror system of claim 13 , wherein each trailer angle estimate in the plurality of trailer angle estimates is determined using a different angle estimation method.

17. 14. The camera mirror system of claim 13, wherein the controller is further configured to add the single trailer estimate to an ordered list of past trailer angle estimates and low-pass filter the ordered list.

18. 18. The camera mirror system of claim 17, wherein the automatic panning is based on the ordered list being low-pass filtered.

Images