Auto panning camera mirror system including weighted trailer angle estimation

Abstract

To provide a method for automatically panning a view for a commercial vehicle that includes determining a plurality of estimated trailer angles and to provide a camera mirror system (CMS).SOLUTION: Each estimated trailer angle is determined using a distinct estimation method, and the method assigns a confidence value to each estimated trailer angle in the plurality of estimated trailer angles. The method determines a weighted sum of the plurality of estimate trailer angles, and automatically pans the view based on the weighted sum and a current vehicle operation at least partially.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present disclosure relates to a camera mirror system (CMS) for use in commercial trucks, and more particularly to a CMS having an automatic panning function including fused trailer angle estimation.

Background Art

[0002] Mirror replacement systems and camera systems for supplementing mirror views have been utilized to enhance a vehicle driver's ability to view the surrounding environment in commercial vehicles. A camera mirror system (CMS) utilizes one or more cameras to provide an extended field of view to a vehicle driver. In some examples, a mirror replacement system covers a wider field of view than a conventional mirror or includes views that are not fully obtainable with a conventional mirror.

[0003] In some operations, such as backing up a trailer, a fixed view as provided by a fixed mirror or a fixed field of view camera may not provide a complete view of the operation, and desirable information that should be presented to the driver may not be presented. A manual panning system where an operator manually adjusts a physical camera or mirror angle requires frequent stops in the operation to adjust the provided view and may provide insufficient accuracy in the adjustment.

[0004] Some exemplary systems attempt to minimize the problems of manual panning by performing automatic or semi-automatic panning. Such systems potentially rely on inaccurate trailer angle estimation, and in particular, kinematic models of vehicle operations during backing maneuvers may have difficulty accounting for potential variations in trailer angle estimation.

Summary of the Invention

Means for Solving the Problems

[0005] An exemplary method for automatically panning a view for a commercial vehicle includes identifying a plurality of estimated trailer angles, each estimated trailer angle being identified using a different estimation method, assigning confidence values ​​to each of the plurality of estimated trailer angles, identifying a weighted sum of the plurality of estimated trailer angles, and automatically panning the view based at least in part on the weighted sum and the current vehicle operation.

[0006] Another example of the above method for automatically panning views for commercial vehicles further involves adding the identified weighted sum to an ordered list of past weighted sums and then applying a low-pass filter to the ordered list.

[0007] In another example of any of the above methods for automatically panning views for commercial vehicles, the automatic panning of the view is based on a filtered ordered list.

[0008] Another example of any of the above methods for automatically panning views for commercial vehicles further includes assigning confidence values ​​to each estimated trailer angle and then discarding estimated trailer angles with confidence values ​​below a predetermined threshold before determining the weighted sum.

[0009] In another example of any of the above methods for automatically panning the view for commercial vehicles, the predetermined threshold is at least 85%.

[0010] In another example of any of the above methods for automatically panning views for commercial vehicles, the weighted sum is determined by multiplying each trailer angle estimate by its corresponding confidence value to determine the weighted estimate, summing the weighted estimates, and dividing the summed weighted estimate by the sum of confidence values.

[0011] In another example of any of the above methods for automatically panning the view for commercial vehicles, a different estimation method includes at least two of the following: image-based wheel detection, image-based trailer rear end angle detection, image-based trailer marking angle detection, image-based wheel angle detection, road edge departure detection, lane marker departure detection, hitch angle sensor, and wheel angle sensor.

[0012] In another example of any of the above methods for automatically panning views for commercial vehicles, automatically panning the view involves adjusting a Class II view within a Class IV view.

[0013] In another example of any of the above methods for automatically panning the view for commercial vehicles, automatically panning the view includes keeping the rear end of the trailer within a Class II view.

[0014] Another example of any of the above methods for automatically panning the view for commercial vehicles further includes continuously repeating this method, thereby generating real-time trailer angle monitoring.

[0015] In another example of any of the above methods for automatically panning views for commercial vehicles, the identification of multiple estimated trailer angles, the assignment of confidence values ​​to each of the multiple estimated trailer angles, and the determination of a weighted average of the multiple estimated trailer angles are performed within the vehicle's camera mirror system controller.

[0016] In another example of any of the above methods for automatically panning the view for commercial vehicles, at least one of the following is performed remotely from the camera mirror system controller and transmitted to the camera mirror system controller: identifying multiple estimated trailer angles, assigning a confidence value to each of the multiple estimated trailer angles, and determining a weighted average of the multiple estimated trailer angles.

[0017] In one exemplary embodiment, a camera mirror system for a vehicle includes a first camera having a first field of view, and a controller configured to receive the first field of view and output a portion of the first field of view to a first display, wherein the controller includes a trailer angle detection module configured to identify a plurality of trailer angle estimates, a confidence module configured to identify a confidence value for each of the trailer angle estimates, and a fusion module configured to fuse the plurality of trailer angle estimates and confidence values ​​into a single trailer angle estimate, and to automatically pan at least one view of the camera mirror system based at least partially on the single trailer angle estimate so that the features of the trailer are maintained within at least one view.

[0018] In another example of the camera mirror system for the above vehicle, the fusion module is configured to determine a weighted sum trailer angle based on multiple trailer angle estimates and corresponding confidence values.

[0019] In one of the other examples of the camera mirror system for the above vehicles, the fusion module is further configured to add the weighted sum trailer angle to a historical weighted sum trailer angle dataset and to perform a low-pass filter on the historical weighted sum trailer angle dataset.

[0020] In another example of the camera mirror system for the vehicle described above, each trailer angle estimate in the multiple trailer angle estimates is determined using a different angle estimation method.

[0021] In one of the other examples of the camera mirror system for the vehicles described above, the controller is further configured to add a single trailer estimate to an ordered list of historical trailer angle estimates and to perform a low-pass filter on the ordered list.

[0022] In another example of the camera mirror system for the vehicle described above, automatic panning is at least partially based on a low-pass filtered ordered list. [Brief explanation of the drawing]

[0023] The present disclosure can be further understood by referring to the following detailed description when considered in connection with the accompanying drawings. [Figure 1A] Schematic front view of a commercial truck having a camera mirror system (CMS) used to provide at least Class II and Class IV views. [Figure 1B] Schematic top view of a commercial truck equipped with a camera mirror system that provides Class II, Class IV, Class V, and Class VI views. [Figure 2] Schematic top perspective view of a vehicle cabin including a display and an interior camera. [Figure 3] FIG. 3A shows the vehicle at the start of a reverse operation without a trailer angle. FIG. 3B shows the vehicle during a reverse operation with a large trailer angle. [Figure 4] Shows a method for obtaining an estimated weighted trailer angle. [Figure 5] Shows a system for identifying an accurate trailer angle from an estimated weighted angle and automatically panning a camera mirror system.

[0024] The foregoing paragraphs, claims, or embodiments, examples, and alternatives of the following description and drawings can be employed independently or in any combination, including any of their various aspects or respective individual features. Features described in connection with one embodiment are applicable to all embodiments unless such features are incompatible.

MODE FOR CARRYING OUT THE INVENTION

[0025] A schematic diagram of a commercial vehicle 10 is shown in Figures 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although this disclosure envisions a commercial truck, the present invention can be applied to other types of vehicles. The vehicle 10 incorporates a camera mirror system (CMS) 15 (Figure 2) having driver-side and passenger-side camera arms 16a, 16b mounted on the outside of the vehicle cab 12. If necessary, the camera arms 16a, 16b may include conventional mirrors integrated with them, but the CMS 15 can be used to completely replace the mirrors. In further examples, each side may include multiple camera arms, each arm housing one or more cameras and / or mirrors.

[0026] Each of the camera arms 16a and 16b includes a base which is fixed to, for example, the cab 12. A slewing arm is supported by the base and can articulate relative to the base. At least one rear-facing camera 20a, 20b is positioned within each camera arm. Each of the external cameras 20a and 20b has an external field of view (FOV) that includes at least one of the Class II view and Class IV view (Figure 1B), which are legally defined views in the commercial truck industry. EX1 FOV EX2 The following is provided: A Class II view of a particular side of the vehicle 10 is part of the Class IV view of the same side of the vehicle 10. If necessary, multiple cameras can also be used in each camera arm 16a, 16b to provide these views. Each arm 16a, 16b can also provide a housing that encloses electronics configured to provide various functions of the CMS 15.

[0027] The first and second video displays 18a and 18b are positioned on or near the A-pillars 19a and 19b on the driver's side and passenger's side, respectively, within the vehicle cab 12, and display Class II and Class IV views of each side of the vehicle 10, which provide a rearward side view along the vehicle 10 captured by external cameras 20a and 20b.

[0028] If video of the Class V and Class VI views is also required, the camera housing 16c and camera 20c can be positioned at or near the front of the vehicle 10 to provide these views (Figure 1B). A third display 18c, positioned near the upper center of the windshield inside the vehicle cab 12, can be used to display the Class V and Class VI views facing forward of the vehicle 10 to the driver.

[0029] If a Class VIII view video is desired, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes part or all of the Class VIII zone of the vehicle 10. In such an example, the third display 18c may include one or more frames displaying the Class VIII view. Alternatively, additional displays can be added near the first, second, and third displays 18a, 18b, and 18c to provide dedicated displays for providing the Class VIII view.

[0030] Continuing to refer to Figures 1A, 1B, and 2, Figures 3A and 3B show vehicle 100 performing a reverse operation. In the initial position (Figure 3A), the trailer 110 has an initial angle of approximately 0 degrees relative to the vehicle cab 120, meaning it is aligned with the orientation of the vehicle cab 120. Alternatively, this angle can be expressed as 180 degrees relative to the cab 120. During the reverse process, especially when reversing while turning, the trailer 110 becomes oblique to the cab 120 (Figure 3B), forming a trailer angle that affects the reverse operation. The particular skew in Figure 3B is exaggerated for illustrative purposes compared to most expected angles.

[0031] To assist the driver in reversing, it is beneficial to ensure that the rear 112 of the trailer 110 is clearly visible to the driver on at least one display throughout the reversing operation. In some specific examples, it is desirable not only to include the rear 112 of the trailer 110, but also to center the rear 112 of the trailer 110 in the Class II view. However, as shown in Figure 3B, a static Class II view may result in the rear 112 of the trailer 110 extending beyond the boundary of the Class II view, even if the rear 112 remains within the Class IV field of view. To prevent the view of the rear 112 of the trailer 110 from being lost in the Class II view, or to maintain the center of the Class II view at the rear 112 of the trailer 110, the vehicles 10, 100 shown herein include an automatic panning function within the camera mirror system.

[0032] The automatic panning function estimates the trailer angle relative to the tractor at any given point in time using a combination of different trailer angle estimation and detection systems. The estimated trailer angle is assigned a "weight" corresponding to how likely it is to be accurate under current driving conditions and is provided to the fusion system within the vehicle controller. For example, a system based on wheel detection may have a high probability of accuracy (over 90%) in daytime conditions where black wheels stand out against the surrounding environment, and a low probability of accuracy (50-70%) in nighttime conditions where black wheels blend into the dark environment. Similarly, detection systems based on lane markers, such as straight-line detection systems using Huff transforms, may have a low probability of accuracy in dim weather conditions (rain, snow, fog, etc.) and a high probability of accuracy in clear conditions, and a bottom-edge detection system may have a high probability of accuracy for container trailers and a low probability of accuracy for tanker trailers.

[0033] The vehicle controller merges multiple trailer angle estimates using a weighted sum based on the confidence values ​​of each detection method. This weighted sum is called the raw estimate. The raw estimate is then combined with past estimates and filtered using a low-pass filter to provide a trailer angle that smoothly transitions to other vehicle systems that can benefit from the CMS's automatic panning function and the trailer angle estimates. In some examples, a camera mirror system can generate metadata corresponding to the side of the trailer, which can be used to prevent false positives and / or to provide further confidence in the trailer angle identified by other means. To reduce the phase delay in identifying the trailer angle estimate, and consequently the reporting delay, to a negligible level, the low-pass filter starts from the first measurement. This reduces the phase delay by providing the initial assigned value. Some delay still exists in such systems, but it is kept to a minimum so as not to affect the reversing system.

[0034] Continuing to refer to Figures 1-3B, Figure 4 shows the process for determining a more accurate trailer angle. First, process 300 identifies angle estimates in step 310, “Identify angle estimates”. The number of angle estimates identified varies depending on the specific system involved. In some examples, at least some of the angle estimates are purely visual, using feature tracking of objects identified in the CMS video feed (e.g., wheels, rear end, trailer markings, etc.) to determine the estimated trailer angle. Similarly, some angle estimates can be determined based on road edge and / or lane departure detection, comparison with positioning satellites and stored maps, hitch angle sensors, trailer end detection, lane detectors, radar sensors, lidar sensors, and other similar trailer angle detection systems.

[0035] Once trailer angle estimates are identified, the controller that identifies the trailer angle estimates assigns a confidence value to each trailer angle estimate in the “Identify Confidence Value” step 320. The method for determining the confidence level of each estimate depends on how the particular estimate is made and can be determined by a person skilled in the art using any appropriate technique. In some examples, the confidence value may be determined by weather conditions, lighting conditions, trailer type, historical accuracy data, and other characteristics that may be related to the likelihood of accuracy. The confidence value is expressed as a percentage of accuracy (for example, a trailer angle based on a 15-degree wheel is 94% accurate).

[0036] After determining the confidence level, the controller discards all estimates below the minimum confidence threshold in step 330, “Discard angles below confidence threshold.” In one example, estimates with less than 85% confidence are considered erroneous or inaccurate in some systems and are excluded from consideration. In another example, estimates with less than 90% confidence are considered erroneous. Discarding estimates below the minimum confidence threshold eliminates outliers that may result from inaccurate sensors, particularly poor conditions for a specific estimation technique, and other similar conditions that lead to inaccurate estimates. Eliminating extreme outliers improves the accuracy of the estimation. In some examples, discarding angles below the threshold can be omitted if a significant number of estimates are provided to the controller and / or if the low confidence levels do not indicate an error.

[0037] After all estimates below the threshold are discarded or step 330 is ignored, the fusion algorithm in the controller identifies the weighted sum of the angle estimates in step 340, which identifies the weighted sum. In one example of weighted average identification, the first trailer angle estimate of 14 degrees has a confidence level of 98%, the second trailer angle estimate of 10 degrees has a confidence level of 86%, and the third trailer angle estimate of 15 degrees has a confidence level of 94%. The predetermined confidence threshold is set to 85%, and all three values ​​are considered acceptable. The fusion algorithm multiplies each angle by its corresponding confidence level, sums the results, and divides the sum by the sum of the confidence levels. In the exemplary case, the fusion algorithm results in ((14*98)+(10*86)+(15*94)) / (98+86+94)=13.10 degrees. Therefore, in the exemplary case, the identified angle (or referred to as the raw measurement) is 13.10 degrees, and this angle is the output to the low-pass filter. The low-pass filtered angle is then output to the automated panning system. In practice, it is possible to use more than three angle estimates, and it is understood that the more angle estimates used, the more accurate the resulting value will be.

[0038] Once the weighted average of the estimated angles is determined, the controller adds the weighted average to a historical dataset that includes previously determined weighted averages for the current operation. In one example, trailer angle estimation is performed approximately every 200ms, and the historical dataset contains each subsequent entry in sequence. A low-pass filter is applied to the historical dataset that includes the newly determined weighted average. The low-pass filter smooths transitions, eliminating "jagged" or "abrupt" trailer angle transitions, providing a more accurate representation of changes in trailer angle over time, allowing the automated panning system and / or other vehicle systems to take these changes into account.

[0039] Continuing with Figure 4, Figure 5 schematically illustrates an exemplary automatic panning system for vehicle 410. Controller 420 receives image and other sensor information from vehicle 410, and Controller 420 uses angle detection module 424 to determine raw angle estimates from the received image and sensor information. In parallel with angle detection module 424, confidence value determination module 422 utilizes received data 421 that indicates conditions and other aspects affecting the confidence of each detected angle. Confidence value determination module 422 determines the confidence of each trailer angle detection, and the detection and confidence values ​​are provided to fusion module 426.

[0040] The fusion module 426 identifies a weighted average of the estimated trailer angles and fuses this weighted average with previous trailer angles stored in the trailer angle history 428. The fusion module 426 also applies a low-pass filter to the combined trailer angle and historical trailer angle data to identify a two-dimensional trailer angle. The two-dimensional trailer angle is an accurate estimate of the current trailer angle on a two-dimensional plane. The two-dimensional trailer angle is then converted to a three-dimensional trailer position based on the trailer angle and geographical features (e.g., hill gradient). The three-dimensional trailer angle is then provided to the auto-panning function, which automatically pans at least one camera view in the image. In one example, the auto-panning is configured so that the rear end of the trailer remains in a Class II view throughout the vehicle's operation. In other implementations, the auto-panning can maintain other objects or parts of objects in the view.

[0041] Merging multiple estimated angles from various sources into a single, more reliable trailer angle estimate can be used across multiple systems. Merging multiple trailer angle estimates into a single value also allows the system to reliably identify and pan for accurate trailer angles while taking into account the unreliability of certain trailer angle estimation techniques for specific conditions and / or types of trailers.

[0042] Although exemplary embodiments are disclosed, those skilled in the art will recognize that some modifications fall within the scope of the claims. Therefore, the following claims should be examined to determine their true scope and content.

Claims

Claim 1 A method for automatically panning a view for a commercial vehicle, comprising: identifying a plurality of estimated trailer angles between a central longitudinal axis of a tractor and a central longitudinal axis of a trailer, each estimated trailer angle being identified using a different estimation method; assigning a confidence value to each estimated trailer angle among the plurality of estimated trailer angles; identifying a weighted sum of the plurality of estimated trailer angles; and automatically panning a view of a camera mirror system on an electronic display based at least in part on the weighted sum and a current vehicle operation. A method as described above. Claim 2 The method according to claim 1, further comprising adding the identified weighted sum to an ordered list of past weighted sums and low-pass filtering the ordered list. Claim 3 The method according to claim 2, wherein automatically panning the view is based on the filtered ordered list. Claim 4 The method according to claim 1, further comprising discarding an estimated trailer angle having a confidence value less than a predetermined threshold after assigning the confidence value to each estimated trailer angle and before identifying the weighted sum. Claim 5 The method according to claim 4, wherein the predetermined threshold is at least 85%. Claim 6 The method according to claim 1, wherein the weighted sum is identified by multiplying a confidence value corresponding to each trailer angle estimate to obtain a weighted estimate, summing the weighted estimates, and dividing the sum of the weighted estimates by the sum of the confidence values. Claim 7 The method according to claim 1, wherein the different estimation methods include at least two of image-based wheel detection, image-based trailer rear end angle detection, image-based trailer marking angle detection, image-based wheel angle detection, road edge deviation detection, lane marker deviation detection, and hitch angle sensors. Claim 8 The method according to claim 1, wherein automatically panning the view includes adjusting a Class II view within a Class IV view. Claim 9 The method according to claim 8, wherein automatically panning the view includes maintaining a rear end of the trailer within the Class II view. Claim 10 The method according to claim 1, further comprising continuously repeating the method to generate real-time trailer angle monitoring. Claim 11 Identifying the plurality of estimated trailer angles, assigning a confidence value to each of the estimated trailer angles among the plurality of estimated trailer angles, and identifying a weighted average of the plurality of estimated trailer angles are performed within the camera mirror system controller of the vehicle, the method according to claim 1.

12. At least one of identifying the plurality of estimated trailer angles, assigning a confidence value to each of the estimated trailer angles among the plurality of estimated trailer angles, and identifying a weighted average of the plurality of estimated trailer angles is performed remotely from a camera mirror system controller and transmitted to the camera mirror system controller, the method according to claim 1.

13. A camera mirror system for a vehicle, A first camera having a first field of view, A controller configured to receive the first field of view and output a part of the first field of view to a first display Comprising, The controller includes a trailer angle detection module configured to identify a plurality of trailer angle estimates for a trailer angle between a central longitudinal axis of a tractor and a central longitudinal axis of a trailer, a confidence value module configured to identify a confidence value for each trailer angle estimate, and a fusion module configured to fuse the plurality of trailer angle estimates and the confidence values into a single trailer angle estimate. A camera mirror system that automatically pans at least one view of the camera mirror system based at least in part on the single trailer angle estimate so that features of the trailer are maintained within the at least one view.

14. The fusion module of the camera mirror system according to claim 13 is configured to identify a weighted sum trailer angle based on the plurality of trailer angle estimates and corresponding confidence values.

15. The fusion module of the camera mirror system according to claim 14 is further configured to add the weighted sum trailer angle to a past weighted sum trailer angle dataset and perform low-pass filtering on the past weighted sum trailer angle dataset.

16. Each of the trailer angle estimates among the plurality of trailer angle estimates is identified using a different angle estimation method, the camera mirror system according to claim 13.

17. The camera mirror system according to claim 13, wherein the controller is further configured to add the single trailer estimate to an ordered list of past trailer angle estimates and low-pass filter the ordered list. **Claim 18** The camera mirror system according to claim 17, wherein the automatic panning is at least partially based on the ordered list that has been low-pass filtered.

Images