Automatic Pan Camera Mirroring System Including Weighted Trailer Angle Estimation

Abstract

The present disclosure provides an automatic pan camera mirroring system including weighted trailer angle estimation. The present disclosure also provides a method for automatically panning a perspective for a commercial vehicle, including determining a plurality of estimated trailer angles. Different estimation methods are used to determine each of the estimated trailer angles, and the method assigns a confidence value to each of the plurality of estimated trailer angles. The method determines a weighted sum of the plurality of estimated trailer angles and automatically pans the perspective at least in part based on the weighted sum and current vehicle operation.

Description

Technical Field

[0001] The present disclosure relates to a camera mirror system (CMS) for commercial trucks, and more particularly to a CMS having an automatic translation feature including fused trailer angle estimation. Background Art

[0002] In commercial vehicles, mirror replacement systems and camera systems for supplementing mirror views are utilized to enhance the ability of a vehicle operator to see the surrounding environment. A camera mirror system (CMS) utilizes one or more cameras to provide an enhanced field of view to a vehicle operator. In some examples, the mirror replacement system covers a larger field of view than a conventional mirror, or includes perspectives that cannot be fully obtained via a conventional mirror.

[0003] In certain operations, such as trailer backing maneuvers, fixed perspectives such as those provided by a fixed mirror or a fixed field of view camera may not provide a complete perspective of the operation and may not present the desired information to the operator. A manual translation system that manually adjusts the physical camera or mirror angle may require frequent stops of the maneuver to adjust the provided perspective and may have insufficient granularity of adjustment settings.

[0004] Some example systems attempt to minimize the problems of manual translation by implementing automatic or semi-automatic translation. Such systems rely on potentially inaccurate estimates of the trailer angle, and the kinematic models of vehicle operation (especially during backing operations) may have difficulty accounting for the potential variability of the trailer angle estimate. Summary of the Invention

[0005] An exemplary method for automatically translating a perspective for a commercial vehicle includes: determining a plurality of estimated trailer angles, each estimated trailer angle being determined using a different estimation method; assigning a confidence value to each of the plurality of estimated trailer angles; determining a weighted sum of the plurality of estimated trailer angles; and automatically translating the perspective at least in part based on the weighted sum and the current vehicle operation.

[0006] Another example of the above method for automatically translating a perspective for a commercial vehicle further includes: adding the determined weighted sum to an ordered list of historical weighted sums and applying a low-pass filter to the ordered list.

[0007] In another example of any of the above methods for automatically translating a perspective for a commercial vehicle, the perspective is automatically translated based on the filtered ordered list.

[0008] Another example of any of the above methods for automatically translating a perspective for a commercial vehicle further includes: discarding any estimated trailer angles having a confidence value below a predefined threshold after assigning a confidence value to each estimated trailer angle and before determining the weighted sum.

[0009] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, the predefined threshold is at least 85%.

[0010] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, a weighted sum is determined by multiplying each trailer angle estimate by a corresponding confidence value, summing the weighted estimates, and dividing the sum of the weighted estimates by the sum of the confidence values.

[0011] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, different estimation methods include at least two of: image-based wheel detection, image-based rear edge angle detection of the trailer, image-based trailer marker angle detection, image-based wheel angle detection, road edge deviation detection, lane marker deviation detection, a towing angle sensor, and a wheel angle sensor.

[0012] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, the automatically translated perspective includes adjusting a Class II perspective within a Class IV perspective.

[0013] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, the automatically translated perspective includes keeping the rear edge of the trailer within a Class II perspective.

[0014] Another example of any of the above methods for automatically translating the perspective for a commercial vehicle further includes continuously iterating the method to generate real-time trailer angle monitoring.

[0015] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, it is performed within the camera mirror system controller of the vehicle: determining a plurality of estimated trailer angles, assigning a confidence value to each of the plurality of estimated trailer angles, and determining a weighted average of the plurality of estimated trailer angles.

[0016] In another example of any of the above methods for automatically translating the perspective for a commercial vehicle, at least one of the following is performed remotely from the camera mirror system controller: determining a plurality of estimated trailer angles, assigning a confidence value to each of the plurality of estimated trailer angles, and determining a weighted average of the plurality of estimated trailer angles, and transmitting it to the camera mirror system controller.

[0017] In an exemplary embodiment, a camera mirroring system for a vehicle includes: a first camera having a first field of view; a controller configured to receive the first field of view and output a subset of the first field of view to a first display, the controller including: a trailer angle detection module configured to determine a plurality of trailer angle estimates; a confidence value module configured to determine a confidence value for each trailer angle estimate; and a fusion module configured to fuse the plurality of trailer angle estimates and the confidence values into a single trailer angle estimate; and automatically translate at least one perspective of the camera mirroring system based at least in part on the single trailer angle estimate such that a feature of the trailer remains within the at least one perspective.

[0018] In another example of the above camera mirroring system for a vehicle, the fusion module is configured to: determine a weighted sum trailer angle based on the plurality of trailer angle estimates and corresponding confidence values.

[0019] In another example of any of the above camera mirroring systems for a vehicle, the fusion module is further configured to: add the weighted sum trailer angle to a historical weighted sum trailer angle data set and perform a low-pass filter on the historical weighted sum trailer angle data set.

[0020] In another example of any of the above camera mirroring systems for a vehicle, each trailer angle estimate of the plurality of trailer angle estimates is determined via a different angle estimation method.

[0021] In another example of any of the above camera mirroring systems for a vehicle, the controller is further configured to: add the single trailer estimate to an ordered list of historical trailer angle estimates and perform a low-pass filter on the ordered list.

[0022] In another example of any of the above camera mirroring systems for a vehicle, the automatic translation is at least partially based on the low-pass filtered ordered list. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present disclosure may be further understood in connection with the following detailed description when taken in conjunction with the accompanying drawings, in which:

[0024] Figure 1A is a schematic front view of a commercial truck having a camera mirroring system (CMS) for providing at least Class II and Class IV perspectives.

[0025] Figure 1B is a schematic top front view of a commercial truck having a camera mirroring system providing Class II, Class IV, Class V, and Class VI perspectives.

[0026] Figure 2 is a schematic top perspective view of a cab including a display and an interior camera.

[0027] Figure 3A The vehicle at the start of a reverse maneuver is shown, without a trailer angle.

[0028] Figure 3B The vehicle mid - reverse maneuver with a high trailer angle is shown.

[0029] Figure 4 A method for obtaining an estimated weighted trailer angle is shown.

[0030] Figure 5 A system for determining an accurate trailer angle based on the estimated weighted angle and automatically translating a camera mirror system is shown.

[0031] Any one of the foregoing paragraphs, claims, or examples, examples, and alternatives of the following description and drawings, including their various aspects or corresponding individual features, may be taken independently or in any combination. Features described in connection with one embodiment apply to all embodiments unless the features are incompatible. Detailed Description

[0032] In Figure 1A and Figure 1B a schematic illustration of a commercial vehicle 10 is shown. The vehicle 10 includes a vehicle cab or tractor 12 for pulling a trailer 14. Although commercial trucks are envisioned in the present disclosure, the invention may also be applied to other types of vehicles. The vehicle 10 includes a camera mirror system (CMS) 15 ( Figure 2 ), the camera mirror system 15 having a driver - side camera arm 16a and a passenger - side camera arm 16b mounted external to the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated with the two arms, although the CMS 15 may be used to completely replace the mirrors. In additional examples, each side may include multiple camera arms, each arm housing one or more cameras and / or mirrors.

[0033] Each of the camera arms 16a, 16b includes a base fixed to, for example, the cab 12. A pivot arm is supported by the base and is pivotable relative to the base. At least one rear - facing camera 20a, 20b is respectively arranged on the camera arms. The external cameras 20a, 20b respectively provide an external field of view FOV EX1 , FOV EX2 , each field of view including at least one of a Class II perspective and a Class IV perspective ( Figure 1B), these perspectives are from the legal requirements perspective in the commercial truck industry. The Class II perspective on a given side of the vehicle 10 is a subset of the Class IV perspective on the same side of the vehicle 10. If desired, multiple cameras can also be used on each camera arm 16a, 16b to provide these perspectives. Each arm 16a, 16b can also provide a housing that encloses electronics configured to provide various features of the CMS15.

[0034] The first video display 18a and the second video display 18b are arranged on each side of the driver's side and the passenger's side within the vehicle cab 12, on or near the A-pillars 19a, 19b, to display the Class II perspective and the Class IV perspective on its respective side of the vehicle 10, which provides a rearward-facing side view of the vehicle 10 captured by the external cameras 20a, 20b.

[0035] If video of the Class V perspective and the Class VI perspective is also desired, the camera housing 16c and the camera 20c can be arranged at or near the front of the vehicle 10 to provide these perspectives ( Figure 1B ). A third display 18c arranged near the top center of the windshield within the cab 12 can be used to display the Class V perspective and the Class VI perspective facing forward of the vehicle 10 to the driver.

[0036] If video of the Class VIII perspective is desired, camera housings can be provided on the sides and rear of the vehicle 10 to provide a field of view that includes part or all of the Class VIII area of the vehicle 10. In such an example, the third display 18c can include one or more frames displaying the Class VIII perspective. Alternatively, additional displays can be added near the first display 18a, the second display 18b, and the third display 18c, and a display dedicated to providing the Class VIII perspective can be provided.

[0037] Continuing to refer to Figure 1A 、 Figure 1B and Figure 2 , Figure 3A and Figure 3B shows the vehicle 100 during a backing maneuver. In the initial position ( Figure 3A ), the trailer 110 has an initial trailer angle of approximately zero degrees relative to the cab 120, meaning the trailer 110 is aligned with the orientation of the cab 120. Alternatively, this angle can be expressed as 180 degrees relative to the cab 120. During the backing process, especially when backing through a turn, the trailer 110 skews relative to the cab 120 ( Figure 3B ), creating a trailer angle that affects the backing maneuver. For illustrative purposes, the specific skew of Figure 3B is exaggerated relative to most expected angles.

[0038] To assist the driver in reverse maneuvers, it is beneficial to ensure that the rear 112 of the trailer 110 is visible to the driver in at least one display during the reverse maneuver. In some specific examples, it is desirable to include not only the rear 112 of the trailer 110, but also to center the Class II view around the rear 112 of the trailer 110. However, as Figure 3B shown in

[0039] , even when the rear 112 remains within the Class IV field of view, a static Class II view can cause the rear 112 of the trailer 110 to extend beyond the boundaries of the Class II view. To prevent the loss of the view of the rear 112 of the trailer 110 in the Class II view or to keep the Class II view centered around the rear 112 of the trailer 110, the vehicles 10, 100 shown herein include an automatic pan feature within the camera mirror system.

[0040] The automatic pan feature uses a combination of different trailer angle estimation and detection systems to estimate the trailer angle relative to the towing vehicle at any given time. The estimated trailer angle is assigned a "weight" and provided to a fusion system within the vehicle controller, and this "weight" corresponds to the likelihood that the estimated trailer angle is accurate under the current operating conditions. For example, a wheel detection-based system can have a high accuracy probability (greater than 90%) under daylight conditions (when the black wheels are prominent against the surrounding environment) and a low accuracy probability (50% - 70%) under nighttime conditions (when the black wheels blend into the dark environment). Similarly, a lane marking-based detection system, such as a line detection system using Hough transforms, can have a lower accuracy probability under fuzzy weather conditions (rain, snow, fog, etc.), a higher accuracy probability under clear weather conditions, and a bottom edge detection system can have a high accuracy probability in the case of a container trailer and a low accuracy probability in the case of an oil tanker trailer.

[0041] Continue to refer to Figures 1A to 3B , Figure 4 which shows a process for determining a more accurate trailer angle. Initially, process 300 determines an angle estimate in "Determine Angle Estimate" step 310. The number of angle estimates determined may vary depending on the particular system involved. In some examples, at least some of the angle estimates are purely vision-based and utilize feature tracking of objects (e.g., wheels, rear edges, trailer markings, etc.) identified in the CMS video transmission to determine the estimated trailer angle. Similarly, some angle estimates can be determined based on deviation from road edges and / or lane detection, comparison of positioning satellites with a stored map, tow angle sensors, trailer edge detection, lane detectors, radar sensors, lidar sensors, and any other similar trailer angle detection systems.

[0042] Once the trailer angle estimate is determined, the controller that determined the trailer angle estimate assigns a confidence value to each trailer angle estimate in "Determine Confidence Value" step 320. The method for determining the confidence for each estimate depends on how that particular estimate is performed and can be determined by those skilled in the art using any appropriate technique. In some examples, the confidence value can depend on weather conditions, lighting conditions, trailer type, historical accuracy data, and any other characteristics that may be related to the likelihood of accuracy. The confidence value is represented as a percentage of accuracy (e.g., a wheel-based trailer angle of 15 degrees has a 94% chance of being accurate).

[0043] After determining the confidence value, the controller discards all estimates below a minimum confidence threshold in "Discard Angles Below Confidence Threshold" step 330. In one example, it is assumed that estimates below 85% confidence are incorrect or inaccurate in some systems and are disregarded. In another example, it is assumed that estimates below 90% confidence are incorrect. Discarding estimates below the minimum confidence value threshold eliminates outliers that may occur due to inaccurate sensors, particularly due to poor conditions for a given estimation technique and any other similar conditions that result in inaccurate estimates. Eliminating extreme outliers increases the accuracy of the estimates. In some examples, the discarding of angles below the threshold can be omitted if a large number of estimates are provided to the controller and / or low confidence values do not indicate an error.

[0044] After discarding all estimates below a threshold or bypassing step 330, the fusion algorithm in the controller determines the weighted sum of the angle estimates in the "Determine Weighted Sum" step 340. In an example of weighted average determination, a first trailer angle estimate of 14 degrees has a confidence of 98%, a second trailer angle estimate of 10 degrees has a confidence of 86%, and a third trailer angle estimate of 15 degrees has a confidence of 94%. A predefined confidence threshold is set at 85%, and all three of the above values are considered acceptable. The fusion algorithm multiplies each angle by its corresponding confidence, sums the results, and divides the sum by the sum of the confidences. In an exemplary case, the fusion algorithm arrives at: ((14 * 98) + (10 * 86) + (15 * 94)) / (98 + 86 + 94) = 13.10 degrees. Thus, in the exemplary case, the determined angle (or the so-called raw measurement) is 13.10 degrees, and this angle is output to a low-pass filter. The low-pass filtered angle is output to the automatic translation system. It should be understood that the actual implementation may utilize substantially more than three angle estimates, and the more angle estimates used, the more accurate the resulting value will be.

[0045] When the weighted average of the estimated angles has been determined, the controller adds this weighted average to a historical dataset that includes previously determined weighted averages of current operations. In one example, the trailer angle estimate is performed approximately every 200 milliseconds, and the historical dataset includes each subsequent entry in sequence. A low-pass filter is applied to the historical dataset, including the newly determined weighted average. The low-pass filter smooths the transitions and eliminates "jerky" or "abrupt" trailer angle transitions, thus providing a more accurate representation of the change in the trailer angle over time and allowing the automatic translation system and / or any other vehicle system to account for the accurate changes.

[0046] Continuing to refer Figure 4 , Figure 5 FIG. schematically shows an example automatic translation system for vehicle 410. The controller 420 receives images and other sensor information from the vehicle 410, and the controller 420 uses the angle detection module 424 to determine the raw angle estimate based on the received images and sensor information. The confidence value determination module 422, together with the angle detection module 424, utilizes the received data 421 indicating the conditions and any other aspects that affect the confidence of each detected angle. The confidence value determination module 422 determines the confidence of each trailer angle detection, and the detection and confidence values are provided to the fusion module 426.

[0047] The fusion module 426 determines a weighted average of the estimated trailer angles and fuses this weighted average with the previous trailer angles stored in the trailer angle history 428. The fusion module 426 also applies a low-pass filter to the combined trailer angle and historical trailer angle data to determine a two-dimensional trailer angle. The two-dimensional trailer angle is an accurate estimate of the current trailer angle on a two-dimensional plane. The two-dimensional trailer angle is converted to a three-dimensional trailer position based on the trailer angle and geographical features (e.g., slope). The three-dimensional trailer angle is then provided to the auto-pan feature, and the camera mirroring system automatically pans at least one camera view within the image. In one example, the auto-pan is configured to ensure that the rear edge of the trailer remains within a Class II view throughout the operation of the vehicle. In other embodiments, the auto-pan may keep other objects or portions of objects within the view.

[0048] Fuse multiple estimated angles from different sources into a single more reliable trailer angle estimate that can be used across multiple systems. Fusing multiple trailer angle estimates into a single value also allows the system to account for the unreliability of certain trailer angle estimation techniques for certain conditions and / or certain types of trailers while still reliably determining and panning an accurate trailer angle.

[0049] Although example embodiments have been disclosed, those skilled in the art will recognize that some modifications fall within the scope of the claims. Accordingly, the appended claims should be studied to determine the true scope and content of the invention.

Claims

1. A method for automatically translating the perspective for a commercial vehicle, the commercial vehicle including a trailer and a tractor for pulling the trailer, the trailer having a trailer angle skewed relative to the tractor, the tractor having a camera mirroring system, the camera mirroring system including a camera disposed outside the tractor for capturing an external field of view and an electronic display disposed inside the tractor for displaying the external field of view, the external field of view including at least one of a Class II perspective and a Class IV perspective, the method comprising: Determining a plurality of estimated trailer angles, each estimated trailer angle being determined using a different estimation method; Assigning a confidence value to each of the plurality of estimated trailer angles; Determining a weighted sum of the plurality of estimated trailer angles; wherein the weighted sum is determined by multiplying each trailer angle estimate by the corresponding confidence value, summing the weighted estimates, and dividing the sum of the weighted estimates by the sum of the confidence values; and Automatically translating the perspective of the camera mirroring system on the electronic display at least in part based on the weighted sum and current vehicle operation; The method further includes adding the determined weighted sum to an ordered list of historical weighted sums and performing a low-pass filter on the ordered list; wherein the automatic translation of the perspective is based on the filtered ordered list; wherein the automatic translation of the perspective includes: adjusting the Class II perspective within the Class IV perspective.

2. The method according to claim 1, further comprising discarding any estimated trailer angle having a confidence value below a predefined threshold after assigning the confidence value to each estimated trailer angle and before determining the weighted sum.

3. The method according to claim 2, wherein, The predefined threshold is at least 85%.

4. The method according to claim 1, wherein, The different estimation methods include at least two of: image-based wheel detection, image-based trailer rear edge angle detection, image-based trailer marker angle detection, image-based wheel angle detection, road edge deviation detection, lane marker deviation detection, and a traction angle sensor.

5. The method according to claim 1, wherein The automatic translation of the perspective further includes: keeping the rear edge of the trailer within the Class II perspective.

6. The method according to claim 1, further comprising continuously iterating the method to generate real-time trailer angle monitoring.

7. The method according to claim 1, wherein Executing within the camera mirroring system controller of the vehicle: determining the plurality of estimated trailer angles, assigning the confidence value to each of the plurality of estimated trailer angles, and determining the weighted average of the plurality of estimated trailer angles.

8. The method according to claim 1, wherein Remotely executing at least one of the following from the camera mirroring system controller: determining the plurality of estimated trailer angles, assigning the confidence value to each of the plurality of estimated trailer angles, and determining the weighted average of the plurality of estimated trailer angles, and transmitting it to the camera mirroring system controller.

9. A camera mirroring system for a vehicle, the vehicle including a trailer and a towing vehicle for towing the trailer, the trailer having a trailer angle skewed relative to the towing vehicle, the camera mirroring system comprising: A first camera having a first field of view; A controller configured to receive the first field of view and output a subset of the first field of view to a first display; The controller includes: a trailer angle detection module configured to determine a plurality of trailer angle estimates for estimating a plurality of the trailer angles; a confidence value module configured to determine a confidence value for each trailer angle estimate; and a fusion module configured to fuse the plurality of trailer angle estimates and the confidence values into a single trailer angle estimate; and Automatically translate at least one perspective of the camera mirroring system at least partially based on the single trailer angle estimate such that features of the trailer remain within the at least one perspective, the at least one perspective including at least one of a Class II perspective and a Class IV perspective; Wherein the fusion module is configured to: determine a weighted sum trailer angle based on the plurality of trailer angle estimates and corresponding confidence values; wherein the weighted estimate is determined by multiplying each trailer angle estimate by the corresponding confidence value, summing the weighted estimates, and dividing the sum of the weighted estimates by the sum of the confidence values to determine the weighted sum; The fusion module is further configured to: add the weighted sum trailer angle to a historical weighted sum trailer angle data set and perform low-pass filtering on the historical weighted sum trailer angle data set; The automatic translation is at least partially based on the low-pass filtered ordered list; Automatically translating the perspective includes: adjusting the Class II perspective within the Class IV perspective.

10. The camera mirroring system according to claim 9, wherein, Each trailer angle estimate in the plurality of trailer angle estimates is determined via different angle estimation methods.

11. The camera mirroring system according to claim 9, wherein, The controller is further configured to: add the single trailer estimate to an ordered list of historical trailer angle estimates and perform low-pass filtering on the ordered list.

Images