Camera monitoring system for commercial vehicles including wheel position estimation

Abstract

A method for estimating trailer wheel positions includes identifying a first set of wheel positions in at least a first image. Each wheel position in the first set of wheel positions is associated with a corresponding trailer angle. A subset of the wheel positions is identified as false alarms in the first set of wheel positions, and the false alarms are removed from the first set of wheel positions. A quadratic regression is applied to the first set of wheel positions, and a parabolic curve is determined that associates a y position in at least the first image with an x position in at least the first image. And determining a trailer angle. A current wheel position is estimated by applying the determined trailer angle to the parabolic curve.

Description

Technical Field

[0001] The present disclosure relates to wheel tracking using a camera monitoring system (CMS) of a vehicle, and more particularly to systems and methods for tracking wheel positions when the wheels are hidden. Background Art

[0002] Camera monitoring systems, such as those used to supplement mirror views, are utilized in commercial vehicles to enhance the vehicle operator's ability to see the surrounding environment. A camera monitoring system (CMS) utilizes one or more cameras to provide an enhanced field of view to the vehicle operator. In some examples, the camera monitoring system covers a larger field of view than conventional mirrors, or includes views that cannot be fully obtained via conventional mirrors.

[0003] Semi-autonomous driver assistance systems, camera monitoring systems, electronic stability program systems, and other vehicle systems use or require knowledge of the location of various vehicle features throughout the operation of the vehicle. These features can be the real-world location or the location in an image of one or more rear wheels of a trailer. There are systems for tracking the location of wheels when the wheels are visible within the field of view of a rear-mounted camera using a camera monitoring system. However, when the trailer is at a small trailer angle, the rear trailer wheels are not visible in the field of view of the driver-side camera or the passenger-side camera, and the real-world location of the wheels as well as the location of the wheels in the image are unknown. In addition, when the field of view of the driver-side camera or the passenger-side camera is obscured or otherwise unavailable, it is impossible to use existing systems to determine the location of the wheels in the image generated by the obscured camera. Summary of the invention

[0004] A method for estimating trailer wheel position according to an example of the present disclosure includes identifying a first set of wheel positions in at least a first image. Each wheel position in the first set of wheel positions is associated with a corresponding trailer angle. A subset of wheel positions is identified as false positives in the first set of wheel positions, and the false positives are removed from the first set of wheel positions. A quadratic regression is applied to the first set of wheel positions, and a parabolic curve is determined that associates at least a y position in the first image with at least an x ​​position in the first image. A trailer angle is determined. A current wheel position is estimated by applying the determined trailer angle to the parabolic curve.

[0005] In another example of the foregoing, the method includes identifying a second set of wheel positions in at least a second image, each wheel position in the second image being associated with a corresponding trailer angle.

[0006] In another example of any of the foregoing, the method includes identifying a subset of wheel positions in the second set of wheel positions as false positives; and removing the false positives from the first set of wheel positions.

[0007] In another example of any of the foregoing, the first image is one of a Class II view and a Class IV view, and the second image is a Class II view and a Class IV view on an opposite side of the vehicle.

[0008] In another example of any of the foregoing, the parabolic curve is defined according to y=ax^2+bx+c, where "a" is not equal to zero, y is the position of the wheel on the y-axis in the image, and x is the position of the wheel on the x-axis in the image.

[0009] In another example of any of the foregoing, the method includes providing the determined wheel position to at least one additional vehicle system.

[0010] In another example of any of the foregoing, the at least one additional vehicle system includes at least one of an advanced driver assistance system, a camera monitoring system, and an electronic stability program.

[0011] In another example of any of the foregoing, the current wheel position is a position within the image.

[0012] In another example of any of the foregoing, the current wheel position is a real-world three-dimensional position of the wheel relative to the vehicle.

[0013] In another example of any of the foregoing, the method includes identifying wheel trajectories relative to the towing vehicle in image space using a parabolic curve.

[0014] In another example of any of the foregoing, the method includes identifying a wheel trajectory relative to the towing vehicle in real-world space using the parabolic curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present disclosure may be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0016] Figure 1A is a schematic front view of a commercial truck having a camera monitoring system (CMS) for providing at least a Class II view and a Class IV view.

[0017] Figure 1B is a schematic top view of a commercial truck with a camera monitoring system providing Class II, Class IV, Class V, and Class VI views.

[0018] Figure 2 is a schematic top perspective view of a vehicle cab including a display and interior cameras.

[0019] Figure 3A A camera monitoring system view is shown including a single view of the vehicle trailer at a medium to large trailer angle.

[0020] Figure 3B A camera monitoring system view is shown including two views of a vehicle trailer at a small trailer angle.

[0021] Figure 4 A dataset of trailer wheel positions within one or more images is shown.

[0022] Figure 5 shows the removal of false positive detections Figure 4 of the dataset.

[0023] Figure 6 A control process for identifying wheel position estimates is shown.

[0024] Equation 1 shows the matrix operations for identifying the quadratic wheel estimation function used to estimate the wheel position.

[0025] The embodiments, examples and alternatives of the preceding paragraphs, claims or the following description and drawings, including any of their various aspects or corresponding individual features, may be adopted independently or in any combination. Features described in conjunction with one embodiment are applicable to all embodiments, unless these features are incompatible. DETAILED DESCRIPTION

[0026] A schematic diagram of a commercial vehicle 10 is shown in Figure 1A and Figure 1B Shown in. Figure 2 1 is a schematic top perspective view of a cab of a vehicle 10 including a display and an internal camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. Although commercial trucks are contemplated in the present disclosure, the present invention may also be applied to other types of vehicles. The vehicle 10 includes a camera monitoring system (CMS) 15 ( Figure 2 ), which has a driver-side camera arm 16a and a passenger-side camera arm 16b mounted to the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, but the CMS 15 may be used to completely replace the mirrors. In another example, each side may include multiple camera arms, each arm housing one or more cameras and / or mirrors.

[0027] Each of the camera arms 16a, 16b includes a base fixed to, for example, the cab 12. The pivot arm is supported by the base and can be articulated relative to the base. At least one rear-facing camera 20a, 20b is disposed within the camera arm, respectively. The external cameras 20a, 20b provide an external field of view FOV, respectively. EX1 FOV EX2 , each field of view includes at least one of a Class II view and a Class IV view ( Figure 1B), which are legally required views in the commercial trucking industry. If desired, multiple cameras may also be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. Any reference to "class" views is not intended to be limiting, but rather is intended as an example of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing that encloses electronic devices configured to provide the various features of the CMS 15.

[0028] A first video display 18a and a second video display 18b are arranged on each of the driver's side and the passenger side within the vehicle cab 12 on or near the A-pillars 19a, 19b to display Class II and Class IV views on the corresponding sides of the vehicle 10, and these video displays provide rearward side views along the vehicle 10 captured by the external cameras 20a, 20b.

[0029] If video of Class V and / or Class VI views is also desired, camera housing 16c and camera 20c may be disposed at or near the front of vehicle 10 to provide those views ( Figure 1B ). A third display 18c disposed within the cab 12 near the top center of the windshield may be used to display to the driver Class V and Class VI views toward the front of the vehicle 10. The displays 18a, 18b, 18c face a driver area 24 within the cab 22 where an operator sits in a driver's seat 26. The location, size, and field(s) of the view streamed to any particular display may vary from the configuration described in this disclosure and still incorporate the disclosed invention.

[0030] If video of a Class VIII view is desired, camera housings may be provided on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII area of ​​the vehicle 10. In such an example, the third display 18c may include one or more frames that display the Class VIII view. Alternatively, an additional display may be added adjacent to the first display 18a, the second display 18b, and the third display 18c and provide a display dedicated to providing a Class VIII view.

[0031] Continue to refer Figure 1A to Figure 2 , Figure 3A The rear view displayed to the vehicle operator via the CMS described above is schematically shown when the trailer 110 is at a medium to large angle (e.g., greater than 10 degrees). When at a medium to large trailer angle, the trailer 110 is minimally visible in the relative view, and the relative view is omitted if the trailer 110 is visible at all. Figure 3BThe trailer 110 is schematically shown in both the driver-side display 102 and the passenger-side display 104 when the trailer is at a small (e.g., less than 10 degrees) angle. When the trailer 110 is at a medium to large angle, the rearmost wheels 112 are visible in the corresponding views 104. In contrast, when the trailer 110 is at a low trailer angle ( Figure 3B ), the wheels 112 are not visible. Because they are not visible, the wheels 112 are said to be hidden. Although the exact angle at which the wheels 112 become hidden will depend on the position of the camera generating the field of view and the length of the trailer 110, the wheels 112 are typically hidden at a small angle (e.g., between 10 degrees and -10 degrees).

[0032] To implement vehicle systems that rely on the position of the wheels 112, such as advanced driver assistance systems, camera monitoring systems, electronic stability programs, and similar vehicle systems, the CMS monitors the views 102, 104 and identifies the wheel positions 112 during any operating condition where the wheels 112 are visible. Existing object tracking systems can identify the wheels 112 when they are visible and track the center points 114 of the wheels 112 as they travel through the image. The positions in the images can then be converted back to real-world 3D positions using known systems. In addition to using these monitored wheel positions, the CMS also generates a data set from each image, where each point in each data set identifies the center point 114 of the wheel 112 in the image, and coordinates the center point 114 of the wheel 112 with the angle of the trailer 110 where the wheel position was detected. The angle of the trailer 110 is detected using one of the trailer angle sensors, CMS image analysis, or a combination of both.

[0033] Based on the relationship established using wheel 112 detection and trailer angle when one or more wheels are visible, CMS 15 is configured to identify a quadratic equation that relates the position of the wheels within the image to the trailer angle. Based on this correlation, CMS 15 can then identify estimated wheel positions during periods when wheels 112 are not visible in the image. For example, this can be done when wheels 112 are Figure 3B Hide occurs during the low angles shown, or when the wheel position is hidden due to obstacles, weather or other external influences.

[0034] The estimated wheel positions are provided to any CMS or other vehicle systems that use this information, thereby providing continuous wheel positions to the CMS or other vehicle systems.

[0035] Continue to refer Figures 1A to 3B , Figure 4 , Figure 5 and Figure 6 Shown for use Figure 6The control process 500 shown generates an on-the-road process of wheel position estimates specific to the currently attached trailer 110. Initially, the raw wheel positions 502 (e.g. Figure 4 The raw wheel positions 502 are generated using any upstream wheel detection process or algorithm and may include one or more error detections 504. Each of the wheel positions 502 and error detections 504 includes a corresponding trailer angle calculated based on the wheel position within the image. The trailer angle is calculated using any conventional trailer angle calculation method, including a trailer angle sensor, a tractor-trailer kinematic model, image-based detection, a combination of sensors and image-based detection, or any other conventional detection method. Figure 4 and Figure 5 As shown, x and y are the image boundaries, and the location of the data point in the xy plane is the location of the wheel detection within the image.

[0036] After receiving the raw wheel positions and trailer angles, a plausibility check is applied to the raw wheel positions 502. The plausibility check uses a kinematic model to determine a ground truth trailer angle and compares the ground truth trailer angle to the received raw wheel positions and trailer angle. When the angle determined via the kinematic model differs from the received angle by at least a certain amount, a false positive 504 is detected. The false positive 504 corresponds to a wheel position that is unachievable and / or impossible given known information inputs (e.g., speed, steering angle, slope, etc.) and information outputs determined from the kinematic model 510 (e.g., trailer angle). The unachievable and / or impossible wheel positions 504 are filtered out from the data set 502 using a filter 520, thereby generating a false positive. Figure 5 Filtered data set 524 is shown.

[0037] In one example, the kinematic model 510 uses only the tractor forward driving maneuvers to ensure accuracy. Although it should be understood that using the kinematic model approach defined herein may inadvertently filter out some accurate wheel detections, the process can operate on less than a complete set of accurate wheel detections as long as there are no or minimal false detections.

[0038] After generating the filtered data set 524, the process 500 applies a quadratic regression 530 to the data set. In one example, the quadratic regression 530 is a matrix simplification according to Equation 1 and provides a resulting quadratic formula of the form y=a*x^2+b*x+c, where a is not equal to 0 and y is the pixel coordinate of the wheel position. In an alternative example, an alternative form of quadratic regression may be utilized to generate a quadratic equation relating trailer angle to the position of the wheels within the image. The controller running the process 500 then receives updated coefficients from the determined equation and provides the new wheel position curve (the resulting quadratic formula) to the CMS 15 and any other vehicle systems that may need to estimate the wheel position.

[0039] While the above examples estimate wheel positions within an image, it will be appreciated that the wheel positions may be converted to 3D world model positions before or after applying the methods described herein, and that the same process may be utilized to estimate actual wheel positions in addition to estimating wheel positions within an image, which describe the real-world three-dimensional position of a wheel relative to a defined point on the vehicle.

[0040] The estimation system and process described above generates estimated wheel positions using images generated by views 102, 104. The CMS controller and / or other vehicle system controller converts the estimated image positions into corresponding three-dimensional real-world positions and may use the corresponding three-dimensional positions as desired.

[0041] In at least one example, the estimated wheel position is provided from the CMS controller to a trailer end detection module within the CMS system. The trailer end detection module may be a software module also located within the controller or a separate software system in communication with the CMS controller. The trailer end detection module uses the wheel position to help identify the trailer end and marks the trailer end position in the CMS display to improve the vehicle operator's situational awareness. In another example, the wheel position may also be used by the CMS to estimate the position of the entire wheelbase, which may then be used within the CMS.

[0042] In another example, the estimated wheel position is provided to an advanced driver assistance system within the vehicle and separate from the CMS system.

[0043] The CMS includes at least one processor and at least one non-transitory electronic storage medium storing instructions that, when executed, cause the CMS to perform the method steps and calculations described herein.

[0044] Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

1. A method for estimating the position of a trailer wheel, comprising: identifying a first set of wheel positions in at least a first image, each wheel position in the first set of wheel positions being associated with a corresponding trailer angle; identifying a subset of wheel positions in the first set of wheel positions as false positives and removing the false positives from the first set of wheel positions; applying a quadratic regression to the first set of wheel positions and determining a parabolic curve relating y positions in the at least first image to x positions in the at least first image; Determine trailer angle; as well as The current wheel position is estimated by applying the determined trailer angle to the parabolic curve. 2 . The method of claim 1 , further comprising identifying a second set of wheel positions in at least a second image, each wheel position in the second image being associated with a corresponding trailer angle. 3 . The method of claim 2 , further comprising identifying a subset of wheel positions in the second set of wheel positions as false positives and removing the false positives from the first set of wheel positions.

4. The method according to claim 2, wherein: The first image is one of a Class II and a Class IV view, and the second image is a Class II and a Class IV view on an opposite side of the vehicle.

5. The method according to claim 1, wherein: The parabolic curve is defined according to y=ax^2+bx+c, where "a" is not equal to zero, y is the position of the wheel on the y-axis in the image, and x is the position of the wheel on the x-axis in the image. 6 . The method of claim 1 , further comprising providing the determined wheel position to at least one additional vehicle system.

7. The method according to claim 6, wherein: The at least one additional vehicle system includes at least one of an advanced driver assistance system, a camera monitoring system, and an electronic stability program.

8. The method according to claim 1, wherein: The current wheel position is a position within the image.

9. The method according to claim 1, wherein: The current wheel position is the real-world three-dimensional position of the wheel relative to the vehicle.

10. The method according to claim 1, further comprising: The parabolic curve is used to identify the wheel trajectory relative to the towing vehicle in image space.

11. The method according to claim 1, further comprising: The parabolic curve is used to identify the wheel trajectory relative to the towing vehicle in real world space.

Images