Camera surveillance system for commercial vehicles including wheel position estimation

Abstract

A method for estimating trailer wheel positions includes identifying a first set of wheel positions in at least a first image. Each wheel position in the first set of wheel positions is associated with a corresponding trailer angle. A subset of wheel positions in the first set of wheel positions is identified as false positives, and the false positives are removed from the first set of wheel positions. A quadratic regression is applied to the first set of wheel positions to determine a parabolic curve relating y positions in at least the first image to x positions in at least the first image. A trailer angle is determined. A current wheel position is estimated by applying the determined trailer angle to the parabolic curve.

Description

[Technical Field]

[0001] The present disclosure relates to wheel tracking using a vehicle's camera surveillance system (CMS), and more particularly to a system and method for tracking wheel position while the wheels are obscured. [Background technology]

[0002] To enhance a vehicle operator's ability to view their surroundings, commercial vehicles utilize camera surveillance systems, such as camera systems that supplement the view in mirrors. Camera surveillance systems (CMS) utilize one or more cameras to provide an enhanced field of view to the vehicle operator. In some instances, camera surveillance systems cover a wider field of view than traditional mirrors or include views that are not fully available through traditional mirrors.

[0003] Semi-autonomous driver assistance systems, camera monitoring systems, electronic stability control program systems, and other vehicle systems use or require knowledge of the location of various vehicle features throughout the vehicle's operation. Among these features may be the real-world location or the location in an image of one or more rear wheels of a trailer. Systems exist that use camera monitoring systems to track the location of wheels while they are visible in the field of view of a rear-facing camera. However, while the trailer is at a low trailer angle, the rear wheels are not visible in the field of view of the driver-side camera or passenger-side camera, and the real-world location of the wheels and their location in the image are unknown. Furthermore, if the field of view of either the driver-side or passenger-side camera is obstructed or unavailable for some reason, the location of the wheels in the image generated by the obstructed camera cannot be determined using existing systems. Summary of the Invention

[0004] A method for estimating trailer wheel positions according to an example of the present disclosure includes identifying a first set of wheel positions in at least a first image. Each wheel position in the first set of wheel positions is associated with a corresponding trailer angle. A subset of wheel positions in the first set of wheel positions is identified as false positives, and the false positives are removed from the first set of wheel positions. A quadratic regression is applied to the first set of wheel positions to determine a parabolic curve relating y positions in at least the first image to x positions in at least the first image. A trailer angle is determined. A current wheel position is estimated by applying the determined trailer angle to the parabolic curve.

[0005] In a further example of the above, the method includes identifying a second set of wheel positions in at least a second image, each wheel position in the second image being associated with a corresponding trailer angle.

[0006] In a further example of any of the above, the method includes identifying a subset of wheel locations in the second set of wheel locations as false positives and removing the false positives from the first set of wheel locations.

[0007] In a further example of any of the above, the first image is one of a Class II and Class IV view and the second image is a Class II and Class IV view of an opposite side of the vehicle.

[0008] In a further example of any of the above, the parabolic curve is defined according to y=ax^2+bx+c, where "a" is not equal to 0, "y" is the position of the wheel in the image on the y-axis, and "x" is the position of the wheel in the image on the x-axis.

[0009] In a further example of any of the above, the method includes providing the determined wheel positions to at least one additional vehicle system.

[0010] In a further example of any of the above, the at least one additional vehicle system includes at least one of an advanced driver assistance system, a camera surveillance system, and an electronic stability control program.

[0011] In a further example of any of the above, the current wheel position is a position within an image.

[0012] In a further example of any of the above, the current wheel position is a real-world three-dimensional position of the wheel relative to the vehicle.

[0013] In a further example of any of the above, the method includes identifying a wheel path for the tractor in image space using the parabolic curve.

[0014] In a further example of any of the above, the method includes identifying a wheel path for the tractor in real world space using the parabolic curve. [Brief explanation of the drawings]

[0015] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0016] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views.

[0017] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera surveillance system providing Class II, Class IV, Class V, and Class VI views.

[0018] [Figure 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an interior camera.

[0019] [Figure 3A]1 shows a view of a camera surveillance system including a single view of a vehicle trailer at a medium to large trailer angle.

[0020] [Figure 3B] 1 shows a view of a camera surveillance system including two views of a vehicle trailer at a low trailer angle.

[0021] [Figure 4] 1 shows a dataset of trailer wheel positions in images.

[0022] [Figure 5] 4 shows the dataset from FIG. 4 with false positive detections removed.

[0023] [Figure 6] 1 shows a control process for identifying wheel position estimates.

[0024] Equation 1 shows the matrix operation that identifies the quadratic wheel estimation function for estimating wheel position.

[0025] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0026] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. FIG. 2 is a schematic top perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While commercial trucks are contemplated in this disclosure, the present invention is applicable to other types of vehicles. The vehicle 10 incorporates a camera monitoring system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, each arm housing one or more cameras and / or mirrors.

[0027] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within each camera arm. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 16a, 16b. If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an illustration of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing enclosing electronics configured to provide various features of the CMS 15.

[0028] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0029] If Class V and / or Class VI view footage is also desired, a camera housing 16c and camera 20c may be positioned at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located within the cab 12 near the top center of the windshield can be used to display Class V and Class VI views forward of the vehicle 10 to the driver. Displays 18a, 18b, and 18c face a driver area 24 within the cab 22, where the driver is seated in a driver's seat 26. The location, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0030] If Class VIII view video is desired, camera housings may be positioned on the sides and rear of vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of vehicle 10. In such an example, third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near first, second, and third displays 18a, 18b, 18c to provide dedicated displays that provide the Class VIII view.

[0031] 1A-2, FIG. 3A schematically illustrates the rear view displayed to the vehicle driver via the CMS described above when the trailer 110 is at a medium to high angle (e.g., greater than 10 degrees). At a medium to high trailer angle, the trailer 110 is only slightly visible in the opposite view, and if visible at all, the opposite view is omitted. FIG. 3B schematically illustrates the trailer 110 on both the driver-side display 102 and the passenger-side display 104 while the trailer is at a low angle (e.g., less than 10 degrees). While the trailer 110 is at a medium to high angle, the rearmost wheels 112 are visible in the corresponding views 104. In contrast, when the trailer 110 is at a low trailer angle (FIG. 3B), the wheels 112 are not visible. Because the wheels 112 are not visible, they are referred to as hidden. The exact angle at which the wheels 112 will be obscured will depend on the position of the camera generating the view and the length of the trailer 110, but the wheels 112 will typically be obscured at a low angle (e.g., between 10 degrees and -10 degrees).

[0032] To facilitate vehicle systems that rely on the position of the wheels 112, such as advanced driver assistance systems, camera monitoring systems, electronic stability control programs, and similar vehicle systems, the CMS monitors the views 102, 104 and identifies wheel positions 112 during any operating condition in which the wheels 112 are visible. Existing object tracking systems can identify the wheels 112 when they are visible and track their center points 114 as they move through the image. The positions within the image can then be converted back to real-world three-dimensional positions using known systems. In addition to using these monitored wheel positions, the CMS generates a dataset from each image, and each point in each dataset identifies the center point 114 of the wheels 112 in the image and adjusts the center point 114 of the wheels 112 by the angle of the trailer 110 where the wheel position is detected. The angle of the trailer 110 is detected using either a trailer angle sensor, CMS image analysis, or a combination of these.

[0033] Based on the relationship established using the detection of the wheels 112 and the trailer angle while the wheel(s) are visible, the CMS 15 is configured to identify a quadratic equation that correlates the wheel position and trailer angle within the image. Based on this correlation, the CMS 15 can identify an estimated wheel position during periods when the wheels 112 are not visible in the image(s). By way of example, this may occur while the wheel(s) 112 are obscured at a low angle as shown in FIG. 3B, or while the wheel position is obscured by an obstacle, weather, or other external influence.

[0034] The estimated wheel positions are provided to any CMS or other vehicle system that uses that information, thereby providing the CMS or other vehicle system with continuous wheel positions.

[0035] With continued reference to FIGS. 1-3B, FIGS. 4, 5, and 6 illustrate an on-road process for generating wheel position estimates specific to the currently attached trailer 110 using a control process 500 shown in FIG. 6. Initially, raw wheel positions 502 (shown in FIG. 4) are provided to the control system 500. The raw wheel positions 502 are generated using any upstream wheel detection process or algorithm and may include one or more false positives 504. Each of the wheel positions 502 and false positives 504 includes a corresponding trailer angle calculated based on the wheel position in the image. The trailer angle is calculated using any conventional trailer angle calculation method, including a trailer angle sensor, a tractor-trailer kinematic model, image-based detection, a combination of sensor and image-based detection, or any other conventional detection method. As shown in FIGS. 4 and 5, x and y are the image boundary, and the location of the data point in the x-y plane is the location of the wheel detection in the image.

[0036] After receiving the raw wheel positions and trailer angles, a plausibility check is applied to the raw wheel positions 502. The plausibility check determines the true trailer angle using a kinematic model and compares the true trailer angle to the received raw wheel positions and trailer angle. If the angle determined via the kinematic model differs from the received angle by at least a certain amount, a false positive 504 is detected. The false positive 504 corresponds to a wheel position that is unlikely and / or unlikely given known information inputs (e.g., speed, steering angle, grade, etc.) and determined information outputs (e.g., trailer angle) from the kinematic model 510. The unlikely and / or unlikely wheel positions 504 are filtered from the dataset 502 using a filter 520, resulting in a filtered dataset 524 shown in FIG. 5.

[0037] In one example, the kinematic model 510 uses only the forward driving motion of the tractor to ensure accuracy. Using the kinematic model approach defined herein, it is possible that some accurate wheel detections may be erroneously filtered out, but it is understood that the process can operate with less than the full set of accurate wheel detections as long as there are no or minimal erroneous detections.

[0038] After generating the filtered data set 524, the process 500 applies a quadratic regression 530 to the data set. In one example, the quadratic regression 530 is a matrix reduction according to Equation 1, providing a resulting quadratic equation in the form y=a*x^2+b*x+c, where a is not equal to 0 and y is the pixel coordinate of the wheel position. In another example, a different form of quadratic regression can be used to generate a quadratic equation relating wheel position in the image to the trailer angle. The controller operating the process 500 then receives updated coefficients from the determined equation, and the new wheel position curve (the resulting quadratic equation) is provided to the CMS 15 and any other vehicle systems that need to estimate wheel position.

[0039] While the above example estimates wheel positions within an image, it should be understood that the wheel positions can be converted to three-dimensional world model positions either before or after application of the methods described herein, and the same process can be utilized to estimate actual wheel positions in addition to estimating wheel positions within an image. The three-dimensional world model positions describe the real-world three-dimensional positions of the wheels relative to defined points on the vehicle.

[0040] The estimation systems and processes described above generate estimated wheel positions using images generated by views 102, 104. The CMS controller and / or other vehicle system controllers can convert the estimated image positions into corresponding 3D real-world positions and use the corresponding 3D positions as needed.

[0041] In at least one example, the estimated wheel positions are provided from the CMS controller to a trailer end detection module within the CMS system. The trailer end detection module may be a software module also located within the controller or a separate software system in communication with the CMS controller. The trailer end detection module uses the wheel positions to assist in identifying the trailer ends, which are marked on a CMS display to improve situational awareness for the vehicle driver. In another example, the CMS may also use the wheel positions to estimate the overall wheelbase position, and the wheelbase distance can then be used within the CMS.

[0042] In another example, the estimated wheel positions are provided to an advanced driver assistance system in the vehicle that is separate from the CMS system.

[0043] The CMS includes at least one processor and at least one non-transitory electronic storage medium that stores instructions that, when executed, cause the CMS to perform the method steps and calculations described herein.

[0044] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for estimating trailer wheel position, comprising: identifying a first set of wheel positions within at least a first image, each wheel position within the first set of wheel positions being associated with a corresponding trailer angle; identifying a subset of wheel locations within the first set of wheel locations as false positives and removing the false positives from the first set of wheel locations; applying a quadratic regression to the first set of wheel positions to determine a parabolic curve relating y positions in the at least first image to x positions in the at least first image; Determining the trailer angle; and estimating the current wheel position by applying the determined trailer angle to the parabolic curve; A method comprising:

2. identifying a second set of wheel positions in at least a second image, each wheel position in the second image being associated with a corresponding trailer angle; The method of claim 1 further comprising:

3. identifying a subset of wheel locations within the second set of wheel locations as false positives and removing the false positives from the first set of wheel locations; The method of claim 2 further comprising:

4. 3. The method of claim 2, wherein the first image is one of a Class II and a Class IV view and the second image is a Class II and a Class IV view of an opposite side of the vehicle.

5. 2. The method of claim 1, wherein the parabolic curve is defined according to y=ax^2+bx+c, where "a" is not equal to 0, "y" is the position of the wheel in the image on the y-axis, and "x" is the position of the wheel in the image on the x-axis.

6. Providing the determined wheel positions to at least one additional vehicle system. The method of claim 1 further comprising:

7. The method of claim 6 , wherein the at least one additional vehicle system includes at least one of an advanced driver assistance system, a camera surveillance system, and an electronic stability control program.

8. The method of claim 1 , wherein the current wheel position is a position within the image.

9. The method of claim 1 , wherein the current wheel positions are real-world three-dimensional positions of the wheels relative to the vehicle.

10. using said parabolic curve to identify wheel paths for a tractor in image space; The method of claim 1 further comprising:

11. using said parabolic curve to identify a wheel path for a tractor in real world space; The method of claim 1 further comprising:

Images