Camera Surveillance System for Commercial Vehicles Including Wheel Position Estimation

Abstract

A method for estimating a trailer wheel position includes identifying a first set of wheel positions in a first image. Each wheel position in the first set of wheel positions is associated with a corresponding trailer angle. The first set of wheel positions is clustered, and a primary cluster in the first set of wheel positions is identified. A best fit curve is applied to the primary cluster. The best fit curve is a curve relating the wheel positions to the trailer angles. An estimated wheel position is determined by applying the determined trailer angle to the best fit curve in response to the wheels being occluded in the first image. The estimated wheel position is output to at least one additional vehicle system.

Description

[Technical field]

[0001] The present disclosure relates to wheel tracking using a camera surveillance system (CMS) on a commercial truck, and more particularly to a system and method for tracking wheel position while the wheels are hidden.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 293199, filed December 23, 2021. [Background technology]

[0003] Mirror replacement systems, and camera systems that supplement the mirror view, are utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. Camera surveillance systems (CMS) utilize one or more cameras to provide an enhanced field of view to the vehicle operator. In some instances, mirror replacement systems cover a wider field of view than a traditional mirror or include views that are not fully available via a traditional mirror.

[0004] Semi-automated driver assistance systems, camera monitoring systems, electronic stability program systems, and other vehicle systems use or require knowledge of the location of various vehicle features throughout the vehicle's operation. Among these features may be the real-world location or location in an image of one or more rear wheels of a trailer. Systems exist that track the location of the wheels while they are visible in the field of view of a rear-facing camera monitoring system camera. However, while the trailer is at a low trailer angle, the rear wheels are not visible in the field of view of the driver's side camera or passenger's side camera, and the real-world location of the wheels and the location of the wheels in the image are unknown. Summary of the Invention

[0005] An exemplary method for estimating a trailer wheel position includes identifying a first set of wheel positions in at least a first image, where each wheel position in the first set of wheel positions is associated with a corresponding trailer angle; clustering the first set of wheel positions and identifying a primary cluster in the first set of wheel positions; generating a best fit curve to be applied to the primary cluster, where the best fit curve is a curve relating wheel positions to trailer angles; in response to the wheels being occluded in the first image, identifying an estimated wheel position by applying a determined trailer angle to the best fit curve; and outputting the estimated wheel position to at least one additional vehicle system.

[0006] Another example of the above method for estimating trailer wheel positions includes identifying a second set of wheel positions in a second image, each wheel position in the second image being associated with a corresponding trailer angle.

[0007] Another example of any of the above methods for estimating a trailer wheel position includes clustering the second set of wheel positions and identifying a second primary cluster within the second set of wheel positions.

[0008] In another example of any of the above methods for estimating a trailer wheel position, the first image is one of a Class II and Class IV view and the second image is a Class II and Class IV view of an opposite side of the vehicle.

[0009] In another example of any of the above methods for estimating trailer wheel position, generating the best fit curve to be applied to the first set of primary clusters includes generating the best fit curve to be applied to both the first set of primary clusters and the second set of primary clusters.

[0010] In another example of any of the above methods for estimating trailer wheel position, extending the best fit curve beyond the primary clusters includes extending the best fit curve from a first end of the first primary cluster to a first end of the second primary cluster, wherein a region of the best fit curve extending from the first end of the first cluster to the first end of the second primary cluster corresponds to wheel positions while the trailer has a trailer angle low enough that the wheels are not visible.

[0011] In another example of any of the above methods for estimating trailer wheel position, the low trailer angle is a trailer angle range of -10° to +10°.

[0012] In another example of any of the above methods for estimating trailer wheel position, the best fit curve is at least a quadratic function.

[0013] In another example of any of the above methods for estimating trailer wheel position, the best fit curve is one of a quadratic function and a cubic function.

[0014] In another example of any of the above methods for estimating trailer wheel position, identifying the first primary cluster includes identifying a cluster having at least one of a maximum number of points in the cluster and a maximum cluster spread.

[0015] Another example of any of the above methods for estimating trailer wheel position includes applying at least one of a Kalman filter, a least squares filter, and a recursive least squares filter to the primary clusters prior to generating a best fit curve.

[0016] In another example of any of the above methods for estimating trailer wheel position, the at least one additional vehicle system includes at least one of an advanced driver assistance system, a camera surveillance system, and an electronic stability program.

[0017] In one exemplary embodiment, a camera surveillance system (CMS) for a commercial vehicle includes a first mirror-replacement camera having a first field of view defining a side view of at least a first side of the vehicle, and a second mirror-replacement camera having a second field of view defining a side view of a second side of the vehicle, and a camera surveillance system controller communicatively connected to each of the first mirror-replacement camera and the second mirror-replacement camera, the camera surveillance system controller receiving a first video feed from the first camera and a second video feed from the second camera, the camera surveillance system controller including a memory and a processor, the memory configured to: identify a first set of wheel positions within at least the first video feed; The memory stores instructions configured to cause the processor to determine a wheel position estimate by: each wheel position in the first set of wheel positions is associated with a corresponding trailer angle; clustering the first set of wheel positions and identifying a primary cluster in the first set of wheel positions; generating a best fit curve to apply to the primary cluster, the best fit curve being a curve relating wheel positions to trailer angles; and the memory further stores instructions configured to cause the controller to respond to a wheel position being indeterminable in at least one of the first field of view and the second field of view by estimating a wheel position based on wheel positions identified while a wheel position was determinable.

[0018] In another example of the camera surveillance system for a commercial vehicle as described above, estimating wheel position based on wheel positions identified while wheel positions were determinable includes identifying a point on a best fit curve that corresponds to a currently detected trailer angle, the point on the best fit curve being the estimated wheel position. [Brief description of the drawings]

[0019] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0020] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views.

[0021] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera surveillance system providing Class II, Class IV, Class V and Class VI views.

[0022] [Diagram 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an in-cabin camera.

[0023] [Figure 3A] 1 shows a camera surveillance system view including a single view of a vehicle trailer at a medium to large trailer angle.

[0024] [Figure 3B] 1 shows a view of a camera surveillance system including two views of a vehicle trailer at a low trailer angle.

[0025] [Figure 4] 1 shows a dataset of trailer wheel positions in an image(s).

[0026] [Diagram 5]The data set in Figure 4 is shown with the data grouped into clusters.

[0027] [Figure 6] We present the dataset from Figure 4 reduced to the primary clusters within each image.

[0028] [Figure 7] A best fit curve for the data sets of Figures 4-7 is shown.

[0029] [Figure 8] A best fit curve isolated from the data set is shown.

[0030] [Figure 9] 9 shows a process flow illustrating the conversion of raw wheel position data into the best-fit curves shown in FIGS.

[0031] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in relation to one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] A schematic diagram of a commercial vehicle 10 is shown in Figures 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although commercial trucks are contemplated in this disclosure, the invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera surveillance system (CMS) 15 (Figure 2) that includes driver and passenger side camera arms 16A, 16B mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16A, 16B may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.

[0033] Each camera arm 16a, 16b includes a base that is fixed to, for example, the cab 12. A pivot arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20A, 20B is disposed within each of the camera arms. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. The reference to a "class" view is not intended to be limiting, but rather as an illustration of the type of view provided on the display by a particular camera. Each of the exterior cameras 20a, 20b has an exterior field of view FOV that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial truck industry. EX1 , FOV EX2 If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. Each arm 16A, 16B may also provide a housing that encloses electronics configured to provide various features of the CMS 15.

[0034] First and second video displays 18A, 18B are positioned on the driver's and passenger's sides, respectively, within the vehicle cab 12 on or near the A-pillars 19A, 19B and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20A, 20B.

[0035] If Class V and Class VI view video is also required, a camera housing 16c and camera 20c may be located at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located in the cab 12 near the top center of the windshield may be used to display to the driver the Class V and Class VI views toward the front of the vehicle 10.

[0036] If Class VIII view footage is required, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of the vehicle 10. As illustrated, the Class VIII view includes a view that surrounds the immediate vicinity of the trailer and a rear close-up view of the vehicle that includes the rear of the trailer. In one example, the rear close-up view of the vehicle is generated by a rear-facing camera positioned at the rear of the vehicle and may include both an immediate rear close-up view and a traditional rear view (e.g., a field of view extending rearward to the horizon provided by a rearview mirror of a vehicle without a trailer). In such an example, the third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near the first, second, and third displays 18a, 18b, 18c to provide a dedicated display that provides a Class VIII view.

[0037] 1A-2, FIG. 3A shows a schematic of the rear view displayed to the vehicle driver via the CMS described above when the trailer 110 is at a medium to large angle (e.g., greater than 10°). At a medium to large trailer angle, the trailer 110 is only slightly visible in the opposite view, and if visible at all, the opposite view is omitted. FIG. 3B shows the trailer 110 in both the driver side display 102 and the passenger side display 104. When the trailer 110 is at a medium to large angle, the rearmost wheels 112 are visible in the corresponding view 104. In contrast, when the trailer 110 is at a low trailer angle (FIG. 3B), the wheels 112 are not visible. Because the wheels 112 are not visible, they are referred to as hidden. The exact angle at which the wheels 112 will be hidden depends on the position of the camera generating the view and the length of the trailer 110, but the wheels 112 are typically hidden at low angles (e.g., between 10 degrees and -10 degrees).

[0038] To facilitate vehicle systems that rely on the position of the wheels 112, such as advanced driver assistance systems, camera monitoring systems, electronic stability programs, and similar vehicle systems, the CMS monitors the views 102, 104 and identifies the wheel positions 112 during any operating condition in which the wheels 112 are visible. Existing object tracking systems can identify the wheels 112 when they are visible and track their center points 114 as they move in the image. The positions in the image can then be translated back into real-world three-dimensional positions using known systems. In addition to using these monitored wheel positions, the CMS generates a data set from each image, and each point in each data set identifies the center point 114 of the wheels 112 in the image and adjusts the center point 114 of the wheels 112 by the angle of the trailer 110 at which the wheel position was detected. The angle of the trailer 110 is detected using either a trailer angle sensor, CMS image analysis, or a combination of these.

[0039] Based on the relationship established with the detection of the wheels 112 and the trailer angle while the wheels 112 are visible, the CMS is configured to determine a best fit curve for estimating the position of the wheels 112 while the wheel(s) 112 are obscured during the low angles shown in Figure 3B and while the wheels are obscured by other external effects. The estimated wheel positions are provided to any CMS or other vehicle systems that use that information, thereby providing continuous wheel positions to the CMS or other vehicle systems.

[0040] With continued reference to Figures 1-3B, Figures 4-8 show an on-road process for generating a wheel position estimate specific to the currently attached trailer 110, and Figure 9 shows a flow of a process 800 for operating on the data sets of Figures 4-7. This process can estimate hidden wheel positions without requiring trailer parameters such as height and length. The examples of Figures 4-9 are shown using data from the driver's side view 102 and passenger's side view 104 (see Figure 3B), resulting in roughly symmetrical images, however, due to the symmetrical nature, it will be appreciated that a similar process can be applied to only a single view to generate a similar wheel position estimate.

[0041] Initially during vehicle operation, wheel position data is collected over time in a "raw wheel detection" step 802 to create the raw wheel position dataset 304 shown in FIG. 4. Raw wheel detection uses image analysis to identify wheel positions (i.e., the location of the wheel center points in the image) in the images provided by the CMS. Once the dataset is sufficiently populated to generate an estimated curve, the dataset 304 is referred to as a complete dataset. In some examples, the complete dataset includes at least 1000 wheel detections 306. In addition to the location of the center points 114 of the wheels 112 in the image, each data point has an associated corresponding trailer angle determined by any available trailer angle detection or estimation system, the corresponding trailer angle being the trailer angle at the time the wheel position was captured. In some examples, once the dataset is sufficiently populated, the process 800 stops adding data to the dataset. In other examples, the process may be continually updated with new detections as they become available, with the accuracy of the resulting estimates continually improving during use of the vehicle.

[0042] In some cases, erroneous wheel position determinations may occur and may be added to the dataset 304, resulting in additional data points 308 that may distort or otherwise affect the resulting estimated curve. To remove erroneous detections from the dataset 304 and improve the resolution of the wheel position estimation, the raw data points 306 are clustered in a "data clustering" step 804. The clustering groups each data point 306 with nearby neighboring data points 306 based on the proximity of the data point 306 to other data points 306 and the density of the data points 306. In some exemplary systems, the clustering is performed using one or more of a k-means clustering process, a density based spatial (dbscan) clustering process, a distribution-based clustering process, a fuzzy clustering process, a mean shift clustering process, and a Gaussian mixture model clustering process. The clustering process results in multiple distinct clusters 310, 312 of wheel detections. It can be seen that in each view 102, 104, a true wheel detection (data points 310) results in a single elongated cluster 310 having an approximate teardrop shape. An incorrect wheel detection 308 results in one or more additional clusters 312, which are randomly shaped.

[0043] Once the data is clustered, the clusters 312 associated with the erroneous wheel detections 308 are discarded in a "cluster culling" step 806. In some examples, the cluster culling may discard the data point 308 entirely, while in other examples, the data point 308 may be retained and flagged as an erroneous detection, and the flagged erroneous detection may be ignored in the remainder of the process 800. If retained, the data point may be reviewed later to improve the wheel detection system or may be used for other diagnostic functions. For ease of reference, the cluster 310 that contains the accurate wheel detection is referred to as the "primary data cluster." In an example using views 102, 104 from each side of the vehicle, such as the example shown in Figures 4-7, there will be two primary data clusters 310 and they will be retained. In another example where only one side (corresponding to a single view 102, 104) is used, a single primary data cluster will be retained.

[0044] After discarding the clusters 312 containing the incorrect wheel detections, a single set of accurate wheel detections 306 remains as shown in Figure 6. The process 800 determines a best fit curve that is applied to all the wheel detections in both primary data clusters 310. In one example, the best fit curve is a parabola defined by a quadratic equation that is the best fit to all the data across both primary data clusters 310. In another example, the best fit curve is a parabola defined by a cubic (cubic) equation. As used throughout, best fit curve refers to a statistically determined curve that most closely approximates the trend of a scatter plot generated by the data.

[0045] The parabola defined by the quadratic best fit curve extends beyond each data cluster 310 and fills in the gaps 322 between the low trailer angles of each data cluster. With continued reference to FIGS. 4-7 and 9, FIG. 8 shows the parabola defining the best fit curve 320 separated from the data, with a trailer angle of 0 degrees located in the center of the chart and trailer angles increasing to the right and decreasing to the left of the chart. Since wheel position corresponds to trailer angle, the CMS estimates that wheel position is on the best fit line when the trailer angle is between negative 10 degrees and positive 10 degrees. In another example, the gaps 322 may be in a different location, but the estimation process may remain the same.

[0046] In yet another example, once a best fit curve is established, the wheel positions can be estimated using the best fit curve 320 whenever the CMS is unable to identify the wheel positions in the image. As an example, if one of the cameras generating the views 102, 104 fails, or if the field of view 102, 104 is fully or partially obstructed, the estimation can continue to provide an estimated wheel position as long as the trailer angle can be determined.

[0047] The estimation systems and processes described above generate estimated wheel positions using images generated by views 102, 104. The CMS controller and / or other vehicle system controllers convert the estimated image positions into corresponding 3D real world positions and can use the corresponding 3D positions as needed.

[0048] In at least one example, the estimated wheel positions are provided from the CMS controller to a trailer end detection module in the CMS system. The trailer end detection module may be a software module also located in the controller or a separate software system in communication with the CMS controller. The trailer end detection module uses the wheel positions to assist in identifying the trailer end, and the trailer end positions are marked on a CMS display to improve the vehicle operator's situational awareness. In another example, the CMS may also use the wheel positions to estimate the overall wheel base position, and the wheel base position may then be used within the CMS.

[0049] In another example, the estimated wheel positions are provided to an advanced driver assistance system in the vehicle, separate from the CMS system.

[0050] Although exemplary embodiments have been disclosed, a person of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for estimating trailer wheel position, comprising: identifying a first set of wheel positions within the first image, each wheel position within the first set of wheel positions being associated with a corresponding trailer angle; clustering the first set of wheel positions and identifying a first cluster within the first set of wheel positions; generating a curve based on the first cluster, the curve relating wheel position to trailer angle; if wheels are not visible in the first image, identifying an estimated wheel position based on the trailer angle and the curve; and outputting the estimated wheel positions to at least one additional vehicle system. A method comprising:

2. identifying a second set of wheel positions within the second image, each wheel position within the second set of wheel positions being associated with a corresponding trailer angle; the first image is a view of one side of a vehicle and the second image is a view of an opposite side of the vehicle; and 2. The method of claim 1, wherein clustering the second set of wheel positions and identifying a second cluster within the second set of wheel positions, and generating the curve based on the first cluster includes generating a curve based on both the first cluster and the second cluster.

3. further comprising extending the curve from a first end of the first cluster to a first end of the second cluster; 3. The method of claim 2, wherein a region of the curve extending from the first end of the first cluster to the first end of the second cluster corresponds to a wheel position during which the trailer has a low trailer angle, the low trailer angle being low enough that the wheels are not visible in the first image or the second image.

4. 4. The method of claim 3, wherein the low trailer angle ranges from -10 degrees to +10 degrees trailer angle.

5. The method of claim 1, wherein generating the curve includes generating a best fit curve.

6. The method described in claim 1, wherein the curve includes at least a quadratic function.

7. The method of claim 1 , wherein the curve is a cubic function.

8. The method of claim 1 , wherein identifying the first cluster comprises identifying a primary cluster that includes at least one of the largest number of points and the largest cluster spread.

9. The method of claim 1 , wherein the at least one additional vehicle system includes at least one of an advanced driver assistance system, a camera surveillance system, or an electronic stability program.

10. Identifying additional clusters within the first set of wheel positions; and discarding said additional clusters. The method of claim 1 further comprising:

11. The first image is captured by a side view camera of a camera surveillance system of a commercial vehicle, and by outputting the first image, Marking the trailer end position on the CMS display; Estimating wheelbase position for advanced driver assistance; and Adjusting the electronic stability program based on estimated wheelbase position The method of claim 1 , further comprising causing the at least one additional vehicle system to perform an action selected from:

12. 1. A camera surveillance system (CMS) for a commercial vehicle, comprising: at least one first camera having a first field of view defining a first side view of a first side of the vehicle; a controller communicatively connected to the at least one first camera, the controller receiving a first image from the at least one first camera; the controller includes a memory and a processor, the memory storing instructions configured to cause the controller to determine a wheel position estimate by identifying a first set of wheel positions in the first image, wherein each wheel position in the first set of wheel positions is associated with a corresponding trailer angle; clustering the first set of wheel positions and identifying a first cluster within the first set of wheel positions; and generating a curve based on the first cluster, the curve relating wheel positions to trailer angles; 11. A camera surveillance system, wherein the memory further stores instructions configured to cause the controller to respond to an inability to determine a wheel position in the first field of view by generating an estimated wheel position based on the curve and the trailer angle.

13. Generating an estimated wheel position includes identifying a point on the curve corresponding to a currently detected trailer angle; The camera surveillance system of claim 12 wherein the points on the curve are the estimated wheel positions.

14. The memory identifying a second set of wheel positions in a second image from a second camera having a second field of view defining a second side view of a second side of the vehicle, each wheel position in the second image being associated with a corresponding trailer angle; and clustering the second set of wheel positions and identifying a second cluster within the second set of wheel positions; 14. The camera surveillance system of claim 13, further storing instructions configured to:

15. A camera surveillance system as described in claim 14, wherein the first image is one of a Class II and Class IV view of the first side of the vehicle, and the second image is one of a Class II or Class IV view of the second side of the vehicle.

16. A camera surveillance system as described in claim 14, wherein generating the curve based on the first cluster includes generating a curve based on the first cluster and the second cluster.

17. The memory extending the curve beyond the first cluster, including extending the curve from a first end of the first cluster to a first end of the second cluster; 17. The camera surveillance system of claim 16, wherein the region of the curve extending from the first end of the first cluster to the first end of the second cluster corresponds to wheel positions while the trailer angle is low enough that the wheels are not visible.

Images