Commercial vehicle camera monitoring system with trailer wheelbase estimation

Abstract

A camera monitoring system (CMS) with trailer wheelbase estimation using image-based wheel position tracking.SOLUTION: A method of determining a wheelbase of a trailer attached to a tractor includes a) identifying a two dimensional trailer wheel position in an image, b) associating the two dimensional trailer wheel position with a corresponding trailer angle, c) converting the two dimensional trailer wheel position to a three dimensional trailer wheel position, d) calculating a wheelbase from the three dimensional trailer wheel position, e) obtaining a new two dimensional trailer wheel position, f) performing steps b) - d) on the new two dimensional trailer wheel position, and g) calculating an average wheelbase based on a previous average wheelbase and the new two dimensional trailer wheel position.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] This disclosure relates to a camera monitor system (CMS) with trailer wheelbase estimation using image-based wheel position tracking. [Background technology]

[0002] To enhance a vehicle operator's ability to view their surroundings, commercial vehicles utilize camera monitor systems, such as camera systems that supplement the view of mirrors. Camera monitor systems (CMS) utilize one or more cameras to provide an enhanced field of view to the vehicle operator. In some instances, camera monitor systems cover a wider field of view than traditional mirrors or include views that are not fully available through traditional mirrors.

[0003] Semi-autonomous driver assistance systems, camera monitoring systems, electronic stability control program systems, and other vehicle systems use or require knowledge of the location of various vehicle features throughout the vehicle's operation. Among these features may be the real-world location or the location within an image of one or more rear wheels of a trailer. Systems exist for using image processing to identify the location of the trailer's wheels while they are visible within the field of view of a rear-facing camera. This wheel location can then be used to estimate the trailer's wheelbase, thereby enabling enhanced CMS functionality such as providing a trailer trajectory overlay on a displayed image.

[0004] Some methods for accurately determining trailer wheel position require large data sets, which in turn require large memory resources to store the data sets, and the algorithms used to process the data sets are relatively slow computationally. Summary of the Invention

[0005] A method for determining the wheelbase of a trailer attached to a tractor includes the steps of: a) identifying two-dimensional trailer wheel positions in an image; b) associating the two-dimensional trailer wheel positions with corresponding trailer angles; c) converting the two-dimensional trailer wheel positions to three-dimensional trailer wheel positions; d) calculating the wheelbase from the three-dimensional trailer wheel positions; e) obtaining new two-dimensional trailer wheel positions; f) performing steps b) through d) on the new two-dimensional trailer wheel positions; and g) calculating the average wheelbase based on the previous average wheelbase and the new two-dimensional trailer wheel positions.

[0006] In a further example of the above, step a) is performed using a rear-facing camera mounted on the side of a tractor, said image being provided by said rear-facing camera, and step e) is performed using the same rear-facing camera.

[0007] In a further example of the above, step b) is performed based on a kinematic model.

[0008] In a further example of the above, step b) includes referencing the steering angle with the tractor in forward gear.

[0009] In a further example of the above, the method includes filtering the two-dimensional trailer wheel positions based on a predicted range of wheel positions relative to the trailer angle.

[0010] In a further example of the above, the filtering step is performed between step b) and step c).

[0011] In a further example of the above, step e) includes storing the new two-dimensional trailer wheel positions in a memory and allowing the stored new two-dimensional trailer wheel positions to be overwritten immediately after performing step g).

[0012] In a further example of the above, the method includes displaying a trailer trajectory based on the average wheelbase.

[0013] In a further example of the above, steps b) to g) are performed repeatedly during a single turn of the trailer, and displaying the trailer trajectory is based on the average wheelbase from the single turn.

[0014] In a further example of the above, steps b) through g) are performed independently with each of first and second rear-facing cameras mounted on the side of the tractor providing a first average wheelbase associated with the first rear-facing camera and providing a second average wheelbase associated with the second rear-facing camera, the method further including comparing the first and second average wheelbases to each other to determine positional accuracy of at least one of the first and second rear-facing cameras.

[0015] A camera monitor system (CMS) for determining the wheelbase of a trailer attached to a tractor includes a plurality of rear-facing cameras configured to capture a plurality of fields of view proximate to the trailer, a plurality of displays configured to display images from the captured fields of view, a sensor configured to provide information regarding trailer angle, and a controller in communication with the plurality of cameras, the plurality of displays, and the sensors, the controller including a memory having vehicle configuration parameters, a kinematic model, and a trailer wheel position determination module. The controller is programmed to perform trailer wheelbase estimation, including the steps of: a) identifying two-dimensional trailer wheel positions in an image of one of the plurality of fields of view; b) associating the two-dimensional trailer wheel positions with corresponding trailer angles; c) converting the two-dimensional trailer wheel positions to three-dimensional trailer wheel positions; d) calculating a wheelbase from the three-dimensional trailer wheel positions; e) obtaining new two-dimensional trailer wheel positions; f) performing steps b) to d) on the new two-dimensional trailer wheel positions; and g) calculating an average wheelbase based on a previous average wheelbase and the new two-dimensional trailer wheel positions.

[0016] In a further example of the above, step a) is performed using one of the multiple rear-facing cameras mounted on the side of a tractor, the image being provided by the one of the multiple rear-facing cameras, and step e) is performed using the same one of the multiple rear-facing cameras.

[0017] In a further example of the above, step b) is performed based on said kinematic model.

[0018] In a further example of the above, step b) includes referencing the steering angle from the sensor with the tractor in forward gear.

[0019] In a further example of the above, the controller performs the step of filtering the two-dimensional trailer wheel positions based on a predicted range of wheel positions relative to the trailer angle.

[0020] In a further example of the above, the filtering step is performed between step b) and step c).

[0021] In a further example of the above, step e) includes storing the new two-dimensional trailer wheel positions in the memory, and the controller allows the stored new two-dimensional trailer wheel positions to be overwritten immediately after performing step g).

[0022] In a further example of the above, the controller performs the step of displaying a trailer trajectory based on the average wheelbase on one of the plurality of displays.

[0023] In a further example of the above, steps b) to g) are performed repeatedly during a single turn of the trailer, and displaying the trailer trajectory is based on the average wheelbase from the single turn.

[0024] In a further example of the above, steps b) through g) are performed independently using each of a first and a second rear-facing camera of the plurality of rear-facing cameras mounted on a side of a tractor, a first average wheelbase associated with the first rear-facing camera is calculated, and a second average wheelbase associated with the first rear-facing camera is calculated, and the controller performs the step of comparing the first and second average wheelbases to each other to determine positional accuracy of at least one of the first and second rear-facing cameras. [Brief explanation of the drawings]

[0025] The present disclosure may be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitor system (CMS) used to provide at least Class II and Class IV views. [Figure 2] FIG. 1 is a schematic top view of a commercial truck equipped with a CMS providing Class II, Class IV, Class V, and Class VI views. [Figure 3] 1 is a schematic top perspective view of a vehicle cab including a display; [Figure 4] 1 shows a view of a CMS including a single view of a vehicle trailer at a medium to large trailer angle. [Figure 5] 5A and 5B show views of the CMS including two views of a vehicle trailer at a low trailer angle. [Figure 6] The kinematic models of the tractor and trailer are shown. [Figure 7] 1 shows a dataset of trailer wheel positions in images. [Figure 8] The data set in Fig. 7 is based on the kinematic model. [Figure 9] 10 illustrates a process for determining trailer wheelbase using identified image-based trailer wheel positions. [Figure 10] We present a method for accurately and efficiently determining trailer wheelbase using trailer wheel positions based on multiple images.

[0026] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0027] 1-3 show schematic diagrams of a commercial vehicle 10. The commercial vehicle 10 includes a vehicle cab or "tractor" 12 for towing a trailer 14, to which the trailer 14 is coupled during turns. In this disclosure, the commercial vehicle 10 is shown as a commercial truck having a single trailer, although it is understood that other commercial vehicle configurations (e.g., different types or quantities of trailers) may be used.

[0028] A pair of camera arms 16A-16B includes respective bases fixed to, for example, the tractor 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20A-B is disposed on or within each of the camera arms 16A-16B. Fixed wings may be used instead of folding camera arms. Each of the exterior cameras 20A-20B has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 2), which represent views legally mandated in the commercial trucking industry. EX1 , FOV EX2 to provide.

[0029] A Class II view of a given side of the commercial vehicle 10 is a subset of a Class IV view of the same side of the commercial vehicle 10. If desired, multiple cameras can be used on each camera arm 16A-16B to provide these views. For example, Class II (narrow angle) and Class IV (wide angle) views are defined in the European R46 regulation, and the United States and other countries may have similar driver visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an example of the type of view provided to the display from a particular camera. For example, SAE J3155 or other regulations may specify specific views.

[0030] Each camera arm 16A-16B may also provide a housing enclosing electronics, e.g., a controller, configured to provide various features of the CMS 15. The camera arms 16A-16B may be attached, for example, to a roof-mounted location above the cab door (as shown) or to a door-mounted bracket or station. If desired, the camera arms 16A-16B may also include conventional mirrors integrated therewith, although the CMS 15 may also be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.

[0031] If Class V and / or Class VI view footage is also desired, camera housing 16c and camera 20c may be positioned at or near the front of commercial vehicle 10 to provide these views (FIG. 2). Generally, Class V covers the passenger side of the vehicle from the corner of the passenger vehicle cab along the rear of the vehicle cab, and Class VI covers the passenger side of the vehicle from the corner of the passenger vehicle cab along the front of the vehicle cab.

[0032] Field of view FOV EX A backup camera 20D may be provided that provides a field of view FOV. The backup camera 20D may be mounted, for example, at the top / centerline of the trailer, at the trailer bumper / bed level, or at a rear top corner of the trailer. Alternatively, or in addition to the rear trailer camera, a backup camera 20D may be mounted at the rear of the tractor 12 and provide a field of view FOV. EX A "fifth wheel camera" 20E may be provided to provide a fifth wheel camera 20E. The fifth wheel camera 20E may be mounted anywhere between the side of the fifth wheel fixture and the top / roof edge of the tractor, for example.

[0033] FIG. 3 is a schematic top view of an exemplary vehicle cab interior 24. With continued reference to FIGS. 1 through 3, electronic displays 18A-18E (which may be, for example, video displays such as LCD displays) and cameras 20A-20E are shown. The various electronic displays 18A-18E and cameras 20A-20E are part of a camera monitor system (CMS) 15 and thus function as CMS displays and CMS cameras. As used herein, a "CMS camera" 20 is a camera configured to record images of the environment surrounding the commercial vehicle 10, and a "CMS display" 18 is an electronic display (e.g., LCD, touchscreen, etc.) configured to capture or display feeds from these cameras.

[0034] CMS 15 includes a CMS controller or electronic control unit (ECU) 22 that includes processing circuitry that functions as a controller and supports the operation of CMS 15. CMS controller or electronic control unit (ECU) 22 is operatively connected to memory 30 (which may include any one or combination of volatile memory elements (e.g., random access memory (RAM) such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). The processing circuitry may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), etc.

[0035] CMS displays 18A-18B are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19A-19B and display Class II and Class IV views on each side of the commercial vehicle 10, which provide rear-facing views along the commercial vehicle 10 captured by exterior cameras 20A-20B.

[0036] As mentioned above, if Class V and Class VI views are also required, camera housing 16c and camera 20c may be located at or near the front of commercial vehicle 10 to provide these views ( FIG. 2 ). In the example of FIG. 3 , additional displays 18C-18E are provided. Display 18C, located in vehicle cab interior 24 near the top center of the windshield, may be used to display Class V and Class VI views toward the front of commercial vehicle 10 or a backup camera view (from camera 20D or 20E) to the driver. Display 18D is located in the center console area of ​​vehicle cab interior 24, generally located in the center of the lower half of the vehicle cab, and may be used for other purposes such as navigation, infotainment, etc. (e.g., a secondary information display). Display 18E may be part of an instrument cluster (i.e., a primary information display) located, for example, behind the steering wheel.

[0037] If Class VIII view video is desired, camera housings can be positioned on the sides and rear of the commercial vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of the commercial vehicle 10. In such an example, one of the displays 18C-18E can include one or more frames that display the Class VIII view. Displays 18A, 18B, 18C face the driver's area within the vehicle cab interior 24, where the driver is seated in the driver's seat.

[0038] In various examples, the ECU 22 includes one or more modules having algorithm(s), equation(s), and / or decision manager(s) that receive input(s) and / or stored values ​​from sensors (e.g., cameras 20A-20E, ultrasonic, LiDAR, radar, etc.), as shown schematically in FIG. 3 . Exemplary modules include a lane detection module 100, an object detection module 102, a trailer end detection module 104, a kinematics module 106, a trailer crash area prediction module 108, a tractor crash area prediction module 110, and a collision warning module 112. Example inputs include one or more sensors 34, such as a steering angle sensor, a vehicle speed sensor, a gear position sensor, and / or other sensor data. Vehicle configuration information 32 that may be stored in the memory 30 relates to vehicle characteristics (e.g., trailer length, axle position, trailer type / wheelbase, tractor configuration / wheelbase, hitch point location, camera internal and external parameters) provided by the manufacturer, operator, and / or determined by one or more modules. During vehicle operation, the ECU 22 may use outputs (e.g., display 18, speaker, etc.) to communicate information to the driver, fleet operator, or others. Exemplary operation and applications of these modules are disclosed in International Application No. PCT / US2023 / 083416, filed December 11, 2023, and entitled "CAMERA MONITOR SYSTEM WITH TRAILER CURB STRIKE ALERT AND TRAILER STRIKING AREA," which is incorporated herein by reference in its entirety.

[0039] The lane detection module 100 also uses image processing of the captured image to identify markings on the road, such as lane markers that visually separate adjacent lanes. An example algorithm is described in U.S. Publication No. 2023 / 117,719, entitled "CAMERA MIRROR SYSTEM DISPLAY FOR COMMERCIAL VEHICLES INCLUDING SYSTEM FOR IDENTIFYING ROAD MARKINGS," which is incorporated by reference in its entirety. This publication describes a lane detection module in which an object detection algorithm identifies lane markings on a road by filtering the lane marking colors from surrounding portions of the captured image. Other techniques based on deep learning or other computer vision methods may also be used, if desired.

[0040] The object detection module 102 includes one or more image processing algorithms configured to identify objects in captured images. The algorithms may be used to identify VRUs (e.g., pedestrians or bicyclists), attributes of the tractor 12 and / or trailer 14, other vehicles, signs, curbs, trees, buildings, and / or other inanimate objects.

[0041] The trailer end detection module 104 is another image processing module that extracts one or more trailer features from the captured image to determine the location of the trailer end in 3D space. These extracted attributes can be used to detect objects such as tractor wheels, trailer ends, and other features. An exemplary wheel detection algorithm technique is disclosed in U.S. Publication No. 2023 / 202,394, entitled "CAMERA MONITOR SYSTEM FOR COMMERCIAL VEHICLES INCLUDING WHEEL POSITION ESTIMATION," which is incorporated herein by reference in its entirety. An example of a trailer end detection algorithm technique is disclosed in U.S. Publication No. 2023 / 125,045, entitled "TRAILER END TRACKING IN CAMERA MONITORING SYSTEM," which is incorporated herein by reference in its entirety. Other techniques can also be used if desired.

[0042] In one example operation, CMS 15 utilizes kinematics module 106 to predict a crash zone for trailer 14 based on the trailer's trajectory during a turning maneuver. A two-dimensional overlay may be generated that is digitally added onto at least one of the displayed Class II / IV images, thereby showing the vehicle operator the expected crash zone for trailer 14 and allowing the vehicle operator to adjust vehicle operation accordingly. CMS 15 uses captured images received from cameras 20A, 20B, as well as other camera and vehicle operation data received from a common vehicle controller via a data connection, such as a CAN bus or LIN bus, to estimate the expected positions of the tractor and / or trailer sides at each of multiple lateral positions and multiple time points. These positions are converted into a geometric region that encompasses all positions. In this way, the shape and size of the geometric region are not fixed but rather reflect the actual expected crash zone of the trailer.

[0043] To avoid accidental collisions, the crash area prediction system uses vehicle data (e.g., steering angle, steering speed, trailer angle, vehicle speed, trailer wheelbase, tractor wheelbase, hitch point location, yaw rate, etc.) to generate a predicted crash area over time. The predicted crash area is a prediction of the path the trailer will take during a turn and is continuously recalculated as the turn progresses. The trailer crash area is also useful in potential "curvecut" scenarios when the vehicle 10 is traveling on curved roads. On curved roads, there is an increased likelihood that the trailer edge will cross the lane markers that mark the boundary between adjacent lanes, creating a potentially dangerous situation.

[0044] CMS 15 includes a decision manager or collision warning module 112 that communicates with modules 100-110 for assessing the proximity between a predicted tractor and / or trailer path (i.e., tractor and trailer collision area) and one or more objects (e.g., predicting an impending curb collision, curb cut, object collision, etc.). The decision manager may take into account estimated time to event, severity (what the object is), proximity rate between objects, etc., and provide overlays and / or alerts.

[0045] While various overlays and alerts are useful in increasing operator awareness and improving safety, it is desirable to provide operators with information to more easily and proactively manage their vehicles to safely navigate their surroundings.

[0046] FIG. 4 schematically illustrates a rear view 200 displayed to a vehicle operator via the CMS described above, with the trailer 14 at a medium-to-large angle (e.g., greater than 10 degrees) with the rearmost wheels 212 visible. Existing image processing techniques can identify the wheels 212 when they are visible and track their center points 214 as they move within the image. To facilitate vehicle systems that rely on wheel position or location, such as advanced driver assistance systems, camera monitoring systems, electronic stability programs, and similar vehicle systems, the CMS 15 monitors left and right views 208, 210 captured by its respective cameras (FIGS. 5A and 5B, respectively) to identify wheel positions during any driving condition in which the wheels are visible. The CMS 15 then uses the wheel position information to predict the trailer's trajectory, for example, using the kinematics module 106.

[0047] An example kinematics module 106, represented diagrammatically in FIG. 6, models the tractor turning radius R1 and the trailer turning radius R2 as being provided by separate, distinct radii. The tractor 12 has front wheels 302 and rear wheels 304. The kinematics model 300 models the front wheels 302 to account for their Ackermann steering characteristics. This is a common steering geometry approach that allows the outer and inner wheels to travel in different radial paths to reduce tire wear. The trailer 14, which is connected to the tractor 12 at a hitch point 308 (i.e., the fifth wheel), has rear wheels 306. V T is the tractor speed (corresponding to the vehicle speed N of 34 in Figure 2), and V n is the trailer speed in its longitudinal direction. V y is the velocity of the trailer end in the same direction as the tractor's movement, and V x is V yis the velocity component of the trailer end in a direction intersecting with the trailer 14. The trailer angle θ, which is the angle between the trailer 14 and the tractor 12, can be determined by a variety of suitable approaches. In one example, the trailer angle is calculated using a kinematic model during forward driving and estimated using image processing methods during reverse driving. Another method may use a LiDAR point cloud to calculate the trailer angle and dimensions. Yet another method may be a relative / absolute angular position sensor mounted at the hitch location.

[0048] In the example kinematic model 300, the vehicle 10 is simplified by using two bicycle or half-truck models. That is, not all wheels need to be represented in the model. In other words, the bicycle model used as the kinematic model in the kinematics module 104 is a simplified representation of a four-wheeled vehicle. Only the inside wheel of a turning motion can be modeled because it is the side of the vehicle 10 that is most at risk of collision. This is used to predict the attitude of the vehicle 10 using the instantaneous positions, angles, velocities, and accelerations acting on and within the system. Furthermore, in this kinematic model, all slip angles are assumed to be zero. As a result, the velocity and future displacement of vehicle components (e.g., wheel positions and / or trailer ends) can be propagated very quickly through the mathematical algorithm. Furthermore, only trailer angle, vehicle speed, and steering angle are required as inputs, providing a simple, accurate, and fast approach to path prediction.

[0049] The disclosed kinematic model is provided by a first bicycle model with Ackermann steering that represents a predicted tractor path 400. A second bicycle model is connected to the first bicycle model by a hitch point 308, and the second bicycle model represents a predicted trailer path 402. The paths of the inner tractor and trailer wheels are the inner boundaries of the respective tractor and trailer paths 400, 402, respectively.

[0050] The kinematics module 106 receives the current trailer angle, steering angle, and vehicle speed to calculate predicted tractor and trailer paths 400, 402. If desired, one or both of the predicted tractor and trailer paths 400, 402 can be shown as overlays on one or more displays 18 (e.g., in at least one of a Class II (narrow angle) view and a Class IV (wide angle) view) to assist the driver in maneuvering the vehicle 10. In one example, the kinematics algorithms run (i.e., calculate) continuously during driving.

[0051] To facilitate vehicle systems that rely on wheel position or location, such as advanced driver assistance systems, camera monitoring systems, electronic stability control programs, and similar vehicle systems, CMS 15 monitors views 102, 104 (FIGS. 5A, 5B) and identifies wheel position 212 during any operating condition in which wheel 112 is visible. Existing object tracking systems can identify wheel 212 when it is visible and track its center point 214 as it moves within the 2D image. The position within the 2D image can then be converted to a real-world 3D position using known systems.

[0052] Wheel positions 500 identified by image processing are shown in Figure 7. Point 502 is the wheel position from in front of center point 214, and point 504 is the wheel position from behind center point 214. A disclosed method 600 for determining trailer wheelbase is shown in Figure 9. Method 600 identifies individual two-dimensional trailer wheel positions 500 within an image (e.g., Figures 5A and / or 5B) and calculates a running average of the wheel positions 500 to calculate an average trailer wheelbase (see 706 in Figure 10) that can be used, for example, to determine trailer trajectory 402.

[0053] An example of a trailer wheelbase calculation method 600 is shown in Figure 9. Initially, raw wheel positions 500 (shown in Figure 7) are provided to ECU 22 for processing one at a time, although multiple wheel positions may be stored at a time if desired. However, using the disclosed methods 600 and 700, it is not necessary to store large data sets. The raw wheel positions 500 are generated using any upstream wheel detection process or algorithm, and each wheel position is associated with a corresponding trailer angle θ. The trailer angle θ may be calculated based on the wheel positions in the image.

[0054] However, the trailer angle may be calculated using any conventional trailer angle calculation method, including a trailer angle sensor, a tractor-trailer kinematics model, image-based detection, a combination of sensor and image-based detection, or any other conventional detection method.

[0055] The kinematics model 610 is a geometric framework that provides the spatial relationship between the camera, the tractor, the trailer, and its wheels. In one example, the tractor-trailer angle is calculated using a kinematics model that uses only the tractor forward driving maneuver, and this angle is used to filter out unreasonable wheel position detections (i.e., the angle formed by the detected wheel position and an anchor point on the tractor should be close to the resulting kinematic angle). To avoid false wheel detections, the ECU 22 optionally performs a step (block 620) of filtering the two-dimensional trailer wheel positions based on the expected range of wheel positions relative to the trailer angle θ. The plausibility check determines the true trailer angle using the kinematics model 610 and compares the true trailer angle to the received raw wheel position and trailer angle. If the angle determined via the kinematics model 610 differs from the received angle by at least a certain amount, a false positive is detected. False positives correspond to wheel positions that are not possible and / or unlikely given the known information inputs (e.g., speed, steering angle, grade, etc.) and determined information outputs (e.g., trailer angle) from the kinematic model 610. Examples of filtered wheel positions are shown at 506 and 508 in FIG.

[0056] After generating the filtered wheel positions 620, the process 600 applies and performs a 2D to 3D transformation that converts the two-dimensional trailer wheel positions in the captured image into three-dimensional trailer wheel positions in the real world. An example technique (block 630) for converting the two-dimensional trailer wheel positions into three-dimensional trailer wheel positions is shown in the following equations:

[0057] [Table 1]

[0058] Various techniques are known for converting 2D positions to 3D positions, for example:

[0059]

number

[0060] The intrinsic parameter K and the extrinsic parameter matrices R_t, t cannot be multiplied directly because they have different sizes.

[0061]

number

[0062] Both have a 4x4 dimension, which allows for direct multiplication. Multiplying a 3D ISO point X, Y, Z successively with intrinsic and extrinsic parameters maps the point to image space uvw, based on computer vision theory. However, depth information is lost, so it is not visible from a 2D screen (x i , y i ) can only be seen. Therefore, equation 1 becomes u=wx i , v=wy i where w is unknown.

[0063] where (x i ,y i ) are pixel coordinates, and w is the depth information lost during the 3D to 2D projection. Then, on both sides of the equation:

[0064]

number

[0065] A 2D to 3D reprojection of the wheel is performed by multiplying it by the inverse of , which can be expressed as follows: This is because, due to the nature of the extrinsic parameter matrix, its inverse and transpose are equal.

[0066]

number

[0067] For the task of wheel detection, it is assumed that the world coordinates of the wheel centers of a commercial truck are always on the plane Z=500mm, as in the following example. Therefore, to preserve the w information, set:

[0068]

number

[0069] next,

[0070]

number

[0071] This solves the following:

[0072]

number

[0073] Once w is solved, the depth information is obtained and a back projection from 2D to 3D is possible based on Equation 2. The reprojection (X,Y) and hitch point position (X h ,Y h ), the trailer length (block 640) is calculated as follows:

[0074]

number

[0075] The estimation systems and processes described above generate estimated wheel positions using images generated by views 102, 104. The CMS controller and / or other vehicle system controllers can convert the estimated image positions into corresponding 3D real-world positions and use the corresponding 3D positions as needed.

[0076] Method 600 is performed for each detected trailer wheel position. When a new trailer wheel position is obtained (block 702), a moving average of the trailer wheelbase is calculated (method 700, FIG. 10).

[0077] The method 600 is performed on the newly acquired trailer wheel positions and calculates the trailer wheelbase (L) based on the newly acquired trailer wheel positions (block 704). The average wheelbase is calculated based on the previous average wheelbase and the new two-dimensional trailer wheel positions (block 706). A simple counter can be used to calculate a running average of the trailer wheelbase, for example, as follows:

[0078]

number

[0079] where n is the number of wheel position samples and L n is the trailer wheelbase calculated from the current wheel position, and L n-1 is the trailer wheelbase calculated earlier.

[0080] This approach allows for efficient determination of trailer wheelbase using limited memory while maintaining accuracy. Thus, new two-dimensional trailer wheel positions may be temporarily stored in memory to perform method 600, but can be overwritten immediately after performing method 600, if desired. Method 700 is sufficiently accurate so that it can be performed repeatedly during a single turn of the trailer, such that the displayed trailer trajectory is sufficiently reliable even based on the average wheelbase from a single turn.

[0081] In one example, the average trailer wheelbase can be used to predict trailer trajectory for use by module(s) 100-112 (block 708). In another example, the average trailer wheelbase is provided to an advanced driver assistance system in the vehicle that is separate from the CMS system.

[0082] Method 700 can also be used to determine the accuracy of one of rear-facing cameras 20A, 20B relative to the other of the rear-facing cameras 20A, 20B. Thus, for example, methods 600 and 700 are performed independently using each of a first and second rear-facing camera of a plurality of rear-facing cameras 20A, 20B. In this manner, a first average wheelbase is calculated and associated with one of cameras 20A, 20B, and a second average wheelbase is calculated and associated with the other of cameras 20A, 20B. ECU 22 then performs a step of comparing the first and second average wheelbases to each other to determine the positional accuracy of cameras 20A, 20B. A large discrepancy between the two cameras 20A, 20B may indicate that one of the cameras is out of position, and an operator can be alerted to check the camera's calibration.

[0083] The CMS includes at least one processor and at least one non-transitory electronic storage medium that stores instructions that, when executed, cause the CMS to perform the method steps and calculations described herein.

[0084] Also, while particular component arrangements are disclosed in the illustrated embodiments, it should be understood that other arrangements would benefit from the present disclosure. Although a particular sequence of steps is shown, described, and claimed, it should be understood that, unless otherwise indicated, the steps may be performed in any order, separated, or combined, and still benefit from the present invention.

[0085] Although the different examples have specific components shown, embodiments of the present invention are not limited to those specific combinations, and some components or features of one example may be used in combination with features or components of another example.

[0086] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for determining the wheelbase of a trailer attached to a tractor, comprising: a) identifying two-dimensional trailer wheel positions within an image; b) correlating said two-dimensional trailer wheel positions with corresponding trailer angles; c) transforming the two-dimensional trailer wheel positions into three-dimensional trailer wheel positions; d) calculating a wheelbase from the three-dimensional trailer wheel positions; e) obtaining new two-dimensional trailer wheel positions; f) performing steps b) to d) on said new two-dimensional trailer wheel positions; g) calculating an average wheelbase based on the previous average wheelbase and the new two-dimensional trailer wheel positions; A method comprising:

2. step a) is performed using a rear-facing camera mounted on the side of a tractor, the image being provided by the rear-facing camera; The method of claim 1 , wherein step e) is performed using the same rear-facing camera.

3. The method of claim 1 , wherein step b) is performed based on a kinematic model.

4. 4. The method of claim 3, wherein step b) includes referencing the steering angle with the tractor in forward gear.

5. 4. The method of claim 3, including filtering the two-dimensional trailer wheel positions based on a predicted range of wheel positions relative to the trailer angle.

6. The method of claim 5 , wherein the filtering step is performed between steps b) and c).

7. 2. The method of claim 1, wherein step e) includes storing the new two-dimensional trailer wheel positions in a memory, and allowing the stored new two-dimensional trailer wheel positions to be overwritten immediately after performing step g).

8. The method of claim 1 including displaying a trailer trajectory based on the average wheelbase.

9. 9. The method of claim 8, wherein steps b) through g) are performed repeatedly during a single turn of the trailer, and wherein displaying the trailer trajectory is based on the average wheelbase from the single turn.

10. steps b) through g) are performed independently with each of first and second rear-facing cameras mounted on the sides of the tractor to provide a first average wheelbase associated with the first rear-facing camera and to provide a second average wheelbase associated with the second rear-facing camera; The method of claim 1 , comprising comparing the first and second average wheelbases to one another to determine positional accuracy of at least one of the first and second rear-facing cameras.

11. 1. A camera monitor system (CMS) for determining the wheelbase of a trailer attached to a tractor, comprising: a plurality of rear-facing cameras configured to capture a plurality of fields of view proximate to the trailer; a plurality of displays configured to display images from the captured plurality of fields of view; a sensor configured to provide information regarding a trailer angle; a controller in communication with the plurality of cameras, the plurality of displays, and the sensors, the controller including a memory having vehicle configuration parameters, a kinematic model, and a trailer wheel position determination module; Equipped with The controller a) identifying two-dimensional trailer wheel positions within an image of one of said plurality of fields of view; b) correlating said two-dimensional trailer wheel positions with corresponding trailer angles; c) transforming the two-dimensional trailer wheel positions into three-dimensional trailer wheel positions; d) calculating a wheelbase from the three-dimensional trailer wheel positions; e) obtaining new two-dimensional trailer wheel positions; f) performing steps b) to d) on said new two-dimensional trailer wheel positions; g) calculating an average wheelbase based on the previous average wheelbase and the new two-dimensional trailer wheel positions; a camera monitor system programmed to perform wheelbase estimation of a trailer including:

12. step a) is performed using one of the plurality of rear-facing cameras mounted on a side of a tractor, the image being provided by the one of the plurality of rear-facing cameras; 12. The camera monitor system of claim 11, wherein step e) is performed using the same one of said plurality of rear-facing cameras.

13. 12. The camera monitor system of claim 11, wherein step b) is performed based on the kinematic model.

14. 14. The camera monitor system of claim 13, wherein step b) includes referencing steering angle from said sensor with said tractor in forward gear.

15. 14. The camera monitor system of claim 13, wherein the controller performs the step of filtering the two-dimensional trailer wheel positions based on a predicted range of wheel positions relative to the trailer angle.

16. 16. The camera monitor system of claim 15, wherein the filtering step is performed between step b) and step c).

17. step e) includes storing the new two-dimensional trailer wheel positions in the memory; 12. The camera monitor system of claim 11, wherein said controller allows said stored new two-dimensional trailer wheel positions to be overwritten immediately after performing step g).

18. The camera monitor system of claim 11 , wherein the controller performs the step of displaying the trailer trajectory based on the average wheelbase on one of the plurality of displays.

19. 20. The camera monitor system of claim 18, wherein steps b) through g) are performed repeatedly during a single turn of the trailer, and wherein displaying trailer trajectory is based on the average wheelbase from the single turn.

20. steps b) through g) are performed independently using each of a first and a second rear-facing camera of the plurality of rear-facing cameras mounted on the side of a tractor; a first average wheelbase associated with the first rear-facing camera is calculated, and a second average wheelbase associated with the second rear-facing camera is calculated; 12. The camera monitor system of claim 11, wherein the controller performs the step of comparing the first and second average wheelbases to one another to determine positional accuracy of at least one of the first and second rear-facing cameras.

Images