Automatic panning camera monitoring system including image based trailer angle detection

Abstract

To provide: a camera monitoring system including an automatic panning function including an image based trailer angle estimation; and a method.SOLUTION: A method of automatically panning a view of a commercial vehicle 10, includes a step of receiving a video feed from at least one of cameras 20a, 20b, and 20c defining a visual field. A plurality of objects in a video feed including at least one wheel and at least one line are identified. A path of each objects in the plurality of objects is tracked via an image plane of the video feed. A trailer angle corresponding to each path is identified, the identified trailer angle is associated with the corresponding path, thereby, generating a plurality of trailer angle measurement values. The plurality of paths are down selected to a single path, and identifying a single trailer angle measurement corresponding to the single path.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present disclosure relates to a camera monitoring system (CMS) for use in commercial trucks or similar vehicles, particularly a CMS having an auto-panning function that includes image-based trailer angle estimation.

Background Art

[0002] Mirror replacement systems, and camera systems that complement the view of mirrors, are utilized in commercial vehicles to enhance the ability of a vehicle driver to view the surrounding environment. A camera monitoring system (CMS) utilizes one or more cameras to provide an enhanced field of view to a vehicle driver. In some examples, a mirror replacement system covers a wider field of view than a conventional mirror or includes views that are not fully obtainable via a conventional mirror.

[0003] For certain operations such as trailer reverse maneuvers, a static view such as that provided by a fixed mirror or a fixed field of view camera may not provide a complete view of the operation and may not present the desired information that can be presented to an operator. A manual panning system where an operator manually adjusts a physical camera or mirror angle requires frequent stops in maneuvers to adjust the view provided and may result in insufficient accuracy of adjustment.

[0004] Some exemplary systems attempt to minimize the problems of manual panning by implementing auto or semi-auto panning. Such systems potentially rely on inaccurate trailer angle estimates, and kinematic models of vehicle motion can be difficult to account for potential variations in trailer angle estimates, particularly in reverse maneuvers.

Summary of the Invention

[0005] In one exemplary embodiment, a method for automatically panning the view of a commercial vehicle includes the steps of receiving a video feed from at least one camera, the camera defining a field of view; identifying a plurality of objects in the video feed, the plurality of objects including at least one wheel and at least one line; tracking the path of each of the plurality of objects through the image plane of the video feed; generating a plurality of trailer angle measurements by identifying a trailer angle corresponding to each path and associating the identified trailer angles with the corresponding paths; and narrowing down the plurality of paths to a single path and identifying a single trailer angle measurement corresponding to the single path.

[0006] Another example of the above method for automatically panning the view of a commercial vehicle further includes the steps of providing the single trailer angle measurement to an image panning system and panning the view based at least in part on the single trailer angle and the current vehicle movement.

[0007] In another example of the above method for automatically panning the view of a commercial vehicle, panning the view includes keeping the trailer end in the view.

[0008] In another example of the above method for automatically panning the view of a commercial vehicle, panning the view includes keeping the trailer end approximately centered in the view.

[0009] Another example of the above method for automatically panning the view of a commercial vehicle includes narrowing down multiple trailer angles by: determining a numerical quantifier for each of the multiple parameters of the track for each track; summing the determined numerical quantifiers for each track to determine a weighted score corresponding to the track; comparing the weighted scores of each track to identify the track with the highest weighted score; and selecting the trailer angle corresponding to the track with the highest weighted score as the single trailer angle.

[0010] In another example of the above method for automatically panning the view of a commercial vehicle, identifying the multiple objects includes identifying multiple lines and edges in the image using one of the Hough transform and a deep neural network.

[0011] In another example of the above method for automatically panning the view of a commercial vehicle, identifying the plurality of lines and edges includes identifying the angle, start point, and end point of each line and edge among the plurality of lines and edges.

[0012] In another example of the above method for automatically panning the view of a commercial vehicle, identifying the multiple objects includes identifying the position of each wheel in the image using one of blob transform image analysis and a deep neural network.

[0013] Another example of any of the above methods for automatically panning the view of a commercial vehicle further includes using the image analysis to identify the wheel angle of each wheel.

[0014] In one exemplary embodiment, a camera system for a vehicle comprises at least one first camera defining a field of view, and a controller communicatively connected to the first camera, the controller including a processor and memory, the memory storing instructions for automatically panning the view of the vehicle by receiving a video feed from the first camera in the controller, the camera defining a field of view, using the controller to identify a plurality of objects in the video feed, the plurality of objects including at least one wheel and at least one line, using the controller to track the path of each of the plurality of objects through the image plane of the video feed, using the controller to generate a plurality of trailer angle measurements by identifying a trailer angle corresponding to each path, and using the controller to narrow down the plurality of paths to a single path and identify a single trailer angle measurement corresponding to the single path.

[0015] In another example of the camera system for the vehicle described above, identifying the trailer angle corresponding to each path includes converting either line detection or wheel detection of each object into a two-dimensional trailer angle, and converting the two-dimensional trailer angle into a real-world angle of the trailer angle relative to the driver's cab.

[0016] In any other example of the camera system for vehicles described above, the controller is configured to apply Kalman filtering to each real-world trailer angle, the Kalman filtering associates the real-world trailer angle with a corresponding trajectory, and updates at least one property of the corresponding trajectory.

[0017] In any other example of the vehicle camera system described above, the characteristics of each trajectory include at least dispersion, elapsed time, direction, and source.

[0018] In any other example of the vehicle camera system described above, narrowing down the identified trailer angle includes, for each track, determining a numerical quantifier for each of several parameters of the track; for each track, summing the determined numerical quantifiers to determine a weighted score corresponding to the track; comparing the weighted scores corresponding to each track to identify the track with the highest weighted score; and selecting the trailer angle corresponding to the track with the highest weighted score as the single trailer angle.

[0019] This disclosure can be further understood by referring to the following detailed description, in conjunction with the attached drawings. [Brief explanation of the drawing]

[0020] [Figure 1A] This is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views. [Figure 1B] This is a schematic top view of a commercial truck equipped with a camera surveillance system that provides Class II, Class IV, Class V, and Class VI views. [Figure 2] This is a schematic top perspective view of the vehicle's driver's cab, including the display and interior cameras. [Figure 3A] This shows the vehicle at the start of reverse steering, with no trailer angle. [Figure 3B] This indicates a large trailer angle and mid-vehicle reverse steering. [Figure 4] This document demonstrates the process of determining the trailer angle using image analysis. [Figure 5] Figure 4 shows a system that determines the trailer angle from an image using the method described therein and automatically pans the camera monitoring system based on the determined angle. [Modes for carrying out the invention]

[0021] The embodiments, examples, and alternatives of the above paragraphs, claims, or the following description and drawings may be employed independently or in any combination, including any of their various aspects or their respective individual features. Features described in relation to one embodiment are applicable to all embodiments, except where such features are incompatible.

[0022] Figures 1A and 1B show schematic external views of a commercial vehicle 10. Figure 2 shows a schematic internal view of the cab of the commercial vehicle 10. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although a commercial truck is contemplated in the present disclosure, the invention can also be applied to other types of vehicles. The vehicle 10 incorporates a camera monitoring system (CMS) 15, to which camera arms 16a, 16b on the driver's and passenger's sides are attached outside the vehicle cab 12. If desired, the camera arms 16a, 16b may include conventional mirrors integrated therewith, although in some examples, the mirrors can be completely replaced using the CMS 15. In additional examples, multiple camera arms may be included on each side, and each arm may house one or more cameras and / or mirrors. <F <F

[0023] <F Each camera arm 16a, 16b includes, for example, a base fixed to the cab 12. A swivel arm is supported by the base and may be pivotally connected thereto. At least one rear-facing camera 20a, 20b is respectively disposed within the camera arm. The external cameras 20a, 20b each include an external field of view FOV that respectively includes at least one of Class II and Class IV views (Figure 1B), which are legally defined views in the commercial truck industry. <F

[0023] , FOV <F EX2 is provided. The Class II view on a given side of the vehicle 10 is a subset of the Class IV view on the same side of the vehicle 10. If desired, multiple cameras may be used on each camera arm 16a, 16b to provide these views. Each arm 16a, 16b may also provide a housing that surrounds electronics configured to provide various features of the CMS 15. <F <F

[0024] The first and second video displays 18a, 18b are respectively disposed on the driver's side and the passenger's side in the vehicle cab 12 on or near the A-pillars 19a, 19b, and display Class II and Class IV views on respective sides of the vehicle 10, which provide rear-side views along the vehicle 10 captured by the outboard cameras 20a, 20b.

[0025] If video images of Class V and Class VI views are also required, the camera housing 16c and the camera 20c may be disposed on or near the front of the vehicle 10 to provide these views (FIG. 1B). Using a third display 18c disposed in the cab 12 near the upper center of the windshield, the driver can be shown Class V and Class VI views facing forward of the vehicle 10.

[0026] If video images of Class VIII views are required, the camera housing can be disposed on the side and rear of the vehicle 10 to provide a field of view including some or all of the Class VIII zone of the vehicle 10. In such an example, the third display 18c may include one or more frames displaying the Class VIII view. Alternatively, an additional display can be added near the first, second, and third displays 18a, 18b, 18c to provide a dedicated display for providing the Class VIII view.

[0027] Referring again to Figures 1A, 1B, and 2, Figures 3A and 3B show the vehicle 100 in the process of performing a reverse maneuver. In the initial position (Figure 3A), the trailer 110 has an initial angle of approximately 0 degrees relative to the cab 120, meaning that the trailer is aligned with the orientation of the cab 120. Alternatively, this angle can also be expressed as 180 degrees relative to the cab 120. During the reverse process, especially when reversing via a turn, the trailer 110 tilts relative to the cab 120 (Figure 3B), forming a trailer angle that affects the reverse maneuver. Furthermore, the reverse speed, trailer yaw, and other operating parameters affect the change in the trailer angle during reverse. The specific tilt in Figure 3B is exaggerated for illustrative purposes relative to the most expected angle.

[0028] To assist the driver in performing reverse maneuvers, it is beneficial to ensure that the rear 112 of the trailer 110 is visible to the driver on at least one display during reverse operation. In some specific examples, it is desirable not only to include the rear 112 of the trailer 110, but also to center the Class II view on the rear 112 of the trailer 110. However, as shown in Figure 3B, a static Class II view may result in the rear 112 of the trailer 110 extending beyond the boundary of the Class II view, even if the rear 112 remains within the Class IV view. To prevent loss of field of view of the rear 112 of the trailer 110 in the Class II view, or to maintain the centering of the Class II view on the rear 112 of the trailer 110, the vehicles 10,100 described herein include an automatic panning function within the camera surveillance system.

[0029] The automatic panning function determines the trailer angle relative to the tractor at any given time using a combination of trailer angle detections based on individual images. The system identifies multiple determined trailer angles and uses a narrowing process to determine which of the new trailer angles is the most accurate. The most accurate determined angle detection is provided to the vehicle controller, which utilizes that angle in any system configured to use the trailer angle, including the automatic panning system. In an example of an automatic panning system, the vehicle controller can automatically pan the view displayed within the full field of view of the corresponding camera (e.g., a Class II view) to ensure that the trailer end remains in view during reverse operation. In some examples, the panning response depends on the vehicle's current operating mode, and the view may pan differently depending on whether the vehicle is moving forward or backward and / or the vehicle's speed, the steering wheel position, or other factors. In some examples, it may be even more advantageous to ensure that the trailer is not only kept in view, but also kept at or near the center of the view. In such cases, the automatic panning system centers the view at the trailer end.

[0030] For example, during operation, the trailer angle detection system receives a series of images from a mirror replacement system (or a similar camera monitoring system) and processes the images for line detection and wheel detection. The position of the line in the image, determined by the line detection system, and the position of the wheel in the image, determined by the wheel detection system, are tracked over time. The path of each wheel and line detected in the image is called a "track" and is used by the angle detection controller module to evaluate the reliability of the corresponding identified trailer angle.

[0031] The angle detection controller module uses a known correlation between position in the image and the trailer angle to determine the trailer angle based on each identified object. The detected trailer angle is correlated to the current position and stored as a trajectory property. In some cases, the trajectory shape can also be used to predict the trailer angle in the near future. For example, if the trajectory shape smoothly follows a path of increasing trailer angle, the prediction for the near future may be that the trailer angle continues to increase at the identified rate.

[0032] Each trajectory has multiple properties, including trailer angle, variance, elapsed time, direction, source, and similar properties. Furthermore, trajectories corresponding to line and edge detection include line angle, interference by other objects, and the start and end points of the line, while trajectories corresponding to wheel detection include the angle of the wheel relative to the trailer (alternatively also called the wheel angle). Each property is quantified using a numerical value (for example, variance may take values ​​from 0, meaning very large variation, to 100, meaning almost no variation). Each quantified numerical value is weighted based on its relative value when determining the accuracy of the angle prediction.

[0033] Once detected by the angle detection controller module, multiple detected angles are provided to the filtering module. The filtering module identifies the most accurately detected trailer angle from the detected trailer angles by compiling a weighted score for each trajectory and determining which trajectory has the highest weighted score. The determined trailer angle is passed to one or more vehicle systems configured to utilize the selected trailer angle. In some cases, trajectories corresponding to wheel detection may be further provided to the wheelbase detection module. The wheelbase detection module uses the wheelbase trajectory to determine the estimated wheelbase position and / or wheelbase length, depending on what values ​​the controller requires for a given operation. Furthermore, (multiple) trajectories may be provided to any other controllers and / or systems that can utilize the trajectory via the image.

[0034] Continuing to refer to Figures 1-3B, Figure 4 shows process 400 for determining the trailer angle. Although process 400 is described in Figure 4 in terms of operation within a single controller, it will be understood that some or all of the steps may be performed within separate controllers and / or using separate processors or threads within controllers that communicate with each other, functioning in the same way.

[0035] First, in step 410, “receive image,” the controller receives an image from the video feed. The image is a single frame from the video feed of the camera surveillance system, and process 400 is repeated for multiple images in the sequence of video feeds. In step 420, “perform line detection,” the controller performs a line detection process on the received image, and in step 430, “perform wheel detection,” it performs a wheel detection process on the received image. Both the line detection and wheel detection processes use image analysis to identify corresponding elements. Detection may be rule-based, neural network-based, or a combination of both. Separate steps 420 and 430 operate simultaneously to provide detection.

[0036] In one example, the line detection process uses the Hough transform process to identify lines and edges in an image. In an alternative example, the same effect can be achieved using an alternative line detection process such as a deep learning neural network or a similar system. In addition to the position of the lines, the line detection process identifies at least the angle of the lines, obstruction by other objects, and the start and end points of the lines. The positions of the identified lines and edges, as well as the corresponding data points for each line, are stored, and in the "identify trajectories" step 422, the path through the image of each identified line and edge is stored in memory. The path through the image frames over time in the video feed is called the object trajectory, and each edge and line trajectory is continuously updated as process 400 is repeated.

[0037] Simultaneously with line detection, wheels in the image are detected using the blob detection process in step 430, "perform wheel detection." In an alternative example, wheels may be detected using an alternative image analysis system such as a deep learning neural network. In addition to the position of the wheels in the image, wheel detection identifies the angle of the wheels relative to the trailer (alternatively also called the wheel angle) in each iteration of process 400. Similar to lines and edges, the path of each identified wheel is tracked over time through the image feed in step 432, "identify trajectories." Each identified wheel is tracked individually, and a separate trajectory is stored for each wheel.

[0038] In some examples, the number of tracked wheels, lines, and edges may be limited to 16 to conserve controller resources. In other examples, an alternative number of objects may be tracked.

[0039] The tracks from each of the track identification steps 422 and 432 are used by the controller to determine the corresponding angle of the trailer in the “determine angle from track” step 440. First, in the “determine angle from track” step 440, the two-dimensional angle of the trailer is determined for each track based on the current position in the image of the element being tracked. For example, for the driver-side rear wheel track, the two-dimensional angle of the driver-side rear wheel is identified. After determining the two-dimensional angle of each element, the two-dimensional angles are converted to three-dimensional coordinates using a conventional converter. The three-dimensional coordinates are used to determine the corresponding three-dimensional trailer angle measurement. The real-world trailer angle relative to the cab is referred to herein as the “3D trailer angle” or “3D angle” or “three-dimensional trailer angle”.

[0040] For each trailer angle measurement based on three-dimensional angle measurements, Kalman filtering is applied to correlate the three-dimensional angle with the corresponding trajectory, and the trajectory characteristics of the angle (e.g., three-dimensional angle, angular velocity, etc.) are updated using the measurement information.

[0041] After determining the 3D angles of each component, a refinement process is applied to identify which 3D angle is the most accurate. As mentioned earlier, each angle track contains multiple properties, each with a weighted numerical quantifier. The weighted numerical qualifiers of each property of a given track are summed up to generate a total score for the given track. The total scores are compared, and the trajectory with the highest score is selected as the most accurate trailer angle estimate. In another example, the weighted numerical qualifiers may be functionally minimized, where a lower score indicates higher accuracy. In such an example, the trajectory with the lowest total is selected as the most accurate trajectory. Each of the other 3D angles is disregarded, and the most accurate 3D angle is adopted as the accurate angle in the “narrow down the angles” step 450.

[0042] Once determined, the narrowed angle is reported to one or more systems within the controller and used in step 460, “Report Identified Angle.” For example, the identified angle is reported to an automated panning system that pans a Class II view presented to the user based on the determined trailer angle and one or more current operating parameters.

[0043] In some examples, identified trajectories corresponding to wheel detection are also provided to the wheelbase determination module, and wheel detection is used in step 470 of “identifying the wheelbase” to estimate or identify the position and / or length of the wheelbase of the trailer attached to the vehicle.

[0044] Continuing with Figure 4, Figure 5 schematically shows an example of an automatic panning system for a vehicle 510. The controller 520 receives an image 512 and other sensor information 514 from the vehicle 510. The controller 520 identifies lines in the received image 512 using the line detection module 522 and identifies wheel positions and angles in the received image 512 using the wheel detection module 524. The trajectories identified by the wheel detection module 524 and the line detection module 522 are provided to the angle conversion module 526. In addition to the detections from the line detection module 522 and the wheel detection module 524, the angle conversion module 526 receives sensor information 514, which includes yaw rate, host speed, and other sensor information indicating the current operation of the vehicle.

[0045] The angle conversion module 526 correlates each track received from the line detection module 522 and the wheel detection module 524 with the corresponding two-dimensional trailer angle using a Kalman filtering process. As described above, an alternative process can be used to convert each track into the corresponding angle detection. Next, the angle conversion module 526 converts each identified 2D trailer angle into the corresponding 3D trailer angle, which is provided to the filtering module 528. The filtering module 528 selects a single best trailer angle estimate based on the weighted scoring described above.

[0046] A selected single trailer angle is provided to the panning system 530. Although shown as a separate controller in the exemplary embodiment, it is understood that the panning system 530 may be included within the controller 520 as a software module in alternative examples. In one example, the automatic panning is configured to ensure that the rear edge of the trailer is maintained within a Class II view while the vehicle is in motion. In other implementations, the automatic panning may maintain other objects or portions of objects within the view as required by the current movement of the vehicle.

[0047] The identification of multiple trailer angles, narrowed down to a single precise trailer angle, can be used across multiple systems and is not limited to application to automated panning systems.

[0048] While exemplary embodiments have been disclosed, those skilled in the art will recognize that certain modifications fall within the scope of the claims. Therefore, the following claims should be considered in order to determine their true scope and content.

Claims

1. A method for automatically panning a displayed view of a commercial vehicle including a tractor, A step of receiving a video feed from at least one camera, wherein the camera defines a field of view including a trailer, A step of identifying a plurality of objects in the video feed, wherein the plurality of objects include both at least one wheel of the trailer and at least one line of the trailer, A step of tracking the path of the at least one wheel through the image plane of the video feed to identify the trajectory of at least one wheel, A step of tracking the path of the at least one line through the image plane of the video feed to identify the at least one line trajectory, The steps include: identifying the trailer angle relative to the tractor corresponding to each of the paths of at least one wheel and the paths of at least one line, and associating each identified trailer angle with the respective path of at least one wheel and the path of at least one line to generate a plurality of trailer angle measurements relative to the tractor; The steps include narrowing down multiple tracks to a single track and identifying a single trailer angle measurement for the tractor corresponding to the single track, and Methods that include...

2. The method according to claim 1, further comprising the steps of providing the single trailer angle measurement to an image panning system and panning the displayed view based at least in part on the single trailer angle and the current vehicle movement.

3. The method according to claim 2, wherein panning the displayed view includes keeping the trailer end in the view.

4. The method according to claim 3, wherein panning the displayed view includes maintaining the trailer end approximately at the center of the view.

5. Narrowing down multiple trajectories is For each orbit, determine the numerical quantifier for each of the multiple parameters of the orbit. For each orbit, the determined numerical quantifiers are summed up to determine the weighted score corresponding to that orbit. Compare the weighted scores of each orbit and identify the orbit with the highest weighted score, and The trailer angle corresponding to the trajectory with the highest weighting score is selected as the single trailer angle. The method according to claim 1, including the method described in claim 1.

6. The method according to claim 1, wherein identifying the plurality of objects includes identifying a plurality of lines and edges in an image using one of the Hough transform and a deep neural network.

7. The method according to claim 6, wherein identifying the plurality of lines and edges includes identifying the angle, start point, and end point of each line and edge among the plurality of lines and edges.

8. The method according to claim 1, wherein identifying the plurality of objects includes identifying the position of each wheel in the image using one of blob transform image analysis and a deep neural network.

9. The method according to claim 8, further comprising using image analysis to identify the wheel angle of each wheel.

10. A first camera that defines a field of view including the trailer, A controller that is communicably connected to the first camera, the controller including a processor and memory A camera system for a vehicle equipped with a tractor that tows the trailer, The aforementioned memory is The controller receives a video feed from the first camera, wherein the first camera defines a field of view including the trailer. Using the controller, identify a plurality of objects in the video feed, wherein the plurality of objects include both at least one wheel of the trailer and at least one line of the trailer. Using the controller, track the path of the at least one wheel through the image plane of the video feed to identify the at least one wheel trajectory. Using the controller, track the path of at least one line through the image plane of the video feed to identify at least one line trajectory. Using the controller, identify the trailer angle relative to the tractor corresponding to each of the paths of the at least one wheel and the at least one line, and generate a plurality of trailer angle measurements relative to the tractor by associating each identified trailer angle with the respective at least one wheel track and the at least one line track, and Using the controller, multiple tracks are narrowed down to a single track, and the trailer angle measurement value for the tractor corresponding to the single track is identified. A camera system for vehicles that stores commands to automatically pan the displayed view of the vehicle by performing certain actions.

11. The camera system according to claim 10, wherein identifying the trailer angle corresponding to each route includes converting either line detection or wheel detection of each object into a two-dimensional trailer angle, and converting the two-dimensional trailer angle into a real-world angle of the trailer angle relative to the driver's cab.

12. The controller is configured to apply Kalman filtering to each real-world trailer angle. The camera system according to claim 11, wherein the Kalman filtering associates the real-world trailer angle with a corresponding trajectory and updates at least one property of the corresponding trajectory.

13. The camera system according to claim 12, wherein the characteristics of each trajectory include at least dispersion, elapsed time, direction, and source.

14. Narrowing down the identified trajectory is For each orbit, determine the numerical quantifier for each of the multiple parameters of the orbit. For each orbit, the determined numerical quantifiers are summed up to determine the weighted score corresponding to that orbit. Compare the weighted scores corresponding to each orbit, identify the orbit with the highest weighted score, and The trailer angle corresponding to the trajectory with the highest weighting score is selected as the single trailer angle. The camera system according to claim 10, including the camera system according to claim 10.

15. The camera system according to claim 10, wherein identifying the plurality of objects includes identifying a plurality of lines and edges in the image using the Hough transform.

16. The camera system according to claim 15, wherein identifying the plurality of lines and edges includes identifying the angle, start point, and end point of each line and edge among the plurality of lines and edges.

17. The camera system according to claim 10, wherein identifying the plurality of objects includes identifying the position of each wheel in the image using blob transform image analysis.

18. The camera system according to claim 17, further comprising using image analysis to identify the wheel angle of each wheel.

Images