Vehicle camera surveillance system including an automatically calibrating camera - Patents.com

Abstract

The camera surveillance system includes a first mirror-replacement camera extending outwardly from the vehicle. The first mirror-replacement camera defines a rear-facing field of view including at least one image feature during at least a first set of operating conditions. The at least one image feature has a fixed position within the field of view during at least the first set of operating conditions. The camera surveillance system controller is configured to automatically calibrate an orientation of the first mirror-replacement camera relative to the vehicle while the vehicle is operating under the first set of operating conditions by comparing a predicted position of the at least one image feature to an actual position of the at least one image feature and identifying a displacement of the camera orientation based on the difference.

Description

[Technical field]

[0001] The present disclosure relates to a camera surveillance system (CMS) for use in a vehicle, and more particularly to a camera surveillance system that includes an auto-calibrating camera.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 293191, filed December 23, 2021. [Background technology]

[0003] Mirror replacement systems, and camera systems that supplement the mirror view, are utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. Camera surveillance systems (CMS) utilize one or more cameras to provide an enhanced field of view to the vehicle operator. In some instances, mirror replacement systems cover a wider field of view than a traditional mirror or include views that are not fully available via a traditional mirror.

[0004] Camera surveillance systems typically include one or more cameras located in wing arms that project outward from the cab at or near where a traditional rear view mirror would be located. Although alternative structures for housing the cameras are available, current camera surveillance systems require the mirror to extend away from the side of the cab to provide a full rearward field of view. As with any mechanical structure, the structure supporting the camera may weaken, loosen, or otherwise displace over time such that the camera angle (orientation of the field of view) relative to the cab may change. This may change the field of view provided by the camera, and images captured by the camera may be different given the same object position relative to the cab. Similarly, the height of the camera relative to the cab and / or the camera angle relative to the cab may displace or change over time. Summary of the Invention

[0005] In one exemplary embodiment, a camera surveillance system includes a first mirror-replacement camera extending outwardly from a vehicle, the first mirror-replacement camera defining a rear-looking field of view, the rear-looking field of view including at least one image feature having a fixed position within the field of view during at least a first set of operating conditions, and a camera surveillance system controller configured to automatically calibrate an orientation of the first mirror-replacement camera relative to the vehicle while the vehicle is operating under the first set of operating conditions by comparing a predicted position of the at least one image feature to an actual position of the at least one image feature and identifying a displacement of camera orientation based on the difference.

[0006] In another example of the above camera surveillance system, the first mirror-replacement camera is positioned in a cab of a commercial vehicle, and the at least one image feature includes at least one feature of a trailer connected to the cab.

[0007] In another example of any of the above camera surveillance systems, the at least one image feature includes at least one member from a set including at least a trailer wheel, a trailer end corner, a trailer trailing edge, and a trailer top edge.

[0008] In another example of any of the above camera surveillance systems, the at least one image feature includes at least two members from the set.

[0009] In another example of any of the above camera surveillance systems, the first set of operating conditions includes a speed of the vehicle exceeding a first threshold and a steering angle of the vehicle falling below a second threshold.

[0010] In another example of any of the above camera surveillance systems, the first set of operating conditions includes a road gradient being below a third threshold.

[0011] In another example of any of the above camera surveillance systems, the at least first set of operating conditions includes a second set of operating conditions.

[0012] In another example of any of the above camera surveillance systems, the camera surveillance system controller is further configured to respond to a displacement in the camera orientation by adjusting the position of a class II view within a class IV view if the displacement falls below a predefined magnitude.

[0013] In another example of any of the above camera surveillance systems, the support structure for the mirror replacement camera includes at least one powered articulation element, the powered articulation element controlled by the camera surveillance system controller, the camera surveillance system controller configured to respond to displacement in the camera's orientation by articulating the powered articulation element such that the displacement in the camera's orientation is reduced relative to a factory calibrated camera orientation.

[0014] In another example of any of the above camera surveillance systems, the camera surveillance system controller is configured to output a warning to a vehicle operator when an orientation or displacement of the camera exceeds a threshold magnitude.

[0015] An exemplary method for automatically calibrating an orientation of a vehicle mirror includes a camera surveillance system (CMS) controller receiving at least one image from a camera mounted on a vehicle, and comparing a predicted position of at least one image feature to an actual position of the at least one image feature while the vehicle is operating under at least a first set of operating conditions, and identifying a displacement of the camera orientation based on the difference.

[0016] In another example of the above method for automatically calibrating a vehicle mirror orientation, a camera mounted on the vehicle is positioned in a vehicle cab, and the at least one image feature includes at least one feature of a trailer connected to the vehicle cab.

[0017] In another example of any of the above methods for automatically calibrating a vehicle mirror orientation, the at least one image feature includes at least one element from a set including at least a trailer wheel, a trailer end corner, a trailer trailing edge, and a trailer top edge.

[0018] In another example of any of the above methods for automatically calibrating an orientation of a vehicle mirror, the at least one image feature includes at least two elements from the set.

[0019] In another example of any of the above methods of automatically calibrating an orientation of a vehicle mirror, the first set of operating conditions includes a speed of the vehicle exceeding a first threshold and a steering angle of the vehicle falling below a second threshold.

[0020] In another example of any of the above methods of automatically calibrating vehicle mirror orientation, the first set of operating conditions includes a road slope being below a third threshold.

[0021] In another example of any of the above methods for automatically calibrating an orientation of a vehicle mirror, the at least first set of operating conditions includes a second set of operating conditions.

[0022] Another example of any of the above methods of automatically calibrating an orientation of a vehicle mirror further includes responding to a displacement in the orientation of the camera by adjusting the position of a class II view within a class IV view if the displacement is below a predefined magnitude.

[0023] Another example of any of the above methods of automatically calibrating a vehicle mirror orientation further includes responding to a displacement in the camera orientation by articulating the powered articulation element such that the displacement in the camera orientation is reduced relative to a factory calibrated camera orientation.

[0024] Another example of any of the above methods of automatically calibrating a vehicle mirror orientation further includes outputting a warning to a vehicle operator when the camera orientation or displacement exceeds a threshold magnitude. [Brief description of the drawings]

[0025] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0026] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views.

[0027] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera surveillance system providing Class II, Class IV, Class V and Class VI views.

[0028] [Diagram 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an in-cabin camera.

[0029] [Diagram 3] 1 illustrates a schematic diagram of an exemplary wing arm structure supporting the camera portion of the CMS.

[0030] [Figure 4] 4 illustrates a schematic diagram of an exemplary view provided by the CMS of FIG. 3;

[0031] [Diagram 5] 4 illustrates a schematic of a process for automatically calibrating the camera angle of the camera in the wing arm of FIG.

[0032] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in relation to one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] A schematic diagram of a commercial vehicle 10 is shown in Figures 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although commercial trucks are contemplated in this disclosure, the invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera surveillance system (CMS) 15 (Figure 2) that includes driver and passenger side camera arms 16A, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16A, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.

[0034] With continued reference to Figures 1A and 1B, Figure 3 illustrates an exemplary camera arm 16a, 16b in a schematic manner. Each of the camera arms 16a, 16b includes a base 102 that is fixed to, for example, the cab 12 (illustrated in Figure 1A and partially illustrated in Figure 3). A pivot arm 104 is supported by the base 102 and may articulate relative thereto via a joint 106. In the simplified example of Figure 3, the pivot arm 104 is a straight arm and the joint 106 is a ball joint that can move up, down, and in and out of the plane of the illustration. At least one rear-facing camera 20a, 20b is disposed within the camera arm 16a, 16b, respectively, and is connected to the pivot arm 104 via a rotation joint (hidden in the illustration of Figure 3). The hidden joints connecting the cameras 20a, 20b to the pivot arm, like the ball joint 106, rotate about the axis of the camera and pivot about the pivot arm 104 to move in and out of the plane shown. It will be appreciated that the pivot arm 104, ball joint 106, and hidden joints connecting the cameras 20a, 20b to the pivot arm 104 have been simplified for purposes of explanation, and that more complex support structures including articulated arms, powered articulation joints, and similar devices may be utilized to achieve the positioning of the cameras 20a, 20b described herein.

[0035] In another embodiment, the camera may be a rear-facing camera fixed to the side of the cab. In such an example, the camera is mounted in a wing housing that is laterally offset from the cab body. This allows the camera to capture the trailer and the environment around the trailer.

[0036] Each of the exterior cameras 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial truck industry. EX1 , FOV EX2For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. The reference to a "class" view is not intended to be limiting, but rather as an illustration of the type of view provided to the display by a particular camera. Multiple cameras may be used in each camera arm 16a, 16b to provide these views, if desired. Each arm 16A, 16b may also provide a housing that encloses electronics configured to provide various features of the CMS 15.

[0037] First and second video displays 18A, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19A, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20A, 20b.

[0038] If Class V and Class VI view video is also required, a camera housing 16c and camera 20c may be located at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located in the cab 12 near the top center of the windshield may be used to display to the driver the Class V and Class VI views toward the front of the vehicle 10.

[0039] If Class VIII view footage is required, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of the vehicle 10. As illustrated, the Class VIII view includes a view that surrounds the immediate vicinity of the trailer and a rear close-up view of the vehicle that includes the rear of the trailer. In one example, the rear close-up view of the vehicle is generated by a rear-facing camera positioned at the rear of the vehicle and may include both an immediate rear close-up view and a traditional rear view (e.g., a field of view extending rearward to the horizon provided by a rearview mirror of a vehicle without a trailer). In such an example, the third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near the first, second, and third displays 18a, 18b, 18c to provide a dedicated display that provides a Class VIII view.

[0040] Certain operations of the camera surveillance system depend on accurate knowledge of the orientation and position of the cameras 20a, 20b relative to the cab 12 of the vehicle 10. Inaccurate understanding of the orientation and / or position can result in an erroneous view being presented to the driver and / or an incomplete view being presented to the driver. As an example, if the CMS controller understands that the cameras 20a, 20b are oriented in a first orientation, but the cameras 20a, 20b are actually oriented in a different orientation, the Class II view, which is a subset of the Class IV view presented to the vehicle driver, may include too much or not enough of the trailer 14. Furthermore, in some examples, images from the CMS cameras 20a, 20b can be used alone or in combination with other sensors to identify nearby objects, ground truth, and other features utilized in driver assistance systems, driver warning systems, and the like. In such examples, accurate knowledge of the camera positions and orientations increases the accuracy and reliability of the system on which the information is relied upon.

[0041] When the vehicle 10 is in new condition and has minimal wear, the joints 106 and the pivot arm 104 are in a calibrated position from a factory calibration and the CMS controller knows exactly the orientation of the cameras 20a, 20b relative to the cab 12. Motion of the vehicle 10 causes vibrations and other normal damage that can loosen and displace the fasteners of the camera arms 16a, 16b and the pivot arm 104, which can displace the position and orientation of the cameras 20a, 20b relative to the cab 12.

[0042] Existing systems manually recalibrate the cameras according to a set recalibration schedule. Manual recalibration involves parking the vehicle in a stationary position and placing one or more physical markers in set locations on or near the vehicle within the field of view of the cameras. With the physical markers placed in known locations and the vehicle stationary, the CMS understands where the markers should be placed in the image relative to the currently understood orientation. The positions of the markers in the image are then used to determine the actual orientation of the cameras 16a, 16b relative to the cab 12. If the actual orientation deviates from the orientation defined in the specifications by more than a set amount, a manual adjustment of the camera orientation is indicated.

[0043] Manual calibration as described above is subject to human error, including longer than expected calibration delays, imprecise placement of markers, and similar sources of error. Furthermore, manual calibration does not account for unexpected events that may increase the rate at which the mirror orientation or position shifts. The CMS in the vehicle 10 replaces or supplements the manual calibration described above with an automatic calibration that uses predicted positions of camera external elements during vehicle operation to confirm or determine the orientation of the cameras 16a, 16b relative to the cab 12. As used herein, camera external elements refer to objects and features whose positions in real-world space are independent of the camera position. By way of example, the trailer 14, trailer ends, trailer wheels, lane markings, curbs, road features, etc. are camera external elements.

[0044] Certain image features of an image captured by the camera (e.g., camera-external elements in the image) have a known or predicted position relative to the camera while the vehicle is operating within certain parameters. As an example, when the vehicle 10 is operating at a speed above a threshold and with a steering angle of less than 5 degrees, the operating parameters ensure that the trailer is maintained in a stable position relative to the cab 12 and thus the cameras 16a, 16b. In another example, while the vehicle is operating on a flat road with a steering angle of less than 5 degrees, the vanishing point of the lane lines adjacent to the trailer is at a predicted position. As used herein, a vanishing point is the intersection of lane lines that are parallel to the vehicle's direction of travel and merge into a single point at the horizon. In another example, another set of operating conditions can be utilized that results in a known positioning of one or more image features in addition to or instead of the described operating conditions.

[0045] The CMS controller automatically initiates camera calibration when a set of operating conditions is met (e.g., the vehicle is traveling above a threshold speed, the steering angle is less than 5°, and the vehicle is traveling on a flat road), or in an example having multiple sets of operating conditions. With continued reference to FIGS. 1A-3, FIG. 4 illustrates an example view provided by the CMS of FIG. 3, including actual views and predicted positions of multiple elements. The field of view 300 illustrated in FIG. 4 includes the trailer 12, multiple wheels 302, trailer end corners 304, trailing edge 304, edge 308 of the trailer 12, and lane vanishing point 307. Additionally, vanishing point 310 of lane boundary line 311 is extrapolated beyond the field of view 300.

[0046] In one example, the camera may undergo an orientation displacement that causes the camera to angle upward relative to the cab 12 and ground 301. This displacement causes the actual positions of the image features 302, 304, 306, 307, 308, 310 in the image to be lower than predicted. The CMS controller may store the set of predicted positions 302', 304', 306', 307', 308', 310' in an internal memory or retrieve the predicted positions 302', 304', 306', 307', 308', 310' from an external memory. The CMS controller compares the actual positions of the image features 302, 304, 306, 307, 308 to the predicted positions 302', 304', 306', 307', 308'. Based on the determined displacement, the CMS controller determines the amount by which the camera orientation has shifted and / or the amount of distance by which the camera has shifted.

[0047] If the actual orientation and / or distance deviates from the factory calibration by a small enough amount that no physical orientation or position change is necessary, the CMS controller can adjust the position of the field of view displayed in software (e.g., displacing a Class II view into a larger Class IV view). Alternatively, if the orientation deviates from the factory calibration by an amount that is too large to compensate for in software, the CMS controller can output a warning or alert to inform the vehicle operator that the CMS needs to make a physical adjustment to the camera orientation.

[0048] 1A-4, FIG 5 generally illustrates a process 400 for performing an automatic calibration. During operation of the vehicle 10, the CMS controller communicates with the vehicle systems via a CAN bus or similar communication system and analyzes the vehicle operating conditions in step 410 "Determine that the Vehicle Satisfies a Set of Operating Conditions." If the vehicle operating conditions satisfy one of a predefined set of operating conditions (e.g., speed above a threshold, steering angle below a threshold, and flat road conditions), the process 400 determines that the vehicle is operating within one of the defined sets and initiates the calibration process.

[0049] Entering the calibration process, the CMS controller analyzes the images provided by the cameras 20a, 20b and identifies at least one image feature corresponding to a set of operating conditions in which the vehicle is operating in step 420 "Identify Camera External Elements in Images." If the CMS controller includes multiple predefined sets of operating conditions, each set may have unique, overlapping, or identical image features depending on the particular set of conditions and the particular vehicle. While the process described herein can be completed and the camera(s) calibrated with only a single image feature, it is understood that additional image features can provide redundancy and validation to the calibration, thereby improving the calibration process 400.

[0050] After identifying the actual location of each corresponding image feature in the image, the CMS controller identifies a predicted location of each image feature in the image in "IDENTIFY PREDICTED POSITION OF IMAGE FEATURE" step 430. The predicted locations are then compared to the actual locations and differences between the locations are identified in "DETERMINE DEVIATION BETWEEN ACTUAL AND PREDICTED POSITION" step 440. The deviation is the magnitude and direction that the actual location is offset from the predicted location.

[0051] The magnitude and direction of the deviation is one-to-one correlated with the physical orientation and / or positional displacement of the camera according to an empirically determinable relationship. This empirically determinable relationship is used by the CMS controller to determine the physical displacement of the camera in "Determine Physical Camera Displacement" step 450. Once it is determined that the camera angle has shifted due to calibration, depending on the magnitude of the physical displacement, the CMS controller may branch to one of three actions.

[0052] If the orientation change is below a predefined threshold (e.g., the orientation shift is such that the class II field of view is still completely contained within the class IV field of view), the CMS controller uses software to adjust the position of the class II view to compensate for the orientation shift in "adjust image" step 462.

[0053] If the orientation displacement exceeds a predefined threshold and the camera arms 16a, 16b include powered orientation control, the CMS controller can communicate with the actuators controlling the camera arms 16a, 16b directly or via a CAN bus or similar communication system to reposition the cameras to their original orientation or to an orientation where the Class II view is entirely within the Class IV view.

[0054] If the orientation displacement exceeds a predefined threshold and the camera arms 16a, 16b are not powered, the CMS controller can output a warning to the driver indicating that the corresponding mirror-replacement camera needs to be repositioned. In some examples, if the camera orientation and / or position displacement is severe enough, this warning can further be used to switch off vehicle systems that rely on images from the mirror-replacement cameras and that may have inaccurate or suboptimal operation due to camera misalignment.

[0055] To ensure that the process runs frequently enough to maintain camera calibration without consuming valuable processing power in the CMS controller, process 400 is executed the first time a set of operating conditions is met during each engine on / off cycle, and is not executed during future times when the set of operating conditions is met within the same engine on / off cycle.

[0056] In some optional implementations, the system may include the ability for a vehicle operator to trigger the calibration process 400 or flag the system to recalibrate even if the set of operating conditions has already been met during the current engine on / off cycle. In still other implementations, the process 400 may be executed on every other engine on / off cycle, every fifth engine on / off cycle, every tenth engine on / off cycle, or any other interval depending on the conditions in which the vehicle 10 is operated.

[0057] Although exemplary embodiments have been disclosed, a person of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. a first mirror-replacement camera extending outwardly from the vehicle, the first mirror-replacement camera defining a rear-facing field of view, the rear-facing field of view including at least one image feature during at least a first set of operating conditions, the at least one image feature having a fixed position within the field of view during the at least first set of operating conditions; a camera surveillance system controller configured to automatically calibrate an orientation of the first mirror-replacement camera relative to the vehicle by comparing a predicted position of the at least one image feature to an actual position of the at least one image feature while the vehicle is operating under the first set of operating conditions, and identifying a displacement of a camera orientation based on the difference; A camera surveillance system comprising:

2. The camera surveillance system of claim 1 , wherein the first mirror-replacement camera is disposed in a cab of a commercial vehicle, and the at least one image feature includes at least one feature of a trailer connected to the cab.

3. The camera surveillance system of claim 2 , wherein the at least one image feature includes at least one member from a set including at least a trailer wheel, a trailer end corner, a trailer trailing edge, and a trailer top edge.

4. The camera surveillance system of claim 3 , wherein the at least one image feature includes at least two members from the set.

5. 2. The camera surveillance system of claim 1, wherein the first set of operating conditions includes the speed of the vehicle exceeding a first threshold and the steering angle of the vehicle falling below a second threshold.

6. The camera surveillance system of claim 5 , wherein the first set of operating conditions includes a road gradient below a third threshold.

7. The camera surveillance system of claim 1 , wherein the at least a first set of operating conditions includes a second set of operating conditions.

8. 2. The camera surveillance system of claim 1, wherein the camera surveillance system controller is further configured to respond to a displacement in the camera orientation by adjusting a position of a class II view within a class IV view if the displacement falls below a predefined magnitude.

9. 2. The camera system of claim 1, wherein the mirror replacement camera support structure includes at least one powered articulation element, the powered articulation element being controlled by the camera surveillance system controller, the camera surveillance system controller being configured to respond to displacements in the camera's orientation by articulating the powered articulation element such that the displacements in the camera's orientation are reduced relative to a factory calibrated camera orientation.

10. The camera system of claim 1 , wherein the camera surveillance system controller is configured to output a warning to a vehicle operator when an orientation or displacement of the camera exceeds a threshold magnitude.

11. 1. A method for automatically calibrating a vehicle mirror orientation, comprising: a camera surveillance system (CMS) controller receiving at least one image from a camera mounted on the vehicle; and comparing a predicted position of at least one image feature to an actual position of the at least one image feature while the vehicle is operating under at least a first set of operating conditions and identifying a displacement of a camera orientation based on the difference; A method comprising:

12. The method of claim 11 , wherein the vehicle mounted camera is disposed in a vehicle cab, and the at least one image feature includes at least one feature of a trailer connected to the vehicle cab.

13. The method of claim 12 , wherein the at least one image feature comprises at least one member from a set including at least a trailer wheel, a trailer end corner, a trailer trailing edge, and a trailer top edge.

14. The method of claim 13 , wherein the at least one image feature includes at least two members from the set.

15. The method of claim 11 , wherein the first set of operating conditions includes a speed of the vehicle exceeding a first threshold and a steering angle of the vehicle falling below a second threshold.

16. The method of claim 15 , wherein the first set of operating conditions includes a road gradient below a third threshold.

17. The method of claim 11 , wherein the at least a first set of operating conditions includes a second set of operating conditions.

18. The method of claim 11 , further comprising: responding to a displacement in the camera orientation by adjusting a position of a class II view within a class IV view if the displacement is below a predefined magnitude.

19. The method of claim 11 , further comprising responding to a displacement in camera orientation by articulating a powered articulation element such that the displacement in camera orientation is reduced relative to a factory calibrated camera orientation.

20. The method of claim 11 , further comprising outputting a warning to a vehicle operator when the camera orientation or displacement exceeds a threshold magnitude.

Images