Camera monitoring system with image processing based camera wing deployment status detection

Abstract

A method of checking a wing position in a CMS, comprising: performing a calibration of a wing position relative to a camera supporting a vehicle by capturing a plurality of images under different lighting conditions to provide a desired field of view; extracting and storing reference features from each of the plurality of images; triggering a wing position check; capturing a current image from the camera having a current position of the reference features; sensing a current lighting condition under which the current image was captured; determining that one of the different lighting conditions is more similar to the current lighting condition; comparing the current position of the reference features to the stored reference features from one of the plurality of images generated under the one of the different lighting conditions; and outputting a result of the wing position check if a difference resulting from the comparing step exceeds a threshold.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 544,319, filed October 16, 2023. Technical Field

[0003] This disclosure relates to a camera surveillance system (CMS) for use in commercial trucks or similar vehicles, and more specifically, to a CMS capable of using image processing to inspect wing positions. Background Technology

[0004] Sight mirror replacement systems and camera systems used to supplement the field of view of sight mirrors are used in commercial vehicles to enhance the vehicle operator's ability to see their surroundings. Camera monitoring systems (CMS) utilize one or more cameras to provide the vehicle operator with an enhanced field of view. In some examples, CMS covers a larger field of view than conventional sight mirrors, or includes views that cannot be fully obtained via conventional sight mirrors.

[0005] In a typical CMS, camera arms or camera wings are arranged on each of the left and right sides of the tractor to provide Category II and Category IV views. Displays are provided on each A-pillar on the driver's and passenger's sides to show the field of view of the camera arms on that side, thus simulating a conventional viewing mirror.

[0006] In some applications, the camera wing can be configured to fold manually or in response to motor actuation. Some customers may expect to automatically determine whether the camera is deployed and in a position that will continue to provide the desired view without driver input. Therefore, it may be desirable to provide wing position verification for some CMSs. One example of an existing technology system involves a physical sensor encapsulated within the camera wing, but this design requires additional components and wiring, thus increasing the cost of the CMS. Attached Figure Description

[0007] This disclosure can be further understood by referring to the following detailed description when considered in conjunction with the accompanying drawings, wherein:

[0008] Figure 1A It is a schematic front view of a commercial truck equipped with a Camera Surveillance System (CMS) for providing views of at least Category II, IV, V and VI.

[0009] Figure 1B It is a schematic top view of a commercial truck with a CMS that provides at least Class II, Class IV, Class V and Class VI views.

[0010] Figure 2 It is a schematic interior view of the carriage, including displays and other schematic CMS features.

[0011] Figure 3A and Figure 3B These correspond to the calibration image and the current image, respectively, and do not correspond to the calibration image.

[0012] Figure 4 This is a method for verifying the wing position in CMS.

[0013] Figure 5A The “acceptable” result is shown after verification at the example wing location.

[0014] Figure 5B Another "acceptable" result is shown after verification at the example wing position.

[0015] Figure 5C The results shown are "unacceptable" after verification at the example wing location.

[0016] Figure 6 Another method for wing position verification for CMS is shown.

[0017] Figure 7 The second calibration image is shown.

[0018] The embodiments, examples, and alternatives described in the foregoing paragraphs, claims, or the following description and drawings, including any of their aspects or corresponding individual features, may be employed independently or in any combination. Features described in conjunction with one embodiment apply to all embodiments unless those features are incompatible. The same reference numerals and names in the various drawings indicate the same elements. Summary of the Invention

[0019] In some examples, a method for checking the wing position in a camera surveillance system includes: performing calibration of the wing position relative to a vehicle-supported camera by capturing multiple images under different lighting conditions to provide a desired field of view; extracting and storing reference features from each of the multiple images; triggering a wing position check; capturing a current image from the camera, the current image having the current position of the reference features; sensing the current lighting conditions for capturing the current image; determining that one of the different lighting conditions is more similar to the current lighting condition; comparing the current position of the reference features with the stored reference features from one of the multiple images generated under that one lighting condition under the different lighting conditions; and outputting the result of the wing position check if the difference obtained from the comparison step exceeds a threshold.

[0020] In a further example of any of the foregoing examples, performing calibration includes: performing a first calibration, including calibrating the wing position of the camera relative to the vehicle support to provide a desired field of view, and the calibration includes generating a first calibration image from the camera under a first illumination condition, and performing a second calibration, including generating a second calibration image from the camera under a second illumination condition different from the first illumination condition.

[0021] In any of the foregoing examples, extracting and storing reference features includes: extracting and storing reference features from a first calibration image; and extracting and storing reference features from a second calibration image.

[0022] In a further example of any of the foregoing examples, the determination step includes: determining which of the first and second lighting conditions is more similar to the current lighting condition.

[0023] In a further example of any of the foregoing examples, the comparison step includes: comparing the current position of the reference feature with a stored reference feature from a calibration image of a first calibration image and a second calibration image generated under the first illumination condition and the second illumination condition.

[0024] In any of the foregoing examples, calibration is performed when the wing is mounted onto the vehicle.

[0025] In another example implementation of any of the foregoing embodiments, the calibration step is performed with the wings in the deployed position and the camera trained within the legally prescribed field of view.

[0026] In any further example of the foregoing example, the reference feature includes the vertical edge of the vehicle's tractor.

[0027] In any additional example of the foregoing example, the method includes: if the difference obtained from the comparison step exceeds a second threshold, storing the captured image as a third calibration image check.

[0028] In another example implementation of any of the foregoing implementations, the triggering step is performed based on a time interval.

[0029] In a further example of any of the foregoing examples, the comparison step includes: comparing at least a portion of the stored reference features of one of a plurality of images generated under one of different lighting conditions with the distance between the reference features and their current position in 2D space.

[0030] In a further example of any of the foregoing examples, distance is the distance between pixels in one of a plurality of images and the captured image of the current image.

[0031] In another example implementation of any of the foregoing implementations, the result is at least one of a visual warning and an audible warning.

[0032] In some examples, a camera surveillance system (CMS) for a vehicle includes a wing that can be pivotally mounted to the vehicle. A camera is mounted to the wing and includes an image capture unit configured to provide a desired field of view for the vehicle. A display depicts at least a portion of the field of view. Input is configured to trigger a wing position check. A controller communicates with the camera and the display, and the controller includes a calibration module that extracts reference features from each of a plurality of images captured under different lighting conditions providing the desired field of view. The controller includes a memory in which the reference features are stored and a wing position verification module responsive to input. The wing position verification module is configured to capture a current image of the current position with the reference features from the camera under the current lighting conditions. One or more sensors communicate with the controller and are configured to sense different lighting conditions and the current lighting condition. The wing position verification module is configured to determine that one of the different lighting conditions is more similar to the current lighting condition, compare the current position reference features with the stored reference features from one of a plurality of images associated with the one of the different lighting conditions, and output the result of the wing position check if the difference obtained from the comparison step exceeds a threshold.

[0033] In a further example of any of the foregoing examples, the reference features are extracted from a first calibration image of a camera under a first illumination condition in which the desired field of view is provided, and the reference features are extracted from a second calibration image of a camera under a second illumination condition in which the desired field of view is provided, the second illumination condition being different from the first illumination condition.

[0034] In any of the foregoing examples, one or more sensors are configured to sense a first lighting condition, a second lighting condition, and the current lighting condition.

[0035] In a further example of any of the foregoing examples, the wing position verification module is configured to determine that one of the first and second lighting conditions is more similar to the current lighting condition.

[0036] In any further example of the foregoing example, the output is a visual warning on the monitor.

[0037] In another example implementation of any of the foregoing embodiments, the desired view corresponds to a legally mandated view that provides at least one of Class II and Class IV views.

[0038] In any further example of the foregoing example, the reference feature includes the vertical edge of the vehicle's tractor.

[0039] In a further example of any of the foregoing examples, the input is at least one of a gear position sensor configured to provide gear position and an engine sensor configured to provide engine operating status.

[0040] In any further example of the foregoing example, the input is a time interval.

[0041] In a further example of any of the foregoing examples, the comparison step includes: comparing at least a portion of a stored reference feature of one of a plurality of images associated with one of the different lighting conditions with the distance between the current position of the reference feature in 2D space, and the distance being the distance between pixels in one of the plurality of images and the captured image of the current image.

[0042] In any of the foregoing examples, the wing includes a fixed portion configured to be fixed to the vehicle, and a foldable portion pivotally mounted on the fixed portion, with the camera mounted to the foldable portion.

[0043] In any of the foregoing examples, the wing includes a motor that communicates with a controller configured to fold and unfold the foldable portion relative to a fixed portion in response to a command from the controller, wherein the controller is configured to send a command to the motor in response to a difference exceeding a threshold in order to actuate the foldable portion and provide a desired field of view. Detailed Implementation

[0044] exist Figure 1A and Figure 1B A schematic diagram of a commercial vehicle 10 is shown. Vehicle 10 includes a vehicle cab or tractor unit 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 can be of any configuration. Although commercial trucks are contemplated in this disclosure, the invention can also be applied to other types of vehicles. Vehicle 10 includes a camera monitoring system (CMS) 15. Figure 2 The system includes a driver-side camera arm and a passenger-side camera arm (i.e., camera wings) 16a, 16b (collectively referred to as camera arm 16 or camera wing 16) mounted to the exterior of the vehicle cab 12. If desired, camera arms 16a, 16b may also include conventional viewing mirrors integrated therewith, although the CMS 15 can be used in some examples to completely replace the viewing mirrors. In additional examples, each side may include multiple camera arms, each arm housing one or more cameras and / or viewing mirrors.

[0045] Each of the camera arms 16a and 16b includes a base fixed to, for example, a driver's cab 12. The pivot arm is supported by the base and can be hinged relative to the base. At least one rear-facing camera 20a and 20b (collectively referred to as camera 20) is arranged within the camera arms 16a and 16b, respectively. The external cameras 20a and 20b each provide an external field of view (FOV). EX1 FOV EX2 Each field of view includes Class II and Class IV views ( Figure 1B At least one of the following is a legally mandated view in the commercial trucking industry: a Class II view on a given side of vehicle 10 is a subset of the Class IV views on the same side of vehicle 10. Multiple cameras may also be used in each camera arm 16a, 16b to provide these views if desired. For example, Class II (narrow) and Class IV (wide) views are defined in European Regulation R46, and similar driving visibility requirements apply to commercial trucks in the United States and other countries. Any reference to “class” views is not intended to be restrictive, but rather to serve as an example of the type of view provided to a display by a particular camera. Each arm 16a, 16b may also provide a housing surrounding electronics (e.g., controllers) configured to provide various features of CMS 15.

[0046] The first video display 18a and the second video display 18b (collectively referred to as display 18) are arranged on or near the A-pillars 21a, 21b (collectively referred to as A-pillar 21) on each of the driver's side and the passenger side within the vehicle cab 12 to display Class II and Class IV views on the respective sides of the vehicle 10, providing rearward-facing side views of the vehicle 10 captured by the external cameras 20a, 20b.

[0047] If video of Class V and / or Class VI views is also desired, camera housing 16c and camera 20c can be positioned at or near the front of vehicle 10 to provide those views. Figure 1B A third display 18c, positioned within the cab 12 near the top center of the windshield, can be used to display Class V and Class VI views toward the driver facing forward of the vehicle 10. Displays 18a, 18b, and 18c (collectively referred to as Display 18) face the driver's area 24 within the cab 22, where the operator sits in the driver's seat 26. The location, size, and field of view of streaming to any particular display may differ from the configuration described in this disclosure, but still incorporate the disclosed invention.

[0048] If a Class III view video is desired, the camera housing can be positioned on the sides and rear of vehicle 10 to provide some or all of the field of view encompassing the Class III region of vehicle 10. In such an example, the third display 18c may include one or more frames displaying the Class III view. Alternatively, additional displays may be added near the first display 18a, second display 18b, and third display 18c, providing a display dedicated to providing the Class III view.

[0049] It should be noted that the controller 30 used in CMS 15 ( Figure 2 The controller 30 can be used to implement various functions disclosed in this application. The controller 30 may include one or more discrete units. For example, a centralized architecture may have a common controller arranged in the vehicle 10, while a distributed architecture may use a controller, for example, disposed in each of the displays 18a, 18b. Furthermore, a portion of the controller 30 may be disposed in the vehicle 10, while another portion of the controller 30 may be located elsewhere, such as camera arm 16. In another example, a master-slave display configuration may be used, where one display includes the controller 30, and the other display receives commands from the controller 30.

[0050] In terms of hardware architecture, such a controller may include a processor, memory (e.g., memory 42, Figure 2 This includes one or more input and / or output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but not limited to, one or more buses and / or other wired or wireless connections. The local interface may have additional components omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communication. Furthermore, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0051] Controller 30 may be used to execute software, particularly stored in memory (e.g., memory 42). Figure 2 The hardware device for the software in the controller 30. The controller 30 may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device typically used to execute software instructions.

[0052] Memory may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard disk drive, magnetic tape, CD-ROM, etc.). Furthermore, memory may incorporate electronic, magnetic, optical, and / or other types of storage media. Memory may also have a distributed architecture, where various components are geographically separated but accessible by a processor.

[0053] Software in memory may include one or more individual programs, each comprising an ordered list of executable instructions for implementing logical functions. System components embodied as software may also be interpreted as source programs, executable programs (object code), scripts, or any other entity containing a set of instructions to be executed. When interpreted as source programs, the programs are translated by compilers, assemblers, interpreters, etc., and may or may not be included in memory.

[0054] The disclosed input and output devices that can be coupled to the system I / O interface may include input devices, such as, but not limited to, keyboards, mice, scanners, microphones, cameras, mobile devices, proximity devices, etc. Output devices may include, but are not limited to, printers, monitors, etc. Finally, input and output devices may also include devices that communicate as input and output, such as, but not limited to, modulators / demodulators (modems; used to access another device, system, or network), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc.

[0055] When controller 30 is in operation, the processor can be configured to execute software stored in memory, transfer data to and from memory, and typically control the operation of the computing device based on the software. The software in memory may be read, in whole or in part, by the processor, possibly buffered within the processor, and then executed.

[0056] exist Figure 1B and Figure 2 In one example schematically shown, camera arm 16 includes a fixed portion 17 attached to vehicle 10, such as the side of tractor 12. A foldable portion 19 is pivotally mounted to the fixed portion 17. In one example, a motor 39 is disposed in camera arm 16 to move the foldable portion 19 between a folded / collapsed position and an unfolded / operational position in response to a command, for example, from controller 30. In the unfolded / operational position, camera 20 is positioned in the desired view, which can provide one or more legally required views to its display 18.

[0057] refer to Figure 2The controller 30 can communicate with various vehicle components via a CAN bus, LIN bus, or other suitable communication architecture. For example, a gear position sensor 40 communicates with the controller 30 and is configured to provide vehicle gear positions, such as park, reverse, neutral, and drive. Engine sensors 44 (such as an engine speed sensor) also communicate with the controller 30 to provide engine operating states, such as engine idling. Other sensors 45 can communicate with the controller 30 to provide one or more operating states to the CMS 15 for use with the wing position verification method described below. In some embodiments, a lighting sensor 47 can communicate with the controller 30 and can be configured to sense lighting conditions associated with the vehicle, such as ambient lighting conditions when images are captured by the camera 20 disclosed herein.

[0058] Unlike existing wing position verification schemes, the disclosed CMS 15 does not use additional dedicated sensors to determine wing position. Instead, CMS 15 uses the same camera 20 and image capture unit, which provides a legally defined view to display 18 for wing position verification features. Image processing algorithm 46 communicates with controller 30 (e.g., as software residing in memory 42) to extract features from the image captured by the image capture unit of camera 20. Image processing algorithm 46 uses known image processing techniques to extract features from the captured image for various CMS functions. For example, image processing algorithm 46 can extract lines, shapes, colors, patterns, and other attributes from the captured image. These extracted attributes can be used to detect objects such as tractor wheels, lane markings, trailer edges, and other features. Exemplary wheel detection algorithm techniques are disclosed in U.S. Application Serial No. 18 / 080,031, filed December 13, 2022, entitled "Camera Monitor System for Commercial Vehicles Including Wheel Position Estimation," and U.S. Provisional Application Serial No. 63 / 405,912, filed September 13, 2022, entitled "Camera Monitor System for Commercial Vehicles Including Wheel Position Estimation," both of which are incorporated herein by reference in their entirety. Exemplary trailer edge detection algorithms are disclosed in U.S. Application Serial No. 17 / 952,459, filed September 26, 2022, entitled “TRAILER END TRACKING IN CAMERA MONITORING SYSTEM”, and U.S. Provisional Application Serial No. 63 / 405,152, filed September 9, 2022, entitled “CAMERA MONITORING SYSTEM INCLUDING TRAILERPRESENCE DETECTION USING OPTICAL FLOW”, the entire contents of which are incorporated herein by reference.Exemplary perception indication algorithm techniques are disclosed in U.S. Application Serial No. 18 / 134,261, filed April 13, 2023, entitled “Camera Mirror System Incloding Automatic Angle Adjustment for Commercial Vehicle Displays”, and U.S. Application Serial No. 18 / 124,646, filed March 22, 2023, entitled “Dynamic Longitudinal and Lateral Adjustment of Perception Lines for a Camera Mirror System for Commercial Vehicles”, the entire contents of which are incorporated herein by reference.

[0059] CMS 15 includes a calibration module 50, which is used after camera 20 has been calibrated when CMS 15 and camera arm 16 are mounted in the vehicle. Calibration module 50 is a routine (e.g., software residing in memory 42) executed by controller 30 once the legally required view is established. Figure 4 As shown in the wing position verification method 100, the calibration module is used to calibrate the wing position and its position relative to the vehicle's supporting camera in order to provide the desired field of view (box 102). The calibration module extracts reference features from the calibration image from the image capture unit of camera 20 (box 104). Figure 3A An example calibration image is shown. Reference features can be vehicle body contours, components (…). Figure 3B 62) Vertical edge ( Figure 3B (60) or marking. When the camera arm 16 is mounted to the tractor 12, it is desirable to extract the reference feature from the tractor 12 because the trailer 14 moves during use, and the trailer 14 may not be attached to the tractor when the calibration of the CMS 15 is installed on the tractor 12 or when the wing position is to be verified.

[0060] CMS 15 also includes a wing position verification module 52, which also uses an image processing algorithm 46. The wing verification module 52 is a routine executed by the controller 30 (e.g., software residing in memory 42). Figure 4 As shown in method 100, wing position verification module 52 (block 106) is utilized in response to an event that warrants a wing position check. The wing position check can be triggered based on the vehicle's operating state. For example, it may be desirable to initiate a wing position check when the engine is started and / or when the vehicle is moved out of parking gear and into a gear (reverse or drive). Alternatively or additionally, the wing position check can occur at predetermined time intervals during vehicle operation, such as every few seconds.

[0061] Once wing position verification has been triggered, the current image ( Figure 3BThe image is captured by camera 20, and image processing algorithm 46 attempts to extract the current position of the same features previously captured and stored during calibration (box 108). The current position of the reference feature is compared with the stored reference feature to see if camera 20 has unintendedly moved from its initial calibration position (box 110), which may result in the camera not providing a legally descriptive view. This comparison can be performed by comparing the distance between at least a portion of the stored reference feature and the current position of the reference feature in two-dimensional space. This distance may correspond to the distance between pixels in the calibration image and the captured image from the image capture unit in the current image.

[0062] It may be desirable to provide some minimal difference between the current position of the reference feature and the stored reference feature to prevent any false alarms related to the wing position being outside its expected position. For example, an example calibration camera provides a 70-degree field of view, while the legally prescribed field of view may only be 50 degrees. Therefore, there may be some tolerance for the camera wing not being in the proper position but still providing a legally described view. Therefore, the output is based on a comparison relative to a threshold (box 112; e.g., the reference feature and the current position of the same feature are within a predetermined number of pixels). The output can be, for example, a visual or audible warning on display 18.

[0063] An example of wing folding verification is in Figures 5A to 5C As shown in the figure. View 200 shows the original calibrated wing position, which provides the legally prescribed view. Figure 5A View 202 in the figure represents wing verification, where the camera arm is determined to be deployed in the original calibration position, as in view 200. Figure 5B View 212 in the diagram represents wing verification, where the camera arm is slightly offset from its original calibration position, but still provides the legally required view. Figure 5C View 222 in the diagram represents a wing verification where the camera arm is offset, in which case the camera arm is no longer able to provide the legally required view. In the example shown, views 200, 202, 212, and 222 are depicted on the Category II (narrow FOV) portion of the co-pilot-side display 18.

[0064] Points 204~211, 204′~211′ and the lines are shown for illustrative purposes only and will not be shown on display 18. Points 204~211 correspond to various reference features from the calibration image from the camera. Points 204′~211′ correspond to the current positions of similarly numbered reference features related to the wing position verification process. For illustrative purposes only, the lines indicate the horizontal matching of the reference features in their calibration positions and their current positions.

[0065] Points 204′~211′ shown in views 202, 212, and 214 are points having a pixel distance sufficiently close to the original reference feature (e.g., within 10 pixels of the original reference feature). If the current positions of a sufficient predetermined number of reference features are maintained, it can be assumed that the wing position provides a legally prescribed view (e.g., Figure 5B However, if this threshold number of reference features is not maintained, the camera arm position is too far (e.g., Figure 5C This indicates that the camera arm problem should be addressed.

[0066] If the wing position is determined to be outside its desired position and the camera arm 16 is powered by the motor 39, the CMS 15 can actuate the foldable portion 19 to a folded / collapsed position and then attempt to redeploy the foldable portion 19 to its unfolded / operational position to restore the desired field of view. At this time, the wing position verification module 52 can re-examine the new current position relative to the stored reference features. This process can capture multiple images during the folding operation and analyze those images in real time to identify the camera arm position, where a sufficient number of current feature positions match the original feature positions in the stored captured images. The motor 39 can then be de-energized, thereby holding the camera arm in the position providing the legally prescribed view.

[0067] Figure 6 It shows something that is basically similar to Figure 4 A flowchart of another example of method 300 of method 100 is shown. Fewer or more steps than those described below may be performed within the scope of this disclosure, and the order of the described steps is not intended to limit this disclosure. In method 300, two or more calibration images may be used to compare with a currently captured image. In some embodiments, two or more calibration images are captured under different lighting conditions, and reference features from each image are stored for future comparison in wing position checks. The captured image can then be compared with a calibration image under similar lighting conditions. In some examples, comparison with a calibration image under similar lighting conditions can lead to improved accuracy in wing position checks. In some examples, performance and / or accuracy may continue and / or improve under different lighting conditions after use.

[0068] Calibration module 50 is used to calibrate the wing position and its support camera relative to the vehicle to provide a desired field of view (box 302). The calibration module extracts reference features from the calibration image from the image capture unit of camera 20 (box 104). Calibration module 50 extracts and stores reference features from a second calibration image from the image capture unit of camera 20 (box 304B). In some embodiments, the second calibration image may be compared with the first calibration image before storing it for reference, for example, in the same manner as disclosed herein. If the positions of one or more reference features in the second calibration image are sufficiently similar to the positions of one or more reference features in the first calibration image, the second calibration image may be stored for reference. In some embodiments, the second image at box 304B is captured under different lighting conditions than the first calibration image. Although two calibration images are captured in this example, in some embodiments, additional calibration images may be captured and stored, including additional images under various lighting conditions.

[0069] Example of a second calibration image in Figure 7 As shown in the figure, and can be substantially similar to Figure 3A The calibration images shown are, except that the second calibration image was captured under darker lighting conditions than the original calibration image. In some embodiments, the original calibration image is captured and stored during assembly in a bright factory environment, and the second calibration image is captured and stored during a darker road environment. In some embodiments, the original calibration image is captured and stored in an environment darker than the second calibration image. In some embodiments, three or more calibration images are captured, each under different lighting conditions. In some examples, a later-captured calibration image can be compared with one or more earlier calibration images and stored if the similarity (such as similarity with respect to one or more reference features) is above a certain threshold. In some examples, a second calibration image can be compared with the original calibration image and stored if the similarity (such as similarity with respect to one or more reference features) is above a certain threshold.

[0070] In response to an event indicating a desired wing position check, the wing position verification module 52 (box 306) is utilized. Once wing position verification has been triggered, the current image ( Figure 3BThe image is captured by camera 20, and image processing algorithm 46 attempts to extract the current position of the same features previously captured and stored during calibration (box 308). In some implementations, illumination sensor 47 senses illumination conditions when capturing the current image. Wing position verification module 52 can then determine which of two or more stored images has illumination conditions most similar to the current image, and select the stored image with the most similar illumination conditions for comparison (box 309).

[0071] The current position of the reference feature is compared with the stored reference feature of the selected stored image to see if the camera 20 has unintendedly moved from its initial calibration position (box 110), which could prevent the camera from providing a legally descriptive view. This comparison can be performed by comparing the distance between at least a portion of the stored reference feature and its current position in two-dimensional space. This distance can correspond to the distance between pixels in the calibration image and the captured image from the image capture unit of the current image.

[0072] There may be some tolerance for a view that provides a legal description even when the camera wing is not in the proper position. Therefore, the output is based on a comparison relative to a threshold (box 312; for example, the current position of a reference feature and the same feature is within a predetermined number of pixels). The output may be, for example, a visual or audible warning on display 18. In some embodiments, if the captured image has at least a threshold similarity to one or more calibration images, the captured image may be stored as an appendage and / or replacement of the calibration images. In those embodiments, the wing position verification module 52 may intelligently update its ground reality features after prolonged use.

[0073] While different lighting conditions have been presented as examples of capturing and storing multiple calibration images, other reasons may exist in other examples. In some examples, the portion of the vehicle shown in the calibration image may have chipped paint or other damage, which could cause an error warning that the camera is not in the proper position. In other examples, the portion of the vehicle shown in the calibration image may have undergone a new paint job. In these examples, the newly captured image can be stored as an additional calibration image, or it can be stored as a replacement for a previously captured calibration image.

[0074] A method for checking the wing position in a camera surveillance system may include: performing calibration of the wing position relative to a vehicle-supported camera by capturing multiple images under different lighting conditions to provide a desired field of view; extracting and storing reference features from each of the multiple images; triggering a wing position check; capturing a current image from the camera, the current image having the current position of the reference features; sensing the current lighting conditions for capturing the current image; determining that one of the different lighting conditions is more similar to the current lighting condition; comparing the current position of the reference features with the stored reference features from one of the multiple images generated under that lighting condition in the different lighting conditions; and outputting the result of the wing position check if the difference from the comparison step exceeds a threshold.

[0075] A camera surveillance system (CMS) for a vehicle may include: a camera pivotally mounted to the wing of the vehicle; a camera mounted to the wing and having an image capture unit configured to provide a desired field of view for the vehicle; a display configured to depict at least a portion of the field of view; an input configured to trigger a wing position check; and a controller communicating with the camera and the display. The controller may include a calibration module in which reference features are extracted from each of a plurality of images captured under different lighting conditions providing the desired field of view, and the controller may have a memory for storing the reference features. The controller may include a wing position verification module responsive to an input, the wing position verification module being configured to capture a current image from the camera showing the current position with reference features under the current lighting conditions. One or more sensors communicating with the controller may be configured to sense different lighting conditions and the current lighting condition. The wing position verification module is configured to determine that one of the different lighting conditions is more similar to the current lighting condition, compare the current position reference features with the stored reference features of one of a plurality of images associated with that lighting condition in the different lighting conditions, and output the result of the wing position check if the difference from the comparison step exceeds a threshold.

[0076] It should also be understood that although a particular arrangement of components is disclosed in the illustrated embodiments, other arrangements will benefit from it. Although a particular sequence of steps has been shown, described, and claimed, it should be understood that, unless otherwise stated, the steps may be performed, separated, or combined in any order and will still benefit from the invention.

[0077] Although the different examples have the specific components shown in the illustrations, embodiments of the invention are not limited to those specific combinations. Some parts or features from one example may be used in combination with features or parts from another example.

[0078] Although example embodiments have been disclosed, those skilled in the art will recognize that certain modifications will fall within the scope of the claims. Therefore, the following claims should be examined to determine their true scope and content.

Claims

1. A method for checking the position of a wing in a camera monitoring system, comprising: By capturing multiple images under different lighting conditions, calibration is performed relative to the wing position of the vehicle-supported camera to provide the desired field of view; Reference features are extracted and stored from each of the plurality of images; Trigger wing position check; Capture a current image from the camera, the current image having the current position of the reference feature; Sensing captures the current lighting conditions of the current image; Determine which of the different lighting conditions is more similar to the current lighting condition; The current position of the reference feature is compared with the stored reference feature of one of the plurality of images generated under the one of the different lighting conditions; as well as If the difference obtained from the comparison step exceeds the threshold, the result of the wing position check is output.

2. The method according to claim 1, wherein, Performing calibration includes: Performing a first calibration includes calibrating the wing position of the vehicle-supported camera relative to provide a desired field of view, wherein the calibration includes generating a first calibration image from the camera under a first illumination condition; and Performing a second calibration includes generating a second calibration image from the camera under a second illumination condition different from the first illumination condition; Extracting and storing reference features includes: Extract and store reference features from the first calibration image; and The reference features are extracted and stored from the second calibration image; The determination steps include: Determine that one of the first and second lighting conditions is more similar to the current lighting condition; and The comparison steps include: The current position of the reference feature is compared with a stored reference feature from one of the calibration images generated under the first and second lighting conditions.

3. The method according to claim 1, wherein, The calibration is performed when the wing is mounted onto the vehicle.

4. The method according to claim 3, wherein, The calibration is performed with the wings in the deployed position and the camera trained within the legally prescribed field of view.

5. The method according to claim 1, wherein, The reference feature includes the vertical edge of the tractor unit of the vehicle.

6. The method according to claim 1, comprising: If the difference obtained from the comparison step exceeds the second threshold, the captured image is stored as a third calibration image for inspection.

7. The method according to claim 1, wherein, Triggering steps are executed based on time intervals.

8. The method according to claim 1, wherein, The comparison step includes comparing at least a portion of a stored reference feature of one of the plurality of images generated under the one of the different lighting conditions with the distance between the reference feature and the current position of the reference feature in 2D space.

9. The method according to claim 8, wherein, The distance is the distance between a pixel in one of the plurality of images and the captured image of the current image.

10. The method according to claim 1, wherein, The result is at least one of a visual warning and an audible warning.

11. A camera monitoring system (CMS) for vehicles, comprising: A wing that can be pivotally mounted to the vehicle; A camera, which is mounted to the wing and has an image capture unit configured to provide the vehicle with a desired field of view; A display configured to depict at least a portion of the field of view; The input is configured to trigger a wing position check; A controller, communicating with the camera and the display, includes a calibration module in which reference features are extracted from each of a plurality of images captured under different lighting conditions providing the desired field of view. The controller also has a memory for storing the reference features. The controller includes a wing position verification module responsive to the input, the wing position verification module being configured to: capture a current image of the current position from the camera under current lighting conditions, having the reference feature; as well as One or more sensors, which communicate with the controller and are configured to sense the different lighting conditions and the current lighting conditions; The wing position verification module is configured to determine that one of the different lighting conditions is more similar to the current lighting condition, compare the current position reference features with stored reference features from one of the plurality of images associated with the one of the different lighting conditions, and output the result of the wing position check if the difference from the comparison step exceeds a threshold.

12. The CMS according to claim 11, wherein, Under a first illumination condition providing the desired field of view, the reference feature is extracted from a first calibration image of the camera, and under a second illumination condition different from the first illumination condition, the reference feature is extracted from a second calibration image of the camera providing the desired field of view; The one or more sensors are configured to sense the first lighting condition, the second lighting condition, and the current lighting condition; and The wing position verification module is configured to determine that one of the first lighting condition and the second lighting condition is more similar to the current lighting condition.

13. The CMS according to claim 11, wherein, The output is a visual warning on the display.

14. The CMS according to claim 11, wherein, The desired view corresponds to a legally mandated view that provides at least one of a Class II view and a Class IV view.

15. The CMS according to claim 11, wherein, The reference feature includes the vertical edge of the vehicle's tractor.

16. The CMS according to claim 11, wherein, The input is at least one of a gear position sensor configured to provide gear position and an engine sensor configured to provide engine operating status.

17. The CMS according to claim 11, wherein, The input is a time interval.

18. The CMS according to claim 11, wherein, The comparison step includes: comparing at least a portion of a stored reference feature of one of the plurality of images associated with the one of the different lighting conditions with the distance between the reference feature and the current position in 2D space, wherein the distance is the distance between pixels in the one of the plurality of images and the captured image of the current image.

19. The CMS according to claim 11, wherein, The wing includes a fixed portion configured to be fixed to the vehicle and a foldable portion pivotally mounted on the fixed portion, and the camera is mounted to the foldable portion.

20. The CMS according to claim 19, wherein, The wing includes a motor that communicates with the controller, the motor being configured to fold and unfold the foldable portion relative to the fixed portion in response to a command from the controller, wherein the controller is configured to send the command to the motor in response to the difference being higher than the threshold, so as to actuate the foldable portion and provide the desired field of view.

Images