Camera monitoring system with image processing-based phase wing expansion state detection

Abstract

A method of checking a wing position in a camera surveillance system includes calibrating a wing position supporting a camera relative to a vehicle to provide a desired field of view; extracting a reference feature from the calibration image from the camera; storing the reference features; triggering a wing position check; capturing a current image from the camera, the current image having a current position of the reference feature; comparing the current position of the reference feature with the stored reference feature; and outputting a result of the wing position check if the difference value from the comparing step exceeds a threshold value.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 544,319, filed October 16, 2023. Technical Field

[0003] This disclosure relates to a camera surveillance system (CMS) for use in commercial trucks or similar vehicles, and more specifically, to a CMS capable of using image processing to inspect wing positions. Background Technology

[0004] Sight mirror replacement systems and camera systems used to supplement the field of view of sight mirrors are used in commercial vehicles to enhance the vehicle operator's ability to see their surroundings. Camera monitoring systems (CMS) utilize one or more cameras to provide the vehicle operator with an enhanced field of view. In some examples, CMS covers a larger field of view than conventional sight mirrors, or includes views that cannot be fully obtained via conventional sight mirrors.

[0005] In a typical CMS, camera arms or camera wings are arranged on each of the left and right sides of the tractor to provide Category II and Category IV views. Displays are provided on each A-pillar on the driver's and passenger's sides to show the field of view of the camera arms on that side, thus simulating a conventional viewing mirror.

[0006] In some applications, the camera wing can be configured to fold manually or in response to motor actuation. Some customers may expect to automatically determine whether the camera is deployed and in a position that will continue to provide the desired view without driver input. Therefore, it may be desirable to provide camera wing position verification for some CMSs. One example of an existing technology system involves a physical sensor encapsulated within the camera wing, but this design requires additional components and wiring, thus increasing the cost of the CMS. Summary of the Invention

[0007] In one exemplary embodiment, a method for checking the wing position in a camera surveillance system includes: calibrating the wing position relative to a vehicle-supported camera to provide a desired field of view; extracting reference features from a calibration image from the camera; storing the reference features; triggering a wing position check; capturing a current image from the camera, the current image having the current position of the reference features; comparing the current position of the reference features with the stored reference features; and outputting the result of the wing position check if the difference obtained from the comparison step exceeds a threshold.

[0008] In another example implementation of any of the foregoing embodiments, a calibration step is performed when the wing is mounted onto the vehicle.

[0009] In another example implementation of any of the foregoing embodiments, the calibration step is performed with the wings in the deployed position and the camera trained within the legally prescribed field of view.

[0010] In another example implementation of any of the foregoing embodiments, the reference feature is set on the tractor unit of the vehicle.

[0011] In another example implementation of any of the foregoing embodiments, the reference feature includes the vertical edge of the tractor.

[0012] In another example implementation of any of the foregoing embodiments, the triggering step is performed based on at least one of the gear position and the engine operating state.

[0013] In another example implementation of any of the foregoing implementations, the triggering step is performed based on a time interval.

[0014] In another example implementation of any of the foregoing embodiments, the comparison step includes: comparing at least a portion of the stored reference feature with the distance between the reference feature and its current position in 2D space.

[0015] In another example implementation of any of the foregoing embodiments, the distance is the distance between pixels in the calibration image and the captured image of the current image.

[0016] In another example implementation of any of the foregoing implementations, the result is at least one of a visual warning and an audible warning.

[0017] In another exemplary embodiment, a camera surveillance system (CMS) for a vehicle includes: a wing having a fixed portion configured to be fixed to the vehicle and a foldable portion pivotally mounted on the fixed portion; a camera mounted to the foldable portion and having an image capture unit configured to provide a desired field of view for the vehicle; a display configured to depict at least a portion of the field of view; an input configured to trigger a wing position check; and a controller communicating with the camera and the display. The controller includes a calibration module in which reference features are extracted from a calibration image of the camera, in which the desired field of view is provided. The controller has a memory in which the reference features are stored. The controller includes a wing position verification module responsive to the input. The wing position verification module is configured to capture a current image of the current position with reference features from the camera, compare the current position reference features with the stored reference features, and output the result of the wing position check if the difference from the comparison step exceeds a threshold.

[0018] In another example embodiment of any of the foregoing embodiments, the wing includes a motor communicating with a controller. The motor is configured to fold and unfold the foldable portion relative to the fixed portion in response to a command from the controller. The controller is configured to send a command to the motor in response to a difference exceeding a threshold, thereby actuating the foldable portion and providing a desired field of view.

[0019] In another example implementation of any of the foregoing implementations, the output is a visual warning on the display.

[0020] In another example implementation of any of the foregoing embodiments, the desired view corresponds to a legally mandated view that provides at least one of Class II and Class IV views.

[0021] In another example implementation of any of the foregoing embodiments, the reference feature is set on the tractor unit of the vehicle.

[0022] In another example implementation of any of the foregoing embodiments, the reference feature includes the vertical edge of the tractor.

[0023] In another example implementation of any of the foregoing embodiments, the input is at least one of a gear position sensor configured to provide gear position and an engine sensor configured to provide engine operating status.

[0024] In another example implementation of any of the foregoing implementations, the input is a time interval.

[0025] In another example implementation of any of the foregoing embodiments, the comparison step includes comparing the distance between at least a portion of the stored reference feature and the current position of the reference feature in 2D space.

[0026] In another example implementation of any of the foregoing embodiments, the distance is the distance between pixels in the calibration image and the captured image of the current image. Attached Figure Description

[0027] This disclosure can be further understood by referring to the following detailed description when considered in conjunction with the accompanying drawings, wherein:

[0028] Figure 1A It is a schematic front view of a commercial truck equipped with a Camera Surveillance System (CMS) for providing views of at least Category II, IV, V and VI.

[0029] Figure 1B It is a schematic top view of a commercial truck with a CMS that provides at least Class II, Class IV, Class V and Class VI views.

[0030] Figure 2 It is a schematic interior view of the carriage, including displays and other schematic CMS features.

[0031] Figure 3A and Figure 3B These correspond to the calibration image and the current image, respectively, and do not correspond to the calibration image.

[0032] Figure 4 This is a method for verifying the wing position in CMS.

[0033] Figure 5A The “acceptable” result is shown after verification at the example wing location.

[0034] Figure 5B Another "acceptable" result is shown after verification at the example wing position.

[0035] Figure 5A The results shown are "unacceptable" after verification at the example wing location.

[0036] The embodiments, examples, and alternatives described in the foregoing paragraphs, claims, or the following description and drawings, including any of their aspects or corresponding individual features, may be employed independently or in any combination. Features described in conjunction with one embodiment apply to all embodiments unless those features are incompatible. The same reference numerals and names in the various drawings indicate the same elements. Detailed Implementation

[0037] exist Figure 1A and Figure 1B A schematic diagram of a commercial vehicle 10 is shown. Vehicle 10 includes a vehicle cab or tractor unit 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 can be of any configuration. Although commercial trucks are contemplated in this disclosure, the invention can also be applied to other types of vehicles. Vehicle 10 includes a camera monitoring system (CMS) 15. Figure 2 The system includes a driver-side camera arm and a passenger-side camera arm (i.e., camera wings) 16a, 16b (collectively referred to as camera arm 16 or camera wing 16) mounted to the exterior of the vehicle cab 12. If desired, camera arms 16a, 16b may also include conventional viewing mirrors integrated therewith, although the CMS 15 can be used in some examples to completely replace the viewing mirrors. In additional examples, each side may include multiple camera arms, each arm housing one or more cameras and / or viewing mirrors.

[0038] Each of the camera arms 16a and 16b includes a base fixed to, for example, a driver's cab 12. The pivot arm is supported by the base and can be hinged relative to the base. At least one rear-facing camera 20a and 20b (collectively referred to as camera 20) is arranged within the camera arms 16a and 16b, respectively. The external cameras 20a and 20b each provide an external field of view (FOV). EX1 FOVEX2 Each field of view includes Class II and Class IV views ( Figure 1B At least one of the following is a legally mandated view in the commercial trucking industry: a Class II view on a given side of vehicle 10 is a subset of the Class IV views on the same side of vehicle 10. Multiple cameras may also be used in each camera arm 16a, 16b to provide these views if desired. For example, Class II (narrow) and Class IV (wide) views are defined in European Regulation R46, and similar driving visibility requirements apply to commercial trucks in the United States and other countries. Any reference to “class” views is not intended to be restrictive, but rather to serve as an example of the type of view provided to a display by a particular camera. Each arm 16a, 16b may also provide a housing surrounding electronics (e.g., controllers) configured to provide various features of CMS 15.

[0039] The first video display 18a and the second video display 18b (collectively referred to as display 18) are arranged on or near the A-pillars 21a, 21b (collectively referred to as A-pillar 21) on each of the driver's side and the passenger side within the vehicle cab 12 to display Class II and Class IV views on the respective sides of the vehicle 10, providing rearward-facing side views of the vehicle 10 captured by the external cameras 20a, 20b.

[0040] If video of Class V and / or Class VI views is also desired, camera housing 16c and camera 20c can be positioned at or near the front of vehicle 10 to provide those views. Figure 1B A third display 18c, positioned within the cab 12 near the top center of the windshield, can be used to display Class V and Class VI views toward the driver facing forward of the vehicle 10. Displays 18a, 18b, and 18c (collectively referred to as Display 18) face the driver's area 24 within the cab 22, where the operator sits in the driver's seat 26. The location, size, and field of view of streaming to any particular display may differ from the configuration described in this disclosure, but still incorporate the disclosed invention.

[0041] If a Class III view video is desired, the camera housing can be positioned on the sides and rear of vehicle 10 to provide some or all of the field of view encompassing the Class III region of vehicle 10. In such an example, the third display 18c may include one or more frames displaying the Class III view. Alternatively, additional displays may be added near the first display 18a, second display 18b, and third display 18c, providing a display dedicated to providing the Class III view.

[0042] It should be noted that the controller 30 used in CMS 15 ( Figure 2The controller 30 can be used to implement various functions disclosed in this application. The controller 30 may include one or more discrete units. For example, a centralized architecture may have a common controller arranged in the vehicle 10, while a distributed architecture may use a controller, for example, disposed in each of the displays 18a, 18b. Furthermore, a portion of the controller 30 may be disposed in the vehicle 10, while another portion of the controller 30 may be located elsewhere, such as camera arm 16. In another example, a master-slave display configuration may be used, where one display includes the controller 30, and the other display receives commands from the controller 30.

[0043] In terms of hardware architecture, such a controller may include a processor, memory (e.g., memory 42, Figure 2 This includes one or more input and / or output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but not limited to, one or more buses and / or other wired or wireless connections. The local interface may have additional components omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communication. Furthermore, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0044] Controller 30 may be used to execute software, particularly stored in memory (e.g., memory 42). Figure 2 The hardware device for the software in the controller 30. The controller 30 may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device typically used to execute software instructions.

[0045] Memory may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard disk drive, magnetic tape, CD-ROM, etc.). Furthermore, memory may incorporate electronic, magnetic, optical, and / or other types of storage media. Memory may also have a distributed architecture, where various components are geographically separated but accessible by a processor.

[0046] Software in memory may include one or more individual programs, each comprising an ordered list of executable instructions for implementing logical functions. System components embodied as software may also be interpreted as source programs, executable programs (object code), scripts, or any other entity containing a set of instructions to be executed. When interpreted as source programs, the programs are translated by compilers, assemblers, interpreters, etc., and may or may not be included in memory.

[0047] The disclosed input and output devices that can be coupled to the system I / O interface may include input devices, such as, but not limited to, keyboards, mice, scanners, microphones, cameras, mobile devices, proximity devices, etc. Output devices may include, but are not limited to, printers, monitors, etc. Finally, input and output devices may also include devices that communicate as input and output, such as, but not limited to, modulators / demodulators (modems; used to access another device, system, or network), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc.

[0048] When controller 30 is in operation, the processor can be configured to execute software stored in memory, transfer data to and from memory, and typically control the operation of the computing device based on the software. The software in memory may be read, in whole or in part, by the processor, possibly buffered within the processor, and then executed.

[0049] exist Figure 1B and Figure 2 In one example schematically shown, camera arm 16 includes a fixed portion 17 attached to vehicle 10, such as the side of tractor 12. A foldable portion 19 is pivotally mounted to the fixed portion 17. In one example, a motor 39 is disposed in camera arm 16 to move the foldable portion 19 between a folded / collapsed position and an unfolded / operational position in response to a command, for example, from controller 30. In the unfolded / operational position, camera 20 is positioned in the desired view, which can provide one or more legally required views to its display 18.

[0050] refer to Figure 2 The controller 30 can communicate with various vehicle components via a CAN bus, LIN bus, or other suitable communication architecture. For example, the gear position sensor 40 communicates with the controller 30 and is configured to provide vehicle gear positions, such as park, reverse, neutral, and drive. Engine sensors 44 (such as an engine speed sensor) also communicate with the controller 30 to provide engine operating status, such as engine idle speed. Other sensors 45 can communicate with the controller 30 to provide one or more operating statuses to the CMS 15 for use with the wing position verification method described below.

[0051] Unlike existing wing position verification schemes, the disclosed CMS 15 does not use additional dedicated sensors to determine wing position. Instead, CMS 15 uses the same camera 20 and image capture unit, which provides a legally defined view to display 18 for wing position verification features. Image processing algorithm 46 communicates with controller 30 (e.g., as software residing in memory 42) to extract features from the image captured by the image capture unit of camera 20. Image processing algorithm 46 uses known image processing techniques to extract features from the captured image for various CMS functions. For example, image processing algorithm 46 can extract lines, shapes, colors, patterns, and other attributes from the captured image. These extracted attributes can be used to detect objects such as tractor wheels, lane markings, trailer edges, and other features. Exemplary wheel detection algorithm techniques are disclosed in U.S. Application Serial No. 18 / 080,031, filed December 13, 2022, entitled "Camera Monitor System for Commercial Vehicles Including Wheel Position Estimation," and U.S. Provisional Application Serial No. 63 / 405,912, filed September 13, 2022, entitled "Camera Monitor System for Commercial Vehicles Including Wheel Position Estimation," both of which are incorporated herein by reference in their entirety. Exemplary trailer edge detection algorithms are disclosed in U.S. Application Serial No. 17 / 952,459, filed September 26, 2022, entitled “TRAILER END TRACKING IN CAMERA MONITORING SYSTEM”, and U.S. Provisional Application Serial No. 63 / 405,152, filed September 9, 2022, entitled “CAMERA MONITORING SYSTEM INCLUDING TRAILERPRESENCE DETECTION USING OPTICAL FLOW”, the entire contents of which are incorporated herein by reference.Exemplary perception indication algorithm techniques are disclosed in U.S. Application Serial No. 18 / 134,261, filed April 13, 2023, entitled “Camera Mirror System Incloding Automatic Angle Adjustment for Commercial Vehicle Displays”, and U.S. Application Serial No. 18 / 124,646, filed March 22, 2023, entitled “Dynamic Longitudinal and Lateral Adjustment of Perception Lines for a Camera Mirror System for Commercial Vehicles”, the entire contents of which are incorporated herein by reference.

[0052] CMS 15 includes a calibration module 50, which is used after camera 20 has been calibrated when CMS 15 and camera arm 16 are mounted in the vehicle. Calibration module 50 is a routine (e.g., software residing in memory 42) executed by controller 30 once the legally required view is established. Figure 4 As shown in the wing position verification method 100, the calibration module is used to calibrate the wing position and its position relative to the vehicle's supporting camera in order to provide the desired field of view (box 102). The calibration module extracts reference features from the calibration image from the image capture unit of camera 20 (box 104). Figure 3A An example calibration image is shown. Reference features can be vehicle body contours, components (…). Figure 3B 62) Vertical edge ( Figure 3B (60) or marking. When the camera arm 16 is mounted to the tractor 12, it is desirable to extract the reference feature from the tractor 12 because the trailer 14 moves during use, and the trailer 14 may not be attached to the tractor when the calibration of the CMS 15 is installed on the tractor 12 or when the wing position is to be verified.

[0053] CMS 15 also includes a wing position verification module 52, which also uses an image processing algorithm 46. The wing verification module 52 is a routine executed by the controller 30 (e.g., software residing in memory 42). Figure 4 As shown in method 100, wing position verification module 52 (block 106) is utilized in response to an event that warrants a wing position check. The wing position check can be triggered based on the vehicle's operating state. For example, it may be desirable to initiate a wing position check when the engine is started and / or when the vehicle is moved out of parking gear and into a gear (reverse or drive). Alternatively or additionally, the wing position check can occur at predetermined time intervals during vehicle operation, such as every few seconds.

[0054] Once wing position verification has been triggered, the current image ( Figure 3BThe image is captured by camera 20, and image processing algorithm 46 attempts to extract the current position of the same features previously captured and stored during calibration (box 108). The current position of the reference feature is compared with the stored reference feature to see if camera 20 has unintendedly moved from its initial calibration position (box 110), which may result in the camera not providing a legally descriptive view. This comparison can be performed by comparing the distance between at least a portion of the stored reference feature and the current position of the reference feature in two-dimensional space. This distance may correspond to the distance between pixels in the calibration image and the captured image from the image capture unit in the current image.

[0055] It may be desirable to provide some minimal difference between the current position of the reference feature and the stored reference feature to prevent any false alarms related to the wing position being outside its expected position. For example, an example calibration camera provides a 70-degree field of view, while the legally prescribed field of view may only be 50 degrees. Therefore, there may be some tolerance for the camera wing not being in the proper position but still providing a legally described view. Therefore, the output is based on a comparison relative to a threshold (box 112; e.g., the reference feature and the current position of the same feature are within a predetermined number of pixels). The output can be, for example, a visual or audible warning on display 18.

[0056] An example of wing folding verification is in Figures 5A to 5C As shown in the figure. View 200 shows the original calibrated wing position, which provides the legally prescribed view. Figure 5A View 202 in the figure represents wing verification, where the camera arm is determined to be deployed in the original calibration position, as in view 200. Figure 5B View 212 in the diagram represents wing verification, where the camera arm is slightly offset from its original calibration position, but still provides the legally required view. Figure 5C View 222 in the diagram represents a wing verification where the camera arm is offset, in which case the camera arm is no longer able to provide the legally required view. In the example shown, views 200, 202, 212, and 222 are depicted on the Category II (narrow FOV) portion of the co-pilot-side display 18.

[0057] Points 204~211, 204′~211′ and the lines are shown for illustrative purposes only and will not be shown on display 18. Points 204~211 correspond to various reference features from the calibration image from the camera. Points 204′~211′ correspond to the current positions of similarly numbered reference features related to the wing position verification process. For illustrative purposes only, the lines indicate the horizontal matching of the reference features in their calibration positions and their current positions.

[0058] Points 204′~211′ shown in views 202, 212, and 214 are points having a pixel distance sufficiently close to the original reference feature (e.g., within 10 pixels of the original reference feature). If the current positions of a sufficient predetermined number of reference features are maintained, it can be assumed that the wing position provides a legally prescribed view (e.g., Figure 5B However, if this threshold number of reference features is not maintained, the camera arm position is too far (e.g., Figure 5C This indicates that the camera arm problem should be addressed.

[0059] If the wing position is determined to be outside its desired position and the camera arm 16 is powered by the motor 39, the CMS 15 can actuate the foldable portion 19 to a folded / collapsed position and then attempt to redeploy the foldable portion 19 to its unfolded / operational position to restore the desired field of view. At this time, the wing position verification module 52 can re-examine the new current position relative to the stored reference features. This process can capture multiple images during the folding operation and analyze those images in real time to identify the camera arm position, where a sufficient number of current feature positions match the original feature positions in the stored captured images. The motor 39 can then be de-energized, thereby holding the camera arm in the position providing the legally prescribed view.

[0060] It should also be understood that although a particular arrangement of components is disclosed in the illustrated embodiments, other arrangements will benefit from it. Although a particular sequence of steps has been shown, described, and claimed, it should be understood that, unless otherwise stated, the steps may be performed, separated, or combined in any order and will still benefit from the invention.

[0061] Although the different examples have the specific components shown in the illustrations, embodiments of the invention are not limited to those specific combinations. Some parts or features from one example may be used in combination with features or parts from another example.

[0062] Although example embodiments have been disclosed, those skilled in the art will recognize that certain modifications will fall within the scope of the claims. Therefore, the following claims should be examined to determine their true scope and content.

Claims

1. A method for checking the position of a wing in a camera monitoring system, comprising: The wing position of the camera relative to the vehicle is calibrated to provide the desired field of view; Extract reference features from the calibration images from the camera; Store the reference feature; Trigger wing position check; Capture a current image from the camera, the current image having the current position of the reference feature; Compare the current position of the reference feature with the stored reference feature; as well as If the difference obtained from the comparison step exceeds the threshold, the result of the wing position check is output.

2. The method according to claim 1, wherein, A calibration procedure is performed when the wing is installed onto the vehicle.

3. The method according to claim 2, wherein, The calibration steps are performed with the wings in the deployed position and the camera trained within the legally prescribed field of view.

4. The method according to claim 1, wherein, The reference feature is provided on the tractor unit of the vehicle.

5. The method according to claim 4, wherein, The reference feature includes the vertical edge of the tractor.

6. The method according to claim 1, wherein, The triggering step is executed based on at least one of the gear position and engine operating state.

7. The method according to claim 1, wherein, Triggering steps are executed based on time intervals.

8. The method according to claim 1, wherein, The comparison step includes: comparing at least a portion of the stored reference feature with the distance between the reference feature and the current position in 2D space.

9. The method according to claim 8, wherein, The distance is the distance between pixels in the calibrated image and the captured image of the current image.

10. The method according to claim 1, wherein, The result is at least one of a visual warning and an audible warning.

11. A camera monitoring system (CMS) for vehicles, comprising: The wing has a fixed portion configured to be fixed to a vehicle and a foldable portion pivotally mounted on the fixed portion; A camera, which is mounted to the foldable portion, and the camera has an image capture unit configured to provide a desired field of view for the vehicle; A display configured to depict at least a portion of the field of view; The input is configured to trigger a wing position check; as well as A controller, communicating with the camera and the display, includes a calibration module that extracts reference features from a calibration image of the camera providing the desired field of view. The controller has a memory in which the reference features are stored. The controller also includes a wing position verification module responsive to the input, configured to capture a current image of the current position of the reference features from the camera, compare the current position of the reference features with the stored reference features, and output the result of the wing position check if the difference obtained from the comparison step exceeds a threshold.

12. The CMS according to claim 11, wherein, The wing includes a motor that communicates with the controller, the motor being configured to fold and unfold the foldable portion relative to the fixed portion in response to a command from the controller, wherein the controller is configured to send the command to the motor in response to the difference being higher than the threshold, so as to actuate the foldable portion and provide the desired field of view.

13. The CMS according to claim 11, wherein, The output is a visual warning on the display.

14. The CMS according to claim 11, wherein, The desired view corresponds to a legally mandated view that provides at least one of a Class II view and a Class IV view.

15. The CMS according to claim 11, wherein, The reference feature is provided on the tractor unit of the vehicle.

16. The CMS according to claim 15, wherein, The reference feature includes the vertical edge of the tractor.

17. The CMS according to claim 11, wherein, The input is at least one of a gear position sensor configured to provide gear position and an engine sensor configured to provide engine operating status.

18. The CMS according to claim 11, wherein, The input is a time interval.

19. The CMS according to claim 11, wherein, The comparison step includes: comparing at least a portion of the stored reference feature with the distance between the reference feature and the current position in 2D space.

20. The CMS according to claim 19, wherein, The distance is the distance between pixels in the calibrated image and the captured image of the current image.

Images