Camera monitoring system including trailer presence detection using optical flow

Abstract

A method for detecting trailer presence includes identifying a region of interest within a video feed from one or more cameras secured to a vehicle. The video feed depicts a scene having a first pixel region, and the region of interest corresponds to a portion of the scene having a second pixel region smaller than the first pixel region. The method also includes determining an optical flow within the region of interest, and determining that a trailer is connected to the vehicle in response to the optical flow within the region of interest matching a predetermined pattern corresponding to the connected trailer. A camera monitoring system for a vehicle is also disclosed.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 405,152, filed on September 9, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to a camera monitoring system (CMS) for a vehicle, and in particular to a method for determining the presence of a trailer using the CMS. Background Art

[0004] Mirror replacement systems and camera systems for supplementing mirror views are utilized in commercial vehicles to enhance the vehicle operator's ability to see the surrounding environment. Camera monitoring systems (CMS) utilize one or more cameras to provide an enhanced field of view to the vehicle operator. In some examples, the camera system covers a larger field of view than conventional mirrors, or includes views that cannot be fully obtained via conventional mirrors.

[0005] In addition to mirror replacement, images provided via cameras in the CMS can be used to detect aspects of the environment and aspects of the vehicle in an image processing based detection process. Among the aspects of the vehicle that can be detected are trailer attributes. The trailer attributes can then be used for any number of systems including attentive driving detection, autonomous driving features, semi-autonomous driver assistance features, jackknife warnings, and any similar elements.

[0006] Accurate detection of trailer attributes using image processing based sensing can be an important feature of the tractor-trailer system described above. However, image processing based sensing is computationally intensive. Running image based sensing to detect trailer characteristics when there is no trailer attached can result in wasted processing power that could be used for other purposes within the camera monitoring system.

[0007] These and other features of the present invention can be best understood from the following specification and drawings, the following of which is a brief description. Summary of the invention

[0008] A method for detecting the presence of a trailer according to an exemplary embodiment of the present disclosure includes identifying a region of interest within a video feed from one or more cameras fixed to a vehicle. The video feed depicts a scene having a first pixel area, and the region of interest corresponds to a portion of the scene having a second pixel area smaller than the first pixel area. The method also includes determining an optical flow within the region of interest, and determining that a trailer is connected to the vehicle in response to the optical flow within the region of interest matching a predetermined pattern corresponding to a connected trailer.

[0009] In another embodiment of the foregoing embodiment, the one or more cameras include a first camera and a second camera; and the method includes generating a video feed by stitching a first video feed from the first camera and a second video feed from the second camera at a stitching interface.

[0010] In a further embodiment of any of the foregoing embodiments, the first video feed and the second video feed are mirror replacement video feeds.

[0011] In another embodiment of any of the foregoing embodiments, the first video feed is a driver side rear view mirror substitute video feed. The second video feed is a passenger side rear view mirror substitute video feed.

[0012] In a further embodiment of any of the foregoing embodiments, the horizontal width of the region of interest is less than a full horizontal width of the video feed; and the region of interest includes a portion of the stitching interface.

[0013] In a further embodiment of any of the foregoing embodiments, the vertical height of the region of interest is less than a full vertical height of the video feed.

[0014] In a further embodiment of any of the preceding embodiments, the method includes determining a set of one or more operating conditions of the vehicle, wherein identifying the region of interest within the video feed is performed based on the set of one or more operating conditions of the vehicle.

[0015] In a further embodiment of any of the foregoing embodiments, the region of interest is centered about an expected position of the trailer within the video feed, and the expected position is based on the determined one or more operating conditions.

[0016] In a further embodiment of any of the foregoing embodiments, the one or more operating conditions include a steering angle of the vehicle.

[0017] In a further embodiment of any of the foregoing embodiments, the one or more operating conditions include a speed of the vehicle.

[0018] In a further embodiment of any of the foregoing embodiments, the region of interest is centered on the stitching interface when a speed of the vehicle is above a predefined threshold speed and a steering angle of the vehicle is less than or equal to a predefined threshold steering angle.

[0019] In a further embodiment of any of the foregoing embodiments, the threshold speed is 9 to 11 miles per hour and the threshold steering angle is 4 to 6 degrees.

[0020] In a further embodiment of any of the preceding embodiments, the method does not determine optical flow outside the region of interest.

[0021] In a further embodiment of any of the preceding embodiments, the method includes outputting a trailer connected notification in response to determining that the trailer is connected to the vehicle.

[0022] A camera monitoring system for a vehicle according to an example embodiment of the present disclosure includes: at least one camera that defines a rearward-facing field of view; and a controller that is connected to the at least one camera so that a video feed generated by the camera is provided to the controller. The controller includes a processor and a memory. The memory stores instructions configured to cause the controller to identify a region of interest within a video feed, wherein the video feed depicts a scene having a first pixel area, and the region of interest corresponds to a portion of the scene having a second pixel area that is smaller than the first pixel area. The memory also stores instructions configured to cause the controller to determine an optical flow within the region of interest, and determine that a trailer is connected to the vehicle in response to the optical flow within the region of interest matching a predetermined pattern corresponding to the connected trailer.

[0023] In another embodiment of the foregoing embodiment, the at least one camera defining a rearward field of view includes: a first mirror replacement camera and a second mirror replacement camera.

[0024] In another embodiment of any of the foregoing embodiments, the one or more cameras include a first camera and a second camera, the first video feed is a driver side rearview mirror substitute video feed, the second video feed is a passenger side rearview mirror substitute video feed, and the controller is configured to generate a video feed by stitching the first video feed from the first camera and the second video feed from the second camera at a stitching interface.

[0025] In a further embodiment of any of the foregoing embodiments, the horizontal width of the region of interest is less than a full horizontal width of the video feed, and the region of interest includes a portion of the stitching interface.

[0026] In a further embodiment of any of the foregoing embodiments, the vertical height of the region of interest is less than a full vertical height of the video feed.

[0027] In a further embodiment of any of the foregoing embodiments, the controller is configured to determine a set of one or more operating conditions of the vehicle, and to perform the identifying of the region of interest within the video feed based on the set of one or more operating conditions of the vehicle.

[0028] The embodiments, examples and alternatives of the preceding paragraphs, claims or the following description and drawings, including any of their various aspects or corresponding individual features, may be taken independently or in any combination. Features described in conjunction with one embodiment are applicable to all embodiments, unless these features are incompatible. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present disclosure may be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0030] Figure 1A is a schematic front view of a commercial truck having a camera monitoring system (CMS) for providing at least Class II and Class IV views.

[0031] Figure 1B is a schematic top view of a commercial truck with a camera monitoring system providing Class II, Class IV, Class V, and Class VI views.

[0032] Figure 2 is a schematic top perspective view of a vehicle cab including a display and interior cameras.

[0033] Figure 3 An image received from a CMS camera is shown.

[0034] Figure 4 Shows from Figure 3 The stitched image exported from the image.

[0035] Figure 5 Shown in Figure 4 The stitched image includes the region of interest.

[0036] Figure 6 Shows Figure 5 Optical flow analysis of regions of interest.

[0037] Figure 7 A method for identifying the presence of a trailer using optical flow within a region of interest is shown. DETAILED DESCRIPTION

[0038] A schematic diagram of a commercial vehicle 10 is shown in Figure 1A and Figure 1B 1. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although commercial trucks are contemplated in the present disclosure, the present invention may also be applied to other types of vehicles. The vehicle 10 includes a camera monitoring system (CMS) 15 ( Figure 2 ), which has a driver-side camera arm 16a and a passenger-side camera arm 16b mounted to the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, but the CMS 15 may be used to completely replace conventional mirrors. In another example, each side may include multiple camera arms, each arm accommodating one or more cameras and / or mirrors.

[0039] Each of the camera arms 16a, 16b includes a base that is secured to, for example, the cab 12. A pivot arm is supported by the base and can be articulated relative to the base. At least one rear-facing camera 20a, 20b is disposed on or within the camera arms 16a, 16b, respectively. The cameras 20a, 20b are secured to the commercial vehicle 10 via the camera arms 16a, 16b. However, it should be understood that the cameras 20a, 20b can be secured to the commercial vehicle in other ways. The external cameras 20a, 20b provide an external field of view FOV, respectively. EX1 FOV EX2 , each external field of view includes at least one of a Class II and a Class IV view ( Figure 1B ), which are legally required views in the commercial trucking industry. If desired, multiple cameras may also be used in each camera arm 16a, 16b to provide these views. Each arm 16a, 16b may also provide a housing that encloses electronic devices configured to provide the various features of the CMS 15.

[0040] A first video display 18a and a second video display 18b are arranged on each of the driver's side and the passenger side within the vehicle cab 12 on or near the A-pillars 19a, 19b to display Class II and Class IV views on their respective sides of the vehicle 10, and these displays provide rearward side views along the vehicle 10 captured by external cameras 20a, 20b.

[0041] If video of Class V and Class VI views is also desired, the camera housing 16c and camera 20c may be disposed at or near the front of the vehicle 10 to provide those views ( Figure 1B ). A third display 18 c disposed within the cab 12 near the top center of the windshield may be used to display Class V and Class VI views toward the front of the vehicle 10 to the driver in the driver's seat 26 .

[0042] If video of a Class VIII view is desired, camera housings may be provided on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII area of ​​the vehicle 10. In such an example, the third display 18c may include one or more frames that display a Class VIII view. Alternatively, an additional display may be added near the first display 18a, the second display 18b, and the third display 18c, and a display dedicated to providing a Class VIII view may be provided. The displays 18a, 18b, 18c face the driver area 24 in the cab 22 where the driver sits on the driver's seat 26.

[0043] The CMS includes a controller 23, which includes a processor and a memory storing instructions for configuring the controller. For example, the processor may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), etc. The memory may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.) and / or non-volatile memory elements (e.g., ROM, hard disk drives, magnetic tapes, CD-ROMs, etc.). In addition, the memory may contain electronic, magnetic, optical, and / or other types of storage media. The memory may also have a distributed architecture in which various components are remote from each other but can be accessed by the processor.

[0044] Figures 3 to 6 A pair of images 210, 220 received from the wing mounted cameras 20a, 20b are shown. When the trailer 14 is connected to the cab 12, the field of view defined by at least one of the cameras 20a, 20b includes an image of the trailer 14. When the vehicle 10 is traveling in a forward direction at a speed above a threshold, the trailer 14 is within the field of view of both cameras 20a, 20b and will appear in both images 210, 220, as in the example shown. In the actual example, the images 210, 220 are generated using a mirror replacement portion of the CMS 15, but alternative examples may be used to achieve the same effect.

[0045] CMS15 uses a stitching algorithm to merge images 210 and 220 into Figure 4 The single images 230 are shown. The single images 230 are joined at a stitching interface 232. The stitching interface 232 is algorithmically determined according to any stitching method. In some examples, the stitching interface 232 is a straight vertical edge of the images 210, 220. In alternative examples, the stitching interface 232 may crop one or both images 210, 220. In further alternatives, the stitching interface 232 may be determined using a neural network-based algorithm and may not be a straight line interface.

[0046] Once the images 210, 220 are stitched together into a single image 230, a region of interest 240 is defined within the single image 230. If a trailer 14 is present, then the region of interest 240 is limited to the area of ​​the single image 230 where the trailer 14 is expected to be present within the image 230 based on current vehicle operating characteristics. The single image 230 depicts a scene having a first pixel area having a vertical height H1 and a horizontal width W1. Figure 5 The region of interest 240 illustrated in FIG. 2 is a rectangle corresponding to a portion of the scene / single image 230 having a second pixel area (having a vertical height H2 and a horizontal width W2) that is smaller than the first pixel area. Figure 5In the example, the region of interest 240 is less than the full horizontal width W1 of a single image 230 and less than the full vertical height H1 (i.e., H2 < H1 and W2 < W1). The exemplary region of interest 240 corresponds to a vehicle 10 traveling in a forward direction at a speed above a predefined threshold speed (e.g., 10 mph), and the steering angle of the vehicle 10 is below a predefined threshold steering angle. In one example, the predefined threshold speed is 9 to 11 mph (i.e., greater than or equal to 9 mph and less than or equal to 11 mph). In another example, the predefined threshold speed is 10 mph. In one example, the predefined threshold steering angle is 4 to 6 degrees (i.e., greater than or equal to 4 degrees and less than or equal to 6 degrees). In another example, the predefined threshold steering angle is 5 degrees. When the operating conditions are different, the region of interest 240 can be defined in an alternative position (e.g., when the steering angle is above the predefined threshold steering angle, the region of interest can be shifted in the steering direction and / or can be horizontally enlarged).

[0047] In an alternative example, when the forward straight driving condition is not met, the trailer presence function can be completely disabled to save processing resources.

[0048] Once the region of interest 240 is established, the CMS 15 performs optical flow analysis within the region of interest 240. To save processing power and prevent wasted computations, the optical flow analysis is limited to the region of interest 240.

[0049] Optical flow refers to the concept of the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and the scene. Optical flow can also be described as the distribution of the apparent speed of movement of the brightness pattern in an image. The CMS 15 measures the pixel velocity 234 within the region of interest 240 (see Figure 6 ), and compares the pattern of the pixel velocity with a known pattern corresponding to the presence of the trailer 14. As an example, when the trailer 14 is present, the portion of the image including the trailer 14 will have minimal optical flow because the trailer 14 is mostly static relative to the towing vehicle, and the portion of the image not including the trailer 14 will have a standard optical flow corresponding to the rate of movement of the vehicle. By defining the region of interest 240 to be slightly larger than the expected size of the trailer 14 and centering the region of interest 240 on the trailer 14, the CMS 15 can capture this difference without having to analyze the optical flow of the entire image 230. In another example, road and engine vibrations may cause the vertical trailing edge of the trailer 14 to oscillate, resulting in an optical flow with a waveform shape at the edge of the trailer.

[0050] The regions of the region of interest where the optical flow has a specific movement and / or the shape of the optical flow changes are referred to as the pattern of the pixel velocity, and these portions are compared with the known patterns corresponding to the presence of the trailer 14.

[0051] When the pixel speed pattern within the region of interest 240 matches the pixel speed pattern corresponding to the presence of the trailer 14, the system determines that the trailer is present and connected to the cab 12. When the pixel speed pattern within the region of interest 240 does not match the pixel speed pattern corresponding to the presence of the trailer 14, the CMS 15 determines that no trailer 14 is connected to the cab 12.

[0052] Continue to refer Figures 1A to 6 , Figure 7 A method 700 of operating the CMS 15 to perform the operations described above is shown. Initially, the CMS 15 receives images from the cameras 20a, 20b in a "Receive CMS Images" step 710. In a "Stitch Images" step 720, the received images are stitched using any established stitching algorithm. In the case where the method is performed using images from a single camera 20a or 20b rather than multiple cameras, the stitch images step 720 may be omitted. Similarly, when images are provided from additional cameras other than the wing-mounted rear view cameras 20a, 20b, the stitch images step 720 may combine all of the images into a single stitched image using a stitching algorithm.

[0053] Once a single image is created, the CMS 15 defines a region of interest within the image in a "limit region of interest" step 730. The region of interest is less than the entire image (i.e., corresponds to a portion of the entire image having a pixel area that is less than the pixel area of ​​the entire image) and corresponds to an area within the image where the trailer 14 is expected to be. To achieve this identification, in some examples, the CMS 15 may receive operating characteristics of the vehicle 10 that may indicate an expected location of the trailer 14 within the image. By way of example, the operating characteristics may include (but are not limited to) a steering angle of the vehicle 10 and a speed of the vehicle 10.

[0054] The region of interest may be limited to less than the entire horizontal portion of the image, less than the entire vertical portion of the image, less than the entire horizontal portion and vertical portion of the image, and / or a specific box defining a limited area. In some examples, the region may be centered on the stitching interface that connects the images into a single image. In other examples, the region may be off-center but still include the stitching interface 232 that connects multiple images into a single image.

[0055] After defining the region of interest, the CMS 15 analyzes the optical flow within the region of interest in an "Analyze Optical Flow in Region of Interest" step 740. In order to save computing resources and provide faster determination, the optical flow is analyzed only within the region of interest. The analysis identifies any patterns of optical flow that exist within the region of interest. The optical flow patterns are compared to known patterns in a "Compare Optical Flow to Pattern" step 750. When the identified optical flow pattern matches a pattern known to correspond to the presence of a trailer, the method 700 outputs a "trailer present" signal to the CMS 15 in an "output trailer present in response to match" step 760. Conversely, when the pattern does not match a pattern known to correspond to the presence of a trailer 14, a "trailer not present" signal is output.

[0056] After being notified of the presence or absence of the trailer 14, the CMS 15 operates in a conventional manner corresponding to the connection status of the trailer 14. In addition, the CMS 15 can provide the trailer detection status to any number of additional vehicle systems that can utilize this information. As an example, the additional systems may include driver assistance systems, object detection systems, etc.

[0057] Although described above as a process within CMS 15, it will be appreciated that the process of trailer detection using optical flow may be performed independently of and provided to CMS 15, performed in a general vehicle controller, or performed in any suitable processing unit within the vehicle.

[0058] Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

1. A method for detecting the presence of a trailer, comprising: identifying a region of interest within a video feed from one or more cameras fixed to a vehicle, wherein the video feed depicts a scene having a first pixel area and the region of interest corresponds to a portion of the scene having a second pixel area that is smaller than the first pixel area; determining optical flow within the region of interest; and Responsive to the optical flow within the region of interest matching a predetermined pattern corresponding to a connected trailer, it is determined that a trailer is connected to the vehicle.

2. The method according to claim 1, wherein: The one or more cameras include a first camera and a second camera; and The method includes generating the video feed by stitching a first video feed from the first camera and a second video feed from the second camera at a stitching interface.

3. The method according to claim 2, wherein: The first video feed and the second video feed are mirror replacement video feeds.

4. The method according to claim 3, wherein: The first video feed is a driver side mirror substitute video feed, and wherein the second video feed is a passenger side mirror substitute video feed.

5. The method according to claim 2, wherein: The horizontal width of the region of interest is less than the full horizontal width of the video feed; and The region of interest includes a portion of the stitching interface.

6. The method according to claim 5, wherein: The vertical height of the region of interest is less than the full vertical height of the video feed.

7. The method according to claim 2, comprising: determining a set of one or more operating conditions of the vehicle; Wherein said identifying the region of interest within the video feed is performed based on a set of one or more operating conditions of the vehicle.

8. The method according to claim 7, wherein: The region of interest is centered about an expected position of the trailer within the video feed, and the expected position is based on the determined one or more operating conditions.

9. The method according to claim 7, wherein: The one or more operating conditions include a steering angle of the vehicle.

10. The method according to claim 9, wherein: The one or more operating conditions include a speed of the vehicle.

11. The method according to claim 10, wherein: When the speed of the vehicle is higher than a predetermined threshold speed and the steering angle of the vehicle is less than or equal to a predetermined threshold steering angle, the region of interest is centered on the splicing interface.

12. The method according to claim 10, wherein: The threshold speed is 9 to 11 miles per hour, and the threshold steering angle is 4 to 6 degrees.

13. The method according to claim 1, wherein: The region of interest is less than the full vertical height of the main video feed.

14. The method according to claim 1, wherein: The method does not determine the optical flow outside the region of interest.

15. The method of claim 1, comprising outputting the trailer connected notification in response to determining that a trailer is connected to the vehicle.

16. A camera monitoring system for a vehicle, comprising: at least one camera, the at least one camera defining a rearward field of view; a controller connected to the at least one camera such that a video feed generated by the camera is provided to the controller; The controller includes a processor and a memory, wherein the memory stores instructions configured to cause the controller to perform the following operations: identifying a region of interest within the video feed, wherein the video feed depicts a scene having a first pixel area and the region of interest corresponds to a portion of the scene having a second pixel area that is smaller than the first pixel area; determining an optical flow within the region of interest; as well as Responsive to the optical flow within the region of interest matching a predetermined pattern corresponding to a connected trailer, it is determined that a trailer is connected to the vehicle.

17. The camera monitoring system of claim 15, wherein: The at least one camera defining a rearward field of view includes a first mirror replacement camera and a second mirror replacement camera.

18. The camera monitoring system of claim 15, wherein: The one or more cameras include a first camera and a second camera; The first video feed is a driver side mirror substitute video feed; The second video feed is a passenger side mirror substitute video feed; and The controller is configured to generate the video feed by stitching a first video feed from the first camera and a second video feed from the second camera at a stitching interface.

19. The camera monitoring system of claim 17, wherein: The horizontal width of the region of interest is less than the full horizontal width of the video feed; and The region of interest includes a portion of the stitching interface.

20. The camera surveillance system of claim 15, wherein: The vertical height of the region of interest is less than the full vertical height of the video feed.

21. The camera monitoring system of claim 15, wherein: The controller is configured to: determining a set of one or more operating conditions of the vehicle; as well as The identifying of the region of interest within the video feed is performed based on the set of one or more operating conditions of the vehicle.

Images