Camera Surveillance System Including Trailer Presence Detection Using Optical Flow

Abstract

A method for detecting the presence of a trailer includes identifying a region of interest within a video feed from one or more cameras fixed to a vehicle. The video feed depicts a scene having a first pixel area, and the region of interest corresponds to a portion of the scene having a second pixel area smaller than the first pixel area. The method also includes determining an optical flow within the region of interest and determining that a trailer is attached to the vehicle in response to the optical flow within the region of interest matching a predetermined pattern corresponding to the attached trailer. A camera surveillance system for a vehicle is also disclosed.

Description

[Technical Field]

[0001] The present disclosure relates to camera surveillance systems (CMS) for vehicles, and more particularly to a process for determining the presence of a trailer using a CMS.

[0002] (Related Applications) This application claims the benefit of U.S. Provisional Application No. 63 / 405,152, filed September 9, 2022, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0003] Mirror replacement systems and camera systems that supplement mirror views are being utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. Camera surveillance systems (CMS) utilize one or more cameras to provide the vehicle operator with an enhanced field of view. In some instances, the camera system covers a wider field of view than a traditional mirror or includes a view not fully accessible through a traditional mirror.

[0004] In addition to mirror replacement, images provided via cameras within the CMS can be used to detect aspects of the environment and aspects of the vehicle in an image processing-based detection process. Among the vehicle aspects that can be detected are trailer characteristics. Trailer characteristics can then be used for any number of systems, including attentive driving detection, autonomous driving features, semi-autonomous driver assistance features, jackknife warning, and any similar elements.

[0005] Accurate detection of trailer characteristics using image-based recognition is one of the key features of the tractor-trailer system described above. However, image-based recognition is computationally intensive. Operating image-based recognition to detect trailer characteristics when no trailer is attached can result in wasted processing power that could be used for other purposes within the camera surveillance system.

[0006] These and other features of the present invention can be best understood from the following specification and drawings. Summary of the Invention

[0007] A method for detecting the presence of a trailer according to an exemplary embodiment of the present disclosure includes identifying a region of interest within a video feed from one or more cameras fixed to a vehicle, the video feed depicting a scene having a first pixel area, the region of interest corresponding to a portion of the scene having a second pixel area smaller than the first pixel area, determining an optical flow within the region of interest, and determining that the trailer is attached to the vehicle in response to the optical flow within the region of interest matching a predetermined pattern corresponding to an attached trailer.

[0008] In a further embodiment of the above embodiment, the one or more cameras include a first camera and a second camera, and the method includes generating the video feed by stitching together a first video feed from the first camera and a second video feed from the second camera in a stitching interface.

[0009] In a further embodiment of any of the above embodiments, the first video feed and the second video feed are mirror replacement video feeds.

[0010] In a further embodiment of any of the above embodiments, the first video feed is a driver's side rearview mirror replacement video feed, and the second video feed is a passenger's side rearview mirror replacement video feed.

[0011] In a further embodiment of any of the above embodiments, the horizontal width of the region of interest is less than the total horizontal width of the video feed, and the region of interest includes a portion of the stitching interface.

[0012] In a further embodiment of any of the above embodiments, the vertical height of the region of interest is less than the total vertical height of the video feed.

[0013] In a further embodiment of any of the above embodiments, the method includes determining a set of one or more operating conditions of the vehicle, and identifying the area of ​​interest within the video feed is performed based on the set of one or more operating conditions of the vehicle.

[0014] In a further embodiment of any of the above embodiments, the region of interest is centered on a predicted position of the trailer within the video feed, the predicted position being based on the determined one or more operating conditions.

[0015] In a further embodiment of any of the above embodiments, the one or more operating conditions include a steering angle of the vehicle.

[0016] In a further embodiment of any of the above embodiments, the one or more operating conditions include a speed of the vehicle.

[0017] In a further embodiment of any of the above embodiments, the region of interest is centered on the stitching interface while the speed of the vehicle is above a predetermined threshold speed and the steering angle of the vehicle is less than or equal to a predetermined threshold steering angle.

[0018] In a further embodiment of any of the above embodiments, the threshold speed is between 9 and 11 miles per hour and the threshold steering angle is between 4 and 6 degrees.

[0019] In a further embodiment of any of the above embodiments, the method does not determine optical flow outside the region of interest.

[0020] In a further embodiment of any of the above embodiments, the method includes outputting a trailer coupled notification in response to determining that the trailer is coupled to the vehicle.

[0021] A camera surveillance system for a vehicle according to an exemplary embodiment of the present disclosure includes at least one camera defining a rearward-facing field of view and a controller connected to the at least one camera such that a video feed generated by the camera is provided to the controller. The controller includes a processor and a memory. The memory stores instructions configured to cause the controller to identify a region of interest within the video feed, the video feed depicting a scene having a first pixel area, the region of interest corresponding to a portion of the scene having a second pixel area smaller than the first pixel area. The memory also stores instructions configured to cause the controller to determine an optical flow within the region of interest and, in response to the optical flow within the region of interest matching a predetermined pattern corresponding to an attached trailer, determine that the trailer is attached to the vehicle.

[0022] In a further embodiment of the above embodiment, the at least one camera defining the rear-facing field of view includes a first mirror-replacement camera and a second mirror-replacement camera.

[0023] In a further embodiment of any of the above embodiments, the one or more cameras include a first camera and a second camera, the first video feed is a driver's side rearview mirror replacement video feed and the second video feed is a passenger's side rearview mirror replacement video feed, and the controller is configured to generate the video feeds by stitching the first video feed from the first camera and the second video feed from the second camera together in a stitching interface.

[0024] In a further embodiment of any of the above embodiments, the horizontal width of the region of interest is less than the total horizontal width of the video feed, and the region of interest includes a portion of the stitching interface.

[0025] In a further embodiment of any of the above embodiments, the vertical height of the region of interest is less than the total vertical height of the video feed.

[0026] In a further embodiment of any of the above embodiments, the controller is configured to determine a set of one or more operating conditions of the vehicle, and perform identification of an area of ​​interest within the video feed based on the set of one or more operating conditions of the vehicle.

[0027] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. [Brief explanation of the drawings]

[0028] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0029] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views.

[0030] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera surveillance system providing Class II, Class IV, Class V, and Class VI views.

[0031] [Figure 2]FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an interior camera.

[0032] [Figure 3] Shows the image received from the CMS camera.

[0033] [Figure 4] A stitched image obtained from the images in Figure 3 is shown.

[0034] [Figure 5] Figure 4 shows that the stitched image contains a region of interest.

[0035] [Figure 6] Fig. 5 shows an optical flow analysis of the region of interest.

[0036] [Figure 7] 1 illustrates a process for identifying the presence of a trailer using optical flow within a region of interest. DETAILED DESCRIPTION OF THE INVENTION

[0037] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. While commercial trucks are contemplated in this disclosure, the present invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera surveillance system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may also be used to replace conventional mirrors entirely. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.

[0038] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed on or within the camera arm 16a, 16b, respectively. The cameras 20a, 20b are fixed to the commercial vehicle 10 via the camera arms 16a, 16b. However, it is understood that the cameras 20a, 20b may be fixed to the commercial vehicle in other manners. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 If desired, multiple cameras may be used on each camera arm 16 a, 16 b to provide these views. Each arm 16 a, 16 b may also provide a housing that encloses electronics configured to provide various features of CMS 15.

[0039] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0040] If Class V and Class VI views are also desired, a camera housing 16c and camera 20c may be located at or near the front of vehicle 10 to provide these views (FIG. 1B). A third display 18c located within cab 12 near the top center of the windshield may be used to display Class V and Class VI views forward of vehicle 10 to the driver in cab 26.

[0041] If Class VIII view video is required, camera housings can be positioned on the sides and rear of vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of vehicle 10. In such an example, third display 18c can include one or more frames displaying the Class VIII view. Alternatively, additional displays can be added near first, second, and third displays 18a, 18b, 18c to provide a dedicated display providing the Class VIII view. Displays 18a, 18b, 18c face toward a driver's area 24 within cab 22, where a driver is seated in driver's seat 26.

[0042] The CMS includes a controller 23 that includes a processor and a memory that stores instructions for configuring the controller. The processor may include, for example, one or more microprocessors, microcontrollers, application-specific integrated circuits (ASICs), etc. The memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). Furthermore, the memory may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory may have a distributed architecture in which various components are located remotely from each other but are accessible by the processor.

[0043] 3-6 show a pair of images 210, 220 received from wing-mounted cameras 20a, 20b. When a trailer 14 is coupled to the cab 12, the field of view defined by at least one of cameras 20a, 20b includes an image of the trailer 14. When the vehicle 10 is traveling in a forward direction at a speed above a threshold, the trailer 14 is within the field of view of both cameras 20a, 20b and appears in both images 210, 220, as in the illustrated example. In a practical example, images 210, 220 are generated using a rearview mirror replacement portion of the CMS 15, although alternatives may be used with the same effect.

[0044] CMS15 uses a stitching algorithm to fuse the images 210, 220 into a single image 230 shown in FIG. 4. The single image 230 is joined at a stitching interface 232. The stitching interface 232 is algorithmically determined according to any stitching method. In some examples, the stitching interface 232 is the vertical edge of a straight line of the images 210, 220. In an alternative, the stitching interface 232 may crop one or both of the images 210, 220. In a further alternative, the stitching interface 232 may be determined using an algorithm based on a neural network and may not be a straight line interface.

[0045] When the images 210, 220 are stitched into the single image 230, a region of interest 240 is defined within the single image 230. The region of interest 240 is limited to the region within the single image 230 where, if the trailer 14 is present, it is predicted to appear within the image 230 based on the current vehicle operating characteristics. The single image 230 shows a scene having a first pixel region (having a vertical height H1 and a horizontal width W1). The region of interest 240 shown in FIG. 5 is a rectangle corresponding to a part of the scene / single image 230 having a second pixel region (having a vertical height H2 and a horizontal width W2) that is smaller than the first pixel region. In the example of FIG. 5, the region of interest 240 is smaller than the full horizontal width W1 of the single image 230 and smaller than the full vertical height H1 (i.e., H2 < H1 and W2 < W1). An exemplary region of interest 240 corresponds to a vehicle 10 traveling forward in a forward direction at a predetermined threshold speed (e.g., 10 mph) with a steering angle less than a predetermined threshold steering angle. In one example, the predetermined threshold speed is 9 - 11 mph (i.e., 9 mph or more and 11 mph or less). In an alternative, the predetermined threshold speed is 10 mph. In one example, the predetermined threshold steering angle is 4 - 6 degrees (i.e., 4 degrees or more and 6 degrees or less). In an alternative, the predetermined threshold steering angle is 5 degrees. If the operating conditions are different, the region of interest 240 may be defined at a different location (e.g., if the steering angle exceeds the predetermined threshold steering angle, the target area may be shifted in the direction of the steering and / or expanded horizontally).

[0046] Alternatively, if the straight ahead condition is not met, the trailer present function may be disabled entirely to conserve processing resources.

[0047] Once the region of interest 240 is established, the CMS 15 performs an optical flow analysis within the region of interest 240. To conserve processing power and prevent unnecessary calculations, the optical flow analysis is limited to only the region of interest 240.

[0048] Optical flow is a concept that refers to the pattern of apparent movement of objects, surfaces, and edges in a visual scene caused by relative motion between the observer and the scene. Optical flow can also be described as the distribution of apparent speeds of movement of brightness patterns in an image. CMS 15 measures pixel velocities 234 within a region of interest 240 (see FIG. 6) and compares the pixel velocity pattern to known patterns corresponding to the presence of a trailer 14. For example, when a trailer 14 is present, the portion of the image containing the trailer 14 will have minimal optical flow because the trailer 14 is nearly stationary relative to the tractor, while the portion of the image not containing the trailer 14 will have standard optical flow corresponding to the vehicle's moving speed. By defining the region of interest 240 slightly larger than the expected size of the trailer 14 and centering it over the trailer 14, CMS 15 can capture this distinction without having to analyze the optical flow of the entire image 230. Alternatively, road and engine vibrations may cause the vertical rear edge of the trailer 14 to vibrate, resulting in optical flow having a wave-like shape at the edge of the trailer.

[0049] Specific movements of the optical flow and / or areas where the shape of the optical flow changes within the region of interest are called pixel velocity patterns, and these portions are compared to patterns known to correspond to the presence of a trailer 14.

[0050] If the pattern of pixel velocities within the region of interest 240 matches the pattern of pixel velocities corresponding to the presence of a trailer 14, the system determines that a trailer is present and coupled to the cab 12. If the pattern of pixel velocities within the region of interest 240 does not match the pattern of pixel velocities corresponding to the presence of a trailer 14, the CMS 15 determines that a trailer 14 is not coupled to the cab 12.

[0051] With continued reference to FIGS. 1 through 6, FIG. 7 illustrates a method 700 for operating the CMS 15 to perform the operations described above. Initially, the CMS 15 receives images from cameras 20a, 20b in a "Receive CMS Images" step 710. The received images are stitched together using any established stitching algorithm in a "Stitch Images" step 720. If the method is performed using images from a single camera 20a or 20b rather than multiple cameras, the stitching images step 720 can be omitted. Similarly, if images are provided from additional cameras in addition to the wing-mounted rear cameras 20a, 20b, the stitching images step 720 can use a stitching algorithm to combine all images into a single stitched image.

[0052] Once the single image is created, CMS 15 defines a region of interest within the image in a "Limit Region of Interest" step 730. The region of interest is smaller than all of the image (i.e., corresponds to a portion of the entire image having a pixel area smaller than the pixel area of ​​the entire image) and corresponds to an area where the trailer 14 is expected to be located within the image. To facilitate this identification, CMS 15 may, in some examples, receive operating characteristics of the vehicle 10 that can indicate the expected location of the trailer 14 within the image. By way of example, the operating characteristics may include (but are not limited to) the steering angle of the vehicle 10 and the speed of the vehicle 10.

[0053] The region of interest may be limited to less than all of the horizontal portion of the image, less than all of the vertical portion of the image, less than all of both the horizontal and vertical portions of the image, and / or a specified box that defines the restricted area. In some examples, the region may be centered on a stitching interface that combines the images into a single image. In other examples, the region may be shifted off-center but still include a stitching interface 232 that combines multiple images into a single image.

[0054] After the region of interest is defined, the CMS 15 analyzes the optical flow within the region of interest in an "Analyze Optical Flow Within Region of Interest" step 740. To conserve computational resources and provide a faster decision, the optical flow is analyzed only within the region of interest. This analysis identifies any patterns of optical flow that exist within the region of interest. The optical flow patterns are compared to known patterns in a "Compare Optical Flow to Pattern" step 750. If the identified optical flow pattern matches a pattern known to correspond to the presence of a trailer, the method 700 outputs a "Trailer Present" signal to the CMS 15 in an "Output Trailer Present in Respond to Match" step 760. Conversely, if the pattern does not match a pattern known to correspond to the presence of a trailer 14, a "Trailer Not Present" signal is output.

[0055] After being notified of the presence or absence of trailer 14, CMS 15 operates in a conventional manner to accommodate the hitched status of trailer 14. Additionally, CMS 15 may provide the trailer detection status to any number of additional vehicle systems that may utilize this information. By way of example, the additional systems may include driver assistance systems, object detection systems, etc.

[0056] Although described above as a process within CMS 15, it will be appreciated that the process for trailer detection using optical flow can be executed independently of CMS 15, can be provided to CMS 15, can be executed in a general vehicle controller, or can be executed in any suitable processing unit within the vehicle.

[0057] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for detecting the presence of a trailer, comprising: identifying a region of interest in a video feed from one or more cameras fixed to a vehicle, the video feed depicting a scene having a first pixel area, the region of interest corresponding to a portion of the scene having a second pixel area that is smaller than the first pixel area; determining optical flow within the region of interest; determining that a trailer is coupled to the vehicle in response to optical flow within the region of interest matching a predetermined pattern corresponding to an coupled trailer; A method comprising:

2. the one or more cameras include a first camera and a second camera; 10. The method of claim 1, wherein the method includes generating the video feed by stitching a first video feed from the first camera and a second video feed from the second camera together in a stitching interface.

3. The method of claim 2 , wherein the first video feed and the second video feed are mirror-displaced video feeds.

4. the first video feed is a driver's side rearview mirror replacement video feed; The method of claim 3 , wherein the second video feed is a passenger side rearview mirror replacement video feed.

5. the horizontal width of the region of interest is less than the total horizontal width of the video feed; The method of claim 2 , wherein the region of interest comprises a portion of the stitching interface.

6. The method of claim 5 , wherein the vertical height of the region of interest is less than the total vertical height of the video feed.

7. determining a set of one or more operating conditions of the vehicle; The method of claim 2 , wherein identifying the region of interest within the video feed is performed based on the set of one or more operating conditions of the vehicle.

8. The method of claim 7 , wherein the region of interest is centered on a predicted location of the trailer within the video feed, the predicted location being based on the determined one or more operating conditions.

9. The method of claim 7 , wherein the one or more operating conditions include a steering angle of the vehicle.

10. The method of claim 9 , wherein the one or more operating conditions include a speed of the vehicle.

11. The method of claim 10 , wherein the region of interest is centered at the stitching interface while the vehicle speed exceeds a predetermined threshold speed and the vehicle steering angle is less than or equal to a predetermined threshold steering angle.

12. 11. The method of claim 10, wherein the threshold speed is between 9 and 11 miles per hour and the threshold steering angle is between 4 and 6 degrees.

12. The method of claim 1 , wherein the region of interest is less than the entire vertical height of the primary video feed.

13. The method of claim 1 , wherein the method does not determine optical flow outside the region of interest.

14. The method of claim 1 , including the step of outputting a trailer coupled notification in response to determining that the trailer is coupled to the vehicle.

15. 1. A camera surveillance system for a vehicle, comprising: at least one camera defining a rearward-facing field of view; a controller connected to the at least one camera such that a video feed generated by the camera is provided to the controller; It is equipped with The controller includes a processor and a memory, and the memory is configured by the controller to: identifying an area of ​​interest within the video feed; the video feed depicts a scene having a first pixel area, and the region of interest corresponds to a portion of the scene having a second pixel area that is smaller than the first pixel area; determining optical flow within the region of interest; determining that a trailer is coupled to the vehicle in response to the optical flow within the region of interest matching a predetermined pattern corresponding to an coupled trailer; a camera monitoring system storing instructions configured to cause the camera to:

16. 16. The camera surveillance system of claim 15, wherein the at least one camera defining the rear-facing field of view includes a first mirror-replacement camera and a second mirror-replacement camera.

17. the one or more cameras include a first camera and a second camera; the first video feed is a driver's side rearview mirror replacement video feed; the second video feed is a passenger side rearview mirror replacement video feed; 16. The camera surveillance system of claim 15, wherein the controller is configured to generate the video feed by stitching together a first video feed from the first camera and a second video feed from the second camera at a stitching interface.

18. the horizontal width of the region of interest is less than the total horizontal width of the video feed; The camera surveillance system of claim 17 , wherein the region of interest comprises a portion of the stitching interface.

19. The camera surveillance system of claim 15 , wherein the vertical height of the region of interest is less than the total vertical height of the video feed.

20. The controller determining a set of one or more operating conditions for the vehicle; and performing identification of the region of interest within the video feed based on the set of one or more operating conditions of the vehicle; 16. A camera surveillance system according to claim 15, configured to:

Images