Method and apparatus for augmenting an image feed for a commercial vehicle

Abstract

A method of augmenting an image feed for a commercial vehicle includes obtaining an image feed from a camera mounted to a commercial vehicle and oriented to have a rearward-facing field of view. The image feed depicts an exterior environment of a trailer of the commercial vehicle. The method also includes displaying the image feed on an electronic display oriented for viewing in a cabin of the commercial vehicle and providing an overlay on the image feed of a vehicle path that would be followed if the commercial vehicle performed a particular driving maneuver. A camera monitoring system is also disclosed.

Description

METHOD AND APPARATUS FOR AUGMENTING AN IMAGE FEED FOR A COMMERCIAL VEHICLETECHNICAL FIELD

[0001] This disclosure relates to a camera monitor system (CMS), and more particularly to a method and apparatus for augmenting an image feed for a commercial vehicle for safe lane changing.BACKGROUND

[0002] Vehicle camera systems for mirror replacement or for supplementing mirror views are utilized in commercial vehicles to enhance the ability of a vehicle operator to see a surrounding environment. Camera monitor systems utilize one or more cameras to provide an enhanced field of view to a vehicle operator. In some examples, the mirror replacement systems cover a larger field of view than a conventional mirror, or include views that are not fully obtainable via a conventional mirror.

[0003] When a commercial vehicle driver is about to perform a vehicle maneuver, such as a lane change or turn, it may be difficult for that driver to determine whether the vehicle maneuver can be safely performed.SUMMARY

[0004] A method of augmenting an image feed for a commercial vehicle according to an example embodiment of the present disclosure includes obtaining an image feed from a camera mounted to a commercial vehicle and oriented to have a rearward-facing field of view. The image feed depicts an exterior environment of a trailer of the commercial vehicle. The method also includes displaying the image feed on an electronic display oriented for viewing in a cabin of the commercial vehicle, and providing an overlay on the image feed of a vehicle path that would be followed if the commercial vehicle performed a particular driving maneuver.

[0005] In a further embodiment of the foregoing embodiment, the particular driving maneuver is a lane change.

[0006] In a further embodiment of any of the foregoing embodiments, providing the overlay is triggered at least partially by activation of a turn signal in the commercial vehicle.

[0007] In a further embodiment of any of the foregoing embodiments, providing the overlay is further triggered by the commercial vehicle starting to perform the particular driving maneuver.

[0008] In a further embodiment of any of the foregoing embodiments, the method includes reducing a size of the overlay as the commercial vehicle performs the particular driving maneuver.

[0009] In a further embodiment of any of the foregoing embodiments, the overlay is a second overlay, and the method includes, prior to providing the second overlay, providing a first overlay on the image feed. The first overlay is different than the second overlay and depicts a future position of the commercial vehicle after completion of the particular driving maneuver.

[0010] In a further embodiment of any of the foregoing embodiments, providing of the first overlay is triggered by activation of a turn signal of the commercial vehicle, and the providing of the second overlay is triggered by the commercial vehicle starting to perform the particular driving maneuver within a predefined time period after activation of the turn signal.

[0011] In a further embodiment of any of the foregoing embodiments, the first overlay includes at least one distance indicator, and each of the at least one distance indicators indicate a respective distance from a rear end of the trailer of the commercial vehicle.

[0012] In a further embodiment of any of the foregoing embodiments, the method includes dimming the first overlay to obtain a dimmed version of the first overlay, and providing the second overlay includes simultaneously providing the second overlay and the dimmed version of the first overlay.

[0013] In a further embodiment of any of the foregoing embodiments, the method includes turning off the first overlay prior to providing the second overlay.

[0014] A camera monitoring system according to an example embodiment of the present disclosure includes a camera, an electronic display, and processing circuitry. The camera is mounted to a commercial vehicle and is configured to provide an image feed that depicts an exterior environment of a trailer of the commercial vehicle. The electronic display is oriented for viewing in a cabin of the commercial vehicle. The processing circuitry is configured to display an image feed from the camera on the electronic display and provide an overlay on the image feed on the electronic display of a vehicle path that would be followed if the commercial vehicle performed a particular driving maneuver.

[0015] In a further embodiment of the foregoing embodiment, the particular driving maneuver is a lane change.

[0016] In a further embodiment of any of the foregoing embodiments, the processing circuitry is configured to provide the overlay at least partially based on activation of a turn signal in the commercial vehicle.

[0017] In a further embodiment of any of the foregoing embodiments, the processing circuitry is configured to detect that the commercial vehicle has started to perform the particular driving maneuver and provide the overlay further based on the detection.

[0018] In a further embodiment of any of the foregoing embodiments, the processing circuitry is configured to reduce a size of the overlay as the commercial vehicle performs the particular driving maneuver.

[0019] In a further embodiment of any of the foregoing embodiments, the overlay is a second overlay, and the processing circuitry is configured to, prior to providing the second overlay, provide a first overlay on the image feed. The first overlay is different than the second overlay and depicts a future position of the commercial vehicle after completion of the particular driving maneuver.

[0020] In a further embodiment of any of the foregoing embodiments, the processing circuitry is configured to provide the first overlay based on activation of a turn signal of the commercial vehicle and provide the second overlay based on the commercial vehicle starting to perform the particular driving maneuver within a predefined time period after activation of the turn signal.

[0021] In a further embodiment of any of the foregoing embodiments, the first overlay includes at least one distance indicator, and each of the at least one distance indicators indicates a respective distance from a rear end of the trailer of the commercial vehicle.

[0022] In a further embodiment of any of the foregoing embodiments, the processing circuitry is configured to dim the first overlay to obtain a dimmed version of the first overlay and simultaneously provide the second overlay and the dimmed version of the first overlay on the image feed.

[0023] In a further embodiment of any of the foregoing embodiments, the processing circuitry is configured to turn off the first overlay prior to providing the second overlay.

[0024] The embodiments, examples, and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or respective individual features, may be taken independently or in anycombination. Features described in connection with one embodiment are applicable to all embodiments, unless such features are incompatible.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The disclosure can be further understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein:

[0026] Figure 1 A is a schematic front view of a commercial truck with a CMS used to provide at least Class II and Class IV views.

[0027] Figure 1 B is a schematic birds-eye view of the commercial truck of Figure 1 A with a CMS providing Class II, Class IV, Class V and Class VI views.

[0028] Figure 2 is a schematic top perspective view of a vehicle cabin including displays and interior cameras.

[0029] Figure 3 is a flowchart of an example method of augmenting an image feed for a commercial vehicle.

[0030] Figure 4 is a schematic view of a CMS image feed.

[0031] Figure 5 is a schematic view of a first example overlay on the CMS image feed of Figure 4.

[0032] Figure 6 is a schematic view of a second example overlay on the CMS image feed of Figure 4.

[0033] Figure 7 is a schematic view of a third example overlay on the CMS image feed of Figure 4.DETAILED DESCRIPTION

[0034] Schematic views of a commercial vehicle 10 are illustrated in Figures 1 A and 1 B. The commercial vehicle 10 includes a vehicle cab or “tractor 12” for pulling a trailer 14, where the trailer 14 pivots with respect to the tractor 12 during turns. Although the vehicle 10 is depicted as a commercial truck with a single trailer in this disclosure, it is understood that other commercial vehicle configurations may be used (e.g., different types or quantities of trailers).

[0035] A pair of camera arms 16A, 16B include a respective base that is secured to, for example, the cab 12. A pivoting arm is supported by the base and may articulate relative thereto. At least one rearward facing camera 20A, 20B (generally, camera 20) is arranged respectively within the camera arms 16A, 16B. The exterior cameras 20A, 20B respectively provide an exterior field of view FOVEXI , FOVEX2 that each include at least one of the Class IIand Class IV views (see Figure 1 B), which are legal prescribed views in the commercial trucking industry. The Class II view on a given side of the vehicle 10 is a subset of the class IV view of the same side of the vehicle 10. Multiple cameras also may be used in each camera arm 16A, 16B to provide these views, if desired. Class II (narrow) and Class IV (wide angle) views are defined in European R46 legislation, for example, and the United States and other countries have similar drive visibility requirements for commercial trucks. Any reference to a “Class” view is not intended to be limiting, but is intended as exemplary for the type of view provided to a display by a particular camera. Each arm 16A-16B may also provide a housing that encloses electronics, e.g., a controller, that are configured to provide various features of the CMS 15 (see Figure 3). The camera arms 16A-B may be mounted either at a roof-mount location over the cab door (as shown), or on a door-mounted bracket or station, for example.

[0036] If video of Class V and Class VI views are also desired, a camera housing 16C and camera 20C may be arranged at or near the front of the vehicle 10 to provide those views (Figure 1 B).

[0037] A rear trailer camera 20D may be provided which provides a field of view FOVEXS- The rear trailer camera 20D may be mounted at a top / centerline of the trailer, at a bumper / bed level of the trailer, or at a top-corner of the back of the trailer, for example. Alternatively, or in addition to the rear trailer camera, a “fifth wheel camera” 20E may be provided that is mounted to a rear of the tractor and that provides a field of view FOVEX4. The fifth wheel camera 20E may be mounted anywhere between the lateral plane of the fifth wheel fixture and the top / roof edge of the tractor, for example.

[0038] Referring now to Figure 2 with continued reference to Figures 1 A-B, a CMS 15 includes driver and passenger side camera arms 16A, 16B mounted to the outside of the vehicle cab 12 (Figure 1A). If desired, the camera arms 16A, 16B may include conventional mirrors integrated with them as well, although the CMS 15 may be used to entirely replace mirrors. In additional examples, each side can include multiple camera arms, each arm housing one or more cameras and / or mirrors. As shown in Figure 2, the CMS 15 includes a CMS controller 13 that includes processing circuitry that supports operation of the CMS 15 and is operatively connected to memory (which may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or nonvolatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). The processing circuitry may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), or the like.

[0039] First and second electronic displays 18A-B, which may be video displays, are arranged on each of the driver and passenger sides within the vehicle cab 12 on or near the A-pillars 19A, 19B to display Class II and Class IV views on its respective side of the vehicle 10, which provide rear facing side views along the vehicle 10 that are captured by the exterior cameras 20A, 20B.

[0040] As discussed above, if video of Class V and Class VI views are also desired, the camera housing 16C and camera 20C may be arranged at or near the front of the vehicle 10 to provide those views (Figure 1 B). A third display 18C arranged within the cab 12 near the top center of the windshield can be used to display the Class V and Class VI views, which are toward the front of the vehicle 10, to the driver.

[0041] If video of Class VIII views is desired, camera housings can be disposed at the sides and rear of the vehicle 10 to provide fields of view including some or all of the Class VIII zones of the vehicle 10. In such examples, the third display 18c can include one or more frames displaying the Class VIII views. Alternatively, additional displays can be added near the first, second, and third displays 18A, 18B, 18C and provide a display dedicated to providing a Class VIII view. The displays 18A, 18B, 18C face a driver region 34 within the cabin 32 where an operator is seated on a driver seat 36, and are correspondingly oriented for viewing by an operator I driver in the driver region 34.

[0042] Figure 3 is a flowchart of an example method 100 of augmenting a CMS image feed (e.g., a video feed, or an image feed provided at a frame rate of less than the 24- 25 frames per second that is commonly associated with video) for a commercial vehicle to facilitate safe performance of a particular driving maneuver, such as a lane change or a turn, while the commercial vehicle is moving forward (i.e., is not in reverse).

[0043] The CMS controller 13 (see Figure 2) obtains an image (step 102) from one or more cameras 20 (e.g., cameras 20A and / or 20B) that are mounted to the commercial vehicle 10 and are oriented to have a rearward-facing field of view. The image feed depicts a surrounding environment of the trailer 14 of the commercial vehicle 10, and may also depict a portion of the trailer 14 as well. The image feed is displayed (step 104) on an electronic display 18 (e.g., electronic display 18A or 18B).

[0044] Referring now to Figure 4, with continued reference to Figure 3, an example of how the image feed obtained in step 102 may be shown the electronic display 18A is provided. As shown, the electronic display 18A includes a first display area 40A corresponding to the Class II view of Figure 1 B, and a second display area 40B corresponding to the Class IV display area of Figure 1 B. Each of the display areas 40A-B display a side 42 of the trailerand its surrounding environment 44. In particular, the surrounding environment 44 includes a lane 46 that is adjacent to the trailer 14. In the example of Figure 4, no overlay is provided in either of the display areas 40A or 40B.

[0045] Referring again to Figure 3, the CMS controller 13 monitors to detect whether a first trigger event has occurred (step 106). In one example, the first trigger event corresponds to activation of a turn signal of the commercial vehicle 10.

[0046] If the first trigger event is not detected (a “no” to step 106), the CMS controller 13 continues to display the image feed on the electronic display (step 104). However, if the first trigger event is detected (a “yes” to step 106), the CMS controller 13 provides a first overlay 48 (see Figure 5) on the image feed that depicts a future position of the commercial vehicle 10 after completion of a particular driving maneuver (e.g., a lane change or turn) (step 108).

[0047] Referring now to Figure 5 with continued reference to Figure 3, an example of the first overlay 48 is provided that corresponds to a lane change maneuver. The first overlay 48 is provided on top of the adjacent lane 46 and is provided in the first display area 40A. The overlay includes three sections 50A, 50B, and 50C, which are separated by dividing lines 52 and 54, respectively. Overlay section 50A represents a future position of the trailer 14 after completion of the lane change. Overlay section 50B represents a first area behind a rear of the trailer 14 after completion of the lane change (e.g., 5-10 meters behind a rear end of the trailer 14), and overlay section 50C represents a second area behind the trailer 14 after completion of the lane change (e.g., 20-30 meters behind the rear end of the trailer 14). In one or more embodiments, each of the display areas 50A-C are displayed in different colors and / or with different shading and / or crosshatching. In one or more embodiments, the display areas 50A-C are all displayed with transparency, so that the underlying depiction of the lane 46 can still be seen through the first overlay 48.

[0048] Referring again to Figure 3, the CMS controller 13 monitors for the occurrence of a second trigger event (step 110). In one or more embodiments, the second trigger event corresponds to the commercial vehicle starting to perform the particular driving maneuver (e.g., lane change or turn).

[0049] If the second trigger event is not detected (a “no” to step 1 10), the CMS controller 13 monitors to see if the first trigger event is still occurring (a “no” to step 1 10). If the first trigger event is no longer occurring (e.g., the turn signal has been turned off), the CMS controller 13 removes the overlay that was added in step 108. However, if the first trigger eventis still occurring (e.g., the turn signal is still activated), the CMS controller 13 continues to provide the first overlay and continues to monitor for the second trigger event (step 110).

[0050] If the CMS controller 13 detects the second trigger event (a “yes” to step 110), the CMS controller 13 provides a second overlay on the image feed of a vehicle path that would be followed if the commercial vehicle 10 performed the particular driving maneuver (step 112). The second overlay 60 is different than the first overlay 48. In one or more embodiments, the second trigger event corresponds to the commercial vehicle 10 starting to perform the particular driving maneuver within a predefined time period (e.g., 30-60 seconds) after activation of the turn signal.

[0051] Referring now to Figure 6, with continued reference to Figure 3, an example of the second overlay 60 is provided that depicts a vehicle path that would be followed if the commercial vehicle 10 performed the particular driving maneuver, which in the example of Figure 6 is a lane change. In one or more embodiments, the second overlay 60 is also displayed with transparency, so that the underlying depiction of the adjacent lane 46 is still visible through the second overlay 60.

[0052] The CMS controller 13 monitors to detect if the particular driving maneuver is complete (step 114). If the particular driving maneuver is not complete (e.g., the vehicle is still changing lanes or turning) (a “no” to step 114), the CMS controller 13 continues providing the second overlay 60. However, if the particular driving maneuver is complete or has been aborted (a “yes” to step 114), the CMS controller 13 removes the second overlay 60, and if it is still being displayed, also removes the first overlay 48.

[0053] In one or more embodiments, the CMS controller 13 turns off (i.e., stops displaying) the first overlay 48 before displaying the second overlay 60.

[0054] In one or more embodiments, upon detecting the second trigger event (a “yes” to step 110), the CMS controller 13 dims (i.e., makes more transparent) the first overlay to obtain a dimmed version of the first overlay, and simultaneously provides the second overlay and the dimmed version of the first overlay on the display.

[0055] In one or more embodiments, upon detecting the second trigger event (a “yes” to step 110), the CMS controller 13 provides the second overlay 60 on top of the first overlay 48 without dimming the first overlay 48.

[0056] As described above, in the example of Figure 3, the providing of the second overlay 60 in step 112 is triggered by the first trigger event and the second trigger event, and the second trigger event alone does not trigger display of the second overlay 60. In one ormore further embodiments, however, the providing of the second overlay 60 is triggered by the second trigger event alone or the first trigger event alone.

[0057] In one or more embodiments, as discussed above, the CMS controller 13 presents one or both of the overlays 48, 60 as transparent on top of the CMS video feed.

[0058] In one or more embodiments, the CMS controller 13 adapts the first overlay 48 and / or the second overlay 60 while the particular driving maneuver is being performed, so that the overlay remains on top of the lane 46 on the electronic display 18, and does not extend onto the trailer 14 or beyond a width of the lane 46 (e.g., as the lane change is being performed and the commercial vehicle 10 is entering the lane 46, reduce a width of the first overlay 48 and / or the second overlay 60 to hide a portion of the overlay(s) that the vehicle trailer 14 would otherwise be on top of).

[0059] In one or more embodiments, if the second trigger event 1 10 is detected on its own, and is not preceded by occurrence of the first trigger event within a predefined time period (e.g., 30-60 seconds), the CMS controller 60 still provides the second overlay 60.

[0060] In one or more embodiments, the second overlay 60 is determined dynamically based on a current speed and steering wheel angle of the commercial vehicle 10, and is dynamically updated as the current speed and / or steering wheel angle change.

[0061] The overlays 48, 60 can help a driver to check more easily if they can safely perform the driving maneuver (e.g., without coming into contact with an obstacle). Optionally, the CMS controller 13 can also perform object detection, and automatically warn a driver about hazards detected in CMS images.

[0062] Although the predefined driving maneuver of a lane change has been described above, it is understood that this is a non-limiting example, and that other driving maneuvers, such as turns, could also trigger display of the second overlay. In one or more embodiments, if the commercial vehicle 10 is traveling above a predefined speed threshold, the CMS controller 13 interprets the turn signal as indicating of an intended lane change, and if the commercial vehicle 10 is traveling below the predefined speed threshold, the CMS controller 13 interprets the turn signal as indicating an intended turn.

[0063] In one or more embodiments, if the CMS controller 13 interprets the turn sign as indicating an intended turn, the first overlay (indicating a future position of the vehicle if the driving maneuver is performed) is omitted (i.e. , steps 106 and 110 are omitted), and only the second overlay 60 is provided, but in a manner that depicts a turn.

[0064] Figure 7 is a schematic view of a third example overlay 70 that corresponds to a turn.

[0065] Although the overlays in Figures 5-7 are only shown as being depicted in the Class II display area 40A, it is understood that they could also, or alternatively, be provided in the Class IV display area 40B.

[0066] Also, although the overlays 48, 60 are only shown for a left turn, it is understood that the same overlay features could be provided on an opposite side of the commercial vehicle (e.g., for rightward lane changes and / or rightward turns).

[0067] Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

CLAIMSWhat is claimed is:1 . A method of augmenting an image feed for a commercial vehicle, comprising: obtaining an image feed from a camera mounted to a commercial vehicle and oriented to have a rearward-facing field of view, the image feed depicting an exterior environment of a trailer of the commercial vehicle; displaying the image feed on an electronic display oriented for viewing in a cabin of the commercial vehicle; and providing an overlay on the image feed of a vehicle path that would be followed if the commercial vehicle performed a particular driving maneuver.

2. The method of claim 1 , wherein the particular driving maneuver is a lane change.

3. The method of claim 1 , wherein said providing the overlay is triggered at least partially by activation of a turn signal in the commercial vehicle.

4. The method of claim 3, wherein said providing the overlay is further triggered by the commercial vehicle starting to perform the particular driving maneuver.

5. The method of claim 4, comprising reducing a size of the overlay as the commercial vehicle performs the particular driving maneuver.

6. The method of claim 1 , wherein the overlay is a second overlay, the method comprising: prior to providing the second overlay, providing a first overlay on the image feed, wherein the first overlay is different than the second overlay and depicts a future position of the commercial vehicle after completion of the particular driving maneuver.

7. The method of claim 6, wherein: the providing of the first overlay is triggered by activation of a turn signal of the commercial vehicle; and the providing of the second overlay is triggered by the commercial vehicle starting to perform the particular driving maneuver within a predefined time period after activation of the turn signal.

8. The method of claim 6, wherein: the first overlay includes at least one distance indicator; and each of the at least one distance indicators indicating a respective distance from a rear end of the trailer of the commercial vehicle.

9. The method of claim 6, comprising: dimming the first overlay to obtain a dimmed version of the first overlay; wherein said providing the second overlay comprises simultaneously providing the second overlay and the dimmed version of the first overlay.

10. The method of claim 6, comprising: turning off the first overlay prior to providing the second overlay.1 1 . A camera monitoring system, comprising: a camera mounted to a commercial vehicle and configured to provide an image feed that depicts an exterior environment of a trailer of the commercial vehicle; an electronic display oriented for viewing in a cabin of the commercial vehicle; and processing circuitry configured to: display an image feed from the camera on the electronic display; and provide an overlay on the image feed on the electronic display of a vehicle path that would be followed if the commercial vehicle performed a particular driving maneuver.

12. The camera monitoring system of claim 1 1 , wherein the particular driving maneuver is a lane change.

13. The camera monitoring system of claim 1 1 , wherein the processing circuitry is configured to provide the overlay at least partially based on activation of a turn signal in the commercial vehicle.

14. The camera monitoring system of claim 13, wherein the processing circuitry is configured to: detect that the commercial vehicle has started to perform the particular driving maneuver; and provide the overlay further based on the detection.

15. The camera monitoring system of claim 14, wherein the processing circuitry is configured to reduce a size of the overlay as the commercial vehicle performs the particular driving maneuver.

16. The camera monitoring system of claim 11 , wherein the overlay is a second overlay, and the processing circuitry is configured to: prior to providing the second overlay, provide a first overlay on the image feed, wherein the first overlay is different than the second overlay and depicts a future position of the commercial vehicle after completion of the particular driving maneuver.

17. The camera monitor system of claim 16, wherein the processing circuitry is configured to: provide the first overlay based on activation of a turn signal of the commercial vehicle; and provide the second overlay based on the commercial vehicle starting to perform the particular driving maneuver within a predefined time period after activation of the turn signal.

18. The camera monitor system of claim 16, wherein: the first overlay includes at least one distance indicator; and each of the at least one distance indicators indicates a respective distance from a rear end of the trailer of the commercial vehicle.

19. The camera monitor system of claim 16, wherein the processing circuitry is configured to: dim the first overlay to obtain a dimmed version of the first overlay; and simultaneously provide the second overlay and the dimmed version of the first overlay on the image feed.

20. The camera monitor system of claim 16, wherein the processing circuitry is configured to: turn off the first overlay prior to providing the second overlay.

Images