Camera monitor system with wide-angle indirect view with multiple cameras and landscape monitor

Abstract

First, second and third cameras respectively capture first, second and third images respectively from first, second and third fields of view that collectively capture a partial surround view of more than 180º at a rear of the vehicle. The first field of view is rearward from the vehicle, and the second and third fields of view are respectively from left-hand and right-hand sides of the vehicle. An image processing module outputs at least a portion of the first, second and third captured images. At least one display is in communication with the controller to receive the output. The at least one display is configured to display the at least portion of the first captured image between the at least portions of the second and third captured images in an un-inverted landscape configuration without stitching the at least portions of the first, second and third captured images to one another.

Description

CAMERA MONITOR SYSTEM WITH WIDE-ANGLE INDIRECT VIEW WITH MULTIPLE CAMERAS AND LANDSCAPE MONITORTECHNICAL FIELD

[0001] This disclosure relates to a camera monitor system (CMS) for a vehicle, and specifically to a CMS and method of capturing and displaying views to provide at least a partial vehicle surround view free of blind spots and configured in an intuitive manner for the operator.BACKGROUND

[0002] Mirror replacement systems, and camera systems for supplementing mirror views, are utilized in commercial on-highway and off-highway (i.e., “machines”) vehicles to enhance the ability of a vehicle operator to see a surrounding environment. Camera monitor systems (CMS) utilize one or more cameras to provide an enhanced field of view to a vehicle operator. In some examples, the camera systems cover a larger field of view than a conventional mirror, or include views that are not fully obtainable via a conventional mirror.

[0003] For machine operators who drive on earth moving, agricultural, and industrial machines, a surround view around their machine is needed. The views surrounding the machine should be free of blind spots as set forth in ISO 5006, ISO 5721 and ISO 14401. Typical indirect viewing systems (i.e., camera systems) use 4:3 (horizontakvertical) aspect ratio for displays placed on the A-pillars of the cabin, thus limiting the direct view of drivers. This is accomplished using four cameras, with each camera placed on each of the four flanks of the machine (i.e., front center of vehicle, right center of vehicle, rear center of vehicle, left center of vehicle).

[0004] Traditional 4:3 displays may offer multiple camera views in a chess board-like pattern (e.g., examples shown in Figs. 8A and 8B). Left-hand and right-hand views are arranged respectively on the left-hand and right-hand sides of the display,1

[0001] 55366340-1extending the full height of the display. The front view is arranged at the center / top of the display, and the rear view is arranged at the center / bottom of the display in an upside down / inverted orientation. Often the views are presented in a disjointed bird’s eye view. This representation does not show the views in an intuitive way, according the real situation around a machine.SUMMARY

[0005] In one exemplary embodiment, a camera monitor system (CMS) for a vehicle includes first, second and third cameras that are respectively configured to capture first, second and third images respectively from first, second and third fields of view that collectively capture a partial surround view of more than 180° at a rear of the vehicle, the first field of view is rearward from the vehicle, and the second and third fields of view are respectively from left-hand and right-hand sides of the vehicle. A controller is in communication with the first, second, and third cameras, the controller includes an image processing module that is configured to output at least a portion of the first, second and third captured images. At least one display is in communication with the controller to receive the output. The at least one display is configured to display the at least portion of the first captured image between the at least portions of the second and third captured images in an un-inverted landscape configuration without stitching the at least portions of the first, second and third captured images to one another.

[0006] In a further embodiment of any of the above, each of the at least portion of first, second and third captured images include a horizon. The horizons of the at least portions of the first, second and third captured images are generally aligned on the at least one display.

[0007] In a further embodiment of any of the above, the at least portions of the first, second and third captured images each include upper image portions above the horizon. The upper image portions are arranged on an upper portion of the at least one display.2

[0001] 55366340-1

[0008] In a further embodiment of any of the above, the at least one display is a common display that is arranged in a landscape orientation.

[0009] In a further embodiment of any of the above, the common display has an aspect ratio of at least 2.5:1

[0010] In a further embodiment of any of the above, the at least portions of the first, second and third captured images are the only images depicted on the common display.

[0011] In a further embodiment of any of the above, the displayed at least portions of the first, second and third captured images include overlapping portions that are unstitched.

[0012] In a further embodiment of any of the above, the first camera is configured to be mounted to the rear of the vehicle, and the second and third cameras are configured to respectively be mounted to left-hand and right-hand sides of the vehicle. The overlapping portions are captured from a different view point.

[0013] In a further embodiment of any of the above, the displayed at least portions of the first, second and third captured images collectively provide a partial surround view of at least 180° including at the rear of the vehicle.

[0014] In a further embodiment of any of the above, the partial surround view includes at least a 180° perimeter of the vehicle at least 1 meter high from the ground, and at least 12 meters from the vehicle.

[0015] In a further embodiment of any of the above, the CMS includes a fourth camera that is configured to capture a fourth image from a fourth field of view from a front of the vehicle. The first, second, third and fourth fields of view together provide an unstitched 360° surround view.

[0016] In a further embodiment of any of the above, the at least one display includes first and second displays. The first display is configured to display the at least portions of the first, second and third captured images, and the second display is configured to display at least a portion of the fourth captured image.3

[0001] 55366340-1

[0017] In a further embodiment of any of the above, the at least one display includes a first display. The first display is configured to display the at least portions of the first, second, third and fourth captured images.

[0018] In another exemplary embodiment, a method of depicting a wide-angle view around a vehicle includes capturing first, second and third images respectively from first, second and third fields of view respectively with first, second and third cameras. The first, second and third images configured to that collectively capture a partial surround view of more than 180° at a rear of the vehicle. The first field of view is rearward from the vehicle, and the second and third fields of view are respectively from left-hand and righthand sides of the vehicle. A display outputs at least a portion of the first, second and third captured images. The at least portion of the first captured image between the at least portions of the second and third captured images is displayed in an un-inverted landscape configuration without stitching the at least portions of the first, second and third captured images to one another.

[0019] In a further embodiment of any of the above, the displaying step includes at least portions of the first, second and third captured images collectively providing a partial surround view of at least 270° including at the rear of the vehicle, and include a step of capturing a fourth image from a fourth field of view from a front of the vehicle with a fourth camera. The first, second, third and fourth fields of view together provide an unstitched 360° surround view.

[0020] These and other features of the present invention can be best understood from the following specification and drawings, the following of which is a brief description.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The disclosure can be further understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein:4

[0001] 55366340-1

[0022] Figure 1 is a perspective view of an earth moving machine with a disclosed camera monitoring system (CMS).

[0023] Figure 2 is a schematic of an example CMS.

[0024] Figure 3 illustrates a cab of an example earth moving machine.

[0025] Figure 4 depicts an image capture unit of a camera.

[0026] Figure 5 is an example display for the earth moving machine and environment shown in Figure 1.

[0027] Figures 6A and 6B are additional examples of display configurations according to the disclosure.

[0028] Figure 7 is a flowchart of an example method used in the disclosed CMS.

[0029] Figures 8A and 8B illustrate example prior art display configurations.

[0030] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, unless such features are incompatible.DETAILED DESCRIPTION

[0031] Figure 1 illustrates a perspective view of a vehicle 10, such as an earth moving machine. The vehicle 10 has a front side 46, a right side 48, a back side 50 and a left side 52. Off-highway machines typically include one or more movable implements, such as a backhoe 54. The disclosed system can be used for a variety of vehicles such as agricultural, industrial and on-highway commercial vehicles, for example.

[0032] As is typical of most applications, the vehicle 10 may operate in a crowded environment with various objects (e.g., people 12a-12d (collectively, “people”) and objects 14a-14d (collectively, “objects”)) with which collisions must be avoided. A reliable camera monitor system (CMS, at 16 in Fig. 2) provides a surround view of the vehicle 10 free from blind spots so that the implements and objects remain in view5

[0001] 55366340-1according to applicable regulations (e.g., ISO 5006, ISO 14401 , ISO 16001). According to one example standard, an operator of the vehicle 10 should have visibility without blind spots of a rectangular vehicle perimeter 13 1 meter high and at ground level out to a radius perimeter 15 at least 12 meters from the vehicle 10.

[0033] Referring to Figures 1 and 2, the CMS 16 includes first, second, third and fourth cameras 18a, 18b, 18c, 18d (collectively, “cameras 18”) in communication with a controller, such as ECU 20. The ECU 20 may be a hardware device for executing software, particularly software stored in memory. The ECU 20 can be a custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chip set) or generally any device for executing software instructions.

[0034] In terms of hardware architecture, such a computing device can include a processor, memory, and one or more input and / or output (I / O) device interface(s) that are communicatively coupled via a local interface. The local interface can include, for example but not limited to, one or more buses (e.g., CAN bus 26) and / or other wired or wireless connections. The local interface may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers to enable communications. Further, the local interface may include address, control, and / or data connections to enable appropriate communications among the aforementioned components.

[0035] The memory can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or nonvolatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). Moreover, the memory may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory can also have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor.6

[0001] 55366340-1

[0036] The software in the memory may include one or more separate programs, each of which includes an ordered listing of executable instructions for implementing logical functions. A system component embodied as software may also be construed as a source program, executable program (object code), script, or any other entity comprising a set of instructions to be performed. When constructed as a source program, the program is translated via a compiler, assembler, interpreter, or the like, which may or may not be included within the memory.

[0037] The disclosed input and output devices that may be coupled to system I / O interface(s) may include input devices, for example but not limited to, a keyboard, mouse, scanner, microphone, camera, mobile device, proximity device, etc. Further, the output devices, for example but not limited to, a printer, display, etc. Finally, the input and output devices may further include devices that communicate both as inputs and outputs, for instance but not limited to, a modulator / demodulator (modem; for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, etc.

[0038] When the ECU 20 is in operation, the processor can be configured to execute software stored within the memory, to communicate data to and from the memory, and to generally control operations of the computing device pursuant to the software. Software in memory, in whole or in part, is read by the processor, perhaps buffered within the processor, and then executed.

[0039] The CMS 16 receives power 24 from the vehicle 10. ECU 20 is in communication with an input device 28, which may be used by the operator to customize and control the CMS 16. The ECU 20 may include an image processing module that can stitch images from the cameras in a known manner to combine more than one field of view into a seamless displayed image, for example, on one or more displays 22. The display(s) 22 may be located in the vehicle’s cab within easy view by the operator.

[0040] It is desirable to display a partial surround view depicting at least the perimeter of the vehicle 10 (e.g., at least 1 meter high from the ground at vehicle perimeter 13, and ground level out to at least 12 meters radius perimeter 15 from the vehicle 10)7

[0001] 55366340-1that is not easily visible to the operator due to visibility restrictions of the particular machine, which varies depending upon the application. This perimeter that must be displayed to the operator free from blind spots is typically greater than 180° in one example, and in another example is at least a 270° Returning to Figure 1 , the vehicle 10 provides the first camera 18a mounted to the rear 50 of the vehicle 10 to capture images in a first field of view FOV1. With the first camera 18a mounted at the rear 50, at best 180° view can be captured. The second and third cameras 18b, 18c are mounted respectively to left-hand and right-hand sides 52, 48 of the vehicle 10 to capture images respectively in second and third fields of view FOV2, FOV3 immediately adjacent to and overlapping with the first field of view FOV1. In this manner, the first, second and third cameras 18a, 18b, 18c capture at least 180° about the vehicle 10.

[0041] A fourth camera 18d at the front 46 of the vehicle 10 captures a fourth image from a fourth field of view FOV4 to provide a 360° surround view in cooperation with the first, second and third cameras 18a, 18b, 18c. Additional cameras may be used, if desired. It should be understood that other types of sensors may be used in addition to the cameras 18. For example, radar sensors, ultrasonic sensors, LiDar sensors, vibration sensors and / or other sensors may be similarly arranged as described in this disclosure to provide a surround “view” free from blind spots.

[0042] Images captured (at 32 in Fig. 4) by the image capture units are warped or distorted by the lenses such that de-warping and / or other image correction is used in an image processing module, as is known, before displaying the corrected images. The image processing module may crop (at 34 in Fig. 4) the captured image such that only a portion of the captured image is used and output to the display 22 for depicting the environment captured by the camera’s field of view.

[0043] Referring to Figure 5, if a common display 22 is used to the display at least a portion of the first, second and third captured images, the display 22 is arranged in a landscape orientation and may have an aspect ratio of at least 2.5:1 (in another example, 8:3) to provide a wide screen able to display three separate views (30a-30c) side-by-side in a manner that is does not require a tall display. Bars 42a, 42b may be 8

[0001] 55366340-1displayed on the screen to separate the views 30a-30c. The display 22 can be mounted centrally, high in the vehicle’s cab (see, Fig. 3) to both provide clear readily accessible images without obstructing direct operator visibility. In the example shown in Figure 5, only the first, second and third captured images are depicted on the common display 22, although other variations are also possible (e.g., see, Figs. 6A-6B).

[0044] From left to right on the display 22 shown in Figures 3 and 5, at least a portion of the captured images the first field of view FOV1 (rear) is arranged centrally in between the at least portion of the captured images from the second field of view FOV2 (left) and the third field of view FOV3 (right). The display portions of the first, second and third captured images are from different view points and are not stitch such that overlapping portions 40a, 40b are displayed. The people 12 and objects 14 are clearly visible to the operator in a situation where they may be lost by stitched images. This approach definitively eliminates blind spots. Unlike many prior art displays, the at least portion of the first captured image is displayed un-inverted, or “right side up” from an operator perspective.

[0045] To make the displayed views more intuitive to the operator as a whole, the horizon 36 (horizon line) in each image is displayed such that they are generally aligned with one another. In the example shown in Figure 5, the upper image portions 38a, 38b, 38c of the captured images (i.e., the portion of the image above the horizon line) are all arranged on an upper portion of the display 22 (i.e., right side up).

[0046] Another example display 122 is shown in Figure 6A, where at least portions of captured images from the first, second and third fields of view FOV1 , FOV2, FOV3 are respectively displayed in viewing areas 130a, 130b, 130c, which are approximately the same dimensions in the example. At least a portion of the captured images from the fourth field of view FOV4 can be displayed in viewing area 130d above the viewing area 130a. In this arrangement, the horizons of the at least portions of captured images from the first, second and third fields of view FOV1 , FOV2, FOV3 may not be perfectly aligned, but may be generally aligned so as to be easily understood to the operator.9

[0001] 55366340-1

[0047] In the example shown in Figure 6B, first and second displays are used to provide the four fields of view to the operator. The first display may be provided by three discrete displays or screens 222a, 222b, 222c to provide the at least portions of the first, second and third captured images. The second display 222d is used to display at least a portion of the fourth captured image in viewing area 230d. Other configurations may also be used.

[0048] In operation, the CMS 16 performs a method 60 depicting a wide-angle view around a vehicle, as illustrated in the flowchart of Figure 7. First, second and third images respectively from first, second and third fields of view FOV1 , FOV2, FOV3 are respectively captured with first, second and third cameras 18a, 18b, 18c (block 62). The first, second and third images collectively capture a partial surround view of more than 180° (e.g., at least 270°) at a rear of the vehicle 10 (see, e.g., Fig. 1 ). The first field of view FOV1 is rearward from the vehicle 10, and the second and third fields of view FOV2, FOV3 are respectively from left-hand and right-hand sides of the vehicle 10. At least a portion of each of the first, second and third captured images are output (block 64), typically after being dewarped, cropped and / or corrected. The output images are displayed on at least one display (e.g., display 22, 122, 222) such that the at least portion of the first captured image is arranged between the at least portions of the second and third captured images in an un-inverted landscape configuration without stitching the at least portions of the first, second and third captured images to one another (block 66; shown, for example, in Fig. 5).

[0049] With the traditional almost square aspect ratio (width:height) of the prior art displays, it is not possible to project an intuitive wide view. Using a bigger display may provide a wider view, but will result in more obstruction of the direct view for the operator. The disclosed approach allows at least three horizontally adjacent images to be projected on a display, representing a natural logical (intuitive to operator) image by placing the images horizontally next to each other, minimizing the height of the display. The disclosed configuration can provide a very wide (rear) view on a horizontal monitor with aspect ratio of at least 2.5:1 (e.g., 8:3). This reduced height enables placement of the display above 10

[0001] 55366340-1the windshields, enabling a wider indirect (rear) view as well as more direct view in front of the operator. By showing this wide view in several adjacent images, ISO 16001 requirements are satisfied relating magnification and sharpness, while ensuring that no visibility blind spots occur. The advantages provided by the disclosed CMS enhance direct visibility in combination with wide view rearwards, which may avoid accidents and enable the operator to execute maneuvers faster while maintain safety.

[0050] It should also be understood that although a particular component arrangement is disclosed in the illustrated embodiment, other arrangements will benefit herefrom. Although particular step sequences are shown, described, and claimed, it should be understood that steps may be performed in any order, separated or combined unless otherwise indicated and will still benefit from the present invention.

[0051] Although the different examples have specific components shown in the illustrations, embodiments of this invention are not limited to those particular combinations. It is possible to use some of the components or features from one of the examples in combination with features or components from another one of the examples.

[0052] Although an example embodiment has been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.11

[0001] 55366340-1

Claims

CLAIMSWhat is claimed is:1 . A camera monitor system (CMS) for a vehicle, comprising: first, second and third cameras respectively configured to capture first, second and third images respectively from first, second and third fields of view that collectively capture a partial surround view of more than 180° at a rear of the vehicle, the first field of view rearward from the vehicle, and the second and third fields of view respectively from lefthand and right-hand sides of the vehicle; a controller in communication with the first, second, and third cameras, the controller including an image processing module configured to output at least a portion of the first, second and third captured images; and at least one display in communication with the controller to receive the output, the at least one display configured to display the at least portion of the first captured image between the at least portions of the second and third captured images in an un-inverted landscape configuration without stitching the at least portions of the first, second and third captured images to one another.

2. The CMS of claim 1 , wherein each of the at least portion of first, second and third captured images include a horizon, the horizons of the at least portions of the first, second and third captured images generally aligned on the at least one display.

3. The CMS of claim 2, wherein the at least portions of the first, second and third captured images each include upper image portions above the horizon, the upper image portions arranged on an upper portion of the at least one display.

4. The CMS of claim 3, wherein the at least one display is a common display arranged in a landscape orientation.12[0001] 55366340-15. The CMS of claim 4, wherein the common display has an aspect ratio of at least 2.5:

16. The CMS of claim 4 or 5, wherein the at least portions of the first, second and third captured images are the only images depicted on the common display.

7. The CMS of any preceding claim, wherein the displayed at least portions of the first, second and third captured images include overlapping portions that are unstitched.

8. The CMS of claim 7, wherein the first camera is configured to be mounted to the rear of the vehicle, and the second and third cameras are configured to respectively be mounted to left-hand and right-hand sides of the vehicle, the overlapping portions captured from a different view point.

9. The CMS of claim 7 or 8, wherein the displayed at least portions of the first, second and third captured images collectively provide a partial surround view of at least 180° including at the rear of the vehicle.

10. The CMS of claim 9, wherein the partial surround view includes at least a 180° perimeter of the vehicle at least 1 meter high from the ground, and at least 12 meters from the vehicle.11 . The CMS of any preceding claim, comprising a fourth camera configured to capture a fourth image from a fourth field of view from a front of the vehicle, the first, second, third and fourth fields of view together providing an unstitched 360° surround view.13[0001] 55366340-112. The CMS of claim 11 , wherein the at least one display includes first and second displays, the first display configured to display the at least portions of the first, second and third captured images, and the second display configured to display at least a portion of the fourth captured image.

13. The CMS of claim 11 or 12, wherein the at least one display includes a first display, the first display configured to display the at least portions of the first, second, third and fourth captured images.

14. A method of depicting a wide-angle view around a vehicle, comprising: capturing first, second and third images respectively from first, second and third fields of view respectively with first, second and third cameras, the first, second and third images configured to that collectively capture a partial surround view of more than 180° at a rear of the vehicle, the first field of view rearward from the vehicle, and the second and third fields of view respectively from left-hand and right-hand sides of the vehicle; outputting to a display at least a portion of the first, second and third captured images; and displaying the at least portion of the first captured image between the at least portions of the second and third captured images in an un-inverted landscape configuration without stitching the at least portions of the first, second and third captured images to one another.

15. The method of claim 14, wherein the displaying step includes at least portions of the first, second and third captured images collectively providing a partial surround view of at least 270° including at the rear of the vehicle, and comprising a step of capturing a fourth image from a fourth field of view from a front of the vehicle with a fourth camera, the first, second, third and fourth fields of view together providing an unstitched 360° surround view.14[0001] 55366340-1

Images