Method and system for vertical image stitching
The CMS method and system address the challenge of integrating multiple camera views by applying perspective transformations and distortion corrections to create a seamless, distortion-free combined image, improving the vehicle operator's field of view.
Patent Information
- Application Number
- JP2025106332
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-07-01
- Filing Date
- 2025-06-24
- Publication Date
- 2026-01-21
AI Technical Summary
Vehicle operators face difficulty in effectively analyzing multiple images from different cameras due to overlapping fields of view and optical axes, leading to challenges in obtaining a cohesive and distortion-free view of the vehicle's surroundings.
A method and system for camera monitor systems (CMS) that involves acquiring images from cameras with different overlapping fields of view and optical angles, applying perspective transformations and distortion corrections, and vertically stitching these images to form a combined, distortion-free image.
Provides a larger, continuous, and cohesive field of view without significant distortion, enhancing the operator's ability to view the vehicle's surroundings by integrating multiple camera feeds into a single, seamless image.
Smart Images
Figure 2026009839000001_ABST
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to camera monitor systems (CMS), and more particularly to a method and system for providing vertical image stitching in a CMS. [Background technology]
[0002] Vehicle camera systems, either to replace mirrors or to supplement mirror views, are being utilized in commercial vehicles to enhance the vehicle operator's ability to view the commercial vehicle's surroundings. Camera monitoring systems (CMS) utilize one or more cameras to provide the vehicle operator with an enhanced field of view. In some instances, mirror replacement systems cover a wider field of view than traditional mirrors or include views not fully available through traditional mirrors. When multiple cameras provide images to vehicle occupants, it can be difficult for the driver to effectively analyze the images. Summary of the Invention
[0003] A method for a camera monitor system (CMS) according to one embodiment of the present disclosure includes acquiring a first image from a first camera and a second image from a second camera, the first image and the second image depicting a side of a commercial vehicle and its surrounding environment, the first camera and the second camera having different overlapping fields of view and different optical axes intersecting a ground plane at different optical angles. The method also includes perspective transforming at least one of the first image and the second image to obtain an updated image set, whereby at least one of the first image and the second image is updated in the updated image set, and vertically stitching images of the updated image set together to form a combined image.
[0004] In a further embodiment of the above embodiment, the respective different optical angles include a first optical angle of the first camera and a second optical angle of the second camera, the first optical angle being less than 90°, and the second optical angle being less than or equal to 90° and greater than the first optical angle.
[0005] In a further embodiment of any of the above embodiments, the first camera has a first focal length and the second camera has a second focal length that is shorter than the first focal length, and the second camera is positioned closer to the front of the commercial vehicle than the first camera.
[0006] In a further embodiment of any of the above embodiments, the first image is rendered over the second image.
[0007] In a further embodiment of any of the above embodiments, the step of performing a perspective transformation is performed on both the first image and the second image, thereby updating both the first image and the second image in the updated image set.
[0008] In a further embodiment of any of the above embodiments, the method includes, before performing the perspective transformation, performing at least one of distortion correction on the first image from the first camera to mitigate image distortion caused by a lens or sensor of the first camera, and distortion correction on the second image from the second camera to mitigate image distortion caused by a lens or sensor of the second camera.
[0009] In a further embodiment of any of the above embodiments, the method includes, before performing the perspective transformation, performing both distortion correction on the first image from the first camera to mitigate image distortion caused by a lens or sensor of the first camera, and distortion correction on the second image from the second camera to mitigate image distortion caused by a lens or sensor of the second camera.
[0010] In a further embodiment of any of the above embodiments, the method includes performing at least one of cropping and zooming of at least one of the first image and the second image before vertical stitching.
[0011] In a further embodiment of any of the above embodiments, the step of vertically stitching images of the updated image set together to form a combined image includes mapping a plurality of points of the first image in the updated image set to the second image in the updated image set, aligning the first image in the updated image set with the second image in the updated image set based on the mapping, and blending the first image in the updated image set with the second image in the updated image set to form the combined image.
[0012] In a further embodiment of any of the above embodiments, the method includes: in a first mode, displaying the combined image on an electronic display within the commercial vehicle; in a second mode, displaying at least one of the first image and the second image separately without being combined; and switching between the first mode and the second mode in response to receiving a toggle command from an occupant of the commercial vehicle.
[0013] A camera monitor system (CMS) according to an exemplary embodiment of the present disclosure includes a first camera and a second camera, each configured to record images of a side of a commercial vehicle and its surrounding environment, the first camera and the second camera having different overlapping fields of view and different optical axes that intersect a ground plane at different optical angles. The camera monitor system includes a processing circuit operatively connected to a memory, the processing circuit configured to: acquire a first image from the first camera and a second image from the second camera; perform a perspective transformation on at least one of the first image and the second image to acquire an updated image set, whereby at least one of the first image and the second image is updated in the updated image set; and vertically stitch images of the updated image set together to form a combined image.
[0014] In a further embodiment of the above embodiment, the respective different optical angles include a first optical angle of the first camera and a second optical angle of the second camera, the first optical angle being less than 90°, and the second optical angle being less than or equal to 90° and greater than the first optical angle.
[0015] In a further embodiment of any of the above embodiments, the first camera has a first focal length and the second camera has a second focal length that is shorter than the first focal length, and the second camera is positioned closer to the front of the commercial vehicle than the first camera.
[0016] In a further embodiment of any of the above embodiments, in the combined image, the first image is depicted below the second image.
[0017] In a further embodiment of any of the above embodiments, the processing circuitry is configured to perform the perspective transformation on both the first image and the second image, whereby both the first image and the second image are updated in the updated image set.
[0018] In a further embodiment of any of the above embodiments, the processing circuitry is configured to, before performing the perspective transformation, perform at least one of distortion correction on the first image from the first camera to mitigate image distortion caused by a lens or sensor of the first camera, and distortion correction on the second image from the second camera to mitigate image distortion caused by a lens or sensor of the second camera.
[0019] In a further embodiment of any of the above embodiments, the processing circuitry is configured to, before performing the perspective transformation, perform both distortion correction on the first image from the first camera to mitigate image distortion caused by a lens or sensor of the first camera, and distortion correction on the second image from the second camera to mitigate image distortion caused by a lens or sensor of the second camera.
[0020] In a further embodiment of any of the above embodiments, the processing circuitry is configured to perform at least one of cropping and zooming of at least one of the first image and the second image before vertical stitching.
[0021] In a further embodiment of any of the above embodiments, to vertically stitch images of the updated image set, the processing circuitry is configured to: map a plurality of points of the first image in the updated image set to the second image in the updated image set; align the first image in the updated image set with the second image in the updated image set based on the mapping; and blend the first image in the updated image set with the second image in the updated image set to form the combined image.
[0022] In a further embodiment of any of the above embodiments, the processing circuitry is configured to: in a first mode, display the combined image on an electronic display within the commercial vehicle; in a second mode, display at least one of the first image and the second image separately without being combined; and switch between the first mode and the second mode in response to receiving a toggle command from an occupant of the commercial vehicle.
[0023] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. [Brief explanation of the drawings]
[0024] The present disclosure may be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitor system (CMS) used to provide at least Class II and Class IV views. [Figure 2]2 is a schematic bird's-eye view of the commercial truck of FIG. 1 with a CMS providing Class II, Class IV, Class V, and Class VI views. [Figure 3] 1 is a schematic top view of an exemplary vehicle cab interior; [Figure 4] FIG. 4 is a perspective view of the inside of the vehicle cab of FIG. 3. [Figure 5A] 2 is a schematic diagram of the side of the commercial vehicle of FIG. 1 and the optical angle of the first CMS camera. [Figure 5B] 2 is a schematic diagram of the side of the commercial vehicle of FIG. 1 and the optical angle of the second CMS camera. [Figure 6A] 5B is an example image of the CMS camera in FIG. 5A. [Figure 6B] Figure 5B is an example image from the CMS camera. [Figure 7A] 6B is an example of the image of FIG. 6A after distortion correction. [Figure 7B] 6C is an example of the image of FIG. 6B after distortion correction. [Figure 8A] 7B is an example of the image of FIG. 7A after perspective transformation. [Figure 8B] 7C is an example of the image of FIG. 7B after perspective transformation. [Figure 9] 8C is a vertically stitched combined image of FIG. 8A and FIG. 8B. [Figure 10] 1 is a flowchart of an exemplary method for a CMS. DETAILED DESCRIPTION OF THE INVENTION
[0025] 1-4 show schematic diagrams of a commercial vehicle 10. The commercial vehicle 10 includes a vehicle cab or "tractor" 12 for towing a trailer 14, which pivots relative to the tractor 12 during turns. In this disclosure, the commercial vehicle 10 is shown as a commercial truck having a single trailer, although it is understood that other commercial vehicle configurations (e.g., different types or quantities of trailers) may be used.
[0026] A pair of camera arms 16A-16B each include a respective base fixed to, for example, the tractor 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20A-20B is disposed on or within each of the camera arms 16A-16B. Each of the exterior cameras 20A-20B has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 2), each of which represents a legally defined view in the commercial trucking industry. EX1 , FOV EX2 to provide.
[0027] A Class II view of a given side of the commercial vehicle 10 is a subset of a Class IV view of the same side of the commercial vehicle 10. If desired, multiple cameras can be used in each camera arm 16A-16B to provide these views. For example, Class II (narrow angle) and Class IV (wide angle) views are defined in the European R46 legislation, and the United States and other countries may have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an example of the types of views provided to the display from particular cameras.
[0028] Each camera arm 16A-16B may also provide a housing that encloses electronics, e.g., a controller, configured to provide various features of CMS 15. Camera arms 16A-16B may be attached, for example, to a roof mount location above the cab door (as shown) or to a door mount bracket or station.
[0029] Camera housing 16C and camera 20C are positioned near the front of commercial vehicle 10 to provide at least a partial Class V view and possibly a Class VI view (FIG. 2). Alternatively, camera 20C may be positioned on or within housing 16B. Camera 20C has a wide-angle lens (focal length less than 35 mm) and, in some cases, a "fish-eye" lens (e.g., focal length on the order of 8-10 mm) with an associated field of view (FOV).EX3 It has.
[0030] field of viewFOVE X4 A backup camera 20D may be provided that provides a field of view FOV. The backup camera 20D may be mounted, for example, at the top / centerline of the trailer, at the trailer bumper / bed level, or at a rear top corner of the trailer. Alternatively, or in addition to the rear trailer camera, a backup camera 20D may be mounted at the rear of the tractor 12 and provide a field of view FOV. EX5 A "fifth wheel camera" 20E may be provided to provide a fifth wheel camera 20E. The fifth wheel camera 20E may be mounted anywhere between the side of the fifth wheel fixture and the top / roof edge of the tractor, for example.
[0031] FIG. 3 is a schematic top view of an exemplary vehicle cab 24, and FIG. 4 is a perspective view of the vehicle cab 24. Referring now to FIGS. 3-4, electronic displays 18A-18E (which may be, for example, video displays such as LCD displays) and cameras 20A-20E are shown. The various electronic displays 18A-18E and cameras 20A-20E are part of a camera monitor system (CMS) 15 and thus function as CMS displays and CMS cameras. As used herein, a "CMS camera" 20 is a camera configured to record images of the environment surrounding the commercial vehicle 10, and a "CMS display" 18 is an electronic display (e.g., an LCD) configured to capture the feed from these cameras.
[0032] CMS 15 includes a CMS electronic control unit (ECU) 22 that includes processing circuitry that functions as a controller and supports the operation of CMS 15. CMS ECU 22 is operatively connected to memory (which may include any one or combination of volatile memory elements (e.g., random access memory (RAM) such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). The processing circuitry may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), etc.
[0033] CMS displays 18A-18B are positioned on the driver's side and passenger's side, respectively, within vehicle cab 12 on or near A-pillars 19A-19B and display Class II and Class IV views on each side of commercial vehicle 10, which provide rear-facing side views along commercial vehicle 10 captured by exterior cameras 20A-20B. As shown in FIG. 4, CMS display 18B has a first display area 21A used to depict the Class II view and a second display area 21B for depicting the Class IV view.
[0034] 3, additional displays 18C-18E are provided. Display 18C, located in vehicle cab 24 near the top center of the windshield, may be used, for example, to display Class V and Class VI views toward the front of commercial vehicle 10 or a backup camera view (from camera 20D or 20E) to the driver. Display 18D is located in the center console area of vehicle cab 24 and may be used for other purposes such as navigation, infotainment, etc. Display 18E may, for example, be part of an instrument cluster.
[0035] If desired, camera arms 16A-16B may also include conventional mirrors integrated therewith, although CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.
[0036] As described in more detail below, CMS ECU 22 is configured to perform vertical image stitching to combine the views of CMS camera 20B ("first camera") and camera 20C ("second camera"). Vertical image stitching may similarly be provided with CMS camera 20A and for additional CMS cameras located in a similar position to camera 20C, but on the same side of commercial vehicle 10 as camera 20A (i.e., the "driver's side" of the commercial vehicle in the United States).
[0037] In the example described below, CMS camera 20B has a first focal length and CMS camera 20C has a second focal length that is shorter than the first focal length, so that camera 20C has a wider angular field of view than camera 20B.
[0038] 5A is a schematic diagram of the side of commercial vehicle 10 and optical angle 52A of CMS camera 20B ("first camera"), which is formed at the intersection between optical axis 50A of camera 20B and ground plane G. Optical angle 52A is less than 90°.
[0039] 5B is a schematic diagram of the side of commercial vehicle 10 and optical angle 52B of CMS camera 20C ("second camera"). Optical angle 52B is formed between optical axis 50B of camera 20C at the intersection with ground plane G. In the example of FIG. 5B, optical angle 52A is less than 90°, but in further examples, it may be equal to 90°. Optical angle 52B of camera 20C is greater than optical angle 52A of camera 20B.
[0040] FIG. 6A is an example image 60A from the CMS camera of FIG. 5A before distortion correction or perspective transformation.
[0041] Figure 6B is an example image 62A from the CMS camera of Figure 5B, which shows more lens distortion than image 60A. This image also shows a significant amount of lens distortion, resulting in image regions 63 lacking image data. In Figures 6B, 8B, and 9, the regions lacking image data are indicated by cross-hatching.
[0042] The CMS ECU 22 performs distortion correction on one or both of the images 60A, 62A. In the following description, it is assumed that distortion correction is performed on both images 60A, 62A.
[0043] Figure 7A shows image 60A of Figure 6A after distortion correction as image 60B. The front portion 66 of the tractor 12 appears generally curved in image 60A, but is less curved in image 60B of Figure 7A.
[0044] 7B shows image 62A after distortion correction as image 62B, where the significant distortion of image 62A has been significantly reduced.
[0045] Figure 8A shows image 60C after a perspective transformation and cropping and / or zooming has been applied to image 60B of Figure 7A. As shown in image 60C, instead of sloping downward, road lines 68 now slope upward due to the altered perspective of image 60C.
[0046] Figure 8B shows an example of image 62B from Figure 7B after a perspective transformation has been applied to crop and / or zoom to produce image 62C. Again, the perspective of image 62B has been transformed in creating image 62C.
[0047] FIG. 9 is a combined image 70 corresponding to vertically stitched images 60C and 62C.
[0048] 10 is a flowchart of an example method 100 for a CMS. Method 100 is executed by CMS ECU 22. A first image 60A is acquired from a first camera, and a second image 62A is acquired from a second camera (step 102). Images 60A and 62A both show a side of commercial vehicle 10 and its surrounding environment. First camera 20B and second camera 20C have different, overlapping fields of view and have different optical axes 50A, 50B that intersect ground plane G at different optical angles 52A, 52B (see, e.g., FIGS. 5A-5B ).
[0049] Distortion correction is performed on at least one of the first image 60A and the second image 62A to obtain a first updated image set (step 104). The distortion correction of the first image 60A, if performed, mitigates image distortion caused by the lens and / or sensor of the first camera 20B. The distortion correction of the second image 62A, if performed, mitigates image distortion caused by the lens and / or sensor of the second camera 20C. In the example of FIGS. 7A-7B above, distortion correction is performed on both images 60A and 62A such that the updated image set includes images 60B and 62B. However, this is not a limiting example, and it is understood that distortion correction can be omitted for one or both of images 60A, 62A.
[0050] At least one of cropping and zooming along with a perspective transformation is performed on at least one of distortion-corrected image 60B and distortion-corrected image 62B, such that at least one of the first image and the second image is updated in the updated image set, to obtain a second updated image set (step 106). In the above example, a perspective transformation is performed on both distortion-corrected images 60B and 62B, such that the second updated image set includes images 60C and 62C. In one or more embodiments, the OpenGL "imwarp" function is used to perform the perspective transformation.
[0051] The first image 60C and / or the second image 62C are then vertically stitched (step 108) to form a combined image 64. In the combined image 64, the first image 60C is rendered on top of the second image 62C.
[0052] In one or more embodiments, the image stitching in step 108 includes the steps of: mapping a plurality of first points of a first image 60C in the updated image set (e.g., along the top of the first image 60C) to a corresponding plurality of second points of a second image 62C in the updated image set (e.g., along the bottom of the second image 62C); aligning the first image 60C in the updated image set with the second image 62C in the updated image set based on the mapping (e.g., so that the points of image 60C overlap with the points of image 62C); and blending the first image 62C in the updated image set with the second image 62C in the updated image set to obtain a combined image.
[0053] Combined image 64 is displayed on an electronic display within commercial vehicle 10, such as display 18B. In one or more embodiments, CMS 15 includes two modes: in a first mode, combined image 64 is displayed on a display (e.g., display 18B across one or both of display areas 21A-21B), and in a second mode, at least one of first image 60 and / or second image 62 (e.g., display area 60 on display 18B and display image 62 on display 18C) are displayed separately without being combined, and CMS 15 toggles between the two modes in response to receiving a toggle command from an occupant of commercial vehicle 10.
[0054] The combined image 64 provides the vehicle occupants with the ability to have a larger, continuous, and cohesive field of view without the expense or significant image distortion associated with adding another camera to provide the combined image 64 field of view.
[0055] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.
Claims
1. 1. A method for a camera monitor system (CMS), comprising: acquiring a first image from a first camera and a second image from a second camera, the first image and the second image depicting a side of a commercial vehicle and its surrounding environment, the first camera and the second camera having different overlapping fields of view and having different optical axes that intersect a ground plane at different optical angles; performing a perspective transformation on at least one of the first image and the second image to obtain an updated image set, whereby at least one of the first image and the second image is updated within the updated image set; vertically stitching the images of the updated image set together to form a combined image; A method comprising:
2. the respective different optical angles include a first optical angle of the first camera and a second optical angle of the second camera; the first optical angle is less than 90°; The method of claim 1 , wherein the second optical angle is less than or equal to 90° and greater than the first optical angle.
3. The method of claim 2 , wherein the first camera has a first focal length and the second camera has a second focal length that is shorter than the first focal length.
4. The method of claim 1 , wherein in the combined image, the first image is rendered over the second image.
5. 2. The method of claim 1, wherein the step of performing a perspective transformation is performed on both the first image and the second image, whereby both the first image and the second image are updated in the updated image set.
6. before the step of performing the perspective transformation, distortion correction for the first image from the first camera to mitigate image distortion caused by the lens or sensor of the first camera; and Distortion correction for the second image from the second camera to mitigate image distortion caused by the lens or sensor of the second camera. The method of claim 1 , comprising performing at least one of the following steps:
7. before the step of performing the perspective transformation, distortion correction for the first image from the first camera to mitigate image distortion caused by the lens or sensor of the first camera; and Distortion correction for the second image from the second camera to mitigate image distortion caused by the lens or sensor of the second camera. The method of claim 1 , comprising performing both of the steps of:
8. Before stitching vertically, The method of claim 1 , comprising performing at least one of cropping and zooming of at least one of the first image and the second image.
9. Vertically stitching the images of the updated image set together to form a combined image comprises: mapping a plurality of first points of the first image in the updated set of images to a plurality of second points of the second image in the updated set of images; registering the first image in the updated image set and the second image in the updated image set based on the mapping; and blending the first image in the updated image set with the second image in the updated image set to form the combined image. The method of claim 1 , comprising:
10. displaying the combined image on an electronic display within the commercial vehicle in a first mode; In a second mode, displaying at least one of the first image and the second image separately without combining them; and switching between the first mode and the second mode in response to receiving a toggle command from an occupant of the commercial vehicle. The method of claim 1 , comprising:
11. a first camera and a second camera each configured to record images of a side of the commercial vehicle and its surroundings, the first camera and the second camera having different overlapping fields of view and having different optical axes that intersect the ground plane at different optical angles; processing circuitry operatively connected to the memory; A camera monitor system (CMS) comprising: The processing circuitry acquiring a first image from the first camera and a second image from the second camera; performing a perspective transformation on at least one of the first image and the second image to obtain an updated image set, whereby at least one of the first image and the second image is updated within the updated image set; and vertically stitching the images of the updated image set together to form a combined image. A camera monitor system configured to:
12. the respective different optical angles include a first optical angle of the first camera and a second optical angle of the second camera; the first optical angle is less than 90°; 12. The camera monitor system of claim 11, wherein the second optical angle is less than or equal to 90 degrees and greater than the first optical angle.
13. 13. The camera monitor system of claim 12, wherein the first camera has a first focal length and the second camera has a second focal length that is shorter than the first focal length.
14. 12. The camera monitor system of claim 11, wherein in the combined image, the first image is depicted below the second image.
15. 12. The camera monitor system of claim 11, wherein the processing circuitry is configured to perform the perspective transformation on both the first image and the second image, whereby both the first image and the second image are updated in the updated image set.
16. The processing circuitry, before performing the perspective transformation, distortion correction for the first image from the first camera to mitigate image distortion caused by the lens or sensor of the first camera; and Distortion correction for the second image from the second camera to mitigate image distortion caused by the lens or sensor of the second camera.
12. The camera monitor system of claim 11, configured to perform at least one of the following:
17. The processing circuitry, before performing the perspective transformation, distortion correction for the first image from the first camera to mitigate image distortion caused by the lens or sensor of the first camera; and Distortion correction for the second image from the second camera to mitigate image distortion caused by the lens or sensor of the second camera.
12. The camera monitor system of claim 11 configured to perform both of the following:
18. The processing circuitry, prior to vertical stitching, 12. The camera monitor system of claim 11 configured to perform at least one of cropping and zooming of at least one of the first image and the second image.
19. To vertically stitch images of the updated image set, the processing circuitry: mapping a plurality of first points of the first image in the updated set of images to a plurality of second points of the second image in the updated set of images; registering the first image in the updated image set and the second image in the updated image set based on the mapping; and blending the first image in the updated image set with the second image in the updated image set to form the combined image.
12. The camera monitor system of claim 11 configured to:
20. The processing circuitry displaying the combined image on an electronic display within the commercial vehicle in a first mode; In a second mode, displaying at least one of the first image and the second image separately without combining them; and switching between the first mode and the second mode in response to receiving a toggle command from an occupant of the commercial vehicle.
12. The camera monitor system of claim 11 configured to: