Panoramic monitoring device and method for vehicle

Abstract

The invention relates to an apparatus and method for panoramic monitoring of a vehicle. According to at least one embodiment, the present disclosure provides an apparatus for panoramic monitoring of a vehicle, the apparatus comprising: a plurality of cameras; a display; and a controller configured to generate a panoramic image including a blind spot by synthesizing a plurality of images obtained from the plurality of cameras, correct the blind spot by using image data of a surrounding environment of the blind spot in the generated panoramic image, and output the corrected panoramic image to the display.

Description

[0001] Related applications

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2024-0043085, filed on March 29, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an apparatus and method for panoramic monitoring of a vehicle, and more particularly, to a technology for displaying blind spots in a panoramic image for a vehicle. Background Art

[0004] The statements herein merely provide background information related to the present disclosure and may not constitute prior art.

[0005] For the convenience of vehicle users, advanced driver assistance systems (ADAS) are being developed and applied in vehicles. In addition, the development of autonomous vehicles is actively underway.

[0006] Surround-view monitoring (SVM) systems provide drivers with a view of their surroundings while driving or parking to enhance vehicle safety and convenience. SVM systems combine images captured by multiple cameras mounted on the front, rear, left, and right sides of the vehicle to show multiple views of the surrounding environment.

[0007] At the same time, because the camera's field of view is obstructed by the camera's installation position, posture, or subject, there are blind spots that the camera cannot capture. The areas corresponding to these blind spots in the SVM image (hereinafter referred to as "blind areas") have no image information to be rendered. Typically, SVM images are generated by masking the blind areas with black or applying a gradient effect to the blind areas.

[0008] However, masking the blind area with black or applying a gradient effect has the disadvantage of making the SVM image look unnatural. Summary of the Invention

[0009] The present disclosure is directed to providing an apparatus and method for rendering a blind spot of an SVM image by utilizing an image of a surrounding environment of the blind spot.

[0010] Aspects of the present disclosure are not limited to the foregoing, and other aspects not mentioned herein will be clearly understood by those skilled in the art from the following description.

[0011] According to at least one embodiment, the present disclosure provides a device for panoramic monitoring of a vehicle, the device comprising: a plurality of cameras; a display; and a controller configured to generate a panoramic image including a blind spot by synthesizing a plurality of images obtained from the plurality of cameras, correct the blind spot by utilizing image data of the surrounding environment of the blind spot in the generated panoramic image, and output the corrected panoramic image to the display.

[0012] The controller can set a specific region of interest around the blind spot in the generated panoramic image, obtain a virtual image by cropping and adjusting the set region of interest, and replace the target sub-region in the blind spot corresponding to the region of interest with the obtained virtual image.

[0013] The controller may also perform filtering on the target sub-region for virtual image replacement.

[0014] The controller may synthesize the panoramic image and the vehicle image so that the vehicle image is arranged in the blind spot.

[0015] When the vehicle is in parking mode, the controller can adjust the brightness of the blind spot based on the brightness of the set area of ​​interest.

[0016] According to another embodiment, the present disclosure provides a method for panoramic monitoring of a vehicle, the method comprising: generating a panoramic image including a blind spot by synthesizing multiple images obtained from multiple cameras; correcting the blind spot by utilizing image data of the surrounding environment of the blind spot in the generated panoramic image; and outputting the corrected panoramic image to a display.

[0017] According to an embodiment of the present invention, the quality of the SVM image can be improved by using images around the blind spot to make the blind area in the SVM image look natural.

[0018] The effects of the present disclosure are not limited to the foregoing, and other effects not mentioned herein will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is an illustrative block diagram of a portion of an autonomous driving vehicle to which an embodiment of the present disclosure is applied.

[0020] Figure 2 is a block diagram of a panoramic monitoring device according to an embodiment of the present disclosure.

[0021] Figure 3 2 is a diagram illustrating a panoramic image and a blind spot generated by a panoramic monitoring device according to an embodiment of the present disclosure.

[0022] Figure 4FIG. 4 is a diagram illustrating a process of correcting a blind spot in a panoramic image by a panoramic monitoring device according to an embodiment of the present disclosure.

[0023] Figure 5 is a flowchart of a method executed by a panoramic monitoring device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying illustrative drawings. In the following description, the same reference numerals preferably represent the same elements, even though the elements are shown in different drawings. In addition, in the following description of some embodiments, for the purpose of clarity and brevity, detailed descriptions of related known components and functions will be omitted when they are considered to obscure the subject matter of the present disclosure.

[0025] Various ordinal numbers or letter codes, such as first, second, i), ii), a), b), etc., are prefixed only to distinguish one component from another and do not imply or indicate the substance, order, or sequence of the components. Throughout this specification, when a part "includes" or "comprises" a component, the part is intended to also include other components, rather than exclude these components, unless otherwise specifically stated. Terms such as "unit" and "module" refer to one or more units for processing at least one function or operation, which can be implemented by hardware, software, or a combination thereof.

[0026] The following description of the present disclosure in conjunction with the accompanying drawings is intended to describe exemplary embodiments of the present disclosure, and is not intended to represent only embodiments in which the technical concept of the present disclosure can be practiced.

[0027] The present disclosure relates to an apparatus and method for panoramic monitoring of a vehicle, which can improve the quality of an SVM image by rendering a blind spot in the image to look natural by using an image of the surrounding environment of the blind spot.

[0028] Figure 1 is an illustrative block diagram of a portion of an autonomous driving vehicle to which an embodiment of the present disclosure is applied.

[0029] refer to Figure 1 , the vehicle according to the embodiment of the present disclosure includes a plurality of cameras 101, 102, 103 and 104, a controller 200 and a display 300. Although Figure 1 The plurality of cameras are shown to include a first camera 101 , a second camera 102 , a third camera 103 , and a fourth camera 104 , but fewer or more than four cameras may be provided on the vehicle.

[0030] Multiple cameras 101, 102, 103, and 104 capture the vehicle's surroundings. The captured images are provided as input to controller 200 for generating a panoramic image. While the multiple cameras are shown as being mounted on the front and rear of the vehicle and on the bottom of the left and right exterior mirrors, they are not limited thereto. The multiple cameras may be ultra-wide-angle cameras with a viewing angle of 180 degrees or greater.

[0031] The first camera 101 can capture images in front of the vehicle. The first camera 101 can be, but is not limited to, disposed in a bumper or grille of the vehicle.

[0032] The second camera 102 can capture images of the left side of the vehicle. The second camera 102 can be, but is not limited to, located on the vehicle's left exterior mirror. Here, a left exterior mirror may include the mirror, various electrical components, a housing surrounding the mirror and electrical components, and the like. The left exterior mirror may be referred to as a left exterior mirror module. The second camera 102 can be located on the vehicle's left front door. A left front door may include a left exterior mirror.

[0033] The third camera 103 can capture images of the right side of the vehicle. The third camera 103 can be, but is not limited to, located on the vehicle's right exterior mirror. Here, a right exterior mirror may include the mirror, various electrical components, a housing surrounding the mirror and electrical components, and the like. The right exterior mirror may be referred to as a right exterior mirror module. The third camera 103 can be located on the vehicle's right front door. A right front door may include a right exterior mirror.

[0034] The fourth camera 104 can capture images of the rear of the vehicle. The fourth camera 104 can be, but is not limited to, located on a rear door. Here, the rear door can include a trunk and a tailgate.

[0035] The controller 200 controls the overall operation of the panoramic monitoring device. The controller 200 can generate a panoramic image by synthesizing multiple images obtained from the multiple cameras 101, 102, 103, and 104. The controller 200 can correct the blind spots included in the generated panoramic image. The controller 200 can output the panoramic image with the corrected blind spots to the display 300.

[0036] The display 300 may display an image output from the controller 200. The display 300 may display a panoramic image and / or a captured image of the surrounding environment from a viewpoint selected by the driver. The display 300 may include, but is not limited to, a digital instrument panel, an AVN display, an infotainment display, and a head-up display (HUD) of the vehicle.

[0037] Figure 2 is a block diagram of a panoramic monitoring device according to an embodiment of the present disclosure.

[0038] refer to Figure 2The panoramic monitoring device according to an embodiment of the present disclosure includes a plurality of cameras 101 , 102 , 103 and 104 , a controller 200 and a display 300 .

[0039] The components of the surround view monitoring device can transmit or receive signals or data using various communication protocols present in the vehicle. The communication protocols may include at least one of CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data Rate), LIN (Local Interconnect Network), FlexRay, and Ethernet.

[0040] A description of the plurality of cameras 101 , 102 , 103 and 104 has been given previously.

[0041] The controller 200 may be disposed within the vehicle and may be an ECU (Electronic Control Unit). The controller 200 may include a hardware processor and a memory, and the memory stores, but is not limited to, commands for executing the controller 20 and may include components for overall operation of the panoramic monitoring device.

[0042] The controller 200 generates a panoramic image including a blind spot by synthesizing multiple images obtained from the multiple cameras 101, 102, 103, and 104. The controller 200 corrects the blind spot by utilizing image data of the surrounding environment of the blind spot in the generated panoramic image. The controller 200 outputs the panoramic image with the blind spot corrected to appear natural to the display 300. To this end, the controller 200 may include an SVM image generation unit 210 and an SVM image correction unit 220.

[0043] The SVM image generation unit 210 generates a panoramic image including blind spots by synthesizing multiple images obtained from the multiple cameras 101, 102, 103, and 104. As used herein, a blind spot refers to an area in a panoramic image where no image data is rendered, corresponding to a blind spot. As used herein, a blind spot refers to an area that cannot be captured by the multiple cameras 101, 102, 103, and 104 because the camera's field of view is obstructed due to the camera's installation position, posture, or subject. For example, a blind spot may be a specific area on the road where a vehicle is located.

[0044] The SVM image generating unit 210 may generate a panoramic image by masking the blind area with black or applying a gradient effect to the blind area.

[0045] Will refer to Figure 3 The panoramic image and the blind area generated by the SVM image generation unit 210 are described.

[0046] refer to Figure 3The SVM image generation unit 210 can generate a panoramic image 300i including the blind spot by synthesizing a first image obtained from the first camera 101, a second image obtained from the second camera 102, a third image obtained from the third camera 103, and a fourth image obtained from the fourth camera 104. Here, the first image may be an image in front of the vehicle. The second image may be an image on the left side of the vehicle. The third image may be an image on the right side of the vehicle. The fourth image may be an image on the rear side of the vehicle.

[0047] Meanwhile, the panoramic image 300i may include images of multiple views such as a top view, a side view, a front view, a rear view, etc. The panoramic image 300i may be implemented as a 2D image or a 3D image.

[0048] Panoramic image 300i includes regions corresponding to the multiple images obtained from multiple cameras 101, 102, 103, and 104, respectively. For example, panoramic image 300i may include a first region 301i, a second region 302i, a third region 303i, and a fourth region 304i. Here, first region 301i may be the region corresponding to the first image. Second region 302i may be the region corresponding to the second image. Third region 303i may be the region corresponding to the third image. Fourth region 304i may be the region corresponding to the fourth image.

[0049] The panoramic image 300i includes a blind area 310i corresponding to a blind spot in the panoramic image where no image data is to be rendered. The blind area 310i may be masked in black or a gradient effect may be applied to the blind area 310i.

[0050] The panoramic image 300i may include a vehicle image 320i corresponding to a vehicle. The SVM image correction unit 220, which will be described below, may render the vehicle image 320i included in the panoramic image 300i.

[0051] Reference again Figure 2 , the SVM image correction unit 220 corrects the blind area 310i by using image data of the surrounding environment of the blind area included in the generated panoramic image.

[0052] The SVM image correction unit 220 sets at least one region of interest (ROI) around the blind spot in the generated panoramic image. The SVM image correction unit 220 obtains a virtual image by cropping and resizing the set ROI to fit the size of the target subregion in the blind spot. The SVM image correction unit 220 replaces the target subregion in the blind spot with the virtual image.

[0053] Also refer to Figure 4 , a process of correcting the blind area of ​​the panoramic image by the SVM image correction unit 220 will be described.

[0054] For example, the SVM image correction unit 220 may set a first region of interest 401 in front of the blind spot 310i, crop and resize the first region of interest 401 to obtain a first virtual image, and replace the first target sub-region 411 in the blind spot 310i with the first virtual image. Here, the first region of interest 401 may be positioned adjacent to the blind spot 310i. The first region of interest 401 may be sized to have the same horizontal length (C) and a predetermined vertical length (A) (e.g., 20 pixels) as the blind spot 310i. The first target sub-region 411 may be sized to have the same horizontal length (C) and a predetermined vertical length (E) (e.g., 1 / 8 of the vertical length of the blind spot 310i). The first virtual image and the first target sub-region 411 are the same size.

[0055] For example, the SVM image correction unit 220 may set a second region of interest 402 to the left of the blind spot 310i, crop and resize the second region of interest 402 to obtain a second virtual image, and replace the second target sub-region 412 in the blind spot 310i with the second virtual image. Here, the second region of interest 402 may be set adjacent to the blind spot 310i. The size of the second region of interest 402 may be set to have a predetermined horizontal length (B) (e.g., 20 pixels) and the same vertical length (D) as the blind spot 310i. Here, the size of the second target sub-region 412 may be set to have a predetermined horizontal length (F) (e.g., ¼ the horizontal length of the blind spot 310i) and a vertical length obtained by subtracting the vertical lengths of the first target sub-region 411 and the fourth target sub-region 414 from the vertical length of the blind spot 310i. Here, the second virtual image is the same size as the second target sub-region 412.

[0056] For example, the SVM image correction unit 220 may set a third region of interest 403 to the right of the blind spot 310i, crop and resize the third region of interest 403 to obtain a third virtual image, and replace the third target sub-region 413 in the blind spot 310i with the third virtual image. Here, the third region of interest 403 may be set adjacent to the blind spot 310i. The size of the third region of interest 403 may be set to have a predetermined horizontal length (B) (e.g., 20 pixels) and the same vertical length (D) as the blind spot 310i. The size of the third target sub-region 413 may be set to have a predetermined horizontal length (F) (e.g., ¼ the horizontal length of the blind spot 310i) and a vertical length obtained by subtracting the vertical lengths of the first target sub-region 411 and the fourth target sub-region 414 from the vertical length of the blind spot 310i. Here, the third virtual image has the same size as the third target sub-region 413.

[0057] For example, the SVM image correction unit 220 may set a fourth region of interest 404 behind the blind spot 310i, crop and resize the fourth region of interest 404 to obtain a fourth virtual image, and replace the fourth target sub-region 414 in the blind spot 310i with the fourth virtual image. Here, the fourth region of interest 404 may be set adjacent to the blind spot 310i. The fourth region of interest 404 may be sized to have the same horizontal length (C) and a predetermined vertical length (A) (e.g., 20 pixels) as the blind spot 310i. Here, the fourth target sub-region 414 may be sized to have the same horizontal length (C) and a predetermined vertical length (E) (e.g., 1 / 8 of the vertical length of the blind spot 310i). Here, the fourth virtual image and the fourth target sub-region 414 have the same size.

[0058] The SVM image correction unit 220 may perform filtering, such as blurring, on each target sub-region replaced with the corresponding virtual image. Filters as used herein may include, but are not limited to, Gaussian filters.

[0059] The SVM image correction unit 220 can synthesize the corrected panoramic image and the vehicle image 320i so that the vehicle image 320i is positioned within the blind spot 310i. Here, the vehicle image 320i can be a specific 2D or 3D image, depending on the specifications of the vehicle, etc. The SVM image correction unit 220 can synthesize the vehicle image 320i and the corrected panoramic image overlaid on a shadow image to reduce the sense of unevenness. Here, the shadow image can be a rounded rectangular image of the same size as the vehicle image 320i, which is blurred at the edges. The shadow image can be applied to minimize the sense of unevenness between the corrected blind spot 310i and the vehicle image 320i.

[0060] When the vehicle is in parking mode, the SVM image correction unit 220 can adjust the brightness of the blind spot based on the brightness of the set region of interest to improve the visibility of the panoramic image. For example, if the brightness of at least two of the first region of interest 401, the second region of interest 402, the third region of interest 403, and the fourth region of interest 404 remains below a first threshold for a first duration, the SVM image correction unit 220 can increase the brightness of the blind spot 310i. For example, if the brightness of at least two of the first region of interest 401, the second region of interest 402, the third region of interest 403, and the fourth region of interest 404 remains above a second threshold for a second duration, the SVM image correction unit 220 can reduce the brightness of the blind spot 310i. Here, the brightness of the region of interest can be the RGB average value of the pixels included in the region of interest.

[0061] The controller 200 may output the panoramic image with the corrected blind spot to the display 300 .

[0062] A description of the display 300 has been given previously.

[0063] Figure 5 is a flowchart of a method executed by a panoramic monitoring device according to an embodiment of the present disclosure.

[0064] refer to Figure 5 The panoramic monitoring device generates a panoramic image including a blind spot by synthesizing multiple images obtained from multiple cameras 101, 102, 103 and 104 (S510). As used herein, a blind spot refers to an area in a panoramic image where no image data is to be rendered, corresponding to a blind spot.

[0065] The panoramic monitoring device may generate a panoramic image by masking the blind area with black or applying a gradient effect to the blind area.

[0066] The panoramic monitoring device sets at least one region of interest around the blind area in the generated panoramic image ( S520 ), in order to correct the corresponding target sub-area in the blind area by using the image data of the at least one region of interest.

[0067] The panoramic monitoring apparatus corrects a blind spot of the panoramic image using the image data of the set region of interest ( S530 ).

[0068] Specifically, the panoramic monitoring device obtains a virtual image by cropping a set region of interest and adjusting its size to fit the size of the target sub-region in the blind spot. The panoramic monitoring device replaces the target sub-region in the blind spot with the virtual image.

[0069] The panoramic monitoring device may perform filtering, such as blurring, on each target sub-region replaced with the corresponding virtual image.

[0070] The panoramic monitoring device may synthesize the corrected panoramic image and the vehicle image 320 i so that the vehicle image 320 i is arranged in the blind spot 310 i .

[0071] When the vehicle is in parking mode, the panoramic monitoring device can adjust the brightness of the blind spot based on the brightness of the set area of ​​interest to improve the visibility of the panoramic image according to the surrounding lighting.

[0072] The panoramic monitoring device may output the panoramic image with the corrected blind spot to the display 300 ( S540 ).

[0073] Each component of the device or method according to the embodiment of the present disclosure can be implemented by hardware, software or a combination of hardware and software. In addition, the function of each component can be implemented by software, and a microprocessor can be implemented to execute the function of the software corresponding to each component.

[0074] Various implementations of the systems and techniques described herein can be implemented using digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which can be a special-purpose processor or a general-purpose processor), at least one input device, and at least one output device, with the at least one programmable processor coupled to receive data and instructions from a storage system and transmit data and instructions to the storage system. A computer program (also referred to as a program, software, software application, or code) includes instructions for the programmable processor and is stored on a "computer-readable recording medium."

[0075] Computer-readable storage media include all types of storage devices that store data readable by a computer system. Computer-readable storage media include non-volatile or non-transitory media (such as ROMs, CD-ROMs, magnetic tapes, floppy disks, memory cards, hard disks, magneto-optical disks, and storage devices), as well as transitory media (such as data transmission media). Furthermore, computer-readable storage media can be distributed among computer systems connected via a network, allowing computer-readable code to be stored and executed in a distributed manner.

[0076] In the flowcharts in this specification, each process is described as occurring in a certain order. However, this is merely an example of the technology of the embodiments of the present disclosure. In other words, a person skilled in the art of the embodiments of the present disclosure can make various modifications and changes by changing the order described in the flowcharts in this specification, or by performing one or more processes simultaneously within the basic features of the embodiments of the present disclosure. Therefore, the flowcharts in this specification are not limited to a chronological order.

[0077] Although exemplary embodiments of the present disclosure have been described for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions may be made without departing from the concept and scope of the claimed invention. Therefore, exemplary embodiments of the present disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of the embodiments of the present disclosure is not limited by the figures. Therefore, those of ordinary skill in the art will understand that the scope of the claimed invention is not limited by the embodiments explicitly described above, but by the claims and their equivalents.

Claims

1. A device for panoramic monitoring of a vehicle, comprising: Multiple cameras; monitor; as well as A controller is configured to generate a panoramic image including a blind spot by synthesizing a plurality of images obtained from the plurality of cameras, correct the blind spot by utilizing image data of a surrounding environment of the blind spot in the generated panoramic image, and output the corrected panoramic image to the display.

2. The device according to claim 1, wherein The controller generates the panoramic image by masking the blind area with black or applying a gradient effect to the blind area.

3. The device according to claim 1, wherein The controller sets a specific region of interest around the blind area in the generated panoramic image, obtains a virtual image by cropping and adjusting the set region of interest, and replaces the target sub-area in the blind area corresponding to the region of interest with the obtained virtual image.

4. The device according to claim 3, wherein The size of the virtual image is adjusted to fit the size of the target sub-region.

5. The apparatus according to claim 3, wherein The controller performs filtering on the target sub-region of the virtual image replacement.

6. The apparatus according to claim 3, wherein The controller synthesizes the panoramic image and the vehicle image so that the vehicle image is arranged in the blind spot.

7. The apparatus according to claim 3, wherein When the vehicle is in a parking mode, the controller adjusts the brightness of the blind spot based on the brightness of the set area of ​​interest.

8. A method for panoramic monitoring of a vehicle, the method comprising: Generating a panoramic image including blind spots by synthesizing multiple images obtained from multiple cameras; Correcting the blind spot by utilizing image data of the surrounding environment of the blind spot in the generated panoramic image; as well as The corrected panoramic image is output to a display.

9. The method according to claim 8, wherein The generating includes generating the panoramic image by masking the blind area with black or applying a gradient effect to the blind area.

10. The method according to claim 9, wherein: The correction includes: Setting a specific region of interest around the blind area in the generated panoramic image; Obtaining a virtual image by cropping and resizing a set region of interest; and The obtained virtual image is used to replace the target sub-region in the blind area corresponding to the region of interest.

11. The method according to claim 10, wherein: The size of the virtual image is adjusted to fit the size of the target sub-region.

12. The method according to claim 10, wherein: The correcting further includes performing filtering on the target sub-region replaced by the virtual image.

13. The method according to claim 10, wherein: The correction further includes synthesizing the panoramic image and the vehicle image so that the vehicle image is arranged in the blind spot.

14. The method according to claim 10, wherein: The correction further includes adjusting the brightness of the blind spot based on the brightness of the set area of ​​interest when the vehicle is in a parking mode.

Images