Device and method for generating a panoramic image

Abstract

The present disclosure provides a device and a method for generating a panoramic image. The device includes: a region of interest (ROI) selection block configured to receive a plurality of images and output ROI images by selecting the ROI of each of the plurality of images; a high dynamic range (HDR) processing block configured to perform HDR processing on the ROI images; and a panoramic image generation block configured to generate a panoramic image by stitching the plurality of images using the matching coordinates of the ROI images on which the HDR processing has been performed. The HDR processing block performs HDR processing on the panoramic image.

Description

Technical Field

[0001] The present disclosure generally relates to an apparatus and method for generating a panoramic image, and more particularly, to an apparatus and method for generating a panoramic image by matching a plurality of images to one image. Background Art

[0002] Since the size of an image included in a photograph is limited when using an ordinary image capturing device, there are cases where a wide image should be generated by capturing each of a number of still images and combining the captured still images. This capturing technique is called panoramic image capturing.

[0003] Due to the increase in the size of displays and the provision of a display environment in which a number of individual displays are connected to each other, panoramic content has recently been widely used. Panoramic content refers to high-definition content having a wide viewing angle, which is used to maximize the feeling of authenticity provided to a user. Panoramic content is used in several parts of real life such as aerial panoramas that enable a user to glance at the ground from the sky at a glance, street panoramas that provide real-time-based information over the ground space of an entire country by applying GPS coordinates to a digital map, and point panoramas that show information on specific locations such as tourist attractions, accommodations, and restaurants in high resolution. In addition, panoramic content is widely used in various industrial fields such as advertising, education, military, performances, and sightseeing. Summary of the Invention

[0004] Embodiments of the present disclosure provide an apparatus and method for generating a panoramic image, in which matching is performed by applying high dynamic range (HDR) to a matching plane of a plurality of images and then selecting points having high correlation as matching points.

[0005] According to an aspect of the present disclosure, there is provided an apparatus for generating a panoramic image, the apparatus including: a region of interest (ROI) selection block configured to receive a plurality of images and output ROI images by selecting an ROI of each of the plurality of images; a high dynamic range (HDR) processing block configured to perform HDR processing on the ROI images; and a panoramic image generation block configured to generate a panoramic image by stitching a plurality of images using matching coordinates of the ROI images on which the HDR processing has been performed, wherein the HDR processing block generates a final panoramic image by performing HDR processing on the panoramic image.

[0006] According to another aspect of the present disclosure, a method for generating a panoramic image is provided, the method including the steps of: respectively setting an overlapping region of a first image and a second image as a first region of interest (ROI) image and a second ROI image; performing high dynamic range (HDR) processing on the first ROI image and the second ROI image; dividing the first ROI image on which the HDR processing has been performed into a plurality of image regions, and searching for matching points by matching a main image region among the plurality of image regions with the second ROI image on which the HDR processing has been performed; and generating a panoramic image by stitching the first image and the second image based on the matching points.

[0007] According to another aspect of the present disclosure, an operation method of an image processor is provided, the operation method including the steps of: performing high dynamic range (HDR) processing on corresponding first and second overlapping regions; and generating a panoramic image by stitching a first image and a second image with reference to points where a selected region within the first overlapping region subjected to HDR processing matches the second overlapping region subjected to HDR processing. The first overlapping region and the second overlapping region may be included in corresponding first and second images and overlap with each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings; however, the embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art.

[0009] In the drawings, for clarity of illustration, dimensions may be exaggerated. It should be understood that when an element is referred to as being "between" two elements, it may be the only element between the two elements, or there may be one or more intervening elements. Like reference numerals always denote like elements.

[0010] Figure 1 is a block diagram illustrating an apparatus for generating a panoramic image according to an embodiment of the present disclosure.

[0011] Figure 2 is an illustration of an embodiment of the present disclosure Figure 1 block diagram of the panoramic image generation block shown.

[0012] Figure 3 is a flowchart illustrating a method for generating a panoramic image according to an embodiment of the present disclosure.

[0013] Figure 4 is an illustration of an embodiment of the present disclosure Figure 3 diagram of operation S310 shown.

[0014] Figure 5 It is a diagram illustrating Figure 3 operation S320 according to an embodiment of the present disclosure.

[0015] Figure 6 It is a diagram illustrating Figure 3 operation S330 according to an embodiment of the present disclosure.

[0016] Figure 7 It is a diagram illustrating Figure 3 operations S340 and S350 according to an embodiment of the present disclosure.

[0017] Figure 8 It is a diagram illustrating Figure 3 operation S360 according to an embodiment of the present disclosure.

[0018] Figure 9 It is a diagram illustrating Figure 3 operation S370 according to an embodiment of the present disclosure. Detailed Embodiments

[0019] The specific structural or functional descriptions disclosed herein are merely illustrative for the purpose of describing embodiments according to the concepts of the present disclosure. Embodiments according to the concepts of the present disclosure may be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0020] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the technical spirit of the present disclosure.

[0021] Figure 1 It is a block diagram of a device for generating a panoramic image according to an embodiment of the present disclosure.

[0022] Referring to Figure 1 , device 100 may include a region of interest (ROI) selection block 110, a high dynamic range (HDR) processing block 120, and a panoramic image generation block 130. The ROI selection block 110, the HDR processing block 120, and the panoramic image generation block 130 may include at least one of circuits, systems, software, firmware, and devices required for their respective operations and functions.

[0023] The ROI selection block 110 receives multiple images from the outside, selects the ROI of each of the multiple received images, and outputs the ROI images. For example, the ROI selection block 110 receives the first image img1, selects the ROI in the first image img1, and outputs the image corresponding to the selected ROI as the first ROI image ROI1. In addition, the ROI selection block 100 receives the second image img2, selects the ROI in the second image img2, and outputs the image corresponding to the selected ROI as the second ROI image ROI2. The ROI selection block 110 selects the overlapping image portions in the first image img1 and the second image img2 as the ROIs of the first image img1 and the second image img2, respectively, and outputs the selected ROIs as the first ROI image ROI1 and the second ROI image ROI2.

[0024] The HDR processing block 120 receives the first ROI image ROI1 and the second ROI image ROI2 from the ROI selection block 110, and generates the first HDR image H_ROI1 and the second HDR image H_ROI2 by performing HDR processing on each of the first ROI image ROI1 and the second ROI image ROI2.

[0025] To provide various illuminances in the image that are more similar to the capacity of a human eye, HDR processing is used to increase the contrast (the difference between the brightest part and the darkest part) of the image or picture by expanding the brightness range of the image or picture from the brightest point to the darkest point. HDR processing refers to a technique that allows the bright points in the image or picture to be brighter and the dark points to be darker.

[0026] In addition, the HDR processing block 120 receives the panoramic image Panorama_img from the panoramic image generation block 130, and performs HDR processing on the received panoramic image Panorama_img. The HDR processing block 120 outputs the panoramic image F_Panorama_img on which HDR processing has been performed.

[0027] The panoramic image generation block 130 receives the first HDR image H_ROI1 and the second HDR image H_ROI2 from the HDR processing block 120, searches for the matching coordinates for stitching the first image img1 and the second image img2 based on the received first HDR image H_ROI1 and the received second HDR image H_ROI2, and generates the panoramic image Panorama_img by stitching the first image img1 and the second image img2 using the searched matching coordinates. Stitching the first image img1 and the second image img2 may indicate combining the first image img1 and the second image img2 with overlapping fields of view to produce a panoramic image or a high-resolution image.

[0028] As described above, the device 100 according to an embodiment of the present disclosure can easily obtain matching points for generating a panoramic image by selecting an area where a plurality of images overlap each other as the ROI of the corresponding image and performing first HDR processing on the selected ROI of the image, so that the matching success rate can be improved. In addition, after generating a panoramic image by stitching a plurality of images, second HDR processing is performed on the generated panoramic image, so that the brightness difference between the matched parts can be minimized.

[0029] Figure 2 Illustrative of an embodiment according to the present disclosure Figure 1 is a block diagram of the panoramic image generation block 130 shown.

[0030] Referring to Figure 2 , the panoramic image generation block 130 may include an image decomposition block 131, a main image area selection block 132, an image matching block 133, a matching coordinate search block 134, and an image stitching block 135. The image decomposition block 131, the main image area selection block 132, the image matching block 133, the matching coordinate search block 134, and the image stitching block 135 may include at least one of circuits, systems, software, firmware, and devices required for their respective operations and functions.

[0031] The image decomposition block 131 receives the first HDR image H_ROI1 and the second HDR image H_ROI2 from the Figure 1 shown HDR processing block 120, and decomposes each of the received first HDR image H_ROI1 and the received second HDR image H_ROI2 into a low-frequency image and a high-frequency image. For example, the image decomposition block 131 may generate a first high-frequency image ROI1_HF by extracting an image in a high-frequency domain higher than a reference frequency from the first HDR image H_ROI1, and generate a second high-frequency image ROI2_HF by extracting an image in a high-frequency domain higher than the reference frequency from the second HDR image H_ROI2. For example, the image decomposition block 131 may generate a first high-frequency image ROI1_HF by performing a high-pass filter (HPF) process on the first HDR image H_ROI1, and generate a second high-frequency image ROI2_HF by performing an HPF process on the second HDR image H_ROI2.

[0032] The main image region selection block 132 receives the first high-frequency image ROI1_HF from the image decomposition block 131, divides the first high-frequency image ROI1_HF into multiple image regions, and then selects one image region including the largest amount of detailed image information among the multiple image regions as the main image region. For example, the main image selection block 132 may divide the first high-frequency image ROI1_HF into multiple image regions of the same size, and select one image region including the largest amount of detailed image information among the multiple image regions as the main image region. The amount of detailed image information of each image region can be obtained by using methods such as standard deviation (STD), mean absolute difference (MAD), or sum of absolute differences (SAD). The selected main image region of the first high-frequency image ROI1_HF may be output as the main image ROI1_MB. The main image region selection block 132 outputs the selected main image region of the first high-frequency image ROI1_HF as the main image ROI1_MB.

[0033] The image matching block 133 selects matching points by matching the main image ROI1_MB of the first high-frequency image ROI1_HF received from the main image region selection block 132 with the second high-frequency image ROI2_HF received from the image decomposition block 131. For example, the image matching block 133 selects matching points by matching the main image ROI1_MB with the second high-frequency image ROI2_HF using template matching technology.

[0034] The matching coordinate search block 134 searches for each of the matching coordinates ROI1_MP of the first ROI image ROI1 and the matching coordinates ROI2_MP of the second ROI image ROI2 based on the matching points selected by the image matching block 133.

[0035] The image stitching block 135 generates a panoramic image Panorama_img by stitching the first image img1 and the second image img2 using the matching coordinates ROI1_MP of the first ROI image ROI1 and the matching coordinates ROI2_MP of the second ROI image ROI2. For example, the image stitching block 135 may perform a blending process of combining the data of the first image img1 and the second image img2 into an overlapping image by forming a linear combination of the image data. The blending process is used to change the space between the images stitched into a single image to make it softer so that the user can match the images. That is, the blending process may be a technique for eliminating the feeling of differences in the boundaries or seams in the overlapping region when two images are combined to overlap each other.

[0036] Figure 3 is a flowchart illustrating a method for generating a panoramic image according to an embodiment of the present disclosure.

[0037] Figure 4 It is a diagram illustrating operation S310 according to an embodiment of the present disclosure. Figure 3

[0038] Figure 5 It is a diagram illustrating operation S320 according to an embodiment of the present disclosure. Figure 3

[0039] Figure 6 It is a diagram illustrating operation S330 according to an embodiment of the present disclosure. Figure 3

[0040] Figure 7 It is a diagram illustrating operations S340 and S350 according to an embodiment of the present disclosure. Figure 3

[0041] Figure 8 It is a diagram illustrating operation S360 according to an embodiment of the present disclosure. Figure 3

[0042] Figure 9 It is a diagram illustrating operation S370 according to an embodiment of the present disclosure. Figure 3

[0043] A method according to an embodiment of the present disclosure will be described with reference to Figures 1 to 9

[0044] In an embodiment of the present disclosure, a method for generating a panoramic image by matching a first image img1 and a second image img2 will be described.

[0045] In operation S310, an overlapping region of each of the plurality of images is selected as an ROI. For example, the ROI selection block 110 receives the first image img1 and the second image img2 from the outside, respectively selects the overlapping image portions in the first image img1 and the second image img2 as the ROIs of the first image img1 and the second image img2, and outputs the selected ROIs as a first ROI image ROI1 and a second ROI image ROI2.

[0046] In operation S320, HDR is applied to the ROI of each of the plurality of images. For example, the HDR processing block 120 receives the first ROI image ROI1 and the second ROI image ROI2 from the ROI selection block 110, and generates a first HDR image H_ROI1 and a second HDR image H_ROI2 by performing HDR processing on each of the first ROI image ROI1 and the second ROI image ROI2. The HDR processing block 120 can perform HDR processing on each of the first ROI image ROI1 and the second ROI image ROI2 by using a tone mapping technique. Since the HDR processing of the first ROI image ROI1 and the second ROI image ROI2 is for matching, a tone mapping curve (TMC) (i.e., a curve that changes the brightness of a plurality of pixels) can be manually set. In addition, by using the contrast-limited adaptive histogram equalization (CLAHE) method, which is a brightness enhancement technique for a local area based on multiple layers, the HDR processing block 120 can perform HDR processing on the first ROI image ROI1 and the second ROI image ROI2.

[0047] In operation S330, each ROI image is decomposed into a low-frequency image and a high-frequency image. For example, the image decomposition block 131 receives the first HDR image H_ROI1 and the second HDR image H_ROI2 from the HDR processing block 120, and decomposes each of the received first HDR image H_ROI1 and the received second HDR image H_ROI2 into a low-frequency image and a high-frequency image. For example, the image decomposition block 131 can generate a first high-frequency image ROI1_HF by extracting an image in a high-frequency domain higher than a reference frequency from the first HDR image H_ROI1, and generate a second high-frequency image ROI2_HF by extracting an image in a high-frequency domain higher than the reference frequency from the second HDR image H_ROI2. For example, the image decomposition block 131 can generate a first high-frequency image ROI1_HF by performing a high-pass filter (HPF) process on the first HDR image H_ROI1, and generate a second high-frequency image ROI2_HF by performing an HPF process on the second HDR image H_ROI2.

[0048] In operation S340, the high-frequency image of the first HDR image H_ROI1 is divided into a plurality of image regions. For example, the main image region selection block 132 receives the first high-frequency image ROI1_HF from the image decomposition block 131, and divides the first high-frequency image ROI1_HF into a plurality of image regions.

[0049] In operation S350, a main image region is selected among the plurality of image regions of the first high-frequency image ROI1_HF, and a matching point is selected by matching the main image region with the high-frequency image ROI2_HF of the second HDR image H_ROI2.

[0050] For example, the main image area selection block 132 selects, as the main image area, one image area among a plurality of image areas that includes the largest amount of detailed image information. For example, the main image selection block 132 may divide the first high-frequency image ROI1_HF into a plurality of image areas of the same size, and select, as the main image area, one image area among the plurality of image areas that includes the largest amount of detailed image information. The amount of detailed image information for each image area can be obtained by using methods such as standard deviation (STD), mean absolute difference (MAD), or sum of absolute differences (SAD). The selected main image area of the first high-frequency image ROI1_HF can be output as the main image ROI1_MB.

[0051] The image matching block 133 selects matching points by matching the main image ROI1_MB of the first high-frequency image ROI1_HF received from the main image area selection block 132 with the second high-frequency image ROI2_HF received from the image decomposition block 131. For example, the image matching block 133 uses template matching technology to match the main image ROI1_MB with the second high-frequency image ROI2_HF to select matching points. The area with the highest correlation between the main image ROI1_MB and the second high-frequency image ROI2_HF in template matching can be selected as the matching points.

[0052] In operation S360, the matching coordinates of each of the plurality of images are searched based on the matching points, and the plurality of images are stitched together to generate the panoramic image Panorama_img.

[0053] For example, the matching coordinate search block 134 obtains the relative coordinate values of each image based on the matching points selected by the image matching block 133, and thus, searches for each of the matching coordinates ROI1_MP of the first ROI image ROI1 and the matching coordinates ROI2_MP of the second ROI image.

[0054] The image stitching block 135 stitches the first image img1 and the second image img2 by using the matching coordinates ROI1_MP of the first ROI image ROI1 and the matching coordinates ROI2_MP of the second ROI image to generate the panoramic image Panorama_img. For example, the image stitching block 135 can perform a blending process of combining the data of the first image img1 and the second image img2 into an overlapping image by forming a linear combination of the image data.

[0055] In operation S370, HDR is applied to the panoramic image Panorama_img generated at operation S360. For example, the HDR processing block 120 receives the panoramic image Panorama_img from the panoramic image generation block 130 and performs HDR processing on the received panoramic image Panorama_img. The HDR processing block 120 outputs the panoramic image F_Panorama_img on which the HDR processing has been performed.

[0056] According to an embodiment of the present disclosure, a panoramic image can be generated by using multiple images to which HDR is applied, the matching success rate in the stitching operation of the multiple images can be improved, and the brightness difference between overlapping portions can be minimized.

[0057] In the above embodiments, all operations can be selectively performed or skipped. Additionally, the operations in each embodiment may not always be executed in the given order successively and may be executed randomly. Furthermore, the embodiments disclosed in this specification and the drawings are intended to assist those of ordinary skill in the art in more clearly understanding the present disclosure and are not intended to limit the scope of the present disclosure. In other words, those of ordinary skill in the art to which the present disclosure pertains will be able to easily understand that various modifications are possible based on the technical scope of the present disclosure and the appended claims. Additionally, the embodiments can be combined to form additional embodiments.

[0058] Cross - reference to related applications

[0059] This application claims the priority of Korean Patent Application No. 10 - 2021 - 0033379, filed on March 15, 2021, the entire disclosure of which is incorporated herein by reference.

Claims

1. An apparatus for generating a panoramic image, the apparatus comprising: a region of interest (ROI) selection block that receives a plurality of images and outputs ROI images by selecting ROIs of each of the plurality of images; a high dynamic range (HDR) processing block that performs HDR processing on the ROI images; and a panoramic image generation block that generates a panoramic image by stitching the plurality of images using matching coordinates of the ROI images on which the HDR processing has been performed, wherein the HDR processing block also performs the HDR processing on the panoramic image, wherein the panoramic image generation block also searches for the matching coordinates of the ROI images on which the HDR processing has been performed, wherein the panoramic image generation block searches for the matching coordinates by: dividing a high-frequency image of one of the ROI images among the ROI images on which the HDR processing has been performed into a plurality of image regions, selecting a main image region including the largest amount of detailed image information among the plurality of image regions, and mapping the selected main image region to high-frequency images of each of the remaining ROI images among the ROI images on which the HDR processing has been performed.

2. The device according to claim 1, wherein, The ROI selection block selects overlapping image portions among the plurality of images as the ROI.

3. The device according to claim 1, wherein The HDR processing block performs the HDR processing on the ROI images by using a tone mapping technique or a contrast limited adaptive histogram equalization (CLAHE) method.

4. The device according to claim 1, wherein The panoramic image generation block includes: an image decomposition block that decomposes high-frequency images of each of the ROI images on which the HDR processing has been performed; a main image region selection block that divides the high-frequency image of one of the ROI images on which the HDR processing has been performed into a plurality of image regions and selects one image region including the largest amount of detailed image information among the plurality of image regions as the main image region; an image matching block that selects matching points by matching the image of the main image region with high-frequency images of each of the remaining ROI images on which the HDR processing has been performed; a matching coordinate search block that searches for the matching coordinates of the ROI images on which the HDR processing has been performed based on the matching points; and an image stitching block that generates the panoramic image by stitching the plurality of images using the matching coordinates of the ROI images on which the HDR processing has been performed.

5. The device according to claim 4, wherein, The image decomposition block decomposes the high-frequency images of each of the ROI images on which the HDR processing has been performed by extracting images in a high-frequency domain having a frequency higher than a reference frequency from each of the ROI images on which the HDR processing has been performed.

6. The apparatus according to claim 4, Among them, wherein the main image region selection block divides the high-frequency image of the one ROI image into a plurality of image regions having the same size.

7. The device according to claim 6, wherein The main image area selection block also obtains detailed image information amounts of each of the plurality of image areas by using a scheme of standard deviation STD, mean absolute difference MAD, or sum of absolute differences SAD.

8. The device according to claim 4, wherein The image matching block matches the image of the main image area with the high-frequency image of each ROI image among the remaining ROI images that have undergone the HDR processing by using a template matching technique.

9. The device according to claim 4, wherein The image stitching block stitches the plurality of images by performing a blending process that combines data of the plurality of images into an overlapping image by forming a linear combination of the image data.

10. A method for generating a panoramic image, the method comprising the following steps: Respectively set an overlapping area of a first image and a second image as a first region of interest ROI image and a second ROI image; Perform high dynamic range HDR processing on the first ROI image and the second ROI image; Divide the high-frequency image of the first ROI image that has undergone the HDR processing into a plurality of image areas; Search for matching points by matching a main image area including the largest detailed image information amount among the plurality of image areas with the high-frequency image of the second ROI image that has undergone the HDR processing; Search for matching coordinates of each of the first ROI image and the second ROI image based on the matching points; And Generate a panoramic image by stitching the first image and the second image based on the matching coordinates.

11. The method according to claim 10, the method further comprising the steps of: Perform the HDR processing on the panoramic image.

12. The method according to claim 10, wherein, Perform the HDR processing on the first ROI image and the second ROI image according to a tone mapping technique or a contrast-limited adaptive histogram equalization CLAHE method.

13. The method according to claim 10, wherein Obtain detailed image information amounts of each of the plurality of image areas by using a scheme of standard deviation STD, mean absolute difference MAD, or sum of absolute differences SAD.

14. The method according to claim 10, Among them, The dividing step includes the following steps: Extract high-frequency images of the first ROI image and the second ROI image that have undergone the HDR processing; and Divide the high-frequency image of the first ROI image that has undergone the HDR processing into the plurality of image areas.

15. An operation method of an image processor, the operation method comprising the following steps: Perform high dynamic range HDR processing on corresponding first and second overlapping areas; And Generate a panoramic image by stitching a first image and a second image by referring to points where a selected area within the first overlapping area that has undergone HDR processing matches the second overlapping area that has undergone HDR processing, wherein the first overlapping area and the second overlapping area are included in the corresponding first image and second image and overlap with each other, wherein the step of stitching the first image and the second image by referring to the points includes: Dividing the high-frequency image of the first overlapping area that has undergone the HDR processing into a plurality of image areas, Selecting a main image area including the largest detailed image information amount among the plurality of image areas, and The point is selected by mapping a selected primary image region to a high-frequency image of the second overlapping region that has been HDR processed.

Images