Method and device for processing image

Abstract

A method and apparatus for processing an image are disclosed. In a method of processing an image that is a video image including a plurality of frame images, a first high dynamic range (HDR) image is generated by merging a first frame image and a second frame image following the first frame image. The first frame image and the second frame image are included in the plurality of frame images. The first frame image has first exposure time. The second frame image has a second exposure time different from the first exposure time. A second HDR image is generated by combining the second frame image and a third frame image following the second frame image. The third frame image is included in the plurality of frame images and has the first exposure time.

Description

[0001] This application is a divisional application of a patent application with an application date of November 5, 2019, application number 201911073478.6, and titled “Method and device for processing images”. Technical Field

[0002] Example embodiments generally relate to image acquisition and image processing. For example, at least some example embodiments relate to an image processing method for acquiring high-quality video images and / or an image processing apparatus for performing the method. Background Art

[0003] Image recording devices (e.g., cameras) are employed in various electronic systems and mobile systems such as computers, mobile phones, tablet computers, virtual reality (VR) devices, and robotic systems. When photographing an object using a camera, if the background of the object is too bright, the object is relatively dark, which is called backlighting. Backlight compensation (BLC) can be used to obtain higher quality images. High dynamic range (HDR) technology merges an image in which bright areas are clearly seen by controlling their exposure with a different image in which dark areas are clearly seen by controlling their exposure, and generates an image in which bright areas and dark areas are seen very clearly. However, when images are merged using conventional HDR technology, frame rate loss may occur as the number of merged frames increases. Summary of the invention

[0004] At least one example embodiment of the present disclosure provides an image processing method capable of efficiently generating an HDR image from a video image without losing a frame rate.

[0005] At least one example embodiment of the present disclosure provides an image processing apparatus capable of efficiently generating an HDR image from a video image without losing a frame rate.

[0006] Some example embodiments relate to a method for processing a video image including a plurality of frame images. In some example embodiments, the method may include: generating a first high dynamic range (HDR) image based on a first frame image and a second frame image among the plurality of frame images, wherein the second frame image is immediately after the first frame image in time among the plurality of frame images, the first frame image has a first exposure time, and the second frame image has a second exposure time different from the first exposure time; and generating a second HDR image based on the second frame image and a third frame image among the plurality of frame images, wherein the third frame image is immediately after the second frame image in time among the plurality of frame images, and the third frame image has the first exposure time.

[0007] Some example embodiments relate to an apparatus for processing a video image including a plurality of frame images, the plurality of frame images including a first frame image, a second frame image, and a third frame image. In some example embodiments, the apparatus may include: a first frame buffer, the first frame buffer being configured to receive the first frame image and the third frame image, the first frame image having a first exposure time, the third frame image being immediately after the second frame image in time among the plurality of frame images, the third frame image having the first exposure time; a second frame buffer, the second frame buffer being configured to receive the second frame image, the second frame image being immediately after the first frame image in time among the plurality of frame images, the second frame image having a second exposure time different from the first exposure time; and a high dynamic range (HDR) processor, the HDR processor being configured to generate a first HDR image based on the first frame image and the second frame image, and to generate a second HDR image based on the second frame image and the third frame image.

[0008] Some example embodiments relate to a method for processing a video image including a plurality of frame images. In some example embodiments, the method includes: generating a first high dynamic range (HDR) image based on a first frame image and a second frame image of the plurality of frame images, the second frame image being immediately after the first frame image in time, the first frame image having a first exposure time, and the second frame image having a second exposure time different from the first exposure time; and generating a second HDR image based on the first HDR image and a third frame image of the plurality of frame images, the third frame image being immediately after the second frame image in time, and the third frame image having the first exposure time.

[0009] Some example embodiments relate to an apparatus for processing a video image including a plurality of frame images, the plurality of frame images including a first frame image, a second frame image, and a third frame image. In some example embodiments, the apparatus includes: a first frame buffer, the first frame buffer being configured to receive the first frame image having a first exposure time, receive the second frame image, and receive the third frame image having the first exposure time, wherein the second frame image is immediately after the first frame image in time among the plurality of frame images, the second frame image has a second exposure time different from the first exposure time, and the third frame image is immediately after the second frame image in time among the plurality of frame images; a high dynamic range (HDR) processor, the HDR processor being configured to generate a first HDR image based on the first frame image and the second frame image, and to generate a second HDR image based on the first HDR image and the third frame image; and a second frame buffer, the second frame buffer being configured to receive the first HDR image and the second HDR image.

[0010] In the method and apparatus for processing an image according to an exemplary embodiment, an HDR image may be generated based on one of a sliding window scheme and an IIR scheme. Therefore, an HDR video image may be effectively generated without losing the frame rate while maintaining the frame rate of the original video image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative, non-limiting example embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0012] Figure 1 is a flowchart illustrating an image processing method according to an example embodiment.

[0013] Figure 2 It is shown Figure 1 Flowchart of the example of generating an HDR image in .

[0014] Figure 3 Is used to describe Figure 2 A view of the operations for generating an HDR image.

[0015] Figure 4 It is shown Figure 2 Flowchart of an example of generating a first HDR image in .

[0016] Figure 5A and Figure 5B Is used to describe Figure 4 A view of the operation of generating a first HDR image.

[0017] Figure 6 It is shown Figure 2A flowchart of another example of generating a first HDR image in FIG.

[0018] Fig. 7A and Figure 7B Is used to describe Figure 6 A view of the operation of generating a first HDR image.

[0019] Figure 8 is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0020] Fig. 9 is a block diagram illustrating an example of an HDR processor included in an image processing apparatus according to example embodiments.

[0021] Fig.10 It is shown Figure 1 Flowchart of another example of generating an HDR image in .

[0022] Fig.11 Is used to describe Fig.10 A view of the operations for generating an HDR image.

[0023] Fig.12 is a block diagram illustrating another example of an HDR processor included in an image processing apparatus according to example embodiments.

[0024] Fig.13 is a flowchart illustrating an image processing method according to an example embodiment.

[0025] Fig.14 It is shown Fig.13 Flowchart of the example of generating an HDR image in .

[0026] Fig.15 Is used to describe Fig.14 A view of the operations for generating an HDR image.

[0027] Fig.16 is a block diagram illustrating still another example of an HDR processor included in an image processing apparatus according to example embodiments.

[0028] Fig.17 It is shown Fig.13 Flowchart of another example of generating an HDR image in .

[0029] Fig.18 Is used to describe Fig.17 A view of the operations for generating an HDR image.

[0030] Fig.19 is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0031] Fig. 20is a block diagram illustrating an electronic system according to example embodiments. DETAILED DESCRIPTION

[0032] Various example embodiments will be described more fully with reference to the accompanying drawings in which the embodiments are shown. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the example embodiments set forth herein. Throughout the application, similar reference numerals represent similar elements.

[0033] Figure 1 is a flowchart illustrating an image processing method according to an example embodiment.

[0034] refer to Figure 1 , the image processing method according to the example embodiment is performed or implemented on a video image including a plurality of frame images. In other words, the image to be processed is a video image including a plurality of frame images. Specifically, the image processing method according to the example embodiment can obtain or acquire an HDR video image without losing a frame rate or with a lossless frame rate by generating a plurality of high dynamic range (HDR) images based on a plurality of frame images.

[0035] In an image processing method according to an example embodiment, in operation S100, a starting frame image is output as a starting HDR image without performing any image processing (including, for example, mixing, compositing, or synthesis, but not limited thereto). The starting frame image is included in a plurality of frame images and is arranged at the first position or the front of the plurality of frame images. The starting HDR image is included in a plurality of HDR images and is arranged at the first position or the front of the plurality of HDR images.

[0036] In operation S200, each HDR image is generated by merging (including, for example, mixing, combining, or synthesizing, but not limited to) N frame images among the plurality of frame images based on a sliding window scheme, where N is a natural number greater than or equal to 2. The N frame images are continuous or sequential frame images among the plurality of frame images and have different exposure times.

[0037] The sliding window scheme is a data transmission method based on a "window" with a desired (or, predetermined) size. For example, when a series of data is to be sent, all data packets included in the window can be sent, and after the transmission of one data packet is completed, the window can be slid by the desired (or, predetermined) unit, and then the next data packet included in the sliding window can be sent. Similarly, in the image processing method according to the example embodiment, a sliding window scheme can be applied or adopted to generate an HDR image. For example, an HDR image can be generated using continuous frame images included in a "window" with a desired (or, predetermined) size (for example, the number N), the window can slide by one frame, and then another HDR image can be generated using other continuous frame images included in the sliding window. Reference will be made to Figures 2 to 12 An image processing operation based on a sliding window scheme and a configuration of an image processing apparatus for performing the image processing operation are described.

[0038] In some example embodiments, the number N representing the window size and the number of frame images merged at a time may be substantially equal to the number of kinds of different exposure times included in the plurality of frame images.

[0039] When an HDR image is generated based on K frame images and the above sliding window scheme having a window size of N, the number of generated HDR images may be (K-N+1) instead of K, where K is a natural number greater than or equal to N. Therefore, as described in operation S100, a starting HDR image may be provided by outputting the starting frame image as it is without merging, so that the number of input frame images is substantially equal to the number of output HDR images (e.g., the frame rate is maintained).

[0040] Figure 2 It is shown Figure 1 Flowchart of the example of generating an HDR image in . Figure 3 Is used to describe Figure 2 A view of the operations for generating an HDR image.

[0041] Figure 2 and Figure 3 An example is shown in which the size of the window and the number of different exposure times is two (eg, N=2). Figure 3 In the embodiment, a plurality of frame images FIMG11, FIMG12, FIMG13, FIMG14, FIMG15, FIMG16, ... included in a video image (e.g., an input image) may be provided from an image sensor or a storage device, etc. Frame images FIMG11, FIMG13, and FIMG15 having a first exposure time L (e.g., a long exposure time) and frame images FIMG12, FIMG14, and FIMG16 having a second exposure time S (e.g., a short exposure time) may be repeated alternately.

[0042] refer to Figure 1 , Figure 2 and Figure 3 As described with reference to operation S100, the starting frame image or the first frame image FIMG11 arranged at the first position of the plurality of frame images FIMG11, FIMG12, FIMG13, FIMG14, FIMG15, and FIMG16 may be output as the starting HDR image HIMG11 among the plurality of HDR images HIMG11, HIMG12, HIMG13, HIMG14, HIMG15, HIMG16, . . . included in the HDR video image (e.g., the output image) without performing any image processing.

[0043] Thereafter, as described with reference to operation S200, when the remaining HDR images HIMG12, HIMG13, HIMG14, HIMG15 and HIMG16 other than the starting HDR image HIMG11 among the multiple HDR images HIMG11, HIMG12, HIMG13, HIMG14, HIMG15 and HIMG16 are generated, two consecutive frame images with different exposure times L and S can be sequentially merged by using a sliding window scheme with a window size of 2.

[0044] Operation S200 may include operations S210 to S230. For example, in operation S210, a first HDR image HIMG12 may be generated by merging a first frame image FIMG11 having a first exposure time L and a second frame image FIMG12 having a second exposure time S.

[0045] The second frame image FIMG12 may be a frame image immediately following the first frame image FIMG11. For example, the first frame image FIMG11 and the second frame image FIMG12 may be two consecutive frame images. The second exposure time S may be different from the first exposure time L. For example, the first exposure time L may be a relatively long time interval, and the second exposure time S may be a relatively short time interval. For example, the first frame image FIMG11 having the first exposure time L as a relatively long time interval may be an image of relatively high illumination or high brightness, and the second frame image FIMG12 having the second exposure time S as a relatively short time interval may be an image of relatively low illumination or low brightness.

[0046] As described above, because the input image is a video image, the first frame image FIMG11 and the second frame image FIMG12 may not show exactly the same scene, so a scene change, scene movement, and / or object movement may occur between the first frame image FIMG11 and the second frame image FIMG12. Therefore, considering the scene change, scene movement, and / or object movement, the first frame image FIMG11 and the second frame image FIMG12 need to be merged. Figure 4 and Figure 6 A detailed operation of merging the first frame image FIMG11 and the second frame image FIMG12 is described.

[0047] After operation S210 is completed, in operation S220, a second HDR image HIMG13 may be generated by merging the second frame image FIMG12 and the third frame image FIMG13 having the first exposure time L. The second HDR image HIMG13 may be an HDR image immediately following the first HDR image HIMG12, and the third frame image FIMG13 may be a frame image immediately following the second frame image FIMG12. In other words, after operation S210 is completed, the window may slide or shift by one frame, and then operation S220 may be performed by using a sliding window.

[0048] Similarly, in operation S230 , a third HDR image HIMG14 subsequent to the second HDR image HIMG13 may be generated by merging the third frame image FIMG13 and a fourth frame image FIMG14 having the second exposure time S subsequent to the third frame image FIMG13 .

[0049] In addition, if Figure 3 As shown, a fourth HDR image HIMG15 after the third HDR image HIMG14 can be generated by merging the fourth frame image FIMG14 and the fifth frame image FIMG15 with the first exposure time L after the fourth frame image FIMG14. A fifth HDR image HIMG16 after the fourth HDR image HIMG15 can be generated by merging the fifth frame image FIMG15 and the sixth frame image FIMG16 with the second exposure time S after the fifth frame image FIMG15.

[0050] As described above, when an HDR image is generated by merging frame images based on a sliding window scheme, the number of multiple frame images FIMG11, FIMG12, FIMG13, FIMG14, FIMG15, and FIMG16 included in an input image and received before an image processing operation may be substantially equal to the number of multiple HDR images HIMG11, HIMG12, HIMG13, HIMG14, HIMG15, and HIMG16 included in an output image and obtained after an image processing operation. Therefore, an HDR video image or an HDR image included in an HDR video image may be efficiently generated without losing a frame rate or with a lossless frame rate.

[0051] Figure 4 It is shown Figure 2 Flowchart of an example of generating a first HDR image in . Figure 5A and Figure 5B Is used to describe Figure 4 A view of the operation of generating a first HDR image.

[0052] refer to Figure 2 , Figure 3 , Figure 4 , Figure 5A and Figure 5B In some example embodiments, operation S210 of generating the first HDR image may include operation S211, operation S213, and operation S215.

[0053] In operation S211 , a global movement or a global motion between a first frame image and a second frame image may be extracted.

[0054] For example, Figure 5A It shows that Figure 3 The first frame image F1A and the second frame image F2A of the first frame image FIMG11 and the second frame image FIMG12 are shown with hatching. Figure 5A For example, the first frame image F1A shown with hatching may have a first exposure time L, and the second frame image F2A not shown with hatching may have a second exposure time S different from the first exposure time L.

[0055] exist Figure 5A , when compared with the first frame image F1A, the second frame image F2A may have a scene including the sun, mountains, and trees as a whole shifted or moved in the right direction. Figure 5AThe dotted lines in the second frame image F2A in FIG. 1A may indicate the positions of the sun, mountains, and trees in the first frame image F1A. The global motion GM between the first frame image F1A and the second frame image F2A may be extracted by comparing and / or analyzing the first frame image F1A and the second frame image F2A. For example, the global motion GM may be obtained in the form of a global motion vector.

[0056] In operation S213, the first frame image may be shifted based on the extracted global motion.

[0057] For example, you can Figure 5A The first frame image F1A in the image is shifted to the right to obtain Figure 5B The shifted first frame image F1A' in the image F1A' is a portion of the image F1A'. Through the shift operation as described above, the shifted first frame image F1A' may include a shift area SA and a virtual area VA. The shift area SA may be substantially the same as a portion of the first frame image F1A. The virtual area VA may be a portion that is not actually included in the first frame image F1A. For example, the virtual area VA may be obtained by performing image interpolation based on the first frame image F1A, etc. For another example, when the size of the virtual area VA is relatively small, the virtual area VA may be obtained by copying a portion of the first frame image F1A as is, or the virtual area VA may be empty without any image processing.

[0058] In operation S215, the shifted first frame image and the second frame image may be merged with each other.

[0059] For example, by Figure 5B The first frame image F1A' and Figure 5A The second frame image F2A in the image is merged to obtain the first HDR image, so the distortion, deterioration or degradation of the first HDR image can be reduced (or prevented).

[0060] although Figure 4 , Figure 5A and Figure 5B An example is shown in which a first frame image is shifted relative to a second frame image and the shifted first frame image and the second frame image are merged with each other (e.g., an example in which a second frame image input later than the first frame image is used as a reference image), but example embodiments are not limited thereto. For example, the second frame image may be shifted relative to a first frame image input before the second frame image, and the first frame image and the shifted second frame image may be merged with each other to generate a first HDR image. When the first frame image input before the second frame image is used as a reference image as described above, the image merging operation may be performed in consideration of the extracted global movement, and the image merging operation may be performed in consideration of a frame delay associated with the input of the second frame image input later than the first frame image.

[0061] Figure 6 It is shown Figure 2 A flowchart of another example of generating a first HDR image in FIG. Fig. 7A and Figure 7B Is used to describe Figure 6 A view of the operation of generating a first HDR image.

[0062] refer to Figure 2 , Figure 3 , Figure 6 , Fig. 7A and Figure 7B In some other example embodiments, operation S210 of generating the first HDR image may include operation S212, operation S214, and operation S216.

[0063] In operation S212, a local movement or a local motion of a first object between a first frame image and a second frame image may be extracted. The first object may be one of a plurality of objects included in the first frame image and the second frame image.

[0064] For example, Fig. 7A It shows that Figure 3 The first frame image FIMG11 and the second frame image FIMG12 are other examples of the first frame image F1B and the second frame image F2B. Figure 5A Described, shown with hatching Fig. 7A The first frame image F1B in FIG. 1 is used to indicate the difference in exposure time.

[0065] exist Fig. 7A , when compared with the first frame image F1B, the second frame image F2B may include the first object OBJ1 moved or shifted in the right direction. Fig. 7A The dotted line in the second frame image F2B in the first frame image F1B may indicate the position of the first object OBJ1 in the first frame image F1B. The remaining portion of the image other than the first object OBJ1 (e.g., the background) may not be shifted, or even if shifted, the remaining portion may be displayed substantially identically without any difference between the first frame image F1B and the second frame image F2B. The local movement LM of the first object OBJ1 between the first frame image F1B and the second frame image F2B may be extracted by comparing and / or analyzing the first frame image F1B and the second frame image F2B. For example, the local movement LM may be obtained in the form of a local motion vector.

[0066] In operation S214, the first object in the first frame image may be shifted based on the extracted local movement.

[0067] For example, you can Fig. 7AThe first object OBJ1 included in the first frame image F1B in the image is shifted to the right to obtain Figure 7B The first frame image F1B' after displacement includes the first object OBJ1' after displacement. Figure 7B As shown in the reference Figure 5B As described, the shifted first frame image F1B' including the shifted first object OBJ1' may include a virtual area by shifting the first object OBJ1. The virtual area may be obtained by performing image interpolation based on the first frame image F1B, may be obtained by copying a portion of the first frame image F1B as is, or may be empty without any image processing.

[0068] In operation S216, the shifted first frame image (including the shifted first object) and the second frame image may be merged with each other.

[0069] For example, by Figure 7B The first frame image F1B' after displacement (including the first object OBJ1' after displacement) and Fig. 7A The second frame image F2B in the image is merged to obtain the first HDR image, so the distortion, deterioration or degradation of the first HDR image can be reduced (or prevented).

[0070] although Figure 6 , Fig. 7A and Figure 7B An example is shown in which the first frame image and the second frame image include one object, but example embodiments are not limited thereto. For example, the first frame image and the second frame image may include two or more objects. When the first frame image and the second frame image include two or more objects, each local movement may be extracted for each object, each object in the first frame image may be shifted based on the corresponding local movement, and the shifted first frame image (including the shifted object) and the second frame image may be merged with each other.

[0071] In addition, despite Figure 6 , Fig. 7A and Figure 7B An example is shown in which a first object in a first frame image is shifted relative to a second frame image and the shifted first frame image (including the shifted first object) and the second frame image are merged with each other (for example, an example in which a second frame image input later than the first frame image is used as a reference image), but example embodiments are not limited thereto. For example, the first object in the second frame image may be shifted relative to the first frame image input before the second frame image, and the first frame image and the shifted second frame image including the shifted first object may be merged with each other to generate a first HDR image.

[0072] Although reference Figure 4, Figure 5A and Figure 5B A first scheme for extracting a global motion between two consecutive frame images and performing an image merging operation based on the global motion is described, and although reference is made to Figure 6 , Fig. 7A and Figure 7B A second scheme for extracting the local movement of an object between two consecutive frame images and performing an image merging operation based on the local movement is described, but example embodiments are not limited thereto. For example, both the first scheme and the second scheme may be used simultaneously to perform the image merging operation. In addition, at least one of the various algorithms for synthesizing frame images may be used to perform the image merging operation.

[0073] Although not in Figure 4 , Figure 5A , Figure 5B , Figure 6 , Fig. 7A and Figure 7B However, the operation of generating each remaining HDR image except the first HDR image may be substantially the same as the operation of generating the first HDR image. Figure 4 , Figure 5A and Figure 5B The first scheme described can be used to perform image merging operations, referring to Figure 6 , Fig. 7A and Figure 7B The described second scheme may be used to perform the image merging operation, or both the first scheme and the second scheme may be used simultaneously to perform the image merging operation.

[0074] Figure 8 is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0075] refer to Figure 8 , the image processing apparatus 100 includes an HDR processor 300. The image processing apparatus 100 may further include a triple automation (3A) processor 200 and an image signal processor 400.

[0076] The HDR processor 300 is located or provided between the 3A processor 200 and the image signal processor 400, and performs a multi-frame processing operation. The HDR processor 300 receives the exposure time information EINF, and performs an image processing method according to an example embodiment.

[0077] In other example embodiments, as shown in FIG. Figures 1 to 7B described, and as will be referred to Fig.10 and Fig.11As described, the HDR processor 300 may generate a plurality of HDR images HIMG by sequentially merging a plurality of frame images FIMG having different exposure times based on a sliding window scheme.

[0078] In other example embodiments, as will be referred to Figures 13 to 15 , Fig.17 and Fig.18 As described, the HDR processor 300 may generate a plurality of HDR images HIMG by sequentially merging a plurality of frame images FIMG having different exposure times based on an infinite impulse response (IIR) scheme.

[0079] The HDR processor 300 may be implemented with at least two frame buffers. Fig. 9 , Fig.12 and Fig.16 The detailed configuration of the HDR processor 300 is described.

[0080] The 3A processor 200 may receive exposure time information EINF, and may perform three automation (3A) operations on a plurality of input images IIMG to generate a plurality of frame images FIMG. The performed auto focus (or auto focus (AF)) operation, auto white balance (or auto white (color) balance (AWB)) operation, and auto exposure (or auto exposure (AE)) operation may be collectively referred to as performing a 3A operation. Like the plurality of frame images FIMG, the plurality of input images IIMG may also have different exposure times. The plurality of frame images FIMG may be acquired from an external image pickup device such as an image sensor (e.g., Fig. 20 ) and / or an external storage device (e.g., Fig. 20 The storage device 1030 in the storage device 1030 provides multiple input images IIMG.

[0081] The 3A processor 200 and the HDR processor 300 may share the exposure time information EINF, and HDR gain compensation may be performed by the 3A processor 200 and the HDR processor 300 based on the exposure time information EINF.

[0082] The image signal processor 400 may perform additional image processing such as image interpolation, color correction, gamma correction, or color conversion on the plurality of HDR images HIMG to generate a plurality of output images OIMG. Like the plurality of HDR images HIMG, each of the plurality of output images OIMG may also be an HDR image. The HDR images HIMG may be displayed to an external user interface such as a display device (e.g., Fig. 20 The user interface 1040 in provides multiple output images OIMG.

[0083] The image processing device 100 may include a processing circuit such as hardware including a logic circuit, a hardware / software combination such as a processor that executes software, or a combination thereof. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a single chip system (SoC), a programmable logic unit, a microprocessor, and an application specific integrated circuit (ASIC).

[0084] For example, in some example embodiments, Figure 8 At least a portion of the 3A processor 200, the HDR processor 300, and the image signal processor 400 in the image processing apparatus 100 according to the example embodiment may be implemented as hardware. For example, at least a portion of the elements included in the image processing apparatus 100 according to the example embodiment may be included in a computer-based electronic system.

[0085] In other example embodiments, Figure 8 At least a portion of the 3A processor 200, the HDR processor 300, and the image signal processor 400 may be implemented as an instruction code or a program routine (e.g., a software program). For example, the instruction code or the program routine may be executed by a computer-based electronic system and may be stored in any storage device located inside or outside the computer-based electronic system.

[0086] Fig. 9 is a block diagram illustrating an example of an HDR processor included in an image processing apparatus according to example embodiments.

[0087] refer to Fig. 9 , the HDR processor 300a includes a first frame buffer 310a, a second frame buffer 320a and an HDR processing unit 350a.

[0088] The first frame buffer 310a and the second frame buffer 320a receive and store a plurality of frame images FIMG. In other words, both the first frame buffer 310a and the second frame buffer 320a may be used as input buffers.

[0089] Two frame buffers 310a and 320a can be used to implement Fig. 9 The HDR processor 300a in the HDR processor 300a is configured to receive a plurality of frame images FIMG having two different exposure times L and S, and store the frame images having the same exposure time in the same frame buffer. For example, the first frame buffer 310a can sequentially receive and store a frame image FIMG_L having a first exposure time L among the plurality of frame images FIMG. The second frame buffer 320a can sequentially receive and store a frame image FIMG_S having a second exposure time S among the plurality of frame images FIMG.

[0090] The HDR processing unit 350a receives the exposure time information EINF and generates a plurality of HDR images HIMG by merging the frame image FIMG_L and the frame image FIMG_S having different exposure times L and S based on a sliding window scheme. In other words, the HDR processing unit 350a may perform Figure 1 Operation S200 and Figure 2 Operation S210, operation S220 and operation S230 in.

[0091] Will combine Figure 3 The example in Fig. 9 Operation of the HDR processor 300a in. Figure 3 The frame images FIMG11, FIMG13 and FIMG15 in may correspond to Fig. 9 The frame image FIMG_L in Figure 3 The frame images FIMG12, FIMG14 and FIMG16 in the Fig. 9 The first frame buffer 310a may receive the frame image FIMG11, and the HDR processing unit 350a may output the frame image FIMG11 as the HDR image HIMG11 without performing any image processing. The second frame buffer 320a may receive the frame image FIMG12, and the HDR processing unit 350a may generate the HDR image HIMG12 by merging the frame image FIMG11 and the frame image FIMG12. The first frame buffer 310a may receive the frame image FIMG13, the previously stored frame image FIMG11 may be deleted or removed from the first frame buffer 310a, and the HDR processing unit 350a may generate the HDR image HIMG13 by merging the frame images FIMG12 and FIMG13. The second frame buffer 320a may receive the frame image FIMG14, the previously stored frame image FIMG12 may be deleted or removed from the second frame buffer 320a, and the HDR processing unit 350a may generate the HDR image HIMG14 by merging the frame image FIMG13 and the frame image FIMG14. Similarly, the frame images FIMG15 and FIMG16 may be sequentially received, and the HDR images HIMG15 and HIMG16 may be sequentially generated.

[0092] The HDR processing unit 350a may include a motion extractor 352a, an image shifter 354a, and an image mixer 356a. For example, the processing circuit included in the image processing apparatus 100 may be hardware that performs the operations of the motion extractor 352a, the image shifter 354a, and the image mixer 356a, or software that converts the processing circuit into a dedicated processor that performs the operations of the motion extractor 352a, the image shifter 354a, and the image mixer 356a may be executed.

[0093] For example, in some example embodiments, Fig. 9 At least a portion of the motion extractor 352a, the image shifter 354a, and the image mixer 356a in may be implemented as hardware.

[0094] In other example embodiments, Fig. 9 At least a portion of the motion extractor 352a, the image shifter 354a, and the image mixer 356a in the embodiment may be implemented as instruction codes or program routines (eg, software programs).

[0095] The motion extractor 352a may extract at least one of a global motion and a local motion between two consecutive frame images. The image shifter 354a may shift one of the two consecutive frame images based on the extracted motion. The image mixer 356a may combine the shifted frame image with the unshifted frame image to generate an HDR image. In other words, the motion extractor 352a may perform Figure 4 Operation S211 and Figure 6 The image shifter 354a may perform at least one of the operations S212 in Figure 4 Operation S213 and Figure 6 At least one of the operations S214 in the above, and the image mixer 356a may perform Figure 4 Operation S215 and Figure 6 At least one operation in operation S216.

[0096] Fig.10 It is shown Figure 1 Flowchart of another example of generating an HDR image in . Fig.11 Is used to describe Fig.10 The operations for generating HDR images in the diagram will be omitted. Figure 2 and Figure 3 Description of duplicate description.

[0097] Fig.10 and Fig.11 An example is shown in which the number of window sizes and different exposure times is three (eg, N=3). Fig.11In the process, a plurality of frame images FIMG21, FIMG22, FIMG23, FIMG24, FIMG25, FIMG26, FIMG27, FIMG28, FIMG29... may be provided from an image sensor or a storage device, etc., and frame images FIMG21, FIMG24 and FIMG27 with a first exposure time L, frame images FIMG22, FIMG25 and FIMG28 with a second exposure time M, and frame images FIMG23, FIMG26 and FIMG29 with a third exposure time S may be repeated alternately.

[0098] refer to Figure 1 , Fig.10 and Fig.11 As described with reference to operation S100, the starting frame image or the first frame image FIMG21 arranged at the first position among the plurality of frame images FIMG21 to FIMG29 may be output as a starting HDR image among a plurality of HDR images HIMG21, HIMG22, HIMG23, HIMG24, HIMG25, HIMG26, HIMG27, HIMG28, HIMG29 . . . without performing any image processing.

[0099] In addition, the second starting HDR image HIMG22 can be generated by merging the starting frame image FIMG21 with the second starting frame image or the second frame image FIMG22 that is arranged immediately after the starting frame image FIMG21. If the number of input frame images is smaller than the size of the window at the time of the initial operation, all input frame images can be merged to generate an HDR image. Figures 4 to 7B The image merging operation for the first frame image FIMG21 and the second frame image FIMG22 is performed in the same manner as described.

[0100] Thereafter, as described with reference to operation S200, when the remaining HDR images HIMG23, HIMG24, HIMG25, HIMG26, HIMG27, HIMG28 and HIMG29 other than the starting HDR images HIMG21 and HIMG22 among the multiple HDR images HIMG21, HIMG22, HIMG23, HIMG24, HIMG25, HIMG26, HIMG27, HIMG28 and HIMG29 are generated, three consecutive frame images with different exposure times L, M and S can be sequentially merged by using a sliding window scheme with a window size of 3.

[0101] For example, in operation S240 , a first HDR image HIMG23 may be generated by merging a first frame image FIMG21 having a first exposure time L, a second frame image FIMG22 having a second exposure time M, and a third frame image FIMG23 having a third exposure time S.

[0102] The second frame image FIMG22 may be a frame image after the first frame image FIMG21, and the third frame image FIMG23 may be a frame image after the second frame image FIMG22. For example, the first frame image FIMG21, the second frame image FIMG22, and the third frame image FIMG23 may be three consecutive frame images. The first exposure time L, the second exposure time M, and the third exposure time S may be different from each other. For example, the second exposure time M may be shorter than the first exposure time L, and may be longer than the third exposure time S.

[0103] You can refer to Figures 4 to 7B The image merging operation for three consecutive frame images is performed in a manner similar to the described manner. For example, the last frame image (e.g., the third frame image FIMG23) of the three consecutive frame images can be used as a reference image, the remaining two frame images (e.g., the first frame image FIMG21 and the second frame image FIMG22) can be shifted relative to the last frame image, and then the image merging operation can be performed. For another example, the middle frame image (e.g., the second frame image FIMG22) of the three consecutive frame images can be used as a reference image, the remaining two frame images (e.g., the first frame image FIMG21 and the third frame image FIMG23) can be shifted relative to the middle frame image, and then the image merging operation can be performed. Alternatively, the front frame image (e.g., the first frame image FIMG21) of the three consecutive frame images can be used as a reference image, the remaining two frame images (e.g., the second frame image FIMG22 and the third frame image FIMG23) can be shifted relative to the front frame image, and then the image merging operation can be performed.

[0104] In operation S250, a second HDR image HIMG24 may be generated by merging the second frame image FIMG22, the third frame image FIMG23, and the fourth frame image FIMG24 having the first exposure time L. The second HDR image HIMG24 may be an HDR image after the first HDR image HIMG23, and the fourth frame image FIMG24 may be a frame image after the third frame image FIMG23. In other words, after operation S240 is completed, the window may slide or shift by one frame, and then operation S250 may be performed using the slid window.

[0105] Similarly, in operation S260 , a third HDR image HIMG25 subsequent to the second HDR image HIMG24 may be generated by merging the third frame image FIMG23 , the fourth frame image FIMG24 , and a fifth frame image FIMG25 having the second exposure time M subsequent to the fourth frame image FIMG24 .

[0106] In operation S270 , a fourth HDR image HIMG26 subsequent to the third HDR image HIMG25 may be generated by merging the fourth frame image FIMG24 , the fifth frame image FIMG25 , and a sixth frame image FIMG26 having a third exposure time S subsequent to the fifth frame image FIMG25 .

[0107] In addition, if Fig.11 As shown, a fifth HDR image HIMG27 after the fourth HDR image HIMG26 can be generated by merging the fifth frame image FIMG25, the sixth frame image FIMG26, and the seventh frame image FIMG27 with the first exposure time L after the sixth frame image FIMG26. A sixth HDR image HIMG28 after the fifth HDR image HIMG27 can be generated by merging the sixth frame image FIMG26, the seventh frame image FIMG27, and the eighth frame image FIMG28 with the second exposure time M after the seventh frame image FIMG27. A seventh HDR image HIMG29 after the sixth HDR image HIMG28 can be generated by merging the seventh frame image FIMG27, the eighth frame image FIMG28, and the ninth frame image FIMG29 with the third exposure time S after the eighth frame image FIMG28.

[0108] Fig.12 is a block diagram showing another example of an HDR processor included in an image processing apparatus according to an example embodiment. Fig. 9 Description of duplicate description.

[0109] refer to Fig.12 The HDR processor 300b includes a first frame buffer 310b, a second frame buffer 320b, a third frame buffer 330b and an HDR processing unit 350b.

[0110] The first frame buffer 310b, the second frame buffer 320b, and the third frame buffer 330b receive and store a plurality of frame images FIMG. The first frame buffer 310b, the second frame buffer 320b, and the third frame buffer 330b may each be used as an input buffer.

[0111] The three frame buffers 310b, 320b and 330b can be used to implement Fig.12The HDR processor 300b in the HDR processor 300b is configured so that the HDR processor 300b receives a plurality of frame images FIMG having three different exposure times L, M, and S, and stores the frame images having the same exposure time in the same frame buffer. For example, the first frame buffer 310b can sequentially receive and store a frame image FIMG_L having a first exposure time L among the plurality of frame images FIMG. The second frame buffer 320b can sequentially receive and store a frame image FIMG_M having a second exposure time M among the plurality of frame images FIMG. The third frame buffer 330b can sequentially receive and store a frame image FIMG_S having a third exposure time S among the plurality of frame images FIMG.

[0112] The HDR processing unit 350b receives the exposure time information EINF and generates a plurality of HDR images HIMG by merging the frame images FIMG_L, FIMG_M and FIMG_S having different exposure times L, M and S based on a sliding window scheme. In other words, the HDR processing unit 350b may perform Figure 1 Operation S200 and Fig.10 Operation S240, operation S250, operation S260 and operation S270 in the embodiment.

[0113] Will combine Fig.11 The example in Fig.12 The operation of the HDR processor 300b in. Fig.11 The frame images FIMG21, FIMG24 and FIMG27 in the Fig.12 The frame image FIMG_L in Fig.11 The frame images FIMG22, FIMG25 and FIMG28 in the Fig.12 The frame image FIMG_M in Fig.11 The frame images FIMG23, FIMG26 and FIMG29 in the Fig.12The first frame buffer 310b may receive the frame image FIMG21, and the HDR processing unit 350b may output the frame image FIMG21 as the HDR image HIMG21 without performing any image processing. The second frame buffer 320b may receive the frame image FIMG22, and the HDR processing unit 350b may generate the HDR image HIMG22 by merging the frame image FIMG21 and the frame image FIMG22. The third frame buffer 330b may receive the frame image FIMG23, and the HDR processing unit 350b may generate the HDR image HIMG23 by merging the frame images FIMG21, FIMG22, and FIMG23. The first frame buffer 310b may receive the frame image FIMG24, the previously stored frame image FIMG21 may be deleted or removed from the first frame buffer 310b, and the HDR processing unit 350b may generate the HDR image HIMG24 by merging the frame images FIMG22, FIMG23, and FIMG24. Similarly, frame images FIMG25, FIMG26, FIMG27, FIMG28, and FIMG29 may be sequentially received, and HDR images HIMG25, HIMG26, HIMG27, HIMG28, and HIMG29 may be sequentially generated.

[0114] The HDR processing unit 350b may include a motion extractor 352b, an image shifter 354b, and an image mixer 356b. Fig.12 The motion extractor 352b, the image shifter 354b and the image mixer 356b in the Fig. 9 The motion extractor 352a, image shifter 354a and image mixer 356a in are basically the same.

[0115] In some example embodiments, as shown in FIG. Fig. 9 and Fig.12 As described above, the number of frame buffers included in the image processing apparatus operating based on the sliding window scheme (for example, included in the HDR processor according to the example embodiment) may be substantially equal to the number of exposure times included in the plurality of frame images FIMG, and may increase as the number of exposure times increases. Fig. 9 As shown, when the plurality of frame images FIMG have two different exposure times L and S, the HDR processor 300a may include two frame buffers 310a and 320a. Fig.12 As shown, when the plurality of frame images FIMG have three different exposure times L, M, and S, the HDR processor 300b may include three frame buffers 310b, 320b, and 330b.

[0116] In reference Figures 1 to 12 In the described image processing method and image processing device, HDR images can be generated for all frame images based on a sliding window scheme without being affected by the number of frames used, so that HDR video images or HDR images included in HDR video images can be efficiently generated without losing frame rate. If there is no restriction on frame delay or computing power (e.g., recording operation), there may be the following advantages: input frames can be used without restriction to generate improved or enhanced HDR video images.

[0117] Fig.13 is a flowchart illustrating an image processing method according to an example embodiment. Figure 1 Description of duplicate description.

[0118] refer to Fig.13 , the image processing method according to the exemplary embodiment is performed or implemented on a video image including a plurality of frame images. In other words, the image to be processed is a video image including a plurality of frame images. Specifically, the image processing method according to the exemplary embodiment can obtain or acquire an HDR video image by generating a plurality of HDR images based on a plurality of frame images without losing the frame rate.

[0119] In an image processing method according to an example embodiment, in operation S1100, a starting frame image is output as a starting HDR image without performing any image processing, combination or synthesis. The starting frame image is included in a plurality of frame images and is arranged at the first position of the plurality of frame images. The starting HDR image is included in a plurality of HDR images and is arranged at the first position of the plurality of HDR images. Fig.13 Operation S1100 in Figure 1 The operation S100 in is basically the same.

[0120] In operation S1200, each HDR image is generated by merging consecutive frame images among a plurality of frame images based on a recursive infinite impulse response (IIR) scheme. The consecutive frame images to be merged have different exposure times.

[0121] The IIR scheme has properties that are applicable to many linear time-invariant systems. Common examples of linear time-invariant systems are most electronic filters and digital filters. Systems with this property are called IIR systems or IIR filters, and are characterized by having an impulse response that does not completely become zero after a certain point but continues indefinitely. This is in contrast to a finite impulse response (FIR) in which the impulse response completely becomes zero after a finite duration. In the IIR scheme, the results of the previous output are used to generate the current output from the current input. Similarly, in the image processing method according to the example embodiment, the IIR scheme can be applied or adopted to generate an HDR image. For example, a previously generated and output HDR image and a currently input frame image can be used to generate a current or current HDR image. Reference will be made to Figures 14 to 18 An image processing operation based on the IIR scheme and a configuration of an image processing apparatus for performing the image processing operation are described.

[0122] Fig.14 It is shown Fig.13 Flowchart of the example of generating an HDR image in . Fig.15 Is used to describe Fig.14 A view of the operations for generating an HDR image.

[0123] Fig.14 and Fig.15 An example is shown in which the number of different exposure times is two. Fig.15 In the process, a plurality of frame images FIMG31, FIMG32, FIMG33, FIMG34, FIMG35, FIMG36 included in a video image (e.g., an input image) can be provided from an image sensor or a storage device, etc. Frame images FIMG31, FIMG33 and FIMG35 with a first exposure time L and frame images FIMG32, FIMG34 and FIMG36 with a second exposure time S can be repeated alternately.

[0124] refer to Fig.13 , Fig.14 and Fig.15 As described with reference to operation S1100, a starting frame image or a first frame image FIMG31 arranged at the first position among a plurality of frame images FIMG31, FIMG32, FIMG33, FIMG34, FIMG35, and FIMG36 may be output as a starting HDR image HIMG31 included in a plurality of HDR images HIMG31, HIMG32, HIMG33, HIMG34, HIMG35, HIMG36, ... included in an HDR video image (e.g., an output image) without performing any image processing.

[0125] Thereafter, as described with reference to operation S1200, when the remaining HDR images HIMG32, HIMG33, HIMG34, HIMG35 and HIMG36 other than the starting HDR image HIMG31 among the multiple HDR images HIMG31, HIMG32, HIMG33, HIMG34, HIMG35 and HIMG36 are generated, the HDR image as the current output can be generated by merging the previously generated and output HDR image and the currently input frame image using the IIR scheme.

[0126] In operation S1210, a first HDR image HIMG32 may be generated by merging the starting HDR image HIMG31 and the second frame image FIMG32. Since the starting HDR image HIMG31 and the starting frame image FIMG31 are substantially identical to each other, the above operation may be interpreted as generating the first HDR image HIMG32 by merging the first frame image FIMG31 having the first exposure time L and the second frame image FIMG32 having the second exposure time S. Figures 4 to 7B The image merging operation on the first frame image FIMG31 and the second frame image FIMG32 is performed in the same manner as described.

[0127] In operation S1220 , a second HDR image HIMG33 may be generated by merging the first HDR image HIMG32 and the third frame image FIMG33 having the first exposure time L. The second HDR image HIMG33 may be an HDR image after the first HDR image HIMG32 , and the third frame image FIMG33 may be a frame image after the second frame image FIMG32 .

[0128] You can refer to Figures 4 to 7B The image merging operation of the first HDR image HIMG32 and the third frame image FIMG33 is performed in a similar manner to that described above. Figure 4 As in the first scheme described above, the global movement between the first HDR image HIMG32 and the third frame image FIMG33 can be extracted, the first HDR image HIMG32 can be shifted based on the global movement, and the shifted first HDR image HIMG32 and the third frame image FIMG33 can be merged with each other. Figure 6As in the second scheme described above, a local movement of the first object between the first HDR image HIMG32 and the third frame image FIMG33 can be extracted, the first object in the first HDR image HIMG32 can be shifted based on the local movement, and the shifted first HDR image HIMG32 and the third frame image FIMG33 including the shifted first object can be merged with each other. In some example embodiments, the first HDR image HIMG32 except the third frame image FIMG33 can be used as a reference image.

[0129] Similarly, in operation S1230, a third HDR image HIMG34 after the second HDR image HIMG33 may be generated by combining the second HDR image HIMG33 with a fourth frame image FIMG34 after the third frame image FIMG33 and having a second exposure time S. Fig.15 As shown, a fourth HDR image HIMG35 after the third HDR image HIMG34 can be generated by merging the third HDR image HIMG34 and a fifth frame image FIMG35 with a first exposure time L after the fourth frame image FIMG34. A fifth HDR image HIMG36 after the fourth HDR image HIMG35 can be generated by merging the fourth HDR image HIMG35 and a sixth frame image FIMG36 with a second exposure time S after the fifth frame image FIMG35.

[0130] As described above, when an HDR image is generated by merging frame images based on the IIR scheme, the number of multiple frame images FIMG31, FIMG32, FIMG33, FIMG34, FIMG35, and FIMG36 received before the image processing operation included in the input image may be substantially equal to the number of multiple HDR images HIMG31, HIMG32, HIMG33, HIMG34, HIMG35, and HIMG36 obtained after the image processing operation included in the output image. Therefore, an HDR video image or an HDR image included in the HDR video image may be efficiently generated without losing a frame rate.

[0131] Fig.16 1 is a block diagram showing another example of an HDR processor included in an image processing apparatus according to an example embodiment. Fig. 9 and Fig.12 Description of duplicate description.

[0132] refer to Fig.16 , the HDR processor 300c includes a first frame buffer 310c, an HDR processing unit 350c and a second frame buffer 360c.

[0133] The first frame buffer 310c receives and stores a plurality of frame images FIMG. The first frame buffer 310c may be used as an input buffer. In the HDR processor 300c operating based on the IIR scheme, regardless of the number of exposure times included in the plurality of frame images FIMG, the number of frame buffers (e.g., the first frame buffer 310c) used as an input buffer may be fixed to one.

[0134] The HDR processing unit 350c receives the exposure time information EINF and generates a plurality of HDR images HIMG by merging a plurality of frame images FIMG with different exposure times based on the IIR scheme. For example, the HDR processing unit 350c may perform Fig.13 Operation S1200 and Fig.14 In the operation S1210, operation S1220 and operation S1230. For another example, the HDR processing unit 350c may perform Fig.13 Operation S1200 and Fig.17 Operation S1240, operation S1250, operation S1260, operation S1270 and operation S1280 in the embodiment.

[0135] In some example embodiments, the HDR processing unit 350c may be implemented with an IIR filter. When merging the plurality of frame images FIMG based on the IIR scheme, the filter coefficient of the IIR filter may be changed based on the exposure time of the plurality of frame images FIMG (eg, based on the exposure time information EINF).

[0136] The second frame buffer 360c receives and stores a plurality of HDR images HIMG. The second frame buffer 360c may be used as an accumulation buffer for sequentially storing a plurality of HDR images HIMG. In the HDR processor 300c operating based on the IIR scheme, the number of frame buffers (e.g., the second frame buffer 360c) used as an accumulation buffer may be fixed to one regardless of the number of exposure times included in the plurality of frame images FIMG.

[0137] As described above, the number of input buffers and the number of accumulation buffers in the HDR processor 300c operating based on the IIR scheme are always fixed, so regardless of the number of exposure times included in the multiple frame images FIMG, the total number of frame buffers included in the HDR processor 300c can be fixed to 2.

[0138] Will combine Fig.15 The example in Fig.16 Operation of the HDR processor 300c in.

[0139] The first frame buffer 310c may receive the frame image FIMG31, the HDR processing unit 350c may output the frame image FIMG31 as the HDR image HIMG31 without performing any image processing, and the second frame buffer 360c may receive the HDR image HIMG31. The first frame buffer 310c may receive the frame image FIMG32, the previously stored frame image FIMG31 may be deleted or removed from the first frame buffer 310c, the HDR processing unit 350c may generate the HDR image HIMG32 by merging the HDR image HIMG31 (e.g., the frame image FIMG31) and the frame image FIMG32, the second frame buffer 360c may receive the HDR image HIMG32, and the previously stored HDR image HIMG31 may be deleted or removed from the second frame buffer 360c. Similarly, the frame images FIMG33, FIMG34, FIMG35, and FIMG36 may be sequentially received, and the HDR images HIMG33, HIMG34, HIMG35, and HIMG36 may be sequentially generated.

[0140] The HDR processing unit 350c may include a motion extractor 352c, an image shifter 354c, and an image mixer 356c. Fig.16 The motion extractor 352c, the image shifter 354c and the image mixer 356c in the embodiment can be respectively connected with Fig. 9 The motion extractor 352a, image shifter 354a and image mixer 356a in are basically the same.

[0141] Fig.17 It is shown Fig.13 Flowchart of another example of generating an HDR image in . Fig.18 Is used to describe Fig.17 The operations for generating HDR images in the diagram will be omitted. Fig.14 and Fig.15 Description of duplicate description.

[0142] Fig.17 and Fig.18 An example is shown in which the number of different exposure times is three. Fig.18 In the process, a plurality of frame images FIMG41, FIMG42, FIMG43, FIMG44, FIMG45, FIMG46... can be provided from an image sensor or a storage device, etc. Frame images FIMG41 and FIMG44 with a first exposure time L, frame images FIMG42 and FIMG45 with a second exposure time M, and frame images FIMG43 and FIMG46 with a third exposure time S can be repeated alternately.

[0143] refer to Fig.13 , Fig.17 and Fig.18 As described with reference to operation S1100, a starting frame image or a first frame image FIMG41 arranged at the first position among a plurality of frame images FIMG41 to FIMG46 may be output as a starting HDR image HIMG41 among a plurality of HDR images HIMG41, HIMG42, HIMG43, HIMG44, HIMG45, HIMG46, . . . without performing any image processing.

[0144] Thereafter, as described with reference to operation S1200, when the remaining HDR images HIMG42, HIMG43, HIMG44, HIM45 and HIMG46 other than the starting HDR image HIMG41 among the multiple HDR images HIMG41, HIMG42, HIMG43, HIMG44, HIM45 and HIMG46 are generated, the HDR image as the current output can be generated by merging the previously generated and output HDR image with the currently input frame image using the IIR scheme.

[0145] For example, in operation S1240, a first HDR image HIMG42 may be generated by merging a first frame image FIMG41 having a first exposure time L and a second frame image FIMG42 having a second exposure time M. Fig.17 Operation S1240 in the above example may be performed with Fig.14 The operation S1210 in is basically the same.

[0146] In operation S1250 , a second HDR image HIMG43 subsequent to the first HDR image HIMG42 may be generated by merging the first HDR image HIMG42 with a third frame image FIMG43 having a third exposure time S subsequent to the second frame image FIMG42 .

[0147] Similarly, in operation S1260 , a third HDR image HIMG44 subsequent to the second HDR image HIMG43 may be generated by merging the second HDR image HIMG43 with a fourth frame image FIMG44 having the first exposure time L subsequent to the third frame image FIMG43 .

[0148] In operation S1270 , a fourth HDR image HIMG45 subsequent to the third HDR image HIMG44 may be generated by merging the third HDR image HIMG44 with a fifth frame image FIMG45 having the second exposure time M subsequent to the fourth frame image FIMG44 .

[0149] In operation S1280 , a fifth HDR image HIMG46 subsequent to the fourth HDR image HIMG45 may be generated by merging the fourth HDR image HIMG45 with a sixth frame image FIMG46 having a third exposure time S subsequent to the fifth frame image FIMG45 .

[0150] As described above, when an HDR image is generated by merging frame images based on the IIR scheme, the number of frame buffers required for image processing operations can be fixed to a constant regardless of the number of exposure times included in a plurality of frame images. Fig.17 and Fig.18 The method described can also be Fig.16 The HDR processor 300c is executed.

[0151] In reference Figures 13 to 18 In the described image processing method and image processing device, HDR images can be generated for all frame images based on the IIR scheme, so HDR video images or HDR images included in HDR video images can be effectively generated without losing frame rate. In the IIR scheme, video HDR can be implemented using only the current input frame and the HDR frame output immediately before the current input frame, and the IIR filter coefficients can be changed based on the exposure time of the input frame. Compared with the sliding window scheme, only two frame buffers can be used in the IIR scheme, so there may be an additional advantage of faster implementation of video HDR without frame delay.

[0152] Although reference Figures 1 to 18 The exemplary embodiment is described based on an example in which a plurality of frame images as input images have a specific number of exposure times and a specific exposure time pattern, but the exemplary embodiment is not limited thereto. For example, the plurality of frame images may have four or more exposure times, and the exposure time pattern included in the plurality of frame images may be changed.

[0153] Fig.19 is a block diagram showing an image processing apparatus according to an example embodiment. Figure 8 Description of duplicate description.

[0154] refer to Fig.19 The image processing apparatus 100a includes a first HDR processor 301 and a second HDR processor 303. The image processing apparatus 100a may further include a 3A processor 200, an image signal processor 400, a first switch 250 and a second switch 450.

[0155] In addition to the image processing apparatus 100a including two HDR processors 301 and 303 and also including two switches 250 and 450, Fig.19 The image processing device 100a in Figure 8 The image processing apparatus 100 in FIG. 1 is basically the same.

[0156] The first HDR processor 301 may be based on a reference Figures 1 to 7B , Fig.10 and Fig.11 For example, the first HDR processor 301 may be Fig. 9 HDR Processor 300a or Fig.12 The second HDR processor 303 can be based on the reference Figures 13 to 15 , Fig.17 and Fig.18 For example, the second HDR processor 303 may be Fig.16 HDR Processor 300c.

[0157] The first switch 250 and the second switch 450 may select one of the first HDR processor 301 and the second HDR processor 303 based on the mode signal MS. For example, when the mode signal MS has a first logic level (e.g., a logic low level), the first switch 250 and the second switch 450 may be connected to the first HDR processor 301, and the image processing apparatus 100a may generate a plurality of HDR images HIMG based on a sliding window scheme. When the mode signal MS has a second logic level (e.g., a logic high level), the first switch 250 and the second switch 450 may be connected to the second HDR processor 303, and the image processing apparatus 100a may generate a plurality of HDR images HIMG based on an IIR scheme.

[0158] As will be appreciated by those skilled in the art, the exemplary embodiments of the present invention may be implemented as systems, methods, computer program products, and / or computer program products implemented in one or more computer-readable media on which computer-readable program codes are implemented. The computer-readable program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be any tangible medium that may contain or store a program used by an instruction execution system, device, or apparatus or a program used in association with an instruction execution system, device, or apparatus. For example, the computer-readable medium may be a non-transitory computer-readable medium.

[0159] Fig. 20 is a block diagram illustrating an electronic system according to example embodiments.

[0160] refer to Fig. 20, the electronic system 1000 includes a processor 1010, a connection (connectivity) 1020, a storage device 1030, a user interface 1040, an image pickup device 1050 and an image processing device 1060. Although not in Fig. 20 Although not shown in FIG. 1 , the electronic system 1000 may further include a power supply.

[0161] The processor 1010 may perform various computational functions, such as specific calculations and tasks. The connection 1020 may communicate with external devices. The storage device 1030 may be used as a data storage or working memory for storing data processed by the processor 1010. The user interface 1040 may include at least one input device such as a keypad, a button, a microphone, and a touch screen, and / or at least one output device such as a speaker or a display device. The power supply may provide power to the electronic system 1000.

[0162] The image pickup device 1050 and the image processing device 1060 are controlled by the processor 1010. The image pickup device 1050 generates a plurality of frame images with different exposure times. For example, the image pickup device 1050 may include a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor, etc. The plurality of frame images may be directly provided to the image processing device 1060, or the plurality of frame images may be stored in the storage device 1030 and provided to the image processing device 1060. The image processing device 1060 may be Figure 8 The image processing device 100 and Fig.19 One of the image processing devices 100a in the image processing apparatus 100a, and can be based on the reference Figures 1 to 19 In some example embodiments, the image processing device 1060 may be included in the processor 1010 .

[0163] The exemplary embodiments of the present invention can be applied to various electronic devices and systems including image processing devices and systems. For example, the exemplary embodiments of the present invention can be applied to systems such as mobile phones, smart phones, tablet computers, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, portable game consoles, music players, portable cameras, video players, navigation devices, wearable devices, Internet of Things (IoT) devices, Internet of Things (IoE) devices, e-book readers, virtual reality (VR) devices, augmented reality (AR) devices, and robotic devices.

[0164] The foregoing is an illustration of example embodiments and should not be construed as limiting thereof. Although several embodiments have been described, those skilled in the art will readily appreciate that many modifications may be made in the example embodiments without materially departing from the novel teachings and advantages of the example embodiments. Therefore, all such modifications are intended to be included within the scope of the example embodiments as defined in the claims. Therefore, it should be understood that the foregoing is an illustration of various example embodiments and is not to be construed as being limited to the specific example embodiments disclosed, and modifications of the disclosed example embodiments as well as other example embodiments are intended to be included within the scope of the appended claims.

Claims

1. A device for processing a video image comprising a plurality of frame images, the device comprising: a first high dynamic range processor configured to combine the plurality of frame images based on a sliding window scheme; a second high dynamic range processor configured to combine the plurality of frame images based on an infinite impulse response scheme; a first switch configured to provide the plurality of frame images to the first high dynamic range processor or the second high dynamic range processor based on a mode signal; a second switch configured to provide an output of the first high dynamic range processor or an output of the second high dynamic range processor as a plurality of high dynamic range images based on the mode signal, The plurality of frame images include frame images with a first exposure time, frame images with a second exposure time, and frame images with a third exposure time that are repeated alternately. wherein the first exposure time, the second exposure time, and the third exposure time are different from each other, Wherein, the first high dynamic range processor is configured as: outputting a first frame image as a first-first high dynamic range image without performing any image processing on the first frame image, the first frame image being at the first position of the plurality of frame images, the first frame image having the first exposure time, generating a first-second high dynamic range image based on the first frame image and the second frame image, wherein the second frame image follows the first frame image in time among the plurality of frame images, and the second frame image has the second exposure time, generating first to third high dynamic range images based on the first frame image, the second frame image and the third frame image, wherein the third frame image follows the second frame image in time among the plurality of frame images, and the third frame image has the third exposure time, generating first to fourth high dynamic range images based on the second frame image, the third frame image and the fourth frame image, wherein the fourth frame image follows the third frame image in time among the plurality of frame images, and the fourth frame image has the first exposure time, The first to fifth high dynamic range images are generated based on the third frame image, the fourth frame image and the fifth frame image, wherein among the plurality of frame images, the fifth frame image is immediately after the fourth frame image in time, and the fifth frame image has the second exposure time, Wherein, the second high dynamic range processor is configured as: outputting the first frame of image as a second-first high dynamic range image without performing any image processing on the first frame of image, generating a second-second high dynamic range image based on the second-first high dynamic range image and the second frame image, generating a second-third high dynamic range image based on the second-second high dynamic range image and the third frame image, generating second to fourth high dynamic range images based on the second to third high dynamic range images and the fourth frame image, Second to fifth high dynamic range images are generated based on the second to fourth high dynamic range images and the fifth frame image.

Images