Image fusion method and data processing device

Abstract

The present application discloses a method and data processing device for fusing images, wherein the method includes: extracting feature points included in an input image and generating matching information indicating a mapping relationship between feature points included in different input images; selecting a reference image from the input image based on the matching information; determining a valid image from the input image by excluding a noise image from the input image based on the matching information; performing two-dimensional bundle adjustment to generate a synchronized image by aligning the valid image to the reference image; and generating a fused image by fusing the reference image and the synchronized image. Image fusion performance is improved by selecting a reference image that is highly correlated with other input images and estimating an accurate homography based on the reference image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from Korean Patent Application No. 10-2020-0052115 filed on April 29, 2020, in the Korean Intellectual Property Office (KIPO), the subject matter of which is hereby incorporated by reference. Technical Field

[0003] Embodiments of the present inventive concept generally relate to image processing, and more particularly, to a method for fusing images and a data processing device capable of performing the method. Background Art

[0004] In the field of computer vision, images are fused using corresponding points between images. The fused images can be applied to various image processing techniques, such as image stitching, noise reduction, dynamic range conversion, etc. Image fusion can be performed by three-dimensional (3D) bundle adjustment, which optimizes camera parameters (e.g., rotation, movement, etc.) for multiple cameras. According to 3D bundle adjustment, a 3D point group shared by multiple images can be used to optimize camera parameters. When two-dimensional (2D) images are fused, it is difficult to obtain a 3D point group because the camera motion is not significant. 3D bundle adjustment is not suitable for fusing 2D images and requires homography between 2D images instead of camera parameters. Summary of the Invention

[0005] Embodiments of the inventive concept provide a method capable of efficiently fusing 2D images and a data processing device capable of executing the method.

[0006] In a method for fusing images according to an embodiment of the present invention, feature points included in input images are extracted, and matching information indicating a mapping relationship between feature points included in different input images is generated. A reference image is selected from the input images based on the matching information. A valid image is determined from the input images by excluding a noise image from the input images based on the matching information. Two-dimensional bundle adjustment is performed to generate a synchronized image by registering the valid image to the reference image. A fused image is generated by fusing the reference image and the synchronized image.

[0007] According to an embodiment of the present invention, a method for fusing images includes: extracting feature points included in an input image and generating matching information indicating a mapping relationship between feature points included in different input images; determining, for each of the input images, a correlation value between the feature points of each input image and feature points of other input images in the input images based on the matching information; selecting an input image with the highest correlation value as a reference image; determining a valid image from the input images by excluding a noise image from the input images based on the matching information; determining a cost function based on the matching information so that coordinates of the feature points of the reference image and a homography matrix indicating a transformation between the reference image and the valid image are set as independent parameters of the cost function; determining an optimal homography matrix and optimal feature points corresponding to a minimum value of the cost function by adjusting the feature points and the homography matrix of the reference image; determining a correction matrix indicating a transformation between the feature points of the reference image and the optimal feature points; generating a synchronized image by aligning the feature points of the valid image to a two-dimensional plane corresponding to the reference image using the correction matrix and the optimal homography matrix; and generating a fused image by fusing the reference image and the synchronized image.

[0008] According to an embodiment of the present invention, a data processing device includes: a feature point generator, a reference image selector, an image filter, a synchronizer, and an image fusion device. The feature point generator extracts feature points included in an input image and generates matching information indicating a mapping relationship between feature points included in different input images. The reference image selector selects a reference image from the input image based on the matching information. The image filter determines a valid image from the input image by excluding a noise image from the input image based on the matching information. The synchronizer performs two-dimensional bundle adjustment to generate a synchronized image by aligning the valid image to the reference image. The image fusion device generates a fused image by fusing the reference image and the synchronized image.

[0009] According to an embodiment of the inventive concept, a method for fusing images and a data processing device for performing the method can improve image fusion performance by selecting a reference image that is highly correlated with other input images and estimating accurate homography based on the reference image.

[0010] In addition, the method for fusing images and the data processing device for executing the method according to the embodiments of the present invention can improve image fusion performance by removing noise images from input images, optimizing the homography matrix and / or homography improvement through two-dimensional bundle adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the inventive concept will be more clearly understood upon consideration of the following detailed description taken in conjunction with the accompanying drawings.

[0012] Figure 1 is a flowchart summarizing a method of fusing images according to an embodiment of the inventive concept.

[0013] Figure 2 is a block diagram illustrating an image processing apparatus according to an embodiment of the inventive concept.

[0014] Figure 3 is a block diagram illustrating a system according to an embodiment of the inventive concept.

[0015] Figure 4A and Figure 4B is a conceptual diagram illustrating a camera arrangement that can provide input images that are later fused according to an embodiment of the inventive concept.

[0016] Figure 5 is an image illustrating a method for extracting feature points used in a method of fusing images according to an embodiment of the inventive concept.

[0017] Figure 6A 、 Figure 6B 、 Figure 6C 、 Figure 6D and Figure 6E is an image illustrating a set of input images that may be used in conjunction with embodiments of the inventive concept.

[0018] Figure 7 is a conceptual diagram illustrating an example of matching information that may be used by a method of fusing images according to an embodiment of the inventive concept.

[0019] Figure 8 is a flowchart summarizing a method of determining a reference image according to an embodiment of the inventive concept.

[0020] Figure 9A 、 Figure 9B 、 Figure 9C and Figure 9D is a schematic diagram illustrating image transformation that may be used by a method of fusing images according to an embodiment of the inventive concept.

[0021] Figure 10 is a set of mathematical expressions describing homography matrices that may be used by a method of fusing images according to an embodiment of the inventive concept.

[0022] Figure 11 、 Figure 12 and Figure 13 is a conceptual diagram further illustrating a method of determining a reference image according to an embodiment of the inventive concept.

[0023] Figure 14 is a conceptual diagram illustrating an example of a reference image and a valid image determined by a method of fusing images according to an embodiment of the inventive concept.

[0024] Figure 15 is a flowchart summarizing a method of determining a valid image according to an embodiment of the inventive concept.

[0025] Figure 16 and Figure 17 is a conceptual diagram describing two-dimensional bundle adjustment of a method for fusing images according to an embodiment of the inventive concept.

[0026] Figure 18 and Figure 19 is a graph illustrating exemplary results of a method of fusing images according to an embodiment of the inventive concept. DETAILED DESCRIPTION

[0027] Certain embodiments of the inventive concept will be described in some additional detail with reference to the accompanying drawings. Throughout the written description and drawings, like reference numerals and labels are used to indicate like or similar elements and / or features.

[0028] Figure 1 is a flowchart summarizing a method of fusing images according to an embodiment of the inventive concept.

[0029] First, feature points included in an input image are extracted, and matching information indicating a mapping relationship of feature points respectively included in different input images is generated ( S100 ).

[0030] Here, the input images may be a set of related images to be fused. For example, the input images may include at least a portion of the same scene.

[0031] In some embodiments, as will be described below with reference to Figure 4A As described above, the input images may be images captured by multiple cameras respectively. Alternatively, the input images may be images captured sequentially by a single camera.

[0032] Various methods can be used to extract feature points and generate matching information. Figure 5 、 Figures 6A to 6E and Figure 7 Examples of possible methods that may be used to extract feature points and generate matching information are described.

[0033] A reference image may be selected from the input image based on the matching information ( S200 ).

[0034] Here, a correlation value may be determined for each of the input images. This may be done, for example, using feature points from the input images, and feature points of other input images may be determined based on the matching information. Once a set of correlation values ​​has been determined for the input images, the input image with the largest (or highest) correlation value may be selected as a reference image.

[0035] In various embodiments of the present inventive concept, different methods can be used to determine the corresponding correlation values ​​of the input images. For example, each correlation value can be determined based on the Euclidean distance between two reprojection points, the two reprojection points being obtained by registering two corresponding feature points respectively included in the two input images to the same two-dimensional plane.

[0036] Will refer to Figure 8 、 9A to 9D 、 Figure 10 、 Figure 11 、 Figure 12 、 Figure 13 and Figure 14 Examples of possible methods are described that may be used to determine correlation values ​​that may later be used to select a reference image from an input image.

[0037] A valid image may be determined from the input image (S300). This may be accomplished, for example, by excluding noise images from the input image based on the matching information.

[0038] In some embodiments, a reference correlation value between a feature point of a reference image and a feature point of each input image may be determined. Then, by using the reference correlation value, it may be determined whether each input image is a noise image or a valid image.

[0039] Here, for example, the reference correlation value may be determined based on the Euclidean distance between two reprojection points obtained by registering two corresponding feature points respectively included in the reference image and each input image to the same two-dimensional plane.

[0040] Will refer to Figure 15 A method for removing a noisy image from an input image is described with some additional detail.

[0041] A two-dimensional bundle adjustment may be performed to generate a synchronized image (S400). This may be accomplished, for example, by registering the active image with the reference image.

[0042] In some embodiments, a cost function value may be determined based on the matching information, and the valid image may be registered with the reference image such that the cost function value is minimized.

[0043] In some embodiments, the cost function may be a reprojection error function corresponding to the sum of Euclidean distances between feature points of the reference image and reprojected points obtained by registering feature points of the valid image to a two-dimensional plane corresponding to the reference image.

[0044] Will refer to Figure 16 and Figure 17 Methods that can be used to perform two-dimensional bundle adjustment are described in some additional detail.

[0045] A fused image may be generated by fusing a reference image and the synchronized image ( S500 ).

[0046] In some embodiments, a fused image may be generated by mixing a reference image and a synchronized image. For example, the mixing of images may be performed using a graph cut segmentation algorithm, a seam cut algorithm, and a multi-band blending algorithm.

[0047] In some embodiments, image stitching can be performed using a reference image and synchronized images to generate a fused image. Due to registration errors or exposure deviations between the images, the boundaries of the stitched portion may be sharpened during the stitching process, and image blending can be performed to smooth the sharpened boundaries. Here, boundary issues of the stitched portion can be minimized, and an accurate fused image can be obtained using synchronized images that are registered to or coordinate-synchronized with the reference image.

[0048] In some embodiments, a high dynamic range (HDR) fused image may be generated using the reference image and the synchronized image.

[0049] The fusion of the reference image and the synchronized image may be performed using various methods (eg, image stitching, noise reduction, HDR conversion, etc.).

[0050] In some embodiments, image fusion can be efficiently performed using matching information that indicates mapping relationships and homographies between feature points, regardless of the 3D point groups used in traditional 3D bundle adjustment.

[0051] Therefore, the method for fusing images and the data processing device for executing the method according to the embodiments of the present invention can improve the image fusion performance by selecting an input image that is highly correlated with other input images as a reference image and estimating accurate homography based on the reference image.

[0052] In addition, the method for fusing images according to an embodiment of the present invention and the data processing device capable of executing the method can improve image fusion performance by removing noise images from input images and optimizing the homography matrix and / or homography improvement using two-dimensional bundle adjustment.

[0053] Figure 2 is a block diagram illustrating an image processing apparatus 10 according to an embodiment of the inventive concept.

[0054] refer to Figure 2 The image processing device 10 may include: a feature point generator (KPG) 100, a reference image selector (RIS) 200, an image filter (IFLT) 300, a synchronizer (SYNC) 400, an image fusion generator (IMRG) 500 and a memory (MEM) 600.

[0055] The feature point generator 100 can be used to extract feature points (or key points) from the set of input images Io-Im and generate matching information indicating the mapping relationship between feature points included in different input images. The feature point generator 100 can provide the matching information to the reference image selector 200 and the synchronizer 400. Here, the matching information can also be provided to the image fuser 500 for use during image fusion.

[0056] In this regard, reference will be made to Figure 7 One possible approach for defining matching information (MINF) used in conjunction with certain embodiments of the present inventive concept is described with some additional details.

[0057] The reference image selector 200 may select a reference image (Ir) from the input image based on the matching information, generate reference image information (RINF) indicating the selected reference image, and provide the reference image information to the image filter 300 , the synchronizer 400 , and the memory 600 .

[0058] In one possible approach, the reference image information may include an image number (or image index) associated with the reference image.

[0059] The image filter 300 can be used to distinguish valid images (Io-In) and noise images from the input image based on the matching information. In this regard, for example, the image filter 300 can generate valid image information (VINF) and provide the valid image information to the synchronizer 400 and the memory 600.

[0060] In one possible method, the valid image information may include an image number (or image index) associated with a valid image in the input images.

[0061] The synchronizer 400 may be configured to perform two-dimensional bundle adjustment (2DBA) to generate a synchronized image by registering the valid image with the reference image. In some embodiments, the synchronizer 400 may provide the bundle adjustment homography matrix (H′ro ˜H′rn) and the correction matrix (Hcor) corresponding to the result of the two-dimensional bundle adjustment to the image fusion unit 500. Figure 16 and Figure 17 Examples of bundle adjustment homography matrices and correction matrices are described with some additional detail.

[0062] The image fuser 500 may generate a fused image (Ic) by fusing the reference image and the synchronized image. The image fuser 500 may generate a synchronized image by registering (or coordinate synchronizing) the valid image with the reference image based on the result of two-dimensional bundle adjustment (e.g., the bundle adjustment homography matrix and the correction matrix). The image fuser 500 may use the reference image and the synchronized image to perform various image processing, such as image stitching, noise reduction, and HDR conversion, as described above.

[0063] The memory 600 can receive and store images provided from an external device (e.g., a camera). In this regard, the memory 600 can provide a set of input images to be fused to the feature point generator 100. The memory 600 can also provide a reference image and a valid image to the image fusion unit 500 based on the reference image information and the valid image information. The memory 600 can be a memory device dedicated to the data processing device 10 or a memory device commonly accessed by components of the system including the data processing device 10.

[0064] Figure 3 is a block diagram illustrating a system 1000 according to an embodiment of the inventive concept.

[0065] refer to Figure 3 , the system 1000 may include a camera module (CAM) 1114 , a transceiver (TRX) 1140 , a control unit 1160 , and a user interface 1150 .

[0066] The camera module 1114 may include at least one camera or image sensor configured to capture and provide an input image. The camera module 1114 may include multiple cameras, each of which provides one or more of the input images. Alternatively, the camera module 1114 may include a single camera that provides an input image.

[0067] The transceiver 1140 may provide any connections required by the system 1000. These connections may include wired and / or wireless links to other networks (eg, the Internet, cellular networks, etc.).

[0068] The user interface 1150 may include an input device (KPD) 1152 (e.g., a keyboard, keypad, etc.) and a display device (DSP) 1112 capable of displaying images captured by the camera module 1114. If suitable for a particular design, a virtual keypad may be integrated into the display device 1112 with touch screen / sensor technology to avoid the input device 1152.

[0069] The control unit 1116 may include a general purpose processor (PRC) 1161, a hardware device (HW) 1162, a firmware device (FW) 1163, a memory (MEM) 1164, an image signal processor (ISP) 1166, a graphics engine (GENG) 1167, and a bus 1177. The control unit 1160 may execute the various methods for fusing images described herein. That is, the control unit 1160 may be configured to execute, for example, Figure 2 The functions of the data processing device 10 are described.

[0070] Here, it should be noted that embodiments of the inventive concept may be variously implemented in hardware, firmware, and / or software.

[0071] In some embodiments, the image fusion method according to an embodiment of the present inventive concept can be executed using the image signal processor 1166. In some embodiments, the image fusion method according to an embodiment of the present inventive concept can be executed according to program instructions executed by a processing device. The program instructions can be stored as software 1165 in the memory 1164, and the program instructions can be executed by the general processor 1161 and / or the image signal processor 1166.

[0072] To execute the program instructions, for example, the general purpose processor 1161 may retrieve or obtain the program instructions from an internal register, an internal cache, or the memory 1164, and decode and execute the program instructions. During or after executing the program instructions, the general purpose processor 1161 may write one or more results (which may be intermediate results or final results) of the program instructions to the internal register, the internal cache, or the memory 1164.

[0073] System 1000 may be a computer system that takes one of many possible forms. For example, system 1000 may be an embedded computer system, a system on a chip (SOC), a single-board computer system (SBC) (e.g., a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system network, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these.

[0074] Program instructions for implementing the method for fusing images according to an embodiment of the present invention may be stored in a computer-readable non-transitory storage medium. Where appropriate, the computer-readable non-transitory storage medium may include one or more semiconductor-based integrated circuits (ICs) or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these items. Where appropriate, the computer-readable non-transitory storage medium may be a volatile storage medium, a non-volatile storage medium, or a combination of a volatile storage medium and a non-volatile storage medium.

[0075] Figure 4A and Figure 4B are corresponding conceptual diagrams illustrating a method for capturing input images that can be fused later in various embodiments of the present inventive concept.

[0076] Figure 4A An example of a camera array including a first camera CAM1 and a second camera CAM2 arranged along an axis (AX) is shown. The above-mentioned input image may include images I1 and I2 captured by the cameras CAM1 and CAM2, respectively. For ease of explanation, in Figure 4A Only two (2) cameras, CAM1 and CAM2, are shown in FIG. However, those skilled in the art will appreciate that three or more cameras may be used in other embodiments.

[0077] In some embodiments, each of cameras CAM1 and CAM2 may include an image sensor configured to capture a single image or a sequence of images (e.g., a video). For example, cameras CAM1 and CAM2 may include a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) active pixel image sensor.

[0078] Each camera in the camera array has a specific field of view (FOV) that depends on various factors such as: relative camera position, focal length, magnification used, camera size, etc. Figure 4A As shown, the first camera CAM1 has a first field of view FOV1, and the second camera CAM2 has a second field of view FOV2 different from the first field of view FOV1.

[0079] In this regard, the field of view of a camera may refer to the horizontal extent, vertical extent, or diagonal extent of a particular scene imaged by the camera. Objects within the field of view of a camera may be captured by the camera's image sensor, and objects outside the field of view may not appear on the image sensor.

[0080] A camera can have an orientation that represents the angle or direction the camera is pointing. Figure 4A As shown, the camera CAM1 may have a first orientation ORT1 and the camera CAM2 may have a second orientation ORT2 different from the first orientation ORT1.

[0081] Depending on the spacing ICS between the cameras, the fields of view FOV1 and FOV2, and the orientations ORT1 and ORT2, the overlap of the first image I1 and the second image I2 captured by the cameras CAM1 and CAM2, respectively, may change. Therefore, before efficiently fusing the images I1 and I2, it is necessary to synchronize or coordinate-synchronize the images I1 and I2 by aligning them to the same two-dimensional plane.

[0082] like Figure 4B As shown, the input images fused by the embodiments of the present inventive concept may include a first image I1 and a second image I2 captured sequentially by a single camera (CAM). For example, images I1 and I2 may be images captured in a sequential capture mode or images that are oversampled to improve image quality. In these cases, there may be a time interval between images I1 and I2, and the overlapping portion of images I1 and I2 may change due to the user's hand movement, etc. Figure 4A As in the case of , before efficiently fusing images I1 and I2, they need to be synchronized (or coordinate synchronized) by registering them to the same two-dimensional plane.

[0083] Figure 5 1 are images illustrating some methods that may be used to extract feature points in a method of fusing images according to an embodiment of the inventive concept.

[0084] Image matching can be achieved by extracting feature points from the images to be fused. Feature points can be understood as key points or points of interest.

[0085] In order to match corresponding image parts, it is necessary to extract appropriate feature points that can be easily identified (or detected) from the image background and distinguished from the image background. For example, conditions associated with appropriate feature points may include: high recognition even when the shape and / or position of the object, camera parameters, illumination, etc. change. An example of an appropriate feature point is a corner point, but many different methods can be used. However, most feature point extraction methods are based on corner point extraction, for example, in

[15] ,

[16] ,

[17] ,

[18] ,

[19] ,

[20] ,

[21] ,

[22] ,

[23] ,

[24] ,

[25] ,

[26] ,

[27] ,

[28] ,

[29] ,

[30] ,

[31] ,

[32] ,

[33] ,

[34] ,

[35] ,

[36] ,

[37] ,

[38] ,

[39] ,

[40] ,

[41] ,

[42] ,

[43] ,

[44] ,

[45] ,

[46] ,

[47] ,

[48] ,

[49] ,

[50] ,

[51] ,

[52] ,

[53] ,

[54] ,

[55] ,

[56] ,

[57] ,

[58] ,

[59] ,

[60] ,

[61] ,

[62] ,

[63] ,

[64] ,

[65] ,

[66] ,

[67] ,

[68] ,

[69] ,

[70] ,

[71] ,

[72] ,

[73] ,

[74] ,

[75] ,

[76] ,

[77] ,

[78] ,

[79] ,

[80] ,

[81] ,

[82] ,

[83] ,

[84] ,

[85] ,

[86] ,

[87] ,

[88] ,

[89] ,

[90] ,

[91] ,

[92] ,

[93] ,

[94] , [ Figure 5Harris angle and scale-invariant feature point (SIFT) angle shown in .

[0086] In some embodiments, feature point detection and point matching can be performed on a grayscale version of an input image, and a specific contrast can be applied to the input image in a separate operation or via a lookup table. In some embodiments, feature point detection can be performed globally on the image using local contrast enhancement. Local contrast enhancement increases "local" contrast while preventing an increase in "global" contrast, thereby protecting large-scale shadow / highlight details. For example, local contrast gradients can indicate edges, corners, or "blobs" corresponding to features. Features of the image can be detected using a feature detection algorithm (e.g., Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented FAST and Rotated BRIEF (ORB), where FAST stands for "Feature from Accelerated Segment Test" and BRIEF stands for "Binary Robust Independent Elementary Features." In some embodiments, the feature point detection process can detect one or more feature points. For example, feature points can be detected by applying a difference of multiple Gaussian smoothing operations. In addition, the locations of the feature points and the contrast values ​​of each feature point for each search area can be stored.

[0087] In some embodiments, the size of the region used to match feature points in different images can be set based on the size of the images. In some embodiments, the geometry of the camera system can be known, and based on the known camera system geometry, the appropriate number of pixels in the search region and the overlap area of ​​the images can be known a priori. For example, the positions and orientations of the cameras of the camera system can be fixed relative to each other, and the overlap between the cameras can be known.

[0088] In some embodiments, a nearest neighbor search algorithm may be used to determine corresponding pairs of feature points respectively included in different input images. For example, a nearest neighbor search algorithm may identify a pattern of feature points in each search area of ​​the overlapping area of ​​one image that matches a corresponding pattern of feature points in each search area of ​​the overlapping area of ​​the other image. In some embodiments, the nearest neighbor algorithm may use a search radius around each feature point to determine the corresponding pairs of feature points. For example, the search area may have a radius of 32 pixels, a radius of 64 pixels, or any suitable radius, or the search area may have a size of 32 pixels × 32 pixels, a size of 64 pixels × 64 pixels, or any suitable size. In some example embodiments, a secondary refinement may be used to realign the corresponding pairs of feature points before the final homography is calculated.

[0089] Figure 6A 、 Figure 6B 、 Figure 6C 、 Figure 6D and Figure 6E (hereinafter Figures 6A to 6E ) is an image illustrating fusion of input images according to an embodiment of the inventive concept.

[0090] Figures 6A to 6E The first to fifth images (I0-I4) to be fused are shown as examples, and Figures 6A to 6E The extracted feature points are shown in . The choice of which image from the first to fifth images to use as the reference image, to which the other images' coordinates are synchronized, highly influences image fusion performance. Intuitively, the third image, I2, has the most feature points, but it's undesirable to select a reference image based on the number of feature points.

[0091] For each of the input images, a correlation value between a feature point of the input image and a feature point of the other input images can be determined based on the matching information. Once the correlation value for each of the input images has been determined, the input image with the largest (or highest) correlation value can be selected from the input images as a reference image. Thus, by selecting a reference image that is highly correlated with the other input images and estimating an accurate homography based on the reference image, image fusion performance can be improved.

[0092] Figure 7 is a conceptual diagram illustrating an example of matching information that may be used by a method of fusing images according to an embodiment of the inventive concept.

[0093] exist Figure 7 , an example of matching information for feature points Xij included in each of the first to fifth images (I0-I4) is shown. Here, feature points in the same row may correspond to the same position of the captured scene. For example, based on the matching information MINF, it will be understood that feature point X02 of the first image I0, feature point X11 of the second image I1, feature point X22 of the third image I2, and feature point X32 of the fourth image I3 correspond to each other, and that the fifth image I4 does not have a corresponding feature point.

[0094] Figure 8 is a flowchart summarizing, in one example, steps of a method for determining a reference image according to an embodiment of the inventive concept.

[0095] refer to Figure 8 , a homography matrix indicating a transformation between the two input images may be determined based on the matching information (S210). Here, the matching information may relate to feature points included in the two input images.

[0096] Then, the Euclidean distance between the two reprojected points may be determined ( S220 ), which may be obtained, for example, by registering two corresponding feature points respectively included in the two input images to the same two-dimensional plane using a homography matrix.

[0097] It is now possible to determine whether the two feature points are a valid pair ( S230 ). This can be done, for example, by determining whether the Euclidean distance between the two reprojected points ( S220 ) is less than a threshold distance.

[0098] For each of the input images, a correlation value may be determined ( S240 ). This may be done, for example, by determining the sum of the number of valid pairs of correspondences between feature points of each input image and feature points of other input images.

[0099] Then, a reference image may be selected by identifying an input image having a highest correlation value from among the input images ( S250 ).

[0100] In the following, reference will be made to Figure 9A 、 Figure 9B 、 Figure 9C 、 Figure 9D 、 Figure 10 、 Figure 11 、 Figure 12 、 Figure 13 and Figure 14 Describe with some additional details Figure 8 An example of the operation of the method outlined in .

[0101] Figure 9A 、 Figure 9B 、 Figure 9C and Figure 9D is an image illustrating image transformation that may be used in a method of fusing images according to an embodiment of the inventive concept.

[0102] Image transformation may indicate a function that applies a specific algorithm to an input image (eg, applies translation, rotation, scale conversion, and / or corresponding conversion to the input image) to obtain a transformed (or modified) image.

[0103] Image transformations can include, for example, Figure 9A The rigid body transformation shown is similar to Figure 9B The similarity transformation of the transformation shown, similar to Figure 9C Affine transformations of the transformations shown, and / or similar Figure 9D The projective transformation of the transformation shown. The converted or transformed coordinates (x', y') can be obtained by applying the image transformation(s) to the original coordinates (x, y).

[0104] Figure 9AA rigid transformation is a basic image transformation that fixes the magnitude of the image. In other words, only translation and rotation are performed by the rigid transformation.

[0105] Figure 9B The similarity transformation also reflects the change of the magnitude or scale of the object, and the shape of the object is preserved through the similarity transformation.

[0106] Figure 9C The affine transformation of adds the linear preservation feature to the similarity transformation. Therefore, the parallelism of lines and the ratio of lengths or distances can be preserved by the affine transformation.

[0107] Figure 9D The projection transformation is a transformation for projecting an image in a 3D space onto an image in a 2D space. The projection transformation indicates the conversion between two images observed from two different perspective points in a 3D space.

[0108] The matrix that represents the image transformation between two images is called a homography matrix or simply a homography. A relationship that represents a general projective transformation to transform a first image into a second image or vice versa can be obtained, and the relationship is a transformation matrix called a homography matrix.

[0109] In image processing, image transformations are required for two images captured from different perspectives or at different times. The required homography is 3x3 in size. A homography can only be determined given eight values. In other words, at least four pairs of corresponding points are required to determine the homography.

[0110] Figure 10 is a set of mathematical expressions describing homography matrices that may be used in a method of fusing images according to an embodiment of the inventive concept.

[0111] therefore, Figure 10 The process of obtaining a homography matrix H corresponding to a pair of points X(u, v, 1) and X′(u′, v′, 1) respectively included in two different images is shown. The homography matrix H can be determined by obtaining components h1 to H9 of the homography matrix H.

[0112] like Figure 10 As shown, the relationship HX=X′ can be transformed into the relationship Ah=0, and the component vector 'h' can be obtained. For a pair of corresponding points, the matrix A has a size of 3X9, and for n pairs of corresponding points, the matrix A has a size of 3nX9.

[0113] Figure 11 、 Figure 12 and Figure 13 is a conceptual diagram further illustrating a method of determining a reference image according to an embodiment of the inventive concept.

[0114] refer to Figure 11 , when the first input image to the fifth input image (I0~I4) are to be fused, the homography matrix Hij (i=0~4, j=0~4) corresponds to the corresponding transformation between each two different input images Ii and Ij. Hij indicates the homography matrix used to transform the coordinates in the i-th input image Ii into coordinates aligned to the two-dimensional plane corresponding to the j-th image Ij. Hji is the inverse matrix of Hij, and Figure 11 The redundant matrices are omitted.

[0115] Therefore, the homography matrix ( Figure 8 S210).

[0116] refer to Figure 12 , the Euclidean distance Dij between two reprojected points Xij and HjiX^ij can be determined using the homography matrix Hji ( Figure 8 S220), wherein the two reprojected points Xij and HjiX^ij are obtained by aligning the two corresponding feature points Xij and Xi^ij respectively included in the two input images Ii and Ij to the same two-dimensional plane (for example, Figure 12 The Euclidean distance between two reprojected points Xij and HjiX^ij can be expressed by Expression 1.

[0117] Expression 1

[0118] Dij=||Xij-HjiX^ij||

[0119] As shown in Expression 2, when the Euclidean distance Dij is shorter than the threshold distance Dth, the two feature points Xij and Xij can be determined as a valid pair ( Figure 8 S230).

[0120] Expression 2

[0121] (Xij, X^ij): Inlier, if Dij < Dth

[0122] : outlier, otherwise

[0123] Figure 13An example of the number of valid pairs between each two images in the input images (I0 to I4) and the sum of the number SNi for each image Ii is shown. Therefore, for each of the input images, a correlation value ( Figure 8 S240).

[0124] exist Figure 13 , a correlation value SN0 of the first input image I0 with respect to the other input images I1, I2, I3, and I4 is 107, a correlation value SN1 of the second input image I1 with respect to the other input images I0, I2, I3, and I4 is 116, a correlation value SN2 of the third input image I2 with respect to the other input images I0, I1, I3, and I4 is 149, a correlation value SN3 of the fourth input image I3 with respect to the other input images I0, I1, I2, and I4 is 95, and a correlation value SN4 of the fifth input image I4 with respect to the other input images I0, I1, I2, and I3 is 101. In this case, the third input image I2 having the largest correlation value (i.e., 149) can be determined as the reference image Ir. Therefore, the input image having the largest (or highest) correlation value among the input images can be determined as the reference image ( Figure 8 S250).

[0125] Therefore, the image fusion performance can be improved by selecting the reference image considering the correlation among all input images.

[0126] Figure 14 is a diagram illustrating an example of a reference image and a valid image determined by a method of fusing images according to an embodiment of the inventive concept.

[0127] exist Figure 14 In the example, the reference image (Ir) and the effective images (I0-In) have been determined according to the above method, and feature points are included in the reference image (Ir) and the effective images (I0-In). The feature point Xrj of the reference image Ir is connected to the feature point X^ij of each effective image Ii by a dotted line. Figure 16 and Figure 17 As described, two-dimensional bundle adjustment according to an embodiment of the inventive concept may be performed using initial homography matrices (Hr0Hrn) indicating transformation between a reference image and a valid image.

[0128] Figure 15 is a flowchart summarizing a method of determining a valid image according to an embodiment of the inventive concept.

[0129] refer to Figure 15, a homography matrix indicating transformation between the reference image and each input image may be determined based on matching information of feature points included in the reference image and feature points included in each input image ( S310 ).

[0130] Then, the Euclidean distance between the two reprojected points may be determined (S320). This may be accomplished, for example, by aligning two corresponding feature points respectively included in the reference image and each input image to the same two-dimensional plane using a homography matrix to obtain the two reprojected points.

[0131] It is possible to determine whether the feature points of each input image fall within a valid range (S330). This can be done, for example, when the Euclidean distance is less than a threshold distance. Feature points within the valid range can indicate that the corresponding two feature points included in the reference image and each input image, respectively, are a valid pair as described in reference expression 2.

[0132] It may be determined whether each input image is a noise image (S340). This may be done, for example, when the number of feature points of the input image that fall within a valid range is less than a threshold number.

[0133] Alternatively or additionally, it may be determined whether each input image is a valid image (S350). This may be done, for example, when the number of feature points of each input image that fall within a valid range is not less than a threshold number.

[0134] The method for determining that a feature point falls within the valid range may be the same as the method described with reference to Expression 1 and Expression 2 above.

[0135] You can refer to Figure 13 The example of describes the determination of a noisy image and / or a valid image. Figure 13 In the example of , the third input image I2 may be determined as the reference image Ir. In this case, the number of valid pairs of the first input image I0 is 37, the number of valid pairs of the second input image I1 is 41, the number of valid pairs of the fourth input image I3 is 32, and the number of valid pairs of the fifth input image I4 is 39. If the threshold number is set to 35, the fourth input image I3, whose number of valid pairs is less than the threshold number, may be determined as a noise image, and the first input image I0, the second input image I2, and the fifth input image I4, whose numbers of valid pairs are greater than the threshold number, may be determined as valid images.

[0136] Therefore, input images that are not suitable for image fusion (eg, noisy images distorted due to stains or noise) may be excluded from the set of input images to be fused.

[0137] Figure 16 and Figure 172 are corresponding diagrams describing two-dimensional bundle adjustment that may be used in a method of fusing images according to an embodiment of the inventive concept.

[0138] Figure 16 The reference image (Ir) and the effective image (I0-In) determined according to the above method are shown, and the reference image (Ir) and the effective image (IO-In) include feature points. In addition, Figure 16 The bundle adjustment homography matrix (H′ro ˜H′rn) and the bundle adjustment feature points X′rj corresponding to the result of two-dimensional bundle adjustment are shown. The bundle adjustment homography matrix indicates the transformation between the bundle adjustment feature points X′Ij of the reference image and the feature points X̂ij of the valid image.

[0139] Two-dimensional bundle adjustment according to an embodiment of the present inventive concept can be performed by determining a cost function based on the matching information and registering the effective image to the reference image so that the cost function value is minimized. Here, registering the effective image to the reference image can refer to transforming the position or coordinates of the effective image to the coordinates of a two-dimensional plane corresponding to the reference image.

[0140] In some embodiments, the cost function value may be determined according to the reprojection error function (Ereproj) represented by Expression 3.

[0141] Expression 3

[0142]

[0143] The reprojection error function Ereproj of Expression 3 corresponds to the sum of the Euclidean distances between the feature point X^ij of the reference image Ir and the reprojection point HriX^ij obtained by aligning the feature point X^ij of the effective image to the two-dimensional plane corresponding to the reference image Ir.

[0144] As shown in Expression 3, the coordinates of the feature points (Xr0~Xrk) of the reference image and the initial homography matrix indicating the transformation between the reference image and the valid image can be set as independent parameters of the cost function (e.g., the reprojection error function Ereproj).

[0145] A bundle adjustment homography matrix corresponding to the initial homography matrix when the value of the cost function becomes a minimum value and a bundle adjustment feature point corresponding to the feature point of the reference image when the value of the cost function becomes a minimum value may be determined.

[0146] In some embodiments, the bundle adjustment homography matrix and the bundle adjustment feature points may be determined by performing partial differential operations on the initial homography matrix and the feature points of the reference image. For example, the bundle adjustment homography matrix and the bundle adjustment feature points may be determined using a Gauss-Newton algorithm or a gradient descent algorithm.

[0147] like Figure 17 As shown, the bundle adjustment feature points (X′r0~X′rk) may deviate from the original feature points (Xr0~Xrk) of the reference image (Ir). Therefore, if the bundle adjustment homography matrix (H′ri) is used to transform the feature points X^ij of the effective image, an error may be caused. The error can be eliminated using a correction matrix (Hcor). In some embodiments, a correction matrix indicating the transformation between the feature points of the reference image and the bundle adjustment feature points can be determined, and a synchronized image can be generated by aligning the feature points of the effective image to a two-dimensional plane corresponding to the reference image using the correction matrix and the bundle adjustment homography matrix. The correction matrix (Hcor) indicating the transformation between the feature points (Xr0~Xrk) of the reference image (Ir) and the bundle adjustment feature points (X′r0~X′rk) can be expressed by Expression 4.

[0148] Expression 4

[0149] Xrj=HcorX′rj

[0150] Therefore, by transforming the feature points X^ij of the effective image using a matrix (HcorH′r0~HcorH′rn) which is the product of the correction matrix (Hcor) and the bundle adjustment homography matrix (H′r0~H′rn), the feature points X^ij of the effective image can be more accurately aligned or coordinate-synchronized to the two-dimensional plane corresponding to the reference image Ir.

[0151] Figure 18 and Figure 19 are graphs illustrating various exemplary results of a method of fusing images according to an embodiment of the inventive concept.

[0152] Figure 18 and Figure 19 1 and 2. The results when the method of fusing images according to an embodiment of the present inventive concept is applied to a set of test images captured using a smartphone are shown. Each test image includes five (5) input images.

[0153] exist Figure 18 , the vertical axis represents the difference (DIFF) corresponding to the structural similarity index (SSIM) obtained by the method according to the embodiment of the present inventive concept and the SSIM obtained by the conventional method for comparison. The SSIM can be represented by Expression 5.

[0154] Expression 5

[0155]

[0156] In Expression 5, 'x' and 'y' indicate coordinates, μx indicates the average value of x, μy indicates the average value of y, σ2x indicates the dispersion of x, σ2y indicates the dispersion of y, and c1 and c2 are variables for stabilizing the denominator. A higher SSIM value indicates a higher similarity to the reference image and better warping performance.

[0157] like Figure 18 As shown, this difference has positive values ​​for almost all test image sets, which means that the image fusion performance according to an embodiment of the inventive concept is better than that of the traditional scheme.

[0158] Figure 19 1 shows an image index of an input image selected as a reference image according to an embodiment of the inventive concept. Figure 19 As shown, for most of the test image sets, either the second image (index=1) or the third image (index=2) is selected as the reference image. The middle image is selected as the reference image, presumably due to user movement when the test scene is substantially stationary.

[0159] As described above, the method for fusing images and the data processing device capable of executing the method according to the embodiments of the present inventive concept can improve image fusion performance by selecting a reference image that is highly correlated with other input images and estimating accurate homography based on the reference image.

[0160] In addition, the method for fusing images according to an embodiment of the present invention and the data processing device capable of executing the method can improve image fusion performance by removing noise images from a set of input images and using two-dimensional bundle adjustment to optimize the homography matrix and / or homography improvement.

[0161] The present invention can be applied to any electronic device or system that benefits from image processing. For example, embodiments of the present invention can be applied to systems such as mobile phones, smart phones, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, camcorders, personal computers (PCs), server computers, workstations, laptop computers, digital televisions, set-top boxes, portable game consoles, navigation systems, wearable devices, Internet of Things (IoT) devices, Internet of Everything (IoE) devices, e-books, virtual reality (VR) devices, augmented reality (AR) devices, etc.

[0162] The foregoing is an illustration of certain embodiments and should not be construed as limiting the scope of the present invention. Although some embodiments of the present invention have been described, those skilled in the art will readily appreciate that many modifications may be made to the embodiments shown without departing substantially from the scope of the present invention.

Claims

1. A method for fusing images, comprising: extracting feature points from input images and generating matching information indicating a mapping relationship between feature points from different input images; selecting a reference image from the input image based on the matching information; determining a valid image from the input image by excluding a noise image based on the matching information; performing two-dimensional bundle adjustment to generate a synchronized image by registering the active image with the reference image; as well as generating a fused image by fusing the reference image and the synchronized image, Wherein, selecting the reference image from the input image based on the matching information includes: determining, for each of the input images, a correlation value between a feature point of each input image and feature points of other input images of the input images based on the matching information; and An input image having the highest correlation value among the input images is selected as the reference image.

2. The method according to claim 1, wherein The correlation value is determined based on a Euclidean distance between two reprojected points obtained by registering two corresponding feature points respectively included in two input images to the same two-dimensional plane.

3. The method according to claim 2, wherein: The correlation value is determined based on the number of corresponding points having a Euclidean distance less than a threshold distance.

4. The method according to claim 1, wherein Determining the correlation value between the feature point of each input image and the feature points of other input images in the input images includes: determining a homography matrix indicating a transformation between the two input images based on matching information of feature points included in the two input images; determining a Euclidean distance between two reprojected points obtained by registering two corresponding feature points respectively included in the two input images to the same two-dimensional plane using the homography matrix; When the Euclidean distance between the two corresponding feature points is shorter than a threshold distance, determining that the two corresponding feature points are a valid pair; and For each of the input images, a correlation value corresponding to the sum of the number of valid pairs between feature points of the each input image and feature points of other ones of the input images is determined.

5. The method according to claim 1, wherein Determining the valid image from the input image includes: determining a reference correlation value between a feature point of the reference image and a feature point of each of the input images; and Based on the reference correlation value, it is determined whether each of the input images is a noise image or a valid image.

6. The method according to claim 5, wherein: The reference correlation value is determined based on a Euclidean distance between two reprojected points obtained by registering two corresponding feature points respectively included in the reference image and each of the input images to the same two-dimensional plane.

7. The method according to claim 6, wherein: The reference correlation value is determined based on the number of corresponding feature points having a Euclidean distance less than a threshold distance.

8. The method according to claim 1, wherein Determining the valid image from the input image includes: determining a homography matrix indicating a transformation between the reference image and each of the input images based on matching information of feature points included in the reference image and feature points included in each of the input images; determining a Euclidean distance between two reprojected points obtained by registering two corresponding feature points respectively included in the reference image and each of the input images to the same two-dimensional plane using the homography matrix; determining whether a feature point of each input image in the input image falls within a valid range, wherein a corresponding Euclidean distance is less than a threshold distance in the valid range; determining that each of the input images is a noise image when the number of feature points falling within the valid range of each of the input images is less than a threshold number; and When the number of feature points of each of the input images that fall within the valid range is not less than the threshold number, each of the input images is determined to be a valid image.

9. The method according to claim 1, wherein: Performing the two-dimensional bundle adjustment includes: determining a cost function based on the matching information; and The effective image is registered to the reference image so that the value of the cost function is minimized.

10. The method according to claim 9, wherein: The cost function is a reprojection error function corresponding to the sum of Euclidean distances between feature points of the reference image and reprojection points obtained by registering feature points of a valid image to a two-dimensional plane corresponding to the reference image.

11. The method according to claim 9, wherein Determining the cost function includes: The coordinates of the feature points of the reference image and an initial homography matrix indicating the transformation between the reference image and the valid image are set as independent parameters of the cost function.

12. The method according to claim 11, wherein Registering the active image to the reference image comprises: A bundle adjustment homography matrix corresponding to the initial homography matrix when the value of the cost function is minimized and bundle adjustment feature points corresponding to the feature points of the reference image when the value of the cost function is minimized are determined.

13. The method according to claim 12, wherein: The bundle adjustment homography matrix and the bundle adjustment feature points are determined by performing a partial differential operation on the initial homography matrix and the feature points of the reference image.

14. The method according to claim 12, wherein: The bundle adjustment homography matrix and the bundle adjustment feature points are determined using a Gauss-Newton algorithm or a gradient descent algorithm.

15. The method according to claim 12, wherein: Performing the two-dimensional bundle adjustment further includes: determining a correction matrix indicating a transformation between feature points of the reference image and the bundle adjusted feature points; and The synchronized image is generated by registering feature points of the valid image to a two-dimensional plane corresponding to the reference image using the correction matrix and the bundle adjustment homography matrix.

16. The method according to claim 1, wherein The input images are images captured by multiple cameras respectively.

17. The method according to claim 1, wherein The input images are images captured sequentially by a single camera.

18. A method for fusing images, comprising: extracting feature points from input images, and generating matching information indicating a mapping relationship between feature points included in different input images; determining, for each of the input images, a correlation value between a feature point of each input image and feature points of other input images of the input images based on the matching information; Selecting an input image with a highest correlation value among the input images as a reference image; determining a valid image from the input image by excluding a noise image from the input image based on the matching information; determining a cost function based on the matching information such that coordinates of feature points of the reference image and a homography matrix indicating a transformation between the reference image and the valid image are set as independent parameters of the cost function; Determining an optimal homography matrix and optimal feature points corresponding to a minimum value of the cost function by adjusting the feature points of the reference image and the homography matrix; determining a correction matrix indicating a transformation between the feature points of the reference image and the optimal feature points; generating a synchronized image by registering feature points of the valid image to a two-dimensional plane corresponding to the reference image using the correction matrix and the optimal homography matrix; as well as A fused image is generated by fusing the reference image and the synchronized image.

19. A data processing device comprising: a feature point generator configured to extract feature points from input images and generate matching information indicating a mapping relationship between feature points included in different input images; a reference image selector configured to: select a reference image from the input image based on the matching information; an image filter configured to: determine a valid image from the input image by excluding a noise image from the input image based on the matching information; a synchronizer configured to: perform two-dimensional bundle adjustment to generate a synchronized image by registering the effective image to the reference image; as well as an image fuser configured to generate a fused image by fusing the reference image and the synchronized image, Wherein, the reference image selector determines, for each input image in the input images, a correlation value between the feature points of the input images and the feature points of other input images in the input images based on the matching information, and selects the input image with the highest correlation value among the input images as the reference image.

Images