Method of matching images to be merged and data processing device performing the method

Abstract

Each of the input images from the multiple input images is divided into multiple image blocks. A feature point map including multiple feature point blocks respectively corresponding to the multiple image blocks is generated by extracting feature points included in each of the multiple image blocks. A descriptor map including multiple descriptor blocks respectively corresponding to the multiple feature point blocks is generated by generating descriptors of the feature points included in the feature point map. Mapping information including matching relationships between the feature points included in different input images of the multiple input images is generated based on the multiple descriptor maps respectively corresponding to the multiple input images. Image merging performance can be improved by dividing the input images into multiple image blocks to increase distribution consistency of the feature points.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0082313, filed on July 3, 2020 with the Korean Intellectual Property Office (KIPO), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The example embodiments generally relate to semiconductor integrated circuits, and more specifically to a method for matching images to be merged and a data processing apparatus for performing the method. Background Technology

[0004] In the field of computer vision, corresponding points between images are used to merge them. The merged image can be applied to various image processing techniques, such as image stitching, noise reduction, and dynamic range conversion. To merge images, the image matching process obtains the mapping relationships between feature points in the multiple images to be merged. This image matching requires significant processing power, and the quality of the merged image may depend on the performance of the image matching process. Summary of the Invention

[0005] Some example embodiments may provide a method and data processing apparatus for matching images that can efficiently match and merge images.

[0006] In a method for matching images according to an example embodiment, each of a plurality of input images is divided into a plurality of image blocks. A feature point map is generated by extracting feature points included in each of the plurality of image blocks, comprising a plurality of feature point blocks corresponding to the plurality of image blocks respectively. A descriptor map is generated by generating descriptors of the feature points included in the feature point map, comprising a plurality of descriptor blocks corresponding to the plurality of feature point blocks respectively. Mapping information, including matching relationships between feature points included in different input images, is generated based on the plurality of descriptor maps corresponding to the plurality of input images.

[0007] According to an example embodiment, a method of matching images includes: dividing each of a plurality of input images into a plurality of image blocks; generating a feature point map including a plurality of feature point blocks respectively corresponding to the plurality of image blocks by extracting feature points included in each of the plurality of image blocks; generating a descriptor map including a plurality of descriptor blocks respectively corresponding to the plurality of feature point blocks by generating descriptors of the feature points included in the feature point map; storing a plurality of descriptor maps respectively corresponding to the plurality of input images in a memory device; reading out descriptors for determining a similarity from the memory device based on addresses of the descriptor blocks included in the descriptor maps corresponding to different input images among the plurality of input images; determining the similarity between the read out descriptors; and generating mapping information containing matching relationships between feature points included in the different input images based on the similarity.

[0008] According to an example embodiment, a data processing device includes a feature point map generator, a descriptor map generator, and an image matcher. The feature point map generator divides each of a plurality of input images into a plurality of image blocks and generates a feature point map including a plurality of feature point blocks respectively corresponding to the plurality of image blocks by extracting feature points included in each of the plurality of image blocks. The descriptor map generator generates a descriptor map including a plurality of descriptor blocks respectively corresponding to the plurality of feature point blocks by generating descriptors of the feature points included in the feature point map. The image matcher generates mapping information containing matching relationships between feature points included in different input images among a plurality of input images based on a plurality of descriptor maps respectively corresponding to the plurality of input images.

[0009] The method of matching images and the data processing device for performing the method according to an example embodiment can improve image merging performance by dividing input images into a plurality of image blocks to increase distribution consistency of feature points.

[0010] In addition, the method of matching images and the data processing device according to an example embodiment can also improve image merging performance by reducing processing amount and processing time for image matching using a feature point map and a descriptor map corresponding to a plurality of image blocks. BRIEF DESCRIPTION OF DRAWINGS

[0011] A more complete understanding of the present disclosure and the many attendant aspects thereof will be achieved by reference to the following detailed description considered in connection with the accompanying drawings, wherein:

[0012] Figure 1 is a flowchart illustrating a method of matching images according to an example embodiment.

[0013] Figure 2is a block diagram showing a system including an image processing apparatus according to an example embodiment.

[0014] Figure 3 is a block diagram showing a system according to an example embodiment.

[0015] Figure 4A and Figure 4B is a diagram showing an example embodiment of providing input images to be matched and merged.

[0016] Figure 5 is a diagram showing a scheme of extracting feature points used by a method of matching images according to an example embodiment.

[0017] Figure 6 is a diagram showing an example embodiment of dividing an image used by a method of matching images according to an example embodiment.

[0018] Figure 7 is a diagram showing an example embodiment of generating a feature point map and a descriptor map used by a method of matching images according to an example embodiment.

[0019] Figure 8 is a diagram showing a mapping relationship between an input image, a feature point map, and a descriptor map used by a method of matching images according to an example embodiment.

[0020] Figure 9 is a diagram showing an example search region for determining correspondence of feature points.

[0021] Figure 10 is a diagram showing an example embodiment of setting a search region according to an example embodiment.

[0022] Figure 11 is a diagram showing an example of mapping information used by a method of matching images according to an example embodiment.

[0023] Figure 12 is a flowchart showing a method of matching images according to an example embodiment.

[0024] Figure 13 is a diagram showing an example embodiment of storing a feature point map and a descriptor map used by a method of matching images according to an example embodiment.

[0025] Figure 14 and Figure 15 is a diagram showing an address assigned to a feature point map and a descriptor map according to a storage order. Figure 13

[0026] Figure 16 is a flowchart showing an example embodiment of reading data used by a method of matching images according to an example embodiment. ​

[0027] Figure 17 is a graph illustrating Figure 16 data reading.

[0028] Figure 18 is a graph illustrating an example embodiment of storing a feature point map and a descriptor map used by a method of matching images according to an example embodiment.

[0029] Figure 19 and Figure 20 is a graph illustrating addresses assigned to a feature point map and a descriptor map according to Figure 18 a storage order.

[0030] Figure 21 is a graph illustrating coordinates of a feature point used by a method of matching images according to an example embodiment.

[0031] Figure 22 is a flowchart illustrating a method of merging images according to an example embodiment.

[0032] Figure 23A to 23D is a graph illustrating image transformation used by a method of merging images according to an example embodiment.

[0033] Figure 24 is a graph for describing a homography matrix used by a method of merging images according to an example embodiment.

[0034] Figure 25 is a graph illustrating a homography matrix between a plurality of input images.

[0035] Figure 26 is a graph illustrating a process of merging images.

[0036] Figure 27 is a block diagram illustrating an electronic system according to an example embodiment. DETAILED DESCRIPTION

[0037] Various example embodiments will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all example embodiments are shown. Like reference numerals can be used to refer to like elements throughout. Repeated description can be omitted.

[0038] Figure 1 is a flowchart illustrating a method of matching images according to an example embodiment.

[0039] Referring to Figure 1 , each of a plurality of input images is divided into a plurality of image blocks (S100). Each input image can be uniformly divided such that the plurality of image blocks can have the same pixel width size and pixel height size. The division of the input images and the image blocks will be described below with reference to Figure 6 .

[0040] A feature point map including a plurality of feature point blocks respectively corresponding to the plurality of image blocks is generated by extracting feature points included in each of the plurality of image blocks (S200). Further, a descriptor map including a plurality of descriptor blocks respectively corresponding to the plurality of feature point blocks is generated by generating descriptors of the feature points included in the feature point map (S300).

[0041] In some example embodiments, the number of feature points included in each image block can be limited to a maximum block feature point number. For example, a maximum image feature point number indicating a maximum number of feature points to be extracted from each input image can be determined. The maximum block feature point number can be determined by dividing the maximum image feature point number by the number of the plurality of image blocks. The number of feature points included in each image block can be limited to the maximum block feature point number. This will be described below with reference to Figure 7 and Figure 8 The feature point map and the descriptor map are described.

[0042] Mapping information containing matching relationships between feature points included in different input images of the plurality of input images is generated based on a plurality of descriptor maps respectively corresponding to the plurality of input images (S400). The mapping information can be used to synchronize the plurality of input images, and a merged image from the plurality of input images can be generated based on the synchronized images.

[0043] The plurality of input images can be a set of related images to be merged. The plurality of input images can include at least a part of the same scene.

[0044] In some example embodiments, as will be described below with reference to Figure 4A The plurality of input images can be images captured by a plurality of cameras. In some example embodiments, as will be described below with reference to Figure 4B The plurality of input images can be images sequentially captured by a single camera.

[0045] Accordingly, the method of matching images according to example embodiments can improve image merging performance by dividing input images into a plurality of image blocks to increase distribution uniformity of feature points. Additionally or alternatively, the method of matching images according to example embodiments can also improve image merging performance by reducing processing amount and processing time for image matching using a feature point map and a descriptor map corresponding to the plurality of image blocks.

[0046] Figure 2 is a block diagram illustrating a system including an image processing apparatus according to an embodiment.

[0047] Referring to Figure 2 , the system 10 can include a data processing apparatus 20 and a memory apparatus 30.

[0048] The data processing device 20 can include a controller CTRL 100, a feature extractor 200, an image matcher 300, and an image merger MRG 400.

[0049] The controller 100 can control the overall operation of the feature extractor 200, the image matcher 300, and the image merger 400. The controller 100 can generate a signal for controlling the feature extractor 200, the image matcher 300, and the image merger 400.

[0050] The feature extractor 200 can include an image tile buffer ITBF 210, a keypoint map generator KMG 220, and a descriptor map generator DMG 230. The image tile buffer 210 can store a plurality of input images into image tile units by dividing each of the plurality of input images into a plurality of image tiles. The keypoint map generator 220 can generate a keypoint map FMP including a plurality of keypoint tiles respectively corresponding to the plurality of image tiles by extracting key points included in each of the plurality of image tiles. The descriptor map generator 230 can generate a descriptor map DMP including a plurality of descriptor tiles respectively corresponding to the plurality of keypoint tiles by generating descriptors of the key points included in the keypoint map FMP.

[0051] The image matcher 300 can generate mapping information MINF between key points of input images based on a plurality of descriptor maps respectively corresponding to the plurality of input images. The image matcher 300 can include a descriptor map buffer DMBF 310 and a matcher MTCH 320. Descriptor tiles for similarity determination can be loaded to the descriptor map buffer 310, and the matcher 320 can generate the mapping information MINF based on similarity between descriptors of the descriptor tiles loaded to the descriptor map buffer 310.

[0052] The image merger 400 can merge the plurality of input images based on the mapping information MINF to generate a merged image. As will be described below with reference to Figure 22 to 26 The image merger 400 can generate a homography matrix for conversion between different input images based on the mapping information MINF. The image merger 400 can generate a synchronized image by synchronizing the plurality of input images based on the homography matrix. The image merger 400 can perform various image processing, e.g., image stitching, noise reduction, dynamic range conversion, etc., using the synchronized image.

[0053] The memory device 30 can receive and store an image provided from an external device (e.g., a camera). The memory device 30 can provide an image set of input images to be merged to the feature extractor 200 and the image merger 400. Also, the memory device 30 can store a merged image generated by the image merger 400.

[0054] In some example embodiments, the memory device 30 can store a feature point map FMP and a descriptor map DMP corresponding to a plurality of input images, respectively. The memory device 30 can provide the stored descriptor map DMP to the image matcher 300. The image matcher 300 can determine a similarity between descriptors based on the descriptor map DMP and generate mapping information MINF based on the similarity.

[0055] In some example embodiments, the image block buffer 210 and the descriptor map buffer 310 can be implemented as a single buffer. The memory device 30 can be a memory device dedicated to the data processing device 20 or a common memory device of the system 10 including the data processing device 20.

[0056] In some example embodiments, the image block buffer 210 and the descriptor map buffer 310 can be implemented with a static random access memory (SRAM) and the memory device 30 can be implemented with a dynamic random access memory (DRAM), but example embodiments are not necessarily limited thereto.

[0057] In some example embodiments, the controller 100 can sequentially store the feature point blocks of the feature point map FMP and the descriptor blocks of the descriptor map DMP in the memory device 30 according to a predetermined storage order. In some examples, the controller can include an address generator ADDG 110. The address generator 110 can generate an access address ACCAD corresponding to the storage order and provide it to the memory device 30. Details will be described below with reference to FIGS. 4 and 5. Figure 12 to 20 The generation of the access address ACCAD corresponding to the storage order will be further described.

[0058] The data processing device 20 performing the method of matching images according to example embodiments can improve image merging performance by dividing input images into a plurality of image blocks to increase distribution consistency of feature points. Also, the data processing device 20 performing the method of matching images according to example embodiments can improve image merging performance by reducing processing amount and processing time for image matching using a feature point map and a descriptor map corresponding to a plurality of image blocks.

[0059] Figure 3 is a block diagram illustrating a system according to example embodiments.

[0060] Reference will now be made to Figure 3The system 1000 can include a camera module CAM 1114, a transceiver TRX 1140, a control unit 1160, and a user interface 1150.

[0061] The camera module 1114 can include a camera and / or an image sensor for capturing and providing images. In some example embodiments, the camera module 1114 can include a plurality of cameras for capturing a plurality of input images to be merged. In some example embodiments, the camera module 1114 can provide a plurality of input images to be merged, wherein the plurality of input images are captured by a single camera.

[0062] The transceiver 1140 can provide a connection to other networks (e.g., the Internet, a cellular network, etc.) through wired or wireless links.

[0063] The user interface 1150 can include an input device KPD 1152 (e.g., a keypad, a key pad, etc.) and a display device DIP 1112 for displaying images. In some examples, a virtual keypad or keyboard can be integrated to the display device 1112 with a touch screen / sensor, etc.

[0064] The control unit 1160 can include a general purpose processor PRC 1161, a hardware device HW 1162, a firmware device FW 1163, a memory MEM 1164, a digital signal processor DSP 1166, a graphics engine GENG 1167, and a bus 1177. The control unit 1160 can perform the method of matching and merging images as described above. In other words, the control unit 1160 can be configured to perform the functions of the data processing device 20. Figure 2

[0065] Example embodiments can be implemented as hardware, software, firmware, or a combination thereof.

[0066] In some example embodiments, the image matching method according to example embodiments can be performed by the digital signal processor 1166. For example, the data processing device 20 as described with reference to Figure 2 may be included in the digital signal processor 1166.

[0067] In some example embodiments, at least a part of the image matching method according to example embodiments can be performed by program instructions executed by a processing device. The program instructions can be stored in the memory 1164 as software SW 1165, and the program instructions can be executed by the general purpose processor 1161 and / or the digital signal processor 1166.

[0068] ​In some example embodiments, to execute program instructions, the general-purpose processor 1161 can retrieve or fetch program instructions from an internal register, an internal cache, or the memory 1164, and decode and execute the instructions. During or after execution of the program instructions, the general-purpose processor 1161 can write one or more results (which can be intermediate or final results) to the internal register, the internal cache, or the memory 1164.

[0069] The system 1000 can be a computer system of any suitable physical form factor. For example, the system 1000 can be an embedded computer system, a system on a chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a grid computing system, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. In some embodiments, the system 1000 can be a virtual machine executing on one or more physical machines.

[0070] Program instructions for implementing the method of merging images can be stored in a computer-readable non-transitory storage medium. The computer-readable non-transitory storage medium can include, where appropriate, one or more semiconductor-based integrated circuits (ICs) or other integrated circuits, such as field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, flash memory, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. The computer-readable non-transitory storage medium can be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0071] Figure 4A and Figure 4B is a diagram illustrating an example embodiment in which input images to be matched and merged are provided.

[0072] Figure 4A An example array of camera CAM1 and camera CAM2 disposed on an axis AX is illustrated. In one example, the multiple input images to be merged as described above can be image I1 and image I2 captured by camera CAM1 and camera CAM2, respectively. In Figure 4A Two cameras CAM1 and camera CAM2 are illustrated in FIG. 1, but example embodiments are not necessarily limited thereto. The multiple input images to be merged can be provided by three or more cameras.

[0073] According to example embodiments, the camera CAM1 and the camera CAM2 can include one or more image sensors configured to capture individual photo images or a sequence of images as a video. For example, the camera CAM1 and the camera CAM2 can include a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) active pixel image sensor.

[0074] A camera can have a field of view (FOV) that depends at least in part on the position, focal length or magnification of the camera and the position or size of the image sensor of the camera. As Figure 4A illustrated, the camera CAM1 can have a first field of view FOV1 and the camera CAM2 can have a second field of view FOV2 that is different from the first field of view FOV1.

[0075] The field of view of a camera can refer to the horizontal, vertical or diagonal range of a particular scene that is visible through the camera. Objects within the field of view of the camera can be captured by the image sensor of the camera and objects outside the field of view can not appear on the image sensor.

[0076] A camera can have an orientation that represents the angle or direction in which the camera is pointing. As Figure 4A illustrated, the camera CAM1 can have a first orientation ORT1 and the camera CAM2 can have a second orientation ORT2 that is different from the first orientation ORT1.

[0077] The overlapping portion of the images I1 and I2 captured by the camera CAM1 and the camera CAM2 can vary by changing the inter-camera spacing ICS, the field of view FOV1 and the field of view FOV2, and the orientation ORT1 and the orientation ORT2. Thus, synchronization or coordinate synchronization can be required by pre-aligning the images I1 and I2 to the same two-dimensional plane to efficiently merge the images I1 and I2.

[0078] As Figure 4B illustrated, the multiple input images to be merged as described above can be the images I1 and I2 captured sequentially by a single camera CAM. For example, the images I1 and I2 can be images captured in a serial capture mode or images oversampled to improve image quality. In these cases, there can be a time interval between the images I1 and I2 and the overlapping portion of the images I1 and I2 captured by the camera CAM1 and the camera CAM2 can vary due to hand motion of a user or the like. As in the case of Figure 4A , synchronization or coordinate synchronization can be required by pre-aligning the images I1 and I2 to the same two-dimensional plane to efficiently merge the images I1 and I2.

[0079] Figure 5FIG. 1 is a diagram illustrating a scheme of extracting feature points used by a method of matching images according to example embodiments.

[0080] Image matching can be achieved by extracting feature points in images to be merged. Feature points can be referred to as key points or interest points.

[0081] In matching corresponding portions of images, it can be desirable to extract discriminative feature points that can be recognized and distinguished from a background. For example, conditions of discriminative feature points can include a high degree of discrimination from a background even if a shape and position of an object, camera parameters, illumination, etc. are changed. An example of discriminative feature points is a corner point, and many feature point extraction methods are based on corner point extraction (e.g., Harris corner points and Scale-Invariant Feature Transform (SIFT) corner points shown in Figure 5

[0082] In some example embodiments, feature point detection and point matching can be performed on a grayscale version of an image, and contrast can be applied to the image in a separate operation or through a lookup table. In some example embodiments, feature point detection can be performed globally on an image using local contrast enhancement. Local contrast enhancement increases "local" contrast while at the same time preventing an increase in "global" contrast, thereby preserving large-scale shadow / highlight details. For example, a local contrast gradient can indicate an edge, a corner, or a "blob" corresponding to a feature. A feature detection algorithm (e.g., Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), or Oriented FAST and Rotated BRIEF (ORB)) can be used to detect features of an image, where FAST stands for "features from accelerated segment test," and BRIEF stands for "binary robust independent elementary features." In some example embodiments, a feature point detection process can detect one or more feature points. For example, a feature point can be detected by taking a difference of a plurality of Gaussian smoothing operations. Additionally or alternatively, a location of a feature point for each search region and a contrast value for each feature point can be stored.

[0083] In some example embodiments, a size of a region used to match feature points in different images can be set according to a size of an image. In some example embodiments, a geometry of a camera system can be known, and based on the known camera system geometry, an appropriate number of pixels for a search region and an overlapping region of images can be known a priori. For example, positions and orientations of cameras of a camera system can be fixed relative to each other, and an overlap between the cameras can be known. ​

[0084] In some example embodiments, determining the corresponding pairs of feature points respectively included in different images can be performed using a nearest neighbor search algorithm. For example, the nearest neighbor search algorithm can identify a corresponding feature point pattern within each search region of the overlapping region of one image that matches a feature point pattern within each search region of the overlapping region of another image. In some example embodiments, the nearest neighbor algorithm can use a search radius around each feature point to determine the corresponding feature point pairs. For example, the radius of the search region can be 32 pixels, 64 pixels, or any suitable radius, or the size of the search region can be 32 pixels x 32 pixels, 64 pixels x 64 pixels, or any suitable size. In some example embodiments, a secondary refinement can be used to realign the corresponding feature point pairs prior to final homography computation.

[0085] Figure 6 is a diagram illustrating an example embodiment of dividing an image used by a method of matching images according to an example embodiment.

[0086] Referring to Figure 6 , each input image Is can be divided into a plurality of image blocks In , s denotes an index of the input image, i denotes a block row index, and j denotes a block column index.

[0087] Each input image Is can be uniformly divided such that the plurality of image blocks respectively include the same number of rows and the same number of columns of pixels. In other words, the plurality of image blocks may have the same row size SZA’ and the same column size SZB’.

[0088] When the row size SZA’ and the column size SZB’ of the image blocks are determined, the number of rows N and the number of columns M of the divided image blocks can be determined by Expression 1, and each input image Is can include N*M image blocks

[0089] Expression 1:

[0090] N = SZA / SZA’

[0091] M = SZB / SZB’

[0092] In Expression 1, N denotes the number of rows of the image blocks, M denotes the number of columns of the image blocks, SZA denotes the row size of each input image, SZB denotes the column size of each input image, SZA’ denotes the row size of each image block, and SZB’ denotes the column size of each image block.

[0093] As will be described below, distribution uniformity of feature points can be improved and processing amount and processing time for merging images can be reduced by uniformly dividing input images.

[0094] Figure 7 FIG. 1 is a diagram illustrating an example embodiment of generating a feature point map and a descriptor map used by a method of matching images according to an example embodiment. Figure 8

[0095] The process of merging multiple input images can be divided into four steps. In one step, multiple feature points or keypoints such as corner points can be extracted from each input image. In another step, a descriptor can be generated for each of the extracted feature points. In another step, mapping information including matching relationships between feature points of different input images can be generated using the descriptors of the feature points. In another step, homographies between the input images can be estimated using the mapping information to generate a merged image.

[0096] Three factors in extracting feature points can affect the performance of the estimated homographies.

[0097] The first factor is extracting feature points that can correspond to feature points of the next image or the next image block. If the extracted feature points from different input images are different, the corresponding feature point pairs can be insufficient, and the performance of the estimated homographies can be degraded. Thus, in some examples, feature points that can correspond to feature points in the next image or the next image block can be described as having a "good" or "high" response. To increase the number of corresponding feature point pairs, it can be necessary to extract feature points of good response. In other words, feature repeatability can be necessary when extracting feature points.

[0098] The second factor is the distribution of the extracted feature points. If only the response of the feature points is considered, the extracted feature points can be concentrated in a certain portion of the image. In this case, the estimated homographies can be biased toward a certain region of the image.

[0099] The third factor is limiting the number of extracted feature points considering the limitations of the processing environment (e.g., image size, processing speed, etc.). The limited number of extracted feature points can have to be uniformly distributed in the entire image portion in addition to having a good response.

[0100] ​Conventionally, feature points are sufficiently extracted, and a sorting algorithm is used to select a limited number of feature points among all the extracted feature points. In this case, feature points with high responses can be extracted, but the processing amount increases and the distribution of the feature points cannot be considered. A spatial data structure scheme such as a quadtree can be used to improve the distribution uniformity. However, the image is repeatedly divided into four regions by the quadtree, and thus the processing complexity significantly increases.

[0101] According to an example embodiment, a predetermined number of feature points can be rapidly extracted using parallel processing by a digital signal processor (DSP), and the extracted feature points can have high distribution uniformity and high responses.

[0102] Figure 7 One of image blocks in an input image Is as described with reference to Figure 6 As described above, the maximum image feature point number Kmax representing the maximum number of feature points included in each input image Is can be determined by the image size, the processing speed, etc. As shown in Expression 2, the maximum block feature point number K is determined by dividing the maximum image feature point number Kmax by the number N*M of the plurality of image blocks , and the number of feature points included in each image block can be limited to the maximum block feature point number K.

[0103] Expression 2:

[0104] K = Kmax / (N*M)

[0105] Referring to Figure 7 , the number of feature points included in each image block can be determined by performing feature point extraction to determine L candidate feature points In , s denotes an input image index, i denotes a block row index, j denotes a block column index, and p denotes a point index in each image block .

[0106] The feature point block can be generated by extracting K feature points with higher responses from the L candidate feature points Here, K is the maximum block feature point number K as described above.

[0107] The K descriptors of the K feature points included in each feature point block corresponding to each image block can be generated by generating the K descriptors of the K feature points ​corresponding descriptor blocks

[0108] According to example embodiments, the descriptors can be generated based on at least one of ORB (Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)), SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and MSD (Maximal Self-Dissimilarities), but the present disclosure is not necessarily limited thereto.

[0109] Figure 8 An input image Is divided into a plurality of image blocks is shown, a plurality of feature point blocks corresponding to the plurality of image blocks is shown, and a descriptor graph DMPs including a plurality of descriptor blocks corresponding to the plurality of image blocks is shown.

[0110] Figure 2 The feature point graph generator 220 in the image processing device 200 can generate each of the plurality of feature point blocks so that each of the plurality of feature point blocks includes the same number of data bits corresponding to the feature point block size. Also, Figure 2 The descriptor graph generator 230 in the image processing device 200 can generate each of the plurality of descriptor blocks so that each of the plurality of descriptor blocks includes the same number of data bits corresponding to the descriptor block size.

[0111] The image blocks, the feature point blocks, and the descriptor blocks can correspond to each other in terms of positions on a two-dimensional plane, respectively. Here, the positions on the two-dimensional plane can correspond to a block row index i and a block column index j. As will be described below, similarity between descriptors can be determined based on the positions of the descriptor blocks on the two-dimensional plane described by the block row index i and the block column index j, regardless of the Euclidean distance between the feature points. In some example embodiments, the similarity between the descriptors can correspond to a Hamming distance between the descriptors.

[0112] Accordingly, according to example embodiments, feature points and descriptors can be generated based on blocks of the same size (i.e., the same data size). Feature points can be obtained with high distribution consistency and high response without repeatedly dividing an image space through a data scheme (e.g., a quad tree), and the number of feature points to be extracted in the entire image can be efficiently limited.

[0113] The feature point block and the descriptor block can include position information corresponding to the same two-dimensional plane as the image block. In other words, the feature point block, the descriptor block, and the image block having the same block row index i and block column index j can correspond to each other.

[0114] In some example embodiments, as will be described below with reference to Figure 12 to 20 the information about the distance between the feature points can be expressed by using an address of a memory device related to the position on the two-dimensional plane corresponding to the block row index i and the block column index j. In this case, the pixel position of the feature point and the data of the corresponding descriptor can be accessed by parsing or processing the address. Further, through this address processing, efficient access to information can be achieved in the process of removing feature point pairs with low correspondence and the homography operation.

[0115] Figure 9 is a diagram illustrating an example search region for determining correspondence of feature points, and Figure 10 is an example embodiment illustrating determination of a search region according to the present disclosure.

[0116] In image merging, corresponding feature point pairs can be searched based on a constraint on the spatial distance between the feature points and the similarity between the descriptors of the feature points.

[0117] Referring to Figure 9 , the similarity between a reference feature point Xr0 in a reference image Ir and feature points Xt1~Xt5 in a target image It can be estimated to determine the feature point with the highest similarity as the feature point corresponding to the reference feature point Xr0. As shown in Figure 9 , a search region SREG centered on the position of the reference point Xr0 on a two-dimensional plane can be determined, and the feature points Xt1~Xt3 in the search region SREG can be selected for similarity calculation and other feature points can be excluded. In some cases, the distance between the feature points can be obtained as a Euclidean distance. When the Euclidean distance is calculated to find the feature points included in the search region SREG, the processing amount can significantly increase.

[0118] Referring to Figure 10 , a reference descriptor block DMPr0 can be determined from among a plurality of descriptor blocks included in a reference descriptor map DMPr corresponding to a reference image among a plurality of input images.

[0119] The central target descriptor block can be determined from among a plurality of descriptor blocks included in a target descriptor map DMPt corresponding to a target image among the plurality of input images so that the central target descriptor block is located at the same position as a reference descriptor block on a two-dimensional plane. The descriptor block having the same block row index and the same block column index as the reference descriptor block can be determined as the central target descriptor block

[0120] The adjacent target descriptor block can be determined from among a plurality of descriptor blocks included in the target descriptor map DMPt and so that the adjacent target descriptor block and is adjacent to the central target descriptor block on a two-dimensional plane. The descriptor block having a block row index and a block column index deviating from the index of the reference descriptor block by less than a predetermined value can be determined as the adjacent target descriptor block. In this example, the adjacent target descriptor blocks are and

[0121] Accordingly, the central target descriptor block and the adjacent target descriptor block determined through comparison of the block row index and the block column index can be set as a search region SREG for similarity calculation.

[0122] Similarity between a reference descriptor included in the reference descriptor block and target descriptors included in the central target descriptor block and the adjacent target descriptor block and can be determined. Based on the similarity, a feature point corresponding to the reference descriptor can be determined from among target feature points corresponding to the target descriptors as matching the reference feature point. As shown in reference Figure 7 , the feature point and the corresponding descriptor can have the same point index p.

[0123] Figure 10 A search region SREG including nine descriptor blocks is shown, but the search region SREG can be set to include different numbers of descriptor blocks according to input image size, the number of image blocks included in each input image, target processing time, etc.

[0124] Accordingly, the search region SREG can be set regardless of the Euclidean distance between the feature points, and the similarity calculation can be performed based on the positions of the descriptor blocks on the two-dimensional plane, which correspond to the block row index and the block column index.

[0125] Figure 11 is a diagram illustrating an example of mapping information used by the method of matching images according to an example embodiment.

[0126] Figure 11 An example of matching information MINF of the feature points Xij included in the first to fifth images I0-I4, respectively, is illustrated. In the matching information MINF, the feature points in the same row can be matched with each other. For example, it will be understood from the matching information MINF that the feature point X02 of the first image I0, the feature point X11 of the second image I1, the feature point X22 of the third image I2, and the feature point X32 of the fourth image I3 correspond to each other, and the fifth image I4 does not have a corresponding feature point. Using this mapping information MINF, the image merging can be performed as described below with reference to Figure 11 Figure 22 to 26

[0127] Hereinafter, a method of matching images based on addresses according to an example embodiment will be described with reference to Figure 12 to 20

[0128] Figure 12 is a flowchart illustrating a method of matching images according to an example embodiment.

[0129] Referring to Figure 12 , each of the plurality of input images is divided into a plurality of image blocks (S100). As described with reference to Figure 6 , each of the input images can be uniformly divided so that each of the plurality of image blocks can include the same number of rows and columns of pixels.

[0130] A feature point map including a plurality of feature point blocks corresponding to the plurality of image blocks, respectively, is generated by extracting feature points included in each of the plurality of image blocks (S200). Further, a descriptor map including a plurality of descriptor blocks corresponding to the plurality of feature point blocks, respectively, is generated by generating descriptors of the feature points included in the feature point map (S300).

[0131] In some example embodiments, as described with reference to Figure 7 and Figure 8 ​​​As shown, the number of feature points included in each image block can be limited to a maximum block feature point number. For example, a maximum image feature point number indicating a maximum number of feature points included in each input image can be determined. The maximum block feature point number can be determined by dividing the maximum image feature point number by the number of image blocks in the plurality of image blocks, and the number of feature points included in each image block can be limited to the maximum block feature point number.

[0132] The plurality of descriptor maps corresponding to the plurality of input images can be stored in the memory device (S350). In some example embodiments, the plurality of feature point blocks corresponding to each input image can be stored in the memory device in a feature point block address offset order starting from a feature point start address corresponding to each input image. In the same manner, the plurality of descriptor blocks corresponding to each input image can be stored in the memory device in a descriptor block address offset order starting from a descriptor start address corresponding to each input image in the same or similar order as an order used to store the plurality of feature point blocks corresponding to each input image.

[0133] Mapping information including matching relationships between feature points included in different input images of the plurality of input images can be generated based on the descriptor maps stored in the memory device.

[0134] Descriptors used to determine the similarity can be read out from the memory device based on addresses of the descriptor blocks included in the descriptor maps corresponding to different input images of the plurality of input images (410). Here, the descriptors used to determine the similarity can refer to a reference descriptor block, a central target descriptor block, and a neighboring target descriptor, as described with reference to Figure 10

[0135] The similarity between the read out descriptors can be determined (S420). Mapping information including matching relationships between feature points included in different input images can be generated based on the similarity (S430). The plurality of input images can be synchronized using the mapping information, and a merged image of the plurality of input images can be generated based on the synchronized images.

[0136] The plurality of input images can be a set of related images to be merged. The plurality of input images can include at least a portion of the same scene.

[0137] In some example embodiments, as will be described below with reference to Figure 4A , the plurality of input images can be images captured by a plurality of cameras, respectively. In some example embodiments, as will be described below with reference to Figure 4B , the plurality of input images can be images sequentially captured by a single camera.

[0138] ​Accordingly, the method of matching images according to an example embodiment can improve image merging performance by dividing an input image into a plurality of image blocks to increase distribution consistency of feature points. Also, the method of matching images according to an example embodiment can improve image merging performance by reducing processing amount and processing time for image matching by using a feature point map and a descriptor map corresponding to the plurality of image blocks.

[0139] Figure 13 is a diagram illustrating an example embodiment of storing a feature point map and a descriptor map used by the method of matching images according to an example embodiment.

[0140] Referring to Figure 13 , a plurality of blocks included in a feature point map FMPs or a descriptor map DMPs can be sequentially stored in a memory device according to a storage order of row units. For example, the plurality of blocks may be a plurality of feature point blocks included in the feature point map FMPs or a plurality of descriptor blocks included in the descriptor map DMPs.

[0141] According to the storage order of row units, blocks of a first row may be stored at addresses of the memory device that increase in order, blocks of a second row may be stored at addresses of the memory device that increase in order, and similarly, blocks of a last row may be stored at addresses of the memory device that increase in order.

[0142] Figure 14 and Figure 15 are diagrams illustrating addresses allocated to a feature point map and a descriptor map according to Figure 13 a storage order.

[0143] Referring to Figure 14 , a plurality of feature point maps FMP0~FMP R may be sequentially stored at feature point start addresses S_PADD0~S_PADD R of a memory device. An address offset POFF1 between the feature point start addresses S_PADD0~S_PADD R may correspond to a number of data bits in each of the feature point maps FMP0~FMP R , and the address offset POFF1 can have the same value for each of the feature point maps FMP0~FMP R .

[0144] Figure 14 illustrates an example of a plurality of feature point blocks S corresponding to one feature point map FMP . A first feature point block address The feature point start address S_PADD of the Feature Point Map (FMPs) S Same. For example... Figure 14 As shown, multiple feature point blocks in the feature point map FMPs corresponding to the input image Is. Starting address of feature point Initially, feature point blocks are stored sequentially in the memory device according to their address offsets POFF2. The feature point block address offset POFF2 can be determined based on the feature point block size corresponding to the data bits included in each feature point block.

[0145] Therefore, according to the storage order of the row units, each feature point block address It can be based on the starting address of the feature points corresponding to the input image Is. The offset of the feature point block, POFF2, is used to determine this, as shown in Expression 3.

[0146] Expression 3:

[0147]

[0148] In expression 3, M represents the number of columns of the image patch included in each input image, which can be the same as the number of columns of the feature point patch included in each feature point map.

[0149] also, Figure 14 The example shown is a block of feature points. Corresponding multiple feature points First feature point address With feature point blocks They have the same starting address. For example... Figure 14 As shown, feature point blocks Multiple feature points in From the starting address Initially, they are stored sequentially in the memory device according to the address offset POFF3.

[0150] refer to Figure 15 Multiple descriptor graphs DMP0 to DMP R It can be sequentially stored at the starting address S_DADD0~S_DADD of the memory device descriptor. R Location. Descriptor starting address S_DADD0~S_DADD R The address offset DOFF1 between these two points can be compared with the descriptor graphs DMP0 to DMP1. R The number of data bits for each descriptor graph in the graph corresponds to the number of data bits for descriptor graphs DMP0 to DMP1. REach of the addresses of the descriptor blocks in the descriptor map DMPs can have the same value.

[0151] Figure 15 The descriptor map DMPs are shown as examples corresponding to one descriptor block S The descriptor map DMPs are shown as examples corresponding to one descriptor block The first descriptor The address of the first descriptor The address of the first descriptor S The address of the first descriptor Figure 15 The descriptor map DMPs are shown as examples corresponding to one descriptor block The descriptor map DMPs are shown as examples corresponding to one descriptor block The descriptor map DMPs are shown as examples corresponding to one descriptor block

[0152] The address of the first descriptor The address of the first descriptor The address of the first descriptor The address of the first descriptor

[0153] Expression 4:

[0154]

[0155] In Expression 4, M denotes the number of columns of the image blocks included in each input image, which can be the same as the number of columns of the descriptor blocks included in each descriptor map.

[0156] Further, Figure 15 The descriptor map DMPs are shown as examples corresponding to one descriptor block The descriptor map DMPs are shown as examples corresponding to one descriptor block The first descriptor The address of the first descriptor The address of the first descriptor The address of the first descriptor Figure 15 The descriptor map DMPs are shown as examples corresponding to one descriptor block The descriptor map DMPs are shown as examples corresponding to one descriptor block The descriptor map DMPs are shown as examples corresponding to one descriptor block The descriptor map DMPs are shown as examples corresponding to one descriptor block

[0157] Accordingly, the addresses according to the storage order of the row units can satisfy the relationship of Expression 3 and Expression 4. Thus, the descriptor start addresses S_DADD0 to S_DADD Rand a descriptor block address offset DOFF2 to read out descriptors for similarity calculation from the memory device, as will be described below with reference to Figure 16 and Figure 17 described.

[0158] Figure 16 is a flowchart showing an example embodiment of reading data used by a method of matching images according to an example embodiment, and Figure 17 is a diagram for describing Figure 16 data reading.

[0159] With reference to Figure 16 , a reference address of a reference descriptor block included in a reference descriptor map among a plurality of descriptor maps can be determined based on a reference descriptor start address and a descriptor block address offset corresponding to the reference descriptor map (S411).

[0160] A target address of a central target descriptor block and adjacent target descriptor blocks included in a target descriptor map among the plurality of descriptor maps can be determined based on a target descriptor start address and a descriptor block address offset corresponding to the target descriptor map, such that the central target descriptor block is located at the same position as the reference descriptor block on a two-dimensional plane and the adjacent target descriptor blocks are adjacent to the central target descriptor block on the two-dimensional plane (S412).

[0161] The reference descriptor block, the central target descriptor block, and the adjacent target descriptor blocks can be read out from the memory device based on the reference address and the target address (S413).

[0162] Figure 17 shows a reference descriptor map DMPr, a target descriptor map DMPt, and addresses corresponding to the reference descriptor map DMPr and the target descriptor map DMPt, which are stored in a memory device according to a storage order of row units as described with reference to Figure 13 , Figure 14 and Figure 15 The reference descriptor map DMPr corresponds to one of a plurality of descriptor maps DMP0~DMP R , and the target descriptor map DMPt corresponds to another one of the plurality of descriptor maps DMP0~DMP R .

[0163] As described with reference to Expression 4, a reference address of a reference descriptor block included in the reference descriptor map DMPr among a plurality of descriptor blocks can be determined based on a reference descriptor start address and a descriptor block address offset DOFF2 corresponding to the reference descriptor map DMPr. As shown in Expression 5. ​

[0164] Expression 5:

[0165]

[0166] In the same way, the starting address of the target descriptor corresponding to the target descriptor graph DMPt can be used. The central target descriptor block is determined by the descriptor block address offset DOFF2. and adjacent target descriptor blocks and target address As shown in expression 6.

[0167] Expression 6:

[0168]

[0169]

[0170]

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178] Figure 2 The address generator 110 in the middle can generate reference addresses according to the storage order of row units. and target address The reference address and target address are provided as access addresses (ACCAD) to the memory device 30. This can be based on the reference address. and target address Read reference descriptor block from memory device Central Target Descriptor Block and adjacent target descriptor blocks and And load it into the descriptor graph buffer 310.

[0179] Figure 2The matcher 320 in the reference descriptor block included in the reference descriptor block and the central target descriptor block and the adjacent target descriptor blocks and included in the target descriptor block. The matcher 320 can determine a feature point that matches a reference feature point corresponding to the reference descriptor from among target feature points corresponding to the target descriptor based on the similarity.

[0180] Accordingly, data for pixel positions of feature points and corresponding descriptors can be accessed by parsing or processing the addresses, and corresponding pairs of feature points can be efficiently estimated. Furthermore, through such address processing, efficient access to information can be achieved in the process of removing pairs of feature points with low correspondence and in homography operation.

[0181] Figure 18 is a diagram illustrating an example embodiment of storing a feature point map and a descriptor map used by a method of matching images according to an example embodiment.

[0182] Referring to Figure 18 , a plurality of blocks included in the feature point map FMPs or the descriptor map DMPs may be sequentially stored in a memory device according to a storage order of column units. The plurality of blocks may be a plurality of feature point blocks included in the feature point map FMPs or a plurality of descriptor blocks included in the descriptor map DMPs.

[0183] The blocks of the first column may be stored at addresses of the memory device that sequentially increase according to the storage order of column units, the blocks of the second column may be stored at addresses of the memory device that sequentially increase, and similarly, the blocks of the last column may be stored at addresses of the memory device that sequentially increase.

[0184] Figure 19 and Figure 20 is a diagram illustrating addresses allocated to a feature point map and a descriptor map according to Figure 18 a storage order.

[0185] Referring to Figure 19 , a plurality of feature point maps FMP0~FMP R may be sequentially stored at feature point start addresses S_PADD0~S_PADD R of a memory device. An address offset POFF1 between the feature point start addresses S_PADD0~S_PADD R may be the same as the feature point maps FMP0~FMP RThe number of data bits in each of the feature point maps corresponds to the number of data bits in the feature point maps FMP0 to FMP0. R For each of them, the address offset POFF1 can have the same value.

[0186] Figure 19 The example shown is a feature point map FMP. S Corresponding multiple feature point blocks First feature point block address With Feature Point Map FMP S Feature point starting address S_PADD S Same. For example... Figure 19 As shown, multiple feature point blocks in the feature point map FMPs corresponding to the input image Is. Starting address of feature point Initially, feature point blocks are stored sequentially in the memory device according to their address offsets POFF2. The feature point block address offset POFF2 can be determined based on the feature point block size corresponding to the data bits included in each feature point block.

[0187] Therefore, according to the storage order of the column cells, each feature point block... address It can be based on the starting address of the feature points corresponding to the input image Is. The offset of the feature point block, POFF2, is used to determine this, as shown in Expression 7.

[0188] Expression 7:

[0189]

[0190] In expression 7, N represents the number of rows of image patches included in each input image, which can be the same as the number of rows of feature point patches included in each feature point map.

[0191] also, Figure 19 The example shown is a block of feature points. Corresponding multiple feature points First feature point address With feature point blocks They have the same starting address. For example... Figure 19 As shown, feature point blocks Multiple feature points in From the starting address Initially, they are stored sequentially in the memory device according to the address offset POFF3.

[0192] refer to Figure 20Multiple descriptor graphs DMP0 to DMP R The descriptors can be sequentially stored from the starting address S_DADD0 to S_DADD0 of the memory device. R Location. Descriptor starting address S_DADD0~S_DADD R The address offset DOFF1 between these two points can be compared with the descriptor graphs DMP0 to DMP1. R The number of data bits for each of them corresponds to the number of data bits in the descriptor graphs DMP0 to DMP0. R For each of them, the address offset DOFF1 can have the same value.

[0193] Figure 20 The example shown is a descriptor graph DMP. S Corresponding multiple descriptor blocks First descriptor block address With Descriptor Graph DMP S The starting address of the descriptor S_DADD S Same. For example... Figure 20 As shown, multiple descriptor blocks in the descriptor graphs (DMPs) corresponding to the input image Is. From the starting address of the descriptor Initially, descriptor blocks are stored sequentially in the memory device according to their address offsets DOFF2. The descriptor block address offsets DOFF2 can be determined based on the size of the descriptor block corresponding to the data bits included in each descriptor block.

[0194] Therefore, according to the storage order of the column cells, each descriptor block address It can be based on the starting address of the descriptor corresponding to the input image Is. It is determined by the descriptor block offset DOFF2, as shown in expression 8.

[0195] Expression 8:

[0196]

[0197] In expression 8, N represents the number of rows of image patches included in each input image, which can be the same as the number of rows of descriptor patches included in each descriptor graph.

[0198] also, Figure 20 The example shown is a block of descriptors. Corresponding multiple descriptors First descriptor address With descriptor block They have the same starting address. For example...Figure 15 As shown, descriptor block Multiple descriptors in From the starting address Initially, they are stored sequentially in the memory device according to the address offset DOFF3.

[0199] Therefore, the addresses based on the storage order of the column cells satisfy the relationship between expressions 7 and 8. Therefore, the descriptor starting addresses S_DADD0~S_DADD can be used. R and descriptor block address offset DOFF2 to reference Figure 16 and Figure 17 The same method described above is used to read descriptors from the memory device for similarity calculation.

[0200] Figure 21 This is a graph showing the coordinates of feature points used by the image matching method according to an example embodiment.

[0201] As described in reference expression 1, the number N of rows of image blocks included in each input image Is can be determined by the row size SZA of each input image Is and the number of rows of each image block. The row size SZA' is determined. Similarly, the number of columns M of image patches included in each input image Is can be determined by the column size SZB of each input image Is and the number of columns M of each image patch. The column size SZB' is used to determine this.

[0202] In some example embodiments, feature points The location can be represented by (A, B) in each input image Is. In other example embodiments, feature points The location can be determined by each image block. In this context, (A', B') represents each image patch. The position can be determined by the block row index i and the block column index j, and one of (A, B) and (A', B') can be calculated based on the other of (A, B) and (A', B'). When the feature point Position by each image block When (A', B') is used to represent the above feature point map used for image matching, the size and processing time can be further reduced by reducing the access time to the memory device.

[0203] Figure 22 This is a flowchart illustrating a method for merging images according to an example embodiment.

[0204] refer to Figure 22The mapping information including matching relationships between feature points included in different ones of the plurality of input images can be generated by dividing each of the plurality of input images into a plurality of image blocks (S10). The mapping information can be efficiently generated by dividing each of the input images into a plurality of image blocks and using a feature point map and a descriptor map as described above.

[0205] A homography matrix for conversion between different input images can be generated based on the mapping information (S20). A synchronized image can be generated by synchronizing the plurality of input images based on the homography matrix (S30), and a merged image can be generated by merging the synchronized image (S40).

[0206] Hereinafter, the following will be described below with reference to Figure 23A to 26 The merged image is generated based on the mapping information, but example embodiments are not necessarily limited thereto.

[0207] Figure 23A to 23D is a diagram illustrating image transformations used by a method of merging images according to an example embodiment.

[0208] The image transformation can include obtaining a converted or modified image by applying a certain algorithmic function to a given image (e.g., by applying a translation, a rotation, a scaling transformation, and / or individual transformations to the given image).

[0209] The image transformation can include a rigid transformation as illustrated in Figure 23A , a similarity transformation as illustrated in Figure 23B , an affine transformation as illustrated in Figure 23C , a projective transformation as illustrated in Figure 23D , etc. The converted or transformed coordinates (x', y') can be obtained by applying these image transformations to the original coordinates (x, y).

[0210] Figure 23A The rigid transformation of is a basic image transformation that fixes the magnitude of an image. In other words, only translation and rotation can be performed by the rigid transformation.

[0211] Figure 23B The similarity transformation of also reflects a change in the magnitude or scale of an object, and the shape of the object is maintained by the similarity transformation.

[0212] Figure 23C The affine transformation of adds a feature of linear preservation to the similarity transformation. Thus, the parallelism of a line, the ratio of the length or distance can be maintained by the affine transformation.

[0213] Figure 23DA projection transformation is a transformation for projecting an image of a 3D space to an image of a 2D space. The projection transformation can include a conversion between two images observed from two different perspective points in the 3D space.

[0214] A matrix representing an image transformation between two images is referred to as a homography matrix or simply a homography. A relationship representing a general projection transformation to transform a first image into a second image or to transform a second image into a first image can be obtained, and the relationship is a transformation matrix referred to as a homography matrix.

[0215] In image processing, two images captured through two different observation points or two different time points can need an image transformation. The size of a homography required is 3X3. If eight values are given, a homography can be determined. In other words, at least four pairs of corresponding points are required to determine a homography.

[0216] Figure 24 is a diagram for describing a homography matrix used by a method of merging images according to an example embodiment.

[0217] Figure 24 A process of obtaining a homography matrix H corresponding to a pair of points X(u, v, 1) and X'(u', v', 1) respectively included in two different images is shown. The homography matrix H can be determined by obtaining components h1~h9 of the homography matrix H.

[0218] As shown in Figure 24 , a relationship HX=X' can be transformed into a relationship Ah=0, and a component vector h can be obtained. In some examples, the matrix A has a size of 3*9 with respect to one pair of corresponding points, and the matrix A has a size of 3n*9 with respect to n pairs of corresponding points.

[0219] Figure 25 is a diagram showing homography matrices between a plurality of input images, and Figure 26 is a diagram showing a process of merging images.

[0220] Referring to Figure 25 , when the first to fifth input images I0~I4 are merged, homography matrices Hij(i=0~4, j=0~4) can correspond to respective transformations between every two different input images Ii and Ij. Hij represents a homography matrix for transforming coordinates in the ith input image Ii to coordinates aligned with a two-dimensional plane corresponding to the jth image Ij. Hji is an inverse matrix of Hij, and redundant matrices in Figure 11 are omitted.

[0221] Accordingly, a homography matrix representing a transformation between two input images can be determined based on matching information of feature points included in the two input images Figure 22S20) in FIG. 10.

[0222] Referring to Figure 26 , synchronized images SI0-SI2 can be generated by synchronizing the plurality of input images I0-I3 based on the homography matrix (S30 in FIG. 10), as described with reference to Figure 22 Figure 23A to 25 One or more of the input images I0-I3 can be a noisy image and additionally or alternatively provided for noise reduction, and thus the number of the synchronized images SI0-SI2 can be less than the number of the input images I0-I3.

[0223] The merged image MI can be generated by merging the synchronized images SI0-SI2 (S40 in FIG. 10). Figure 22

[0224] In some example embodiments, image stitching can be performed using the synchronized images SI0-SI2 to generate the merged image MI. Due to an alignment error or an exposure bias between the synchronized images SI0-SI2, a boundary of a stitched portion can be sharpened during the stitching process, and image blending can be performed to smooth the sharpened boundary. The image stitching can be performed using a plurality of the synchronized images SI0-SI2 to minimize the boundary and thus the merged image MI can be obtained. In some example embodiments, the merged image MI of a high dynamic range (HDR) can be generated using a plurality of the synchronized images SI0-SI2.

[0225] The merging of the synchronized images SI0-SI2 can be implemented using various schemes (e.g., image stitching, noise reduction, HDR conversion, etc.), and example embodiments are not limited thereto.

[0226] Figure 27 is a block diagram illustrating an electronic system according to an example embodiment.

[0227] Referring to Figure 27 , the electronic system 2000 can include a processor 2010, a connection device 2020, a memory device 2030, a user interface 2040, a data processing device 2060, and an image pickup device 2050. In addition, the electronic system 2000 can include a power supply.

[0228] ​​The processor 2010 can perform various computing functions, e.g., specific computations and tasks. The connection device 2020 can communicate with external devices. The memory device 2030 can operate as a data memory for data processed by the processor 2010 and the data processing device 2060. The user interface 2040 can include at least one input device (e.g., a keypad, buttons, a microphone, a touch screen, etc.) and / or at least one output device (e.g., a speaker or a display device, etc.). The power supply can supply power to the electronic system 2000.

[0229] The image pickup device 2050 and the data processing device 2060 can be controlled by the processor 2010. The image pickup device 2050 can provide an input image to the data processing device 2060. For example, the image pickup device 2050 can include a complementary metal-oxide semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, etc. The data processing device 2060 can be a data processing device of Figure 2 and can perform image matching and image merging according to example embodiments as described with reference to Figure 1 to 26

[0230] As described above, the method of matching images and the data processing device performing the method according to example embodiments can improve image merging performance by dividing an input image into a plurality of image blocks to increase distribution consistency of feature points. In addition, image merging performance can be further improved by reducing processing amount and processing time for image matching by using a feature point map and a descriptor map corresponding to the plurality of image blocks.

[0231] The inventive concept can be applied to any electronic device and system requiring image processing. For example, the inventive concept can be applied to systems such as a mobile phone, a smart phone, a personal digital assistant (PDA), a portable multimedia player (PMP), a digital camera, a camcorder, a personal computer (PC), a server computer, a workstation, a laptop computer, a digital TV, a set-top box, a portable game console, a navigation system, a wearable device, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, an e-book, a virtual reality (VR) device, an augmented reality (AR) device, a vehicle camera, a medical camera, etc.

[0232] The above-described content is a description of example embodiments, and should not be construed as limiting thereof. Although some example embodiments have been described, those skilled in the art will readily understand that many modifications can be made in the example embodiments without departing from the inventive concept in its essence.​

Claims

1. A method of matching images, comprising: dividing each of a plurality of input images into a plurality of image blocks; generating a feature point map by extracting feature points from each of the plurality of image blocks, wherein the feature point map comprises a plurality of feature point blocks corresponding to the plurality of image blocks; generating a descriptor map by generating descriptors from the feature points of the feature point map, wherein the descriptor map comprises a plurality of descriptor blocks corresponding to the plurality of feature point blocks; and generating mapping information including matching relationships between feature points in different ones of the plurality of input images based on a plurality of descriptor maps corresponding to the plurality of input images, wherein generating the mapping information comprises: determining a reference descriptor block from a plurality of descriptor blocks included in a reference descriptor map corresponding to a reference image of the plurality of input images; determining a central target descriptor block from a plurality of descriptor blocks included in a target descriptor map corresponding to a target image of the plurality of input images such that the central target descriptor block is located at a same position as the reference descriptor block on a two-dimensional plane; determining an adjacent target descriptor block from the plurality of descriptor blocks included in the target descriptor map such that the adjacent target descriptor block is adjacent to the central target descriptor block on the two-dimensional plane; determining a similarity between a reference descriptor included in the reference descriptor block and target descriptors included in the central target descriptor block and the adjacent target descriptor block; and determining, based on the similarity, a feature point from target feature points corresponding to the target descriptors that matches a reference feature point corresponding to the reference descriptor.

2. The method of claim 1, wherein, generating the feature point map comprises: limiting a number of feature points extracted from each image block to a maximum block feature point number.

3. The method of claim 1, wherein, generating the feature point map comprises: determining a maximum image feature point number indicating a maximum number of feature points to be extracted from each input image; determining a maximum block feature point number by dividing the maximum image feature point number by a number of image blocks; and limiting a number of feature points included in each image block to the maximum block feature point number.

4. The method of claim 1, wherein, dividing each input image comprises: uniformly dividing each input image such that each of the plurality of image blocks has a same pixel width size and a same pixel height size.

5. The method of claim 1, wherein, generating the feature point map comprises: generating each of the plurality of feature point blocks such that each of the plurality of feature point blocks includes a same number of data bits corresponding to a feature point block size, and wherein generating the descriptor map comprises: generating each of the plurality of descriptor blocks such that each of the plurality of descriptor blocks includes a same number of data bits corresponding to a descriptor block size.

6. The method of claim 1, wherein, determining a similarity between descriptors independent of Euclidean distances between feature points based on positions of descriptor blocks on a two-dimensional plane.

7. The method of claim 6, wherein, a position of each descriptor on the two-dimensional plane corresponds to a block row index and a block column index of each descriptor block.

8. The method of claim 1, wherein, The plurality of descriptor maps corresponding to the plurality of input images are stored in a memory device, and a similarity between descriptors is determined based on addresses of descriptor blocks of the memory device at which the plurality of descriptor maps are stored.

9. The method of claim 1, wherein, A position of each descriptor on the two-dimensional plane corresponds to a block row index and a block column index of each descriptor block, and wherein a central target descriptor block and neighboring target descriptor blocks of the target image corresponding to a reference descriptor block of the input image are determined based on the block row index and the block column index.

10. The method of claim 1, wherein, The similarity between descriptors corresponds to a Hamming distance between the descriptors.

11. The method of claim 1, further comprising: storing a plurality of feature point blocks corresponding to each input image sequentially in a memory device starting from a feature point block address offset from a feature point starting address corresponding to each input image; and storing a plurality of descriptor blocks corresponding to each input image sequentially in the memory device starting from a descriptor starting address corresponding to each input image followed by the descriptor block address offset.

12. The method of claim 11, wherein, The feature point block address offset is determined based on a feature point block size corresponding to a number of data bits included in each feature point block, and the descriptor block address offset is determined based on a descriptor block size corresponding to a number of data bits included in each descriptor block.

13. The method of claim 11, wherein, Generating the mapping information includes: reading descriptors for determining a similarity from the memory device based on a plurality of descriptor starting addresses corresponding to the plurality of input images, respectively, and the descriptor block address offset.

14. The method of claim 11, wherein, Generating the mapping information includes: determining a reference address of a reference descriptor block included in a reference descriptor map of the plurality of descriptor maps based on a reference descriptor starting address corresponding to the reference descriptor map and the descriptor block address offset; determining target addresses of a central target descriptor block and neighboring target descriptor blocks included in a target descriptor map of the plurality of descriptor maps based on a target descriptor starting address corresponding to the target descriptor map and the descriptor block address offset, such that the central target descriptor block is located at a same position on a two-dimensional plane as the reference descriptor block and the neighboring target descriptor blocks are adjacent to the central target descriptor block on the two-dimensional plane; and reading the reference descriptor block, the central target descriptor block, and the neighboring target descriptor blocks from the memory device based on the reference address and the target addresses.

15. The method of claim 14, wherein, Generating the mapping information further includes: determining a similarity between a reference descriptor included in the reference descriptor block and target descriptors included in the central target descriptor block and the neighboring target descriptor blocks; and determining a feature point matching a reference feature point corresponding to the reference descriptor from target feature points corresponding to the target descriptors based on the similarity.

16. The method of claim 1, wherein, generate the descriptors based on at least one of: features from accelerated segment test (FAST) and oriented FAST and rotated BRIEF (ORB), scale-invariant feature transform (SIFT), speeded up robust features (SURF), and maximally stable extremal regions (MSD).

17. A method of matching images, comprising: dividing each of a plurality of input images into a plurality of image blocks; generating a feature point map by extracting feature points from each of the plurality of image blocks, wherein the feature point map includes a plurality of feature point blocks corresponding to the plurality of image blocks; generating a descriptor map by generating descriptors of the feature points from the feature point map, wherein the descriptor map includes a plurality of descriptor blocks corresponding to the plurality of feature point blocks; storing a plurality of descriptor maps respectively corresponding to the plurality of input images in a memory device; reading out descriptors for determining a similarity from the memory device based on addresses of the descriptor blocks included in the descriptor maps corresponding to different input images among the plurality of input images; determining a similarity between the read out descriptors; and generating mapping information including matching relationships between feature points included in the different input images based on the similarity, wherein the generating the mapping information includes: determining a reference descriptor block from a plurality of descriptor blocks included in a reference descriptor map corresponding to a reference image among the plurality of input images; determining a central target descriptor block from a plurality of descriptor blocks included in a target descriptor map corresponding to a target image among the plurality of input images such that the central target descriptor block is located at a same position as the reference descriptor block on a two-dimensional plane; determining an adjacent target descriptor block from the plurality of descriptor blocks included in the target descriptor map such that the adjacent target descriptor block is adjacent to the central target descriptor block on the two-dimensional plane; determining a similarity between a reference descriptor included in the reference descriptor block and target descriptors included in the central target descriptor block and the adjacent target descriptor block; and determining a feature point matching a reference feature point corresponding to the reference descriptor from target feature points corresponding to the target descriptors based on the similarity.

18. The method of claim 17, wherein, the generating the feature point map includes: limiting a number of the feature points included in each image block to a maximum block feature point number.

19. A data processing device, comprising: a feature point map generator configured to divide each of a plurality of input images into a plurality of image blocks, and to generate a feature point map including a plurality of feature point blocks respectively corresponding to the plurality of image blocks by extracting feature points included in each of the plurality of image blocks; a descriptor map generator configured to generate a descriptor map including a plurality of descriptor blocks respectively corresponding to the plurality of feature point blocks by generating descriptors of the feature points included in the feature point map; and a matching information generator configured to generate mapping information including matching relationships between feature points included in different input images among the plurality of input images based on a similarity between the different input images. an image matcher configured to generate mapping information including matching relationships between feature points included in different ones of the plurality of input images based on a plurality of descriptor maps respectively corresponding to the plurality of input images, wherein generating the mapping information includes: determining a reference descriptor block from a plurality of descriptor blocks included in a reference descriptor map corresponding to a reference image of the plurality of input images; determining a central target descriptor block from a plurality of descriptor blocks included in a target descriptor map corresponding to a target image of the plurality of input images such that the central target descriptor block is located at a same position in a two-dimensional plane as the reference descriptor block; determining an adjacent target descriptor block from the plurality of descriptor blocks included in the target descriptor map such that the adjacent target descriptor block is adjacent to the central target descriptor block in the two-dimensional plane; determining a similarity between a reference descriptor included in the reference descriptor block and target descriptors included in the central target descriptor block and the adjacent target descriptor block; and determining, based on the similarity, a feature point corresponding to the target descriptor that matches a reference feature point corresponding to the reference descriptor from target feature points corresponding to the target descriptors.

Images