Method for extracting features, method for image matching, and method for processing images

Abstract

In a method of extracting features from an image, a plurality of initial key points are estimated based on an input image. A plurality of descriptors are generated based on a reduced image generated by reducing the input image. A plurality of feature points are obtained by respectively matching the plurality of initial key points with the plurality of descriptors. Also provided is a method of image matching and a method of processing an image.

Description

[0001] [Cross - Reference to Related Applications]

[0002] Korean Patent Application No. 10 - 2018 - 0124329, filed on October 18, 2018 with the Korean Intellectual Property Office (KIPO) and entitled "Method of Extracting Features from Image, Method of Matching Images Using the Extraction Method, and Method of Processing Images Using the Matching Method", is incorporated herein by reference in its entirety. Technical Field

[0003] Each exemplary embodiment relates to image processing technology, and more particularly to a method of extracting features from an image, a method of matching images using the extraction method, and a method of processing images using the matching method. Background Art

[0004] Computer vision technology is related to artificial intelligence technology that utilizes various electronic devices to implement the visual recognition ability among the five human senses. Image matching technology in computer vision technology has always been an important technical field because image matching technology can be applied to various applications such as intelligent robots, augmented reality, virtual reality, three - dimensional structure information, and security technology based on face recognition. With the significant development of mobile communication devices (e.g., smart phones and tablet personal computers (PCs)) in mobile communication technology, there is a continuous need for real - time image matching technology in a mobile environment. However, when capturing an image in a mobile environment, various influencing factors (e.g., size, illumination, obstacles, and rotation, etc.) may affect the captured image. In addition, when the captured image has a large motion or movement or when the captured image is captured in a dark environment, it may be difficult to accurately identify the features in the captured image. Summary of the Invention

[0005] Each embodiment relates to a method of extracting features from an image. The method may include: estimating a plurality of initial key points based on an input image; generating a plurality of descriptors based on a reduced image generated by reducing the input image; and obtaining a plurality of feature points by respectively matching the plurality of initial key points with the plurality of descriptors.

[0006] Each embodiment relates to a method for image matching. The method may include: detecting a plurality of first feature points based on a first input image; detecting a plurality of second feature points based on a second input image; and obtaining a plurality of corresponding points between the first input image and the second input image based on the plurality of first feature points and the plurality of second feature points. Detecting the plurality of first feature points includes: estimating a plurality of first initial key points based on the first input image; generating a plurality of first descriptors based on a first downsampled image generated by downsampling the first input image; and obtaining the plurality of first feature points by respectively matching the plurality of first initial key points with the plurality of first descriptors.

[0007] Each embodiment relates to a method for processing an image. The method may include: setting a plurality of corresponding points between a first input image and a second input image; and performing image processing on at least one of the first input image and the second input image based on the plurality of corresponding points. Setting the plurality of corresponding points includes: detecting a plurality of first feature points based on the first input image; detecting a plurality of second feature points based on the second input image; and obtaining the plurality of corresponding points based on the plurality of first feature points and the plurality of second feature points. Detecting the plurality of first feature points includes: estimating a plurality of first initial key points based on the first input image; generating a plurality of first descriptors based on a first downsampled image generated by downsampling the first input image; and obtaining the plurality of first feature points by respectively matching the plurality of first initial key points with the plurality of first descriptors. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] By describing the exemplary embodiments in detail with reference to the drawings, the features will become apparent to those skilled in the art, in which:

[0009] Figure 1 shows a method for extracting features from an image according to an exemplary embodiment.

[0010] Figure 2 shows a feature extraction device according to an exemplary embodiment.

[0011] Figure 3 shows Figure 1 the step of generating a plurality of descriptors in

[0012] Figure 4A , Figure 4B , Figure 5A and Figure 5B shows Figure 3 the operation of generating the plurality of descriptors in

[0013] Figure 6 shows a method for extracting features from an image according to an exemplary embodiment.

[0014] Figure 7 Shows a feature extraction device according to an exemplary embodiment.

[0015] Figure 8 Shows Figure 6 An example of converting an input image into a one-channel image.

[0016] Figure 9 Shows Figure 8 An example of generating a plurality of one-channel pixel data.

[0017] Figure 10A And Figure 10B Shows Figure 8 An operation of converting an input image into a one-channel image.

[0018] Figure 11 Shows Figure 6 Another example of converting an input image into a one-channel image.

[0019] Figure 12 Shows Figure 11 An example of generating a plurality of one-channel pixel data.

[0020] Figure 13 Shows a method of image matching according to an exemplary embodiment.

[0021] Figure 14 Shows an image matching device according to an exemplary embodiment.

[0022] Figure 15 Shows Figure 13 An example of obtaining a plurality of corresponding points.

[0023] Figure 16 Shows Figure 15 An operation of obtaining the plurality of corresponding points.

[0024] Figure 17 Shows a method of processing an image according to an exemplary embodiment.

[0025] Figure 18 Shows an image processing device according to an exemplary embodiment.

[0026] Figure 19 Shows an electronic system according to an exemplary embodiment.

[0027] [Description of symbols]

[0028] 100, 100a, 100b: Feature extraction device;

[0029] 110: Key point estimator;

[0030] 120: Descriptor generator;

[0031] 122: Scaler;

[0032] 124: Key point calculator;

[0033] 126: Descriptor calculator;

[0034] 130: Feature point generator;

[0035] 140: Image converter;

[0036] 200: Image matching device;

[0037] 210: Similarity comparator;

[0038] 220: Corresponding point generator;

[0039] 300, 1060: Image processing device;

[0040] 310: Processing unit;

[0041] 1000: Electronic system;

[0042] 1010: Processor;

[0043] 1020: Connectivity;

[0044] 1030: Memory device;

[0045] 1040: User interface;

[0046] 1050: Image capture device;

[0047] A, A': Region;

[0048] B22, B24, B26, B42, B44, B46: Original pixel data / Blue pixel data;

[0049] COMP: Comparison signal;

[0050] CP: Corresponding point signal;

[0051] D1: First direction;

[0052] D2: Second direction;

[0053] DSC: Descriptor signal;

[0054] FP: Feature point signal;

[0055] FP1: First feature point signal;

[0056] FP2: Second feature point signal;

[0057] G12, G14, G16, G21, G23, G25, G32, G34, G36, G41, G43, G45: Original pixel data / Green pixel data;

[0058] GIMG: 1-channel image;

[0059] IKP: Initial key point signal;

[0060] IKP11, IKP12, IKP13: Initial key point / First initial key point;

[0061] IKP21, IKP22, IKP23: Second initial key point;

[0062] IMG1: Input image / First input image;

[0063] IMG2: Second input image;

[0064] IS1: First image signal;

[0065] IS2: Second image signal;

[0066] IS_1C: 1-channel image signal;

[0067] IS_R: Original image signal;

[0068] OBJ11, OBJ12, OBJ21, OBJ22, SOBJ11, SOBJ12: Objects;

[0069] OIS: Output image signal;

[0070] P11, P12, P13, P14, P15, P16, P21, P22, P23, P24, P25, P26, P31, P32, P33, P34, P35, P36, P41, P42, P43, P44, P45, P46: 1-channel pixel data;

[0071] PA, PA', PB, PC, PD, PD', PE, PE': Pixels;

[0072] PIDIFF, PIDIFF': Intensity difference;

[0073] R11, R13, R15, R31, R33, R35: Original pixel data / Red pixel data;

[0074] RIMG: Original image;

[0075] S100, S110, S120, S122, S124, S130, S132, S134, S136, S138, S200, S200a, S400, S400a, S600, S410, S420, S430, S1100, S1200, S1300, S1310, S1320, S1330, S2100, S2200: Steps;

[0076] SF: Scaling factor;

[0077] SIMG1: Scaled-down image;

[0078] SIS: Scaled-down image signal;

[0079] SKP: Scaled-down key point signal;

[0080] SKP11, SKP12, SKP13: Scaled-down key points. Detailed implementation manners

[0081] Hereinafter, each exemplary embodiment will be explained in detail with reference to the accompanying drawings.

[0082] Figure 1 A method for extracting features from an image according to an exemplary embodiment is shown. Refer to Figure 1 , a plurality of initial key points may be estimated based on an input image (step S200). A key point may represent a point associated with a feature or characteristic of the input image, and may be referred to as an interest point. For example, a key point may be located at the contour of an object in the input image. The plurality of initial key points may be estimated from the input image based on at least one of various algorithms for image analysis, as will be elaborated later.

[0083] In some exemplary embodiments, the input image may be a one-channel image. A one-channel image may not contain color information and may only contain luminance information. A one-channel image may be referred to as a gray image, a grayscale image, or a monochrome image. When the input image is not a one-channel image, the input image may be converted into a one-channel image through an image conversion process, as will be elaborated with reference to Figure 6 described.

[0084] A plurality of descriptors may be generated based on a scaled-down image generated by scaling down the input image (step S400). Each descriptor may represent a feature or characteristic of a corresponding key point. The plurality of descriptors may also be generated based on at least one of various algorithms for image analysis, as will be elaborated later.

[0085] For example, when the input image is converted into a downscaled image, the intensity difference between adjacent pixels may increase. In other words, the intensity difference between adjacent pixels in the downscaled image may be greater than the intensity difference between adjacent pixels in the input image. Therefore, when generating the plurality of descriptors based on the downscaled image, each descriptor may be robust with respect to noise and have improved singularity.

[0086] A plurality of feature points are obtained by respectively matching the plurality of initial key points with the plurality of descriptors (step S600). For example, one initial key point and one descriptor may be matched with each other to obtain one feature point. The plurality of feature points may be used to obtain a plurality of corresponding points between two images, as will be described with reference to Figure 13 below.

[0087] Figure 2 FIG. shows a feature extraction apparatus according to an exemplary embodiment. Referring to Figure 2 , the feature extraction apparatus 100a may include a key point estimator 110, a descriptor generator 120, and a feature point generator 130.

[0088] The key point estimator 110 may receive the 1-channel image signal IS_1C as the input image, estimate a plurality of initial key points based on the 1-channel image signal IS_1C, and output an initial key point signal IKP indicating the plurality of initial key points. In other words, the key point estimator 110 may perform or execute Figure 1 step S200 in. The input image may be a 1-channel image, for example, a gray image, a grayscale image, or a monochrome image.

[0089] In some exemplary embodiments, the 1-channel image signal IS_1C may be provided from an external image capturing device (e.g., Figure 19 the image capturing device 1050 in) and / or an external memory device (e.g., Figure 19 the memory device 1030 in). In other exemplary embodiments, the 1-channel image signal IS_1C may be provided from a separate image converter (e.g., Figure 7 the image converter 140 in).

[0090] The descriptor generator 120 may perform or execute Figure 1 step S400 in. For example, the descriptor generator 120 may receive the 1-channel image signal IS_1C, the initial key point signal IKP, and the scaling factor SF, generate a plurality of descriptors based on the downscaled image generated by downscaling the 1-channel image signal IS_1C, and output a descriptor signal DSC indicating the plurality of descriptors.

[0091] For example, the descriptor generator 120 may include a scaler 122, a key point calculator 124, and a descriptor calculator 126. The scaler 122 may receive a 1-channel image signal IS_1C and a scaling factor SF, may generate a scaled-down image by scaling down the 1-channel image signal IS_1C, and may output a scaled-down image signal SIS indicating the scaled-down image. The scaling factor SF may be provided from a controller located inside or outside the feature extraction device 100a.

[0092] The key point calculator 124 may receive the scaled-down image signal SIS, the scaling factor SF, and an initial key point signal IKP, may calculate a plurality of scaled-down key points included in the scaled-down image signal SIS and corresponding to the plurality of initial key points, and may output a scaled-down key point signal SKP indicating the plurality of scaled-down key points.

[0093] The descriptor calculator 126 may receive the scaled-down image signal SIS and the scaled-down key point signal SKP, may calculate the plurality of descriptors, and may output a descriptor signal DSC.

[0094] The feature point generator 130 may implement or execute Figure 1 step S600 in. For example, the feature point generator 130 may receive the initial key point signal IKP and the descriptor signal DSC, may obtain a plurality of feature points by respectively matching the plurality of initial key point signals IKP with the descriptor signal DSC, and output a feature point signal FP indicating the plurality of feature points.

[0095] In some exemplary embodiments, at least a part of the key point estimator 110, the descriptor generator 120, and the feature point generator 130 may be implemented in hardware. For example, at least a part of the elements included in the feature extraction device 100a according to the exemplary embodiment may be included in a computer-based electronic system. In another embodiment, all of the key point estimator 110, the descriptor generator 120, and the feature point generator 130 may be implemented in hardware. In other exemplary embodiments, at least a part of the key point estimator 110, the descriptor generator 120, and the feature point generator 130 may be implemented in software (e.g., instruction codes or program routines). For example, the instruction codes or program routines may be executed by a computer-based electronic system and may be stored in any storage device located inside or outside the computer-based electronic system. In another embodiment, all of the key point estimator 110, the descriptor generator 120, and the feature point generator 130 may be implemented in software.

[0096] Figure 3 Shows Figure 1 an example of generating a plurality of descriptors in. Figure 4A 、 Figure 4B 、 Figure 5Aand Figure 5B shows Figure 3 the operations that generate the multiple descriptors. Refer to Figure 1 、 Figure 3 、 Figure 4A and Figure 4B Before generating the multiple descriptors, multiple initial key points IKP11, IKP12, and IKP13 can be estimated based on the input image IMG1 (step S200).

[0097] As Figure 4A shown, the input image IMG1 may include two objects OBJ11 and OBJ12. For example, two corner points of the object OBJ11 and one corner point of the object OBJ12 can be estimated as (or referred to as) the multiple initial key points IKP11, IKP12, and IKP13. This way of estimating corner points as key points can be referred to as corner detection or a corner detection scheme.

[0098] The corner points of an object in an image can be defined as the intersection points of at least two edges of the object in the image. In addition, the corner points of an object in the image can also be defined as the points where there are at least two main and different edges in the local neighborhood of the points. The key points or interest points in an image can be points with well-defined positions that are to be robustly detected (i.e., to be well detected in a noisy background). For example, the corner points of an object in an image can be used as key points or interest points in the image. In addition, the vertices of an object in the image can be used as key points or interest points in the image.

[0099] In some exemplary embodiments, the corner points in an image can be detected based on at least one of various algorithms such as Features from Accelerated Segment Test (FAST), Harris, etc. The detected corner points can be estimated as the initial key points IKP11, IKP12, and IKP13.

[0100] In other exemplary embodiments, the blob points or blob regions ( Figure 4A not shown in

[0101] Blob detection or blob detection schemes may relate to detecting regions in a digital image that are different in nature (e.g., brightness or color) compared to the surrounding area. For example, a blob may be a region in the image where some properties are constant or approximately constant. All points in a blob may be similar to each other. For example, a blob may be used as a key point. The blob region may be referred to as a region of interest (ROI).

[0102] In some exemplary embodiments, blobs may be detected based on at least one of various algorithms such as Maximally Stable Extremal Region (MSER). The detected blobs may be estimated as initial key points.

[0103] As described above, the plurality of initial key points may be estimated based on at least one of corner detection and blob detection. For example, the plurality of initial key points may be estimated based on at least one of other detection schemes such as ridge detection, edge detection, etc.

[0104] Hereinafter, exemplary embodiments will be described based on a case where the corners of an object in an image are estimated as the plurality of initial key points IKP11, IKP12, and IKP13.

[0105] As described above, in step S400, the plurality of descriptors are generated based on the downscaled image SIMG1. In step S410, the downscaled image SIMG1 may be generated by downscaling the input image IMG1 based on a scaling factor. For example, step S410 may be performed by Figure 2 the scaler 122 therein.

[0106] As Figure 4B shown, the downscaled image SIMG1 may be an image generated by downscaling the input image IMG1 by approximately half (i.e., about 1 / 2). The scaling factor for downscaling the input image IMG1 may be about 2. The objects SOBJ11 and SOBJ12 in the downscaled image SIMG1 may be downscaled from the objects OBJ11 and OBJ12 in the input image IMG1. For example, the size of the edges of the objects SOBJ11 and SOBJ12 in the downscaled image SIMG1 may be half (i.e., 1 / 2) the size of the corresponding edges of the objects OBJ11 and OBJ12 in the input image IMG1.

[0107] The plurality of shrunk key points SKP11, SKP12, and SKP13 in the shrunk image SIMG1 can be calculated based on a scaling factor and the plurality of initial key points IKP11, IKP12, and IKP13 (step S420). For example, the plurality of shrunk key points SKP11, SKP12, and SKP13 in the shrunk image SIMG1 can be calculated by applying the scaling factor to the plurality of initial key points IKP11, IKP12, and IKP13. The plurality of shrunk key points SKP11, SKP12, and SKP13 in the shrunk image SIMG1 can respectively correspond to the plurality of initial key points IKP11, IKP12, and IKP13 in the input image IMG. Step S420 can be implemented by Figure 2 the key point calculator 124 in

[0108] In some exemplary embodiments, when the pixel positions of n initial key points in the input image IMG1 are x i ={x i , y i}, where i = 1, 2, …, n and n is a natural number greater than or equal to 2, the pixel positions of the n shrunk key points in the shrunk image SIMG1 can be obtained by Equation 1

[0109] [Equation 1]

[0110]

[0111] In Equation 1, Δs can correspond to the scaling factor.

[0112] For example, the upper left pixel position in the input image IMG1 can be (1, 1), and the upper left pixel position in the shrunk image SIMG1 can be (1, 1). In this case, when the scaling factor is about 2, the pixel position of the initial key point IKP11 in the input image IMG1 is (60, 20), and by Equation 1, the pixel position of the shrunk key point SKP11 in the shrunk image SIMG1 can be (30, 10).

[0113] As described above, when calculating the pixel positions of the shrunk key points in the shrunk image by applying the scaling factor to the pixel positions of the initial key points in the input image, loss of the position information of the initial key points in the input image can be prevented because the pixel positions of the initial key points are estimated from the input image (e.g., the original image), and thus the position accuracy of the shrunk key points can be maintained because the pixel positions of the shrunk key points are calculated based on the lossless position information of the initial key points.

[0114] The multiple descriptors of the reduced image SIMG1 can be calculated based on multiple adjacent points adjacent to the multiple reduced key points SKP11, SKP12, and SKP13 in the reduced image SIMG1 (step S430). The multiple adjacent points can represent pixels adjacent to the pixel positions of the multiple reduced key points SKP11, SKP12, and SKP13. Step S430 can be implemented by Figure 2 the descriptor calculator 126 in

[0115] In some exemplary embodiments, at least one of various algorithms such as the following can be used to generate the multiple descriptors: Oriented Features from Accelerated Segment Test (FAST) and Rotated Binary Robust Independent Elementary Features (BRIEF) (ORB), Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Maximal Self-Dissimilarities (MSD).

[0116] SIFT can be a feature detection algorithm for extracting features in an image. The extracted feature points generated by SIFT can be applied to an image processing system such as a surveillance camera or an autonomous navigation system. The image processing system can identify the object by deriving high-order descriptors from the extracted feature points of the object in the image.

[0117] In addition, SURF can be another method for extracting features in an image. The extracted feature points generated by SURF can be applied to an image processing system such as an object tracking system or a panoramic image generation system. The image processing system can identify the object in the image by deriving feature points and high-order descriptors according to each scale of the integral image, which is generated by summing or accumulating the pixel values of the input image. Although SIFT and SURF have the advantage of being robust to the size of the image (or object), illumination changes, and image changes caused by rotation, SIFT and SURF have the disadvantage of requiring complex calculations because the generated descriptors are real-valued or floating vectors.

[0118] The ORB can be the oriented FAST and rotated BRIEF algorithms. For example, when using the ORB to identify an object in an image, the feature points of the object in the image can be extracted by the oriented FAST or rotated BRIEF algorithm, and the binary descriptor can be generated based on the extracted feature points generated by the oriented FAST or rotated BRIEF algorithm. Since the generated binary descriptor is a binary vector, the computational amount can be reduced and the object recognition speed can be increased.

[0119] Based on the above-described algorithm, n reduced key points in the reduced image SIMG1 can be Generated n descriptors

[0120] As described above, a plurality of descriptors can be generated based on at least one of ORB, SIFT, SURF, and MSD. For example, the plurality of descriptors can be generated based on at least one of other algorithms such as the following: BRIEF, Binary Robust Invariant Scalable Key point (BRISK), Fast Retina Key point (FREAK).

[0121] After generating the plurality of descriptors, the plurality of feature points are obtained by respectively matching the plurality of initial key points IKP11, IKP12, and IKP13 in the input image IMG1 with the plurality of descriptors (step S600).

[0122] For example, the initial key point IKP11 can be matched with the descriptor of the reduced key point SKP11 to obtain a feature point. Similarly, the n initial key points x i ={x i , y i}, i = 1, 2,..., n can be matched with the n descriptors To obtain n feature points

[0123] Refer to Figure 5A And Figure 5B , Figure 5A Shows Figure 4A An enlarged view of region A in Figure 5B Shows Figure 4B An enlarged view of region A' in

[0124] For example, in Figure 5AIn this case, pixel PA may have a light gray color (e.g., high gray level), pixels PB and PC adjacent to pixel PA may have a dark gray color (e.g., low gray level), and pixels PD and PE adjacent to pixels PB and PC may have a black color (e.g., zero gray level). For example, in Figure 5B since the image is reduced by a scaling factor of approximately half, pixel PA' (corresponding to pixel PA in Figure 5A ) may have a relatively light gray color, pixels corresponding to pixels PB and PC in Figure 5A may be omitted or deleted, and pixels PD' and PE' (corresponding to pixels PD and PE in Figure 5A ) may have a black color.

[0125] For example, Figure 5B the intensity difference PIDIFF' between adjacent pixels PA' and PD' in the reduced image SIMG1 in Figure 5A may be greater than the intensity difference PIDIFF between adjacent pixels PA and PB in the input image IMG1 in

[0126] That is to say, as the image is reduced, the intensity difference between adjacent pixels may increase. As described above, the plurality of initial key points IKP11, IKP12, and IKP13 can be estimated from the input image IMG1 so that the position accuracy of the plurality of initial key points can be maintained. In addition, when the plurality of descriptors are generated from the reduced image SIMG1 in which the intensity difference PIDIFF' between adjacent pixels PA' and PD' is relatively large, the plurality of descriptors can be stable with respect to noise and can have improved singularity. For example, when a low-light image captured in a dark environment (e.g., a night photo) and / or a blurred image captured under a large movement of an object is reduced, the intensity difference (or brightness difference) between adjacent pixels in the low-light image or the blurred image can increase. Therefore, the descriptors obtained in the reduced image can have improved specificity or singularity compared to the descriptors obtained in the original image.

[0127] Figure 6 shows a method for extracting features from an image according to an exemplary embodiment. Referring to Figure 6 , when the input image is not a 1-channel image, the input image can be converted into a 1-channel image (step S100).

[0128] In some exemplary embodiments, the input image may be a 3-channel image with a Bayer pattern (e.g., a red-green-blue (RGB) image). It may be difficult to extract feature points from the original image whose pixel data is in a Bayer pattern. Therefore, the original image can be converted into a 1-channel image for feature extraction.

[0129] For example, a plurality of initial key points may be estimated based on a one-channel image (step S200a), and a plurality of descriptors may be generated based on a reduced image generated by reducing the one-channel image (step S400a). Figure 6 Steps S200a and S400a in Figure 1 may be substantially the same as steps S200 and S400 in

[0130] except that steps S200a and S400a are implemented based on a one-channel image. For example, a plurality of feature points may be obtained by respectively matching the plurality of initial key points with the plurality of descriptors (step S600). Figure 6 Step S600 in Figure 1 may be substantially the same as step S600 in

[0131] Figure 7 shows a feature extraction device according to an exemplary embodiment. Referring to Figure 7 , the feature extraction device 100b may include a key point estimator 110, a descriptor generator 120, a feature point generator 130, and an image converter 140.

[0132] Figure 7 The feature extraction device 100b in Figure 2 may be substantially the same as the feature extraction device 100a in

[0133] except that the feature extraction device 100b further includes an image converter 140. The image converter 140 may receive the original image signal IS_R indicating the input image, may convert the input image into a one-channel image, and may output the one-channel image signal IS_1C indicating the one-channel image. The input image may be an unprocessed original image. In other words, the image converter 140 may implement or execute Figure 6 step S100 in

[0134] In some exemplary embodiments, the original image signal IS_R may be provided from an external image capturing device (e.g., Figure 19 the image capturing device 1050 in Figure 19 ) and / or an external memory device (e.g.,

[0135] In some exemplary embodiments, at least a part of the image converter 140 may be implemented in hardware. In other exemplary embodiments, at least a part of the image converter 140 may be implemented in software (e.g., instruction codes or program routines).

[0136] Figure 8 shows Figure 6 an example of converting an input image into a one-channel image in Figure 9 showsFigure 8 Instances of generating multiple one-channel pixel data are produced. Figure 10A and Figure 10B illustrates an example for explaining Figure 8 the operation of converting an input image into a one-channel image in

[0137] Referring to Figure 6 , Figure 8 , Figure 9 , Figure 10A and Figure 10B , an input image is converted into a one-channel image ( Figure 6 step S100 in Figure 8 ). An unprocessed original image RIMG can be received as the input image (

[0138] As Figure 10A shown in

[0139] For example, referring to Figure 10B, a plurality of single-channel pixel data P11, P12, P13, P14, P15, P16, ..., P21, P22, P23, P24, P25, P26, ..., P31, P32, P33, P34, P35, P36, ..., P41, P42, P43, P44, P45, P46, ... included in the single-channel image GIMG can be generated based on all of the plurality of original pixel data R11, G12, R13, G14, R15, G16, ..., G21, B22, G23, B24, G25, B26, ..., R31, G32, R33, G34, R35, G36, ..., G41, B42, G43, B44, G45, B46, ... included in the original image RIMG (step S120). The plurality of original pixel data can represent pixel values (e.g., grayscale) of a color image obtained from the plurality of color pixels, and the plurality of single-channel pixel data can represent pixel values of a grayscale image.

[0140] In some exemplary embodiments, the original image RIMG and the single-channel image GIMG can have substantially the same size. In other words, the number of the plurality of original pixel data (or the number of the plurality of color pixels) in the original image RIMG can be substantially the same as the number of the plurality of single-channel pixel data (or the number of multiple pixels or grayscale pixels) in the single-channel image GIMG.

[0141] For example, when generating the plurality of single-channel pixel data (step S120), one single-channel pixel data (e.g., the first single-channel pixel data) of the plurality of single-channel pixel data can be generated based on X original pixel data of the plurality of original pixel data, where X is a natural number greater than or equal to 2. The X original pixel data can be obtained from X pixels adjacent to each other. For example, the summed original pixel data can be generated by adding the X original pixel data (step S122). The one single-channel pixel data can be generated by dividing the summed original pixel data by X (step S124).

[0142] In some exemplary embodiments, the X original pixel data can be obtained from four pixels arranged in a 2*2 matrix form. The pixel data of the four pixels can be a pixel data group. In other words, since the RGGB color pattern including one red pixel data, two green pixel data, and one blue pixel data arranged in a 2*2 matrix form is repeatedly included in the original image RIMG, X is 4, and each of the plurality of single-channel pixel data can be generated by the one pixel data group.

[0143] For example, the original pixel data R11 at the corresponding position in the original image RIMG and the original pixel data G12, G21, and B22 adjacent to the original pixel data R11 can be used to calculate the one-channel pixel data P11 in the one-channel image GIMG. For example, the value of the one-channel pixel data P11 can be obtained by the following equation: P11 = (R11 + G12 + G21 + B22) / 4.

[0144] Similarly, the original pixel data G12, R13, B22, and G23 arranged in a 2*2 matrix form in the original image RIMG can be used to calculate the one-channel pixel data P12 in the one-channel image GIMG. The original pixel data G21, B22, R31, and G32 arranged in a 2*2 matrix form in the original image RIMG can be used to calculate the one-channel pixel data P21 in the one-channel image GIMG, and the remaining part of the one-channel pixel data in the one-channel image GIMG can be calculated in a similar manner.

[0145] In some exemplary embodiments, the one-channel pixel data arranged in the last row and the last column in the one-channel image GIMG may not have a corresponding group of pixel data arranged in a 2*2 matrix form in the original image RIMG. Therefore, one original pixel data at the corresponding position in the original image RIMG, or one original pixel data at the corresponding position in the original image RIMG and another original pixel data adjacent to the one original pixel data can be used to calculate the one-channel pixel data arranged in the last row and the last column in the one-channel image GIMG. For example, the one-channel pixel data arranged in the first row and the last column in the one-channel image GIMG may be substantially equal to the original pixel data arranged in the first row and the last column in the original image RIMG, or may be calculated by adding the two original pixel data arranged in the first row and the second row and the last column in the original image RIMG and dividing the added original pixel data by 2.

[0146] Figure 11 Show Figure 6 Another example of converting the input image into a one-channel image. Figure 12 Show Figure 11 An example of generating a plurality of one-channel pixel data. Refer to Figure 6 , Figure 10A , Figure 10B , Figure 11 And Figure 12 , when converting the input image into a one-channel image (step S100), the unprocessed original image RIMG can be received as the input image (step S110). Figure 11 The step S110 in Figure 8 May be substantially the same as the step S110 in

[0147] Referring to Figure 10A and Figure 10B , a plurality of 1-channel pixel data P11, P12, P13, P14, P15, P16,... P21, P22, P23, P24, P25, P26,... P31, P32, P33, P34, P35, P36,... P41, P42, P43, P44, P45, P46,... included in the 1-channel image GIMG can be generated based on a part of a plurality of original pixel data R11, G12, R13, G14, R15, G16,... G21, B22, G23, B24, G25, B26,... R31, G32, R33, G34, R35, G36,... G41, B42, G43, B44, G45, B46,... included in the original image RIMG ( Figure 11 step S130) in

[0148] When generating the plurality of 1-channel pixel data ( Figure 11 step S130) in Figure 12 the first original pixel data in the plurality of original pixel data can be selected (step S132). For example, the first original pixel data can be obtained from the first pixel corresponding to the same color. The plurality of 1-channel pixel data can be obtained by magnifying the first original pixel data ( Figure 12 step S134) in Figure 12 For example, the first original pixel data can be set as a part of the plurality of 1-channel pixel data ( Figure 12 step S136) in Figure 12 and the remaining part of the plurality of 1-channel pixel data can be calculated by interpolation operations performed based on the first original pixel data ( Figure 12 step S138) in

[0149] In some exemplary embodiments, referring to Figure 10A and Figure 10B , the red pixel data R11, R13, R15,... R31, R33, R35,... in the original image RIMG can be selected as the first original pixel data, and the values of the 1-channel pixel data P11, P13, P15,... P31, P33, P35,... at the corresponding positions in the 1-channel image GIMG can be set to be equal to the values of the red pixel data R11, R13, R15,... R31, R33, R35,... respectively.

[0150] The remaining one-channel pixel data P12, P14, P16, ..., P21, P22, P23, P24, P25, P26, ..., P32, P34, P36, ..., P41, P42, P43, P44, P45, P46, ... in the one-channel image GIMG can be calculated through interpolation operations. For example, the one-channel pixel data P12 can be obtained through P12 = (P11 + P13) / 2 = (R11 + R13) / 2.

[0151] Similarly, the one-channel pixel data P21 in the one-channel image GIMG can be calculated using the one-channel pixel data P11 and P31, the one-channel pixel data P22 in the one-channel image GIMG can be calculated using the one-channel pixel data P11 and P33, and the remaining part of the one-channel pixel data in the one-channel image GIMG can be calculated in a similar manner.

[0152] In other exemplary embodiments, referring to Figure 10A and Figure 10B , the blue pixel data B22, B24, B26, ..., B42, B44, B46, .. in the original image RIMG can be selected as the first original pixel data, and the multiple one-channel pixel data in the one-channel image GIMG can be obtained based on the blue pixel data B22, B24, B26, ..., B42, B44, B46, ... in the original image RIMG.

[0153] Figure 13 A method for image matching according to an exemplary embodiment is shown. Referring to Figure 13 , a plurality of first feature points are detected based on a first input image (step S1100), and a plurality of second feature points are detected based on a second input image (step S1200). The first input image and the second input image can be different images. For example, the first input image and the second input image can be two consecutive frame images in a moving image, that is, they can be the (t - 1)th frame image and the tth frame image respectively.

[0154] Each of steps S1100 and S1200 can be implemented or executed based on a method for extracting features from an image according to an exemplary embodiment, as referring to Figure 13 in Figures 1 to 12As described above. For example, a plurality of first initial key points may be estimated based on a first input image. A plurality of first descriptors may be generated based on a first downscaled image generated by downscaling the first input image. The plurality of first feature points may be obtained by respectively matching the plurality of first initial key points with the plurality of first descriptors. Similarly, a plurality of second initial key points may be estimated based on a second input image. A plurality of second descriptors may be generated based on a second downscaled image generated by downscaling the second input image. The plurality of second feature points may be obtained by respectively matching the plurality of second initial key points with the plurality of second descriptors.

[0155] A plurality of corresponding points between the first input image and the second input image are obtained based on the plurality of first feature points and the plurality of second feature points (step S1300). For example, a corresponding point may be obtained by matching a first initial key point in the first input image with a second initial key point in the second input image. This matching operation may be performed using the original image (e.g., the original scale image).

[0156] As described above, when steps S1100 and S1200 are performed based on the method of extracting image features according to an exemplary embodiment, descriptors that are robust to noise and have improved singularity may be obtained. In addition, when the plurality of corresponding points are obtained using the plurality of first descriptors and second descriptors having improved singularity, the accuracy of estimating the corresponding points between the first input image and the second input image (e.g., the matching accuracy between the first input image and the second input image) may be improved.

[0157] Figure 14 An image matching apparatus according to an exemplary embodiment is shown. Referring to Figure 14 , the image matching apparatus 200 may include a feature extraction apparatus 100, a similarity comparator 210, and a corresponding point generator 220.

[0158] The feature extraction apparatus 100 may receive a first image signal IS1 indicating a first input image and a second image signal IS2 indicating a second input image, may detect a plurality of first feature points and a plurality of second feature points based on the first input image and the second input image, and may output a first feature point signal FP1 indicating the plurality of first feature points and a second feature point signal FP2 indicating the plurality of second feature points.

[0159] Figure 14 The feature extraction apparatus 100 in Figure 2 may correspond to the feature extraction apparatus 100a in Figure 7 or the feature extraction apparatus 100b in Figure 14The feature extraction device 100 in Figure 2 may be the feature extraction device 100a in Figure 14 . When each of the first input image and the second input image is an original image, Figure 7 the feature extraction device 100 in

[0160] The similarity comparator 210 may receive the first feature point signal FP1 and the second feature point signal FP2, compare the similarity between the plurality of first descriptors included in the plurality of first feature points and the plurality of second descriptors included in the plurality of second feature points, and output a comparison signal COMP indicating the comparison result.

[0161] The corresponding point generator 220 may receive the first feature point signal FP1, the second feature point signal FP2, and the comparison signal COMP, obtain a plurality of corresponding points between the first input image and the second input image, and output a corresponding point signal CP indicating the plurality of corresponding points.

[0162] In some exemplary embodiments, at least a part of the similarity comparator 210 and the corresponding point generator 220 may be implemented in hardware. In other exemplary embodiments, at least a part of the similarity comparator 210 and the corresponding point generator 220 may be implemented in software (e.g., instruction codes or program routines).

[0163] Figure 15 illustrates Figure 13 an example of obtaining a plurality of corresponding points in Figure 16 illustrates an example for explaining Figure 15 the operations of obtaining the plurality of corresponding points in

[0164] Referring to Figure 4A , Figure 13 , Figure 15 and Figure 16 , before obtaining the plurality of corresponding points, a plurality of first feature points are obtained by matching the plurality of first initial key points IKP11, IKP12, and IKP13 in the first input image IMG1 (e.g., the (t - 1)-th frame image) with the plurality of first descriptors as Figure 4A shown in as Figure 16 shown in

[0165] When obtaining the plurality of corresponding points (step S1300), one of the plurality of first descriptors among the plurality of first feature points may be selected (step S1310), and one of the plurality of second descriptors among the plurality of second feature points may be selected by comparing the similarity between the selected first descriptor and the plurality of second descriptors (step S1320). The selected second descriptor may have the highest similarity with the selected first descriptor. Steps S1310 and S1320 may be implemented by Figure 14 the similarity comparator 210 in

[0166] One of the plurality of first initial key points corresponding to the selected first descriptor among the plurality of first feature points and one of the plurality of second initial key points corresponding to the selected second descriptor among the plurality of second feature points may be set as one of the plurality of corresponding points (step S1330). Step S1330 may be implemented by Figure 14 the corresponding point generator 220 in

[0167] For example, the first descriptor corresponding to the first initial key point IKP11 may be selected, the similarity between the selected first descriptor and all the second descriptors corresponding to the second initial key points IKP21, IKP22, and IKP23 may be compared, and a second initial key point corresponding to the second descriptor having the highest degree of similarity with the selected first descriptor may be selected. For example, the first initial key point IKP11 and the second initial key point IKP21 may be set as a corresponding point.

[0168] Similarly, the similarity comparison may be performed on the remaining first descriptors, the first initial key point IKP12 and the second initial key point IKP22 may be set as another corresponding point, and the first initial key point IKP13 and the second initial key point IKP23 may be set as another corresponding point. Therefore, all the plurality of corresponding points may be obtained by comparing the descriptors of the plurality of first feature points with the similarity between the descriptors of the plurality of second feature points

[0169] In some exemplary embodiments, the similarity between the plurality of first descriptors and the plurality of second descriptors may be determined based on at least one of the Hamming distance and the Euclidean distance. For example, the similarity between the plurality of first descriptors and the plurality of second descriptors may be determined based on at least one of various other algorithms.

[0170] Reference may be made to Figure 15The exemplary embodiment is described based on a case in which one corresponding point is set for a first descriptor (e.g., by comparing the similarity between one first descriptor and all second descriptors). For example, a second descriptor may be first selected, the similarity between the selected second descriptor and all first descriptors may be compared to select one first descriptor having the highest degree of similarity with the selected second descriptor, and one first initial key point and one second initial key point corresponding to the selected first descriptor and second descriptor may be set as one corresponding point.

[0171] Figure 17 A method of processing an image according to an exemplary embodiment is shown. Figure 17 , a plurality of corresponding points between the first input image and the second input image may be set (step S2100).

[0172] The method of matching images according to the exemplary embodiment may be implemented or performed. Figure 17 Step S2100 in Figures 13 to 16 For example, a plurality of first feature points may be detected based on the first input image. A plurality of second feature points may be detected based on the second input image. The plurality of corresponding points between the first input image and the second input image may be obtained based on the plurality of first feature points and the plurality of second feature points.

[0173] Image processing may be performed on at least one of the first input image and the second input image based on the plurality of corresponding points (step S2200). For example, the image processing may include various functions such as camera motion, noise reduction, video digital image stabilization (VDIS), high dynamic range (HDR), etc. In addition, the image processing may further include various functions such as motion detection, image registration, video tracking, image mosaicing, panoramic stitching, three-dimensional (3D) modeling, object recognition, etc.

[0174] Figure 18 An image processing apparatus according to an exemplary embodiment is shown. Figure 18 , the image processing device 300 may include an image matching device 200 and a processing unit 310.

[0175] The image matching apparatus 200 may receive a first image signal IS1 indicating a first input image and a second image signal IS2 indicating a second input image, may set a plurality of corresponding points between the first input image and the second input image, and may output a corresponding point signal CP indicating the plurality of corresponding points. Figure 18The image matching device 200 in Figure 14 may correspond to the image matching device 200 in

[0176] The processing unit 310 may receive a first image signal IS1, a second image signal IS2, and a corresponding point signal CP, may perform image processing on at least one of the first input image and the second input image, and may output an output image signal OIS indicating the result of the image processing.

[0177] In some exemplary embodiments, at least a part of the processing unit 310 may be implemented in hardware. In other exemplary embodiments, at least a part of the processing unit 310 may be implemented in software (e.g., instruction codes or program routines).

[0178] As those skilled in the art should understand, the present invention may be implemented as a system, a method, a computer program product, and / or a computer program product implemented in one or more computer-readable media having computer-readable program code embodied thereon. The computer-readable program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, the computer-readable media may be a non-transitory computer-readable media.

[0179] Figure 19 An electronic system according to an exemplary embodiment is shown. Referring to Figure 19 , the electronic system 1000 may include a processor 1010, connectivity 1020, a memory device 1030, a user interface 1040, an image capture device 1050, and an image processing device 1060. In addition, the electronic system 1000 may include a power supply.

[0180] The processor 1010 may perform various computing functions, e.g., specific calculations and tasks. The connectivity 1020 may communicate with external devices. The memory device 1030 may be used as a data storage for data processed by the processor 1010, or as a working memory. The user interface 1040 may include at least one input device (e.g., a keypad, buttons, a microphone, a touch screen, etc.) and / or at least one output device (e.g., a speaker or a display device, etc.). The power supply may supply power to the electronic system 1000.

[0181] The image capturing device 1050 and the image processing device 1060 can be controlled by the processor 1010. The image capturing device 1050 can provide an input image to the image processing device 1060. For example, the image capturing device 1050 may include a complementary metal oxide semiconductor (CMOS) image sensor, a charged coupled device (CCD) image sensor, etc. The image processing device 1060 can be Figure 18 the image processing device 300 in Figures 1 to 18 and can operate according to the exemplary embodiment described with reference to

[0182] As a summary and review, when key point estimation is performed at the same image scale for image matching and descriptor generation, noise can be induced.

[0183] In the method of extracting features from an image, the method of matching images, and the method of processing images according to the exemplary embodiment, a plurality of key points can be estimated based on the input image, and a plurality of descriptors can be generated based on the downsampled image. The position accuracy of the key points can be maintained by estimating the key points using the original image, and descriptors that are robust to noise and have improved singularity can be obtained by generating descriptors using the downsampled image in which the intensity difference between adjacent pixels is relatively large. In addition, when using the plurality of descriptors with improved singularity to obtain a plurality of corresponding points, the accuracy of corresponding point estimation (e.g., the matching accuracy of two input images) between two input images can be improved.

[0184] Specifically, a low-illumination image captured in a relatively dark environment (e.g., a night photo) and / or a blurred image in which the object motion is significant can be provided as the input image. In such a case, when the descriptor is obtained based on the downsampled image of the low-illumination image or the blurred image instead of its original image, the descriptor obtained in the downsampled image can have improved specificity or singularity compared to the descriptor obtained in the original image.

[0185] Each exemplary embodiment can be applied to various electronic devices and electronic systems including an image processing device. For example, each exemplary embodiment can be applied to systems such as mobile phones, smart phones, tablet computers, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, portable game consoles, music players, camcorders, video players, navigation devices, wearable devices, Internet of Things (IoT) devices, Internet of Everything (IoE) devices, e-book readers, virtual reality (VR) devices, augmented reality (AR) devices, robot devices, and the like. In addition, each exemplary embodiment can be applied to various devices and systems that require image processing, such as automotive cameras, medical cameras, and the like.

[0186] Exemplary embodiments have been disclosed herein, and although specific terms have been employed, they are to be used and interpreted in a general and illustrative sense only and not for purposes of limitation. In some instances, as will be apparent to those of ordinary skill in the art from the filing of this application, unless otherwise specifically indicated, features, characteristics, and / or elements described in connection with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in connection with other embodiments. Accordingly, those skilled in the art will appreciate that various changes may be made in form and detail without departing from the spirit and scope of the invention as set forth in the claims above.

Claims

1. A method for extracting features from an image, the method comprises: estimating a plurality of initial key points based on a 1-channel image; reducing the 1-channel image by a scaling factor to produce a reduced 1-channel image; calculating a plurality of reduced key points from the plurality of initial key points using the scaling factor; generating a plurality of descriptors for the reduced 1-channel image based on the plurality of reduced key points; and obtaining a plurality of feature points by respectively matching the plurality of initial key points with the plurality of descriptors, wherein: the 1-channel image is generated from an original image having X color channels arranged in groups of 4 pixels as a 2×2 matrix, the 4 pixels corresponding to two or more colors, where X is a natural number greater than or equal to 2, and the 1-channel image is produced by adding the values of the 4 pixels of the 2×2 matrix and then dividing by 4, and the plurality of initial key points are estimated based on the produced 1-channel image.

2. The method according to claim 1, wherein generating the plurality of descriptors comprises: calculating the plurality of descriptors for the reduced 1-channel image based on a plurality of adjacent points adjacent to the plurality of reduced key points in the reduced 1-channel image.

3. The method according to claim 2, wherein when the 1-channel image is converted into the reduced 1-channel image, the intensity difference between adjacent pixels increases.

4. The method according to claim 1, wherein the plurality of initial key points are estimated using at least one of corner detection and blob detection.

5. The method according to claim 1, wherein the plurality of descriptors are generated using at least one of: Oriented FAST and Rotated BRIEF, Scale Invariant Feature Transform, Speeded Up Robust Features, and Maximum Non-Self Similarity.

6. A method for matching images, the method comprises: detecting a plurality of first feature points based on a first input image; detecting a plurality of second feature points based on a second input image; and obtaining a plurality of corresponding points between the first input image and the second input image based on the plurality of first feature points and the plurality of second feature points, wherein detecting the plurality of first feature points includes extracting features from the first input image according to the method of claim 1.

7. The method according to claim 6, wherein detecting the plurality of second feature points comprises: estimating a plurality of second initial key points based on the second input image; generating a plurality of second descriptors based on a second reduced image produced by reducing the second input image; and obtaining the plurality of second feature points by respectively matching the plurality of second initial key points with the plurality of second descriptors.

8. The method according to claim 7, wherein obtaining the plurality of corresponding points comprises: selecting one of the plurality of first descriptors among the plurality of first feature points from the first input image; One of the plurality of second descriptors is selected by comparing the similarity between the selected first descriptor and the plurality of second descriptors among the plurality of second feature points, and the selected second descriptor has the highest similarity with the selected first descriptor; and One of the plurality of first initial key points corresponding to the selected first descriptor among the plurality of first feature points and one of the plurality of second initial key points corresponding to the selected second descriptor among the plurality of second feature points are set as one of the plurality of corresponding points.

9. The method according to claim 8, wherein the similarity between the selected first descriptor and the selected second descriptor is determined based on at least one of a Hamming distance and an Euclidean distance.

10. A method for processing an image, the method comprises: setting a plurality of corresponding points between a first input image and a second input image; and performing image processing on at least one of the first input image and the second input image based on the plurality of corresponding points, wherein setting the plurality of corresponding points comprises: detecting a plurality of first feature points based on the first input image; detecting a plurality of second feature points based on the second input image; and obtaining the plurality of corresponding points based on the plurality of first feature points and the plurality of second feature points, wherein detecting the plurality of first feature points comprises extracting features from the first input image according to the method of claim 1.

Images