Image matching method and image matching device

Abstract

An image matching method and an image matching device are provided. An image matching method comprises: extracting a marker block including a marker point of the object from a first image of the object; extracting a target block corresponding to the marker block from a second image of the object; and determining a target point in the second image corresponding to the marker point based on a match between the marker block and the target block.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2018-0017303, filed on February 12, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to an image matching technique. Background Art

[0003] Recently, due to the continuous development of various mobile devices and wearable devices (such as smart phones), the importance of security authentication has continued to increase. Authentication technology using biometrics uses fingerprints, irises, voices, faces, blood vessels, etc. to authenticate users. The biological characteristics used for authentication are unique to individuals, convenient to carry, and stable throughout a person's life. In addition, biological characteristics are difficult to steal or forge. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In a general aspect, an image matching method is provided, comprising: extracting a marker block including a marker point of the object from a first image of the object; extracting a target block corresponding to the marker block from a second image of the object; and determining a target point in the second image corresponding to the marker point based on a match between the marker block and the target block.

[0006] The image matching method may further include: using a color image sensor to acquire a color image as the first image; and using an infrared (IR) image sensor to acquire an IR image as the second image.

[0007] The image matching method may further include determining whether the object is an organizational structure of the user based on the target point.

[0008] The image matching method may further include allowing access to one or more features of the device through a user interface of the device in response to the object being determined to be a living tissue structure and / or an identified user.

[0009] The step of extracting the target block may include determining the target block in the second image based on a position of the marker block in the first image.

[0010] The extracting of the target block may include extracting the target block in response to a marker point being detected in a determined area of ​​the first image.

[0011] The extracting of the target block may include determining the determined area based on a difference between a field of view (FOV) of a first image sensor for capturing the first image and a FOV of a second image sensor for capturing the second image.

[0012] The step of determining the target point may include: retrieving a local area matching the marker block from the target block; and determining a center point of the retrieved local area as the target point.

[0013] The step of retrieving the local area may include: calculating the degree of similarity between the marker block and each of the multiple local areas of the target block; and determining the local area with the highest calculated degree of similarity among the multiple local areas as the local area matching the marker block.

[0014] The calculating of the degree of similarity may include calculating a degree of correlation between a value of a pixel in each of the plurality of local areas included in the target block and a value of a pixel included in the marker block as the degree of similarity.

[0015] The first image may be a color image, and the second image may be an infrared (IR) image; the step of extracting the marker block includes: selecting a channel image from the first image; and extracting the marker block from the selected channel image.

[0016] The first image may include a plurality of channel images; and the step of extracting the marker block may include extracting the marker block from a channel image having a minimum wavelength difference between the channel image and the second image among the plurality of channel images.

[0017] The step of extracting the marker block may include extracting the marker block from the first image for each of a plurality of markers of the object, the step of extracting the target block may include extracting the target block from the second image for each of the plurality of markers, and the step of determining the target point may include determining a target point corresponding to the marker point for each of the plurality of markers.

[0018] The step of extracting the marker block may include: determining an object region corresponding to the object in the first image; identifying a marker point of the object in the object region; and extracting the marker block including the identified marker point.

[0019] The image matching method may further include matching the first image with the second image based on the marker points of the first image and the target points of the second image.

[0020] The image matching method may further include: preprocessing the marker block and the target block using a Gaussian filter; and matching the preprocessed marker block and the preprocessed target block.

[0021] The step of extracting the target block may include determining the target block in the second image based on a position of the marker block in the first image and a distance between the image matching device and the object.

[0022] The image matching method may further include, in response to detecting a plurality of markers of the object in the first image, determining points in the second image corresponding to the remaining markers based on a target point associated with one of the plurality of markers of the object.

[0023] The image matching method may further include recognizing an object present in the second image based on the target point.

[0024] The image matching method may further include verifying activity of an object present in the second image based on the target point.

[0025] A non-transitory computer-readable storage medium may store instructions, wherein when the instructions are executed by a processor, the processor is caused to perform the image matching method.

[0026] In another general aspect, an image matching device is provided, comprising: one or more processors configured to: obtain a first image of an object and a second image of the object; extract a marker block including a marker point of the object from the first image, extract a target block corresponding to the marker block from the second image, and determine a target point in the second image corresponding to the marker point based on a match between the marker block and the target block.

[0027] The image matching apparatus may further include: one or more image sensors configured to acquire the first image and the second image for the process of obtaining the first image and the second image.

[0028] In another general aspect, an image matching method is provided, comprising: extracting a first block including first feature points of the object from a first image of the object; extracting a second block from a second image of the object based on the first block; determining second feature points in the second block; and identifying the object or verifying the identity of the object based on the second feature points and the second image.

[0029] Other features and aspects will be apparent from the following detailed description and accompanying drawings. Additional aspects will be set forth in part in the following description and in part will be apparent from the description, or may be learned by practice of the provided embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 and Figure 2 is a flowchart illustrating an example of an image matching method.

[0031] Figure 3is a flow chart illustrating an example of object recognition and liveness verification based on determined target points.

[0032] Figure 4 is a flow chart illustrating an example of a method of matching a color image and an infrared (IR) image.

[0033] Figure 5 An example of image matching processing and feature point extraction is shown.

[0034] Figure 6 and Figure 7 An example of image matching results is shown.

[0035] Figure 8 is a diagram illustrating an example of an image matching device.

[0036] Fig. 9 and Fig.10 An example of applying the image matching device is shown.

[0037] Unless otherwise described or provided, throughout the drawings and detailed description, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0038] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example, however, these orders are not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of known features may be omitted.

[0039] The features described herein can be implemented in different forms and will not be interpreted as being limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many possible ways to implement the methods, devices and / or systems described herein, which will be clear after understanding the disclosure of this application.

[0040] Throughout the specification, when an element (such as a layer, a region, or a substrate) is described as being “on,” “connected to,” or “coupled to” another element, the element may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present between them. Conversely, when an element is described as being “directly” “on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements present between them.

[0041] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0042] Although terms such as "first", "second" and "third" may be used herein to describe various components, assemblies, regions, layers or parts, these components, assemblies, regions, layers or parts are not limited by these terms. More specifically, these terms are only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teachings of the examples described herein, the first component, first component, first region, first layer or first part referred to in the examples may also be referred to as the second component, second component, second region, second layer or second part.

[0043] The terms used herein are only used to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "include", "comprises", and "have" indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0044] The features of the examples described herein may be combined in various ways as will be clear after understanding the disclosure of the present application. In addition, although the examples described herein have various configurations, other configurations are also feasible as will be clear after understanding the disclosure of the present application.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in view of the present disclosure and the art to which the present disclosure belongs. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as an idealized or overly formal meaning.

[0046] Furthermore, in the description of the embodiments, when it is considered that a detailed description of a well-known related structure or function will cause vague interpretation of the present disclosure, such description will be omitted.

[0047] Figure 1 and Figure 2 is a flowchart illustrating an example of an image matching method.

[0048] Figure 1 is a flowchart illustrating an example of an image matching method.

[0049] Reference Figure 1 In operation 110, the image matching device extracts a landmark block including a landmark point of an object from a first image. The first image represents an image captured by a first image sensor. For example, the first image may be a color image. However, the first image is not limited thereto. In addition, the first image may be a black and white image and / or a depth image.

[0050] The color image includes a plurality of color channel images. The color channel images may represent the intensity of light having a wavelength band corresponding to the corresponding color acquired by the first image sensor (e.g., a color sensor). For example, the color image includes a red channel image, a green channel image, and a blue channel image. The red channel image represents the intensity of light having a wavelength band corresponding to red being received. The green channel image represents the intensity of light having a wavelength band corresponding to green being received. The blue channel image represents the intensity of light having a wavelength band corresponding to blue being received.

[0051] The depth image represents an image captured by a first image sensor (e.g., a depth sensor). Each pixel of the depth image has a value indicating the distance between the corresponding pixel and the point corresponding to the pixel. The depth sensor may include a sensor based on a time of flight (TOF) scheme, a sensor based on a structured light scheme, etc. However, the depth sensor is not limited thereto.

[0052] Here, the term "landmark" may represent a desired part about an object. For example, if the object is a person's face, the landmark may be an eye, a nose, a mouth, an eyebrow, a lip, a pupil, etc. In addition, the term "landmark point" may represent a feature point representing a feature of an object. For example, a landmark point may be used to indicate both ends of an eye, the center of an eye, both ends of a mouth, a tip of a nose, etc. However, the landmark point is not limited thereto. The term "landmark block" may represent an image block including the landmark point in the first image. The term "image block" may represent a subset of pixels of an image or a cropped or processed portion of an image.

[0053] In operation 120, the image matching device extracts a target block corresponding to the marker block from the second image. The second image represents an image captured by a second image sensor. For example, the second image may be an infrared (IR) image. However, the second image is not limited thereto.

[0054] The target block may be an image block including a point corresponding to the position of the marker block. The image matching device roughly determines a position in the second image corresponding to the position of the marker point or the position of the marker block in the first image. The image matching device extracts a target block having a size larger than that of the marker block. The operation of determining the target block will be further described below.

[0055] Here, the image block may represent a set of pixels of a local area in the entire image. For example, the marker block may be a set of pixels of a local area in the first image. The target block may be a set of pixels of a local area in the second image.

[0056] The image type of the first image and the image type of the second image may be different. For example, the first image may be a color image and the second image may be an IR image. The first image may be an IR image and the second image may be a color image. The first image may be a depth image and the second image may be an IR image. The first image may be an IR image and the second image may be a depth image. However, the image type of each of the first image and the second image is not limited thereto. The image type of the first image and the image type of the second image may be the same. Both the first image and the second image may be color images. Both the first image and the second image may be IR images. For example, the image type of the first image and the image type of the second image may be classified based on the wavelength of light that can be received by each of the first image sensor and the second image sensor configured to capture the respective corresponding images. The first band of light that can be received by the first image sensor and the second band of light that can be received by the second image sensor may be different. Although the use of color images and IR images as image types is described herein, they are provided only as examples. For ease of description, the following description is made based on an example in which a color image is used as the first image and an IR image is used as the second image. However, the first image and the second image are not limited thereto.

[0057] In operation 130, the image matching device determines a target point in the second image corresponding to the marker point based on the matching between the marker block and the target block. The image matching device retrieves a local area matching the marker block from the target block. For example, the image matching device may divide the target block into a plurality of local areas, and may select a local area most similar to the marker block from the plurality of local areas. The image matching device may determine a point (e.g., a center point) of the local area selected from the second image as the target point corresponding to the marker point in the first image.

[0058] The image matching device may adjust the target block and the marker block at the same resolution by performing scaling on at least one of the target block and the marker block, and may perform block matching. For example, the image matching device may generate a plurality of local areas by dividing the target block into a plurality of windows each having a desired size. The image matching device may perform scaling on at least one of the local area and the marker block so that the size, resolution, and number of pixels of the local area may become the same as the size, resolution, and number of pixels of the marker block, respectively.

[0059] Therefore, the image matching apparatus can effectively determine feature points for the second image using feature points identified in the first image without additionally extracting feature points from the second image.

[0060] Figure 2 is a flow chart showing an image matching method.

[0061] Reference Figure 2 In operation 210, the image matching apparatus acquires a first image. For example, the image matching apparatus may acquire the first image by photographing or capturing an object using a first image sensor.

[0062] In operation 220, the image matching device determines the object region in the first image. In one example, the image matching device may determine the object region corresponding to the object in the first image based on an object model. The object model may be a model configured or trained to output a region corresponding to the object from an input image, for example, the object model may be a trained neural network.

[0063] A neural network represents a recognition model using a large number of nodes that can be connected by edges (e.g., by weighted connections) and / or to which trained kernels can be applied, for example, in an implemented convolution operation. The neural network is implemented by hardware or a combination of hardware and instructions (e.g., by instructions stored in a non-transitory memory of an image matching device, wherein when the instructions are executed by one or more processors of the image matching device, the one or more processors are caused to implement the recognition model). The recognition model can be trained by using data of various data structures stored in the memory of the image matching device. The various data structures may include different data structures that store the obtained training parameters (e.g., including trained connection weights and / or kernels) in vectors, matrices, volumes, or other single-dimensional or multi-dimensional data structures. In addition, although the recognition model is discussed using an example neural network structure, in other examples, alternative machine learning structures are also available.

[0064] In operation 230, the image matching apparatus extracts a landmark block based on the object region. In one example, the image matching apparatus may identify a landmark point of the object in the object region. For example, the image matching apparatus may determine the landmark point in the object region based on a landmark model. The landmark model may be a model that outputs landmark points from the object region, for example, the landmark model may also be a trained neural network (as a non-limiting example, a neural network trained for a landmark block extraction target). The image matching apparatus may extract a landmark block including the identified landmark point. For example, the image matching apparatus may use the landmark point as a center point to extract an image block as a landmark block.

[0065] In operation 240, the image matching device acquires the second image. For example, the image matching device acquires the second image by photographing the object using the second image sensor. The time point of capturing the first image may be different from the time of capturing the second image. For example, the image matching device may respectively acquire the first image and the second image at different time sequences. In addition, the image matching device may acquire the first image and the second image at the same time. In addition, as a non-limiting example, the image matching device may adjust the zoom between the two images based on the respective fields of view (FOV) of the first image sensor and the second image sensor and the relative position difference between the first image sensor and the second image sensor.

[0066] In operation 250, the image matching device determines a target block in the second image. The image matching device may determine the target block in the second image based on the position of the marker block in the first image. In addition, the image matching device may determine the target block in the second image based on the position of the marker block in the first image and the distance between the object and the image matching device. For example, the image matching device may determine an image block in the second image mapped to the position of the marker block in the first image as the target block. The image matching device may estimate an area in the second image where a feature point corresponding to the marker is expected to exist based on the position of the marker block in the first image.

[0067] In addition, the image matching device extracts the target block in response to extracting the marker point from the desired area (e.g., the determined area) of the first image. In one example, the image matching device determines the desired area based on the difference between the FOV of the first image sensor used to capture the first image and the FOV of the second image sensor used to capture the second image. In another example, the image matching device determines the desired area based on the arrangement between the first image sensor capturing the first image and the second image sensor capturing the second image, the FOV of the first image sensor, and the FOV of the second image sensor. For example, the image matching device may determine the area corresponding to the FOV of the second image sensor in the first image as the desired area based on the separation distance between the first image sensor and the second image sensor and the distance between the object and the image matching device. The distance between the object and the image matching device may be a predetermined (or, optionally, a desired) distance (e.g., 15 cm to 30 cm). For example, the image matching device may use a depth sensor to measure the distance to the object, or use the parallax between the left image and the right image acquired by the stereo camera to estimate the distance to the object.

[0068] The image matching device terminates the image matching operation in response to a failure to identify the marker point. In addition, the image matching device terminates the image matching operation in response to the absence of the marker point identified in the first image in the desired area.

[0069] In operation 260, the image matching device matches the marker block with the target block. The image matching device retrieves a local area matching the marker block from the target block. Figure 5 An example of retrieving a local region matching a landmark block is further described.

[0070] In operation 270, the image matching device determines the target point based on the image matching result. For example, the image matching device determines the center point of the retrieved local area as the target point. As described above, the first image and the second image can be acquired respectively at different time sequences. Therefore, the form, position and posture of the object present in the first image may be different from the form, position and posture of the object present in the second image. In one example, the image matching device can accurately match images captured at different time points by matching the first image with the second image based on a marker.

[0071] Figure 3 is a flow chart illustrating an example of object recognition and liveness verification based on determined target points.

[0072] Reference Figure 3In operation 340, the image matching device identifies the object or verifies the activity of the object based on the target point. For example, the image matching device may identify the object present in the second image based on the target point of the second image. As another example, the image matching device may verify the validity of the activity of the object present in the second image based on the target point of the second image. For example, the image matching device may determine whether the object is a tissue structure of the user (e.g., the face of the user, etc.) based on the target point of the second image.

[0073] Here, the term "identification" may include verification and identification. As a non-limiting example, verification may refer to an operation of determining whether input data is true or false. Identification may refer to an operation of determining a label indicated by input data among a plurality of labels.

[0074] Activity indicates whether the object is a living being rather than an attempt at deception or a false / forged object. For example, the image matching device determines the activity of the object based on an activity parameter determined for each frame of at least one of the first image and the second image. The activity parameter is used to determine whether each frame is based on an image effectively captured from a real user. For example, if the image corresponding to the frame is captured from a real user, the activity parameter indicates "true". If the image is forged, the activity parameter indicates "false". The image matching device may calculate the activity parameter based on at least one of the feature points of the first image and the feature points of the second image. In one example, in response to the object being determined to be a living tissue structure and / or an identified user (i.e., the activity parameter indicates "true"), the image matching device may allow access to one or more features and / or functions of the image matching device through a user interface of the image matching device.

[0075] Figure 4 is a flow chart illustrating an example of a method of matching a color image with an IR image.

[0076] In operation 410, the image matching device acquires a color image as a first image. In operation 440, the image matching device acquires an IR image as a second image. The image matching device may capture the color image and the IR image from the same object. That is, the same object may be included in both the color image and the IR image.

[0077] In operation 420, the image matching device detects a face and facial feature points. For example, the image matching device may detect a facial region based on a color image. The image matching device may detect facial feature points in the facial region.

[0078] In operation 430, the image matching apparatus extracts a block around the facial feature point. For example, the image matching apparatus may extract a landmark block including the facial feature point detected in operation 420.

[0079] In operation 450, the image matching apparatus extracts a block around a position corresponding to the facial feature point. For example, the image matching apparatus may extract a target block corresponding to the facial feature point detected in operation 420 from the IR image.

[0080] In operation 431, the image matching device preprocesses the color image. In operation 451, the image matching device preprocesses the IR image. The image matching device may apply a Gaussian filter to the color image and the IR image. The Gaussian filter may output the magnitude of the gradient of the pixel of the image. The pixel value of each pixel of the image to which the Gaussian filter is applied may represent the Gaussian magnitude. The color image to which the Gaussian filter is applied may be blurred. In the IR image to which the Gaussian filter is applied, the edge of the IR image may be highlighted. In one example, the image matching device may preprocess the marker block and the target block. For example, the image matching device may preprocess the marker block and the target block using a Gaussian filter. The image matching device may perform a morphological operation on the IR image before applying the Gaussian filter. The morphological operation may reduce the effect of the light spot or spotlight that may exist in the IR image due to the reflection of the glasses.

[0081] Although after operation 430 and operation 450, respectively Figure 4 The image matching apparatus may perform the preprocessing operation in operation 431 and operation 451, but it is provided only as an example. The image matching apparatus may perform the preprocessing operation before operation 430 and operation 450, and may also perform the preprocessing operation before detecting the face in operation 420.

[0082] In operation 460, the image matching device matches blocks of two images (i.e., the color image and the IR image). For example, the image matching device may determine a local area of ​​a target block in the IR image that matches a marker block in the color image. The image matching device may match a preprocessed marker block and a preprocessed target block.

[0083] In operation 470, the image matching device obtains facial feature points for the IR image. In one example, the image matching device may determine facial feature points for the IR image based on the matching between the aforementioned marker block and the target block. The image matching device may determine a point included in the determined local area as a target point. For example, a center point of the local area may be determined as the target point. The target point may be a facial feature point.

[0084] Therefore, even if the IR image may be susceptible to reflections from accessories (e.g., glasses) having total reflection characteristics (and therefore it may be difficult to accurately determine feature points directly from the IR image), the image matching device may accurately determine feature points for the IR image by mapping feature points extracted based on the color image to the IR image. In one example, the image matching device may identify an object and / or verify the identity of an object based on the feature points of the IR image and the IR image.

[0085] Figure 5 An example of image matching processing and feature point extraction is shown.

[0086] Although Figure 5 An example in which a color image 510 is used as a first image and an IR image 520 is used as a second image is shown, but it is provided only as an example. For example, the first image may be an IR image, and the second image may be a color image. Both the first image and the second image may be IR images. Both the first image and the second image may be color images. In addition, the first image may be a depth image, and the second image may be a color image. For example, the first image may be one of a color image, an IR image, and a depth image, and the second image may be one of a color image, an IR image, and a depth image.

[0087] The image matching device obtains the color image 510 as the first image. Figure 5 , the first image sensor of the image matching device generates a color image 510 by photographing the face of a person as a subject.

[0088] The image matching device identifies the landmark points of the object for the color image 510. For example, the image matching device extracts the feature points of the face of a person from the color image 510 based on the object model. In one example, the image matching device may determine whether the landmark points are identified in a predetermined (or, optionally, desired) area 511. Figure 5 , the landmark points are represented as dots. Region 511 in color image 510 is a region corresponding to the FOV of IR image 520. Therefore, in response to the absence of the landmark points extracted from color image 510 in region 511, the landmark of the object (e.g., the nose of the face) may not exist in IR image 520. In response to the absence of the landmark of the object in IR image 520, landmark-based matching may not be performed. Therefore, the image matching device terminates the matching operation.

[0089] The image matching device acquires an IR image 520 as a second image. The second image sensor of the image matching device generates the IR image 520 by photographing the face of a person as an object. The first image sensor may be a camera sensor, and the second image sensor may be an IR sensor. The FOV of the IR sensor may be narrower than or wider than the FOV of the camera sensor. The camera sensor acquires a color image in a visible band, and the IR sensor acquires an IR image in an IR band. Figure 5 , compared with the color image 510 , the proportion of the face in the IR image 520 is relatively large.

[0090] The first image acquired by the image matching device includes multiple color channel images. For example, the multiple color channel images may include a red channel image, a green channel image, and a blue channel image. In addition, the first image may include a brightness channel image (e.g., a Y channel image) and a chrominance channel image (e.g., a U channel image and a V channel image).

[0091] The image matching device extracts the marker block 531 from the channel image 530 having the smallest wavelength difference with respect to the second image among the plurality of color channel images in response to the first image including the plurality of color channel images. For example, if the first image includes a red channel image, a green channel image, and a blue channel image and the second image is an IR image 520, the color channel image having the smallest wavelength difference is the red channel image. Therefore, the image matching device selects the red channel image as the channel image 530 from the first image. The image matching device extracts the marker block 531 from the channel image 530. Figure 5 As shown, the channel image 530 may be a preprocessed color image. For example, the preprocessing of the color image 510 may be performed using a Gaussian filter.

[0092] The image matching apparatus extracts a target block 541 at a position corresponding to the marker block 531 of the color image 510 from the IR image 520. For example, the image matching apparatus extracts the target block 541 from the pre-processed IR image 540.

[0093] The image matching device retrieves a local area that matches the marker block 531 from the target block 541. The image matching device determines the center point of the retrieved local area as the target point. For example, the image matching device may calculate the degree of similarity (e.g., degree of correlation) between the marker block 531 and each of the multiple local areas of the target block 541. The image matching device may determine the local area with the highest calculated degree of similarity (e.g., degree of correlation) among the multiple local areas as the local area that matches the marker block 531.

[0094] The image matching apparatus calculates the degree of similarity between the pixels in each of the plurality of local regions included in the target block 541 and the pixels included in the marker block 531. The image matching apparatus calculates the degree of correlation (e.g., a mutual correlation score) as the degree of similarity. For example, the image matching apparatus may calculate a normalized mutual correlation value as the mutual correlation score by performing a normalized mutual correlation operation on the two blocks. Figure 5 , the image matching apparatus calculates a block 550 of cross-correlation between a local area of ​​the target block 541 and the marker block 531. Statistics (eg, average and median) of pixel values ​​of pixels included in the cross-correlation block 550 may be determined as a correlation degree.

[0095] In order to improve the calculation speed, the image matching device samples a portion of the target block 541 and calculates the degree of similarity between the sampled portion and the marker block 531. For example, the degree of similarity may be a value corresponding to the degree of correlation.

[0096] The image matching device determines a target point 561 for the IR image 520 based on the match between the marker block 531 and the target block 541. The target point 561 may represent a feature point of an object (e.g., a person's face) present in the IR image 520. The image matching device recognizes the object or verifies the activity of the object based on the second IR image 560 in which the target point 561 is determined. The second IR image 560 may be the same image as the IR image 520 (however, the second IR image 560 is not limited thereto and the second IR image may be different from the IR image 520).

[0097] In one example, the image matching device may match the first image and the second image based on the marker points of the first image and the target points of the second image. For example, the image matching device may match the marker points and the target points based on the marker points of the first image and the target points of the second image by transforming at least one of the first image and the second image. The image matching device may match the marker points and the target points by applying, for example, transformation, movement, and rotation to at least one of the first image and the second image. For the remaining pixels in the first image and the second image other than the feature points (e.g., the marker points and the target points), matching may be performed based on the marker points and the target points by applying transformation.

[0098] Figure 6 and Figure 7 An example of image matching results is shown.

[0099] Figure 6An example is shown in which a portion of a sign of an object exists in a predetermined (or, alternatively, desired) area 611 in a color image 610 captured by an image matching device. Since the FOV of the IR image 620 is smaller than the FOV of the color image 610, only a portion of the object may exist in the IR image 620. In one example, in response to only a portion of the sign being identified in the desired area 611 of the color image 610, the image matching device may determine a target point in the IR image 620 corresponding to the identified sign. Therefore, although only a portion of the sign exists in the FOV of the IR image 620, the image matching device may accurately determine a feature point corresponding to the sign based on the sign point of the color image 610.

[0100] Figure 7 An example is shown in which a plurality of markers are present in a predetermined (optionally, desired) area 711 in a first image 710 captured by an image matching device. The image matching device extracts a marker block for each of the plurality of markers of an object from the first image 710. The image matching device extracts a target block for each of the plurality of markers from the second image 720. The image matching device determines a target point corresponding to the marker point for each of the plurality of markers.

[0101] Without being limited thereto, the image matching device determines points corresponding to the remaining markers in the second image 720 based on target points associated with one of the multiple markers in response to detecting multiple markers for the object from the first image 710. Therefore, although the target points for all the markers in the second image 720 are not determined, the image matching device can estimate feature points corresponding to the remaining markers based on the target points determined for a portion of the markers.

[0102] In one example, the image matching device may calculate the distance between the marker point for the marker in the first image and the target point corresponding to the marker in the second image. The image matching device may estimate the position of the feature point in the second image by reflecting the calculated distance to the feature point corresponding to the remaining marker in the first image.

[0103] In one example, the image matching device may determine the corrected position of the feature point corresponding to the remaining marker in the color image as the position of the feature point in the IR image based on the distance calculated for the marker in the color image. In response to the pixels in the desired area in the color image (e.g., the area corresponding to the FOV of the IR image) being simply mapped to the pixels in the IR image based on a one-to-one correspondence (e.g., pixel by pixel), parallax may occur between the object existing in the color image and the object existing in the IR image based on the relative position between the object and the image matching device. Therefore, the pixel determined as the feature point in the IR image may have an error. The image matching device can accurately determine the feature point in the IR image by correcting the parallax.

[0104] In one example, the image matching device can determine the target point of the second image based on the match between the marker block and the target block. Therefore, compared with the case where the pixel position of the first image is simply transformed into the pixel position of the second image, the image matching device can accurately determine the feature point of the second image.

[0105] Figure 8 is a diagram showing an example of an image matching device. In one example, the image matching device may be the one described above. Figures 1 to 7 The image matching apparatus discussed in any one or any combination thereof, wherein the examples are not limited thereto.

[0106] Reference Figure 8 The image matching device 800 includes an image acquirer 810 and a processor 820. In addition, the image matching device 800 also includes a memory 830. The processor 820 and the memory 830 also represent one or more processors 820 and one or more memories 830, respectively.

[0107] The image acquirer 810 acquires a first image and a second image. The image acquirer 810 includes a first image sensor and a second image sensor. The first image sensor acquires a first image, and the second image sensor acquires a second image. The first image sensor and the second image sensor may have different sensor characteristics (for example, may have different FOVs). The bands (for example, a visible band and an IR band) at which the first image and the second image are respectively captured may be different. In addition, the time points at which the first image and the second image are respectively captured may be different. However, they are provided only as examples. The types of the first image and the second image may be the same. Optionally, the time points at which the first image and the second image are respectively captured may be the same.

[0108] The processor 820 extracts a marker block including a marker point of the object from the first image, and extracts a target block corresponding to the marker block from the second image. The processor 820 determines a target point corresponding to the marker point in the second image based on a match between the marker block and the target block. However, the operation of the processor 820 is not limited thereto, and the processor 820 may refer to Figures 1 to 7 Perform the above operations. In addition, without limiting the above reference Figures 1 to 7 In the context of the descriptions presented above, the base operations may be performed in various orders in various examples.

[0109] The memory 830 temporarily or semi-permanently stores data for performing the image matching method. For example, the memory 830 stores images generated during the image matching process. In addition, the memory 830 stores models (e.g., object models, etc.) and parameters for recognition.

[0110] In one example, the image matching apparatus 800 does not need a face detector and a face feature point detector for each image type. If the types of images are different (for example, if the image is a color image or an IR image), the image matching apparatus 800 uses a feature point detector developed for a single image to detect feature points for different types of images.

[0111] Furthermore, the image matching apparatus 800 performs matching for local areas that are expected to be feature points, thereby saving computing power. Furthermore, the image matching apparatus 800 can reduce the amount of time used for matching.

[0112] Fig. 9 and Fig.10 An example of applying an image matching device is shown. In one example, the image matching device can be the one described above. Figures 1 to 8 The image matching apparatus discussed in any one or any combination of, wherein the examples are not limited thereto.

[0113] Reference Fig. 9 , the image matching device represents or is applied to a device 900 (e.g., a smart phone) including a multi-image sensor. For example, the image matching device may represent or be applied to a verification system using a multi-mode input image. The multi-image sensor may detect one or both of a color image and an IR image.

[0114] Reference Fig.10 The image matching device may represent or be applied to a security management system 1000 (eg, a closed circuit television (CCTV)). For example, the image matching device may compositely use the first image and the second image to accurately determine feature points of an object despite limited light (eg, at night).

[0115] This is achieved through hardware components Figures 1 to 10The image matching device 800, image acquisition device 810, processor 820, memory 830, device 900 and other components described are described. Examples of hardware components that can be used to perform the operations described in the appropriate position in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (for example, by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0116] The method for performing the operations described in the present application is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as executing instructions or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors or a processor and a controller. One or more operations may be performed by one or more processors or a processor and a controller, and one or more other operations may be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may perform a single operation or two or more operations.

[0117] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and the methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein disclose algorithms for performing the operations performed by the hardware components and the methods described above.

[0118] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device, wherein any other device is configured to: store the instructions or software and any associated data, data files, and data structures in a non-transitory manner, and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0119] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered in a descriptive sense only, and not for the purpose of limitation. The description of the features or aspects in each example is considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, appropriate results can be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.

Claims

1. An image matching method, comprising: Extracting a marker block including a marker point of the object from a first image of the object, wherein the first image represents an image captured by a first image sensor, wherein the step of extracting the marker block including the marker point of the object comprises: determining an object region corresponding to the object in the first image, identifying the marker point of the object in the object region, and extracting the marker block including the identified marker point; Extracting a target block corresponding to the marker block from a second image of the object, wherein the second image represents an image captured by a second image sensor, and an image type of the first image and an image type of the second image are different, wherein the step of extracting the target block corresponding to the marker block comprises: determining the target block in the second image based on a position of the marker block in the first image; Based on the match between the marker block extracted from the first image and the target block extracted from the second image, a target point in the second image corresponding to the marker point is determined, wherein the step of determining the target point in the second image corresponding to the marker point includes: retrieving a local area matching the marker block from the target block, and determining a point in the retrieved local area as the target point.

2. The image matching method according to claim 1, wherein: The first image sensor is a color image sensor, and the first image is a color image; The second image sensor is an infrared (IR) image sensor, and the second image is an IR image.

3. The image matching method according to claim 1, further comprising: Based on the target point, it is determined whether the object is an organizational structure of the user.

4. The image matching method according to claim 1, further comprising: In response to the object being determined to be a living tissue structure and / or an identified user, access to one or more features and / or functions of the device is permitted through a user interface of the device.

5. The image matching method according to claim 1, wherein: The step of extracting the target block corresponding to the marker block is performed in response to detecting the marker point in the determined area of ​​the first image.

6. The image matching method according to claim 5, wherein: The step of extracting the target block corresponding to the marker block further includes determining the determined area based on a difference between a field of view FOV of a first image sensor for capturing the first image and a FOV of a second image sensor for capturing the second image.

7. The image matching method according to claim 1, wherein: The point of the retrieved local area is the center point of the retrieved local area.

8. The image matching method according to claim 1, wherein: The step of retrieving a local area matching the marker block comprises: Calculating the similarity between the marker block and each of the multiple local areas of the target block; A local region having the highest calculated degree of similarity among the plurality of local regions is determined as a local region that matches the marker block.

9. The image matching method according to claim 8, wherein: The step of calculating the degree of similarity between the marker block and each of the multiple local areas of the target block includes: calculating the degree of correlation between the value of the pixel in each of the multiple local areas of the target block and the value of the pixel included in the marker block as the degree of similarity.

10. The image matching method according to claim 1, wherein: The first image is a color image, and the second image is an infrared IR image; The step of determining an object area corresponding to the object in the first image comprises: selecting a channel image from the first image; An object region corresponding to the object is determined from the selected channel image.

11. The image matching method according to claim 1, wherein: The first image includes a plurality of channel images; The step of determining the object region corresponding to the object in the first image includes determining the object region corresponding to the object from a channel image having a minimum wavelength difference between the channel image and the second image among the plurality of channel images.

12. The image matching method according to claim 1, wherein: The step of extracting a landmark block including landmark points of the object is performed for each of a plurality of landmarks of the object, The step of extracting a target block corresponding to the marker block is performed for each marker in the plurality of markers, The step of determining a target point in the second image corresponding to the marker point is performed for each marker of the plurality of markers.

13. The image matching method according to claim 1, further comprising: The first image and the second image are matched based on the marker points of the first image and the target points of the second image.

14. The image matching method according to claim 1, further comprising: Preprocessing the marker block and the target block using a Gaussian filter; Match the preprocessed marker block with the preprocessed target block.

15. The image matching method according to claim 1, wherein: The step of extracting a target block corresponding to the marker block includes determining the target block in the second image based on the position of the marker block in the first image and the distance between the image matching device and the object.

16. The image matching method according to claim 1, further comprising: In response to detecting a plurality of landmarks of the object in the first image, based on a target point associated with one of the plurality of landmarks of the object, points in the second image corresponding to the remaining landmarks are determined.

17. The image matching method according to claim 1, further comprising: An object present in the second image is recognized based on the target point.

18. The image matching method according to claim 1, further comprising: The activity of the object present in the second image is verified based on the target point.

19. A non-transitory computer-readable storage medium storing instructions, wherein: When the instructions are executed by a processor, the processor is caused to execute the image matching method according to claim 1.

20. An image matching device, comprising: One or more processors configured to: obtaining a first image of the object and a second image of the object, wherein the first image represents an image captured by a first image sensor, the second image represents an image captured by a second image sensor, and an image type of the first image and an image type of the second image are different; Extracting a marker block including a marker point of the object from the first image, wherein the step of extracting the marker block including the marker point of the object comprises: determining an object area corresponding to the object in the first image, identifying the marker point of the object in the object area, and extracting the marker block including the identified marker point, Extracting a target block corresponding to the marker block from the second image, wherein the step of extracting the target block corresponding to the marker block comprises: determining the target block in the second image based on the position of the marker block in the first image, Based on the match between the marker block extracted from the first image and the target block extracted from the second image, a target point in the second image corresponding to the marker point is determined, wherein the step of determining the target point in the second image corresponding to the marker point includes: retrieving a local area matching the marker block from the target block, and determining a point in the retrieved local area as the target point.

21. An image matching method, comprising: Extracting a first block including first feature points of the object from a first image of the object, wherein the first image represents an image captured by a first image sensor, wherein the step of extracting the first block including the first feature points of the object comprises: determining an object region corresponding to the object in the first image, identifying first feature points of the object in the object region, and extracting the first block including the identified first feature points; extracting a second block from a second image of the object based on the first block, wherein the second image represents an image captured by a second image sensor, and an image type of the first image and an image type of the second image are different, wherein the step of extracting the second block comprises: determining the second block in the second image based on a position of the first block in the first image; Determining second feature points in the second block based on a match between a first block extracted from the first image and a second block extracted from the second image, wherein the step of determining the second feature points in the second block includes: retrieving a local area matching the first block from the second block, and determining points of the retrieved local area as second feature points; Based on the second feature points and the second image, the object is identified and / or the identity of the object is verified.

Images