Method and apparatus for performing combined authentication

Abstract

A method and apparatus for performing combined authentication. The method and apparatus for performing combined authentication performs single authentication based on a first modality among a plurality of modalities; in response to a failure of the single authentication, determines whether to perform combined authentication through a combination of the plurality of modalities; and in response to a determination to perform the combined authentication, performs the combined authentication.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2018-0018666, filed on February 14, 2018, Korean Patent Application No. 10-2018-0028707, filed on March 12, 2018, and Korean Patent Application No. 10-2018-0094439, filed on August 13, 2018, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entireties by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to a method and apparatus for performing combined authentication. BACKGROUND

[0003] With the development of various mobile devices including smart phones and user devices such as wearable devices, the importance of secure authentication is increasing, and interest in biometric recognition is also rapidly growing. Biometric recognition enhances the security of user devices and enables more secure use of various applications such as mobile payment applications. Biometric recognition exhibits a relatively high recognition rate, and thus is widely used. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is neither intended nor

[0005] In one general aspect, a method of performing combined authentication includes performing single authentication based on a first modality among a plurality of modalities, determining whether to perform combined authentication through a combination of the plurality of modalities in response to a failure of the single authentication, and performing the combined authentication in response to a determination to perform the combined authentication.

[0006] The determining can include determining whether a second condition for the combined authentication is satisfied. The second condition can be different from a first condition for the single authentication.

[0007] The performing of the single authentication can include determining whether a first feature of the first modality satisfies a first condition for the single authentication.

[0008] The determining can include determining whether the first feature of the first modality satisfies a second condition different from the first condition for the single authentication.

[0009] The plurality of modalities can include a first modality and a second modality different from the first modality, and the determining can include determining whether a first feature of the first modality, a second feature of the second modality, or a combination of the first feature and the second feature satisfies a second condition different from a first condition for the single authentication.

[0010] The performing of the combined authentication can include generating a third feature by fusing the first feature of the first modality and the second feature of the second modality included in the plurality of modalities in response to the determining of the performing of the combined authentication, and performing the combined authentication based on the third feature.

[0011] The performing of the combined authentication can include determining whether the third feature satisfies a third condition for the combined authentication.

[0012] The second condition can be determined differently for each combination of the plurality of modalities.

[0013] The second condition can be determined based on a false acceptance rate (FAR) of a modality having the highest security among the plurality of modalities.

[0014] The second condition can be determined based on a FAR of a modality having the highest convenience among the plurality of modalities.

[0015] The combination of the plurality of modalities can be determined based on security or convenience of the combined authentication.

[0016] The plurality of modalities can include any one or any combination of the following: a face image, a fingerprint image, an iris image, a vein image, a palm print image, a signature, a voice, a gait, and a DNA structure of a user.

[0017] In another general aspect, a method of performing combined authentication includes determining whether to perform combined authentication based on any one or any combination of a first feature of a first modality and a second feature of a second modality, and performing the combined authentication based on the first feature and the second feature in response to the determining of the performing of the combined authentication.

[0018] The performing can include generating a third feature by fusing the first feature and the second feature, and performing the combined authentication based on the third feature.

[0019] The determining can include determining whether the first feature, the second feature, or a combination of the first feature and the second feature satisfies a second condition different from a first condition for the single authentication, and determining that the combined authentication is performed by a combination of the first modality and the second modality in response to the second condition being satisfied.

[0020] The second condition can be determined based on a first FAR of the first modality, a second FAR of the second modality, or a combination of the first FAR and the second FAR.

[0021] The step of performing the combined authentication based on the third feature can include determining whether the third feature satisfies a third condition for the combined authentication.

[0022] In another general aspect, a method of performing combined authentication includes determining whether a first entry condition corresponding to a first combination of a plurality of modalities is satisfied; in response to the first entry condition being satisfied, performing authentication through the first combination; in response to the authentication through the first combination failing, determining whether a second entry condition corresponding to a second combination of the plurality of modalities is satisfied; and in response to the second entry condition being satisfied, performing authentication through the second combination.

[0023] The first entry condition and the second entry condition can be determined differently for each combination of the plurality of modalities.

[0024] The step of determining whether the first entry condition is satisfied can include determining whether the first entry condition is satisfied based on whether the first combination of the plurality of modalities satisfies a second condition different from a first condition for single authentication.

[0025] The first combination of the plurality of modalities can include a first modality and a second modality different from the first modality, and the step of determining whether the first entry condition is satisfied can include determining whether any one or any combination of a first feature of the first modality and a second feature of the second modality satisfies the second condition; and in response to the second condition being satisfied, determining that the first entry condition is satisfied.

[0026] In another general aspect, a method of performing combined authentication includes performing single authentication based on a first modality among a plurality of modalities; and in response to the single authentication failing, performing combined authentication through a combination of the plurality of modalities.

[0027] The method can further include, in response to the single authentication failing, performing single authentication based on a second modality among the plurality of modalities before performing the combined authentication.

[0028] The step of performing the combined authentication can include generating a third feature by fusing a first feature of the first modality and a second feature of a second modality among the plurality of modalities; and performing the combined authentication based on the third feature.

[0029] In another general aspect, a biometric authentication method of authenticating a user using a first biometric modality and a second modality different from each other includes: determining whether biometric information of the user satisfies one of a first condition reflecting a characteristic of the first biometric modality and a second condition reflecting a characteristic of the second biometric modality; determining whether the biometric information of the user satisfies a combined condition reflecting a characteristic of the first biometric modality and a characteristic of the second biometric modality; in response to determining that the biometric information of the user satisfies one of the first condition and the second condition and satisfies the combined condition, determining that a combined authentication is successful.

[0030] The combined condition can be a condition reflecting a characteristic in which the characteristic of the first biometric modality and the characteristic of the second biometric modality are fused.

[0031] The combined condition can be a condition in which a score calculated based on the characteristic of the first biometric modality and a score calculated based on the characteristic of the second biometric modality are combined.

[0032] The determining whether the biometric information of the user satisfies the combined condition can include generating a third characteristic by fusing the characteristic of the first biometric modality and the characteristic of the second biometric modality, and determining whether the combined condition reflecting the third characteristic is satisfied.

[0033] The determining whether the biometric information of the user satisfies the combined condition can include generating a third score by combining a first score calculated based on the characteristic of the first biometric modality and a second score calculated based on the characteristic of the second biometric modality, and determining whether the combined condition based on the third score is satisfied.

[0034] The first biometric modality can be an iris modality, and the second biometric modality can be a face modality.

[0035] Other features and aspects will be apparent from the following specific description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 An example of a method of performing a combined authentication is illustrated.

[0037] Figure 2 is a flowchart illustrating an example of a method of performing a combined authentication.

[0038] Figure 3A and Figure 3B An example of a first condition and a second condition is illustrated.

[0039] Figure 4 is a block diagram illustrating an apparatus for performing a combined authentication.

[0040] Figure 5An example illustrating an operation of a classifier performing combined authentication.

[0041] Figure 6A 、 Figure 6B and Figures 7 to 11 are flowcharts illustrating examples of a method of performing combined authentication.

[0042] Figure 12 An example illustrating a user interface.

[0043] Figure 13A and Figure 13B illustrate examples of a biometric registration process.

[0044] Figure 14 An example illustrating performing combined authentication by exploiting correlation between different modalities.

[0045] Figure 15 is a block diagram illustrating an example of an apparatus for performing combined authentication.

[0046] Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals are to be understood to represent same elements, features, and structures. The drawings can not be to scale and the dimensions, proportions, and other particulars shown in the drawings can have been exaggerated for the purpose of clarity, illustration, and convenience. DETAILED DESCRIPTION

[0047] The following structural or functional descriptions are exemplary to describe only example embodiments, and the scope of the example embodiments is not limited to the descriptions provided in this specification. Various changes and modifications can be made thereto by those having ordinary skill in the art.

[0048] Although the terms "first" or "second" are used to explain various components, the components are not limited by these terms. The terms are used only to distinguish one component from another component. For example, within the scope of the right of the idea according to the present disclosure, a "first" component can be referred to as a "second" component, or similarly, and a "second" component can be referred to as a "first" component.

[0049] It will be understood that when a component is referred to as being "connected to" another component, the component can be directly connected or coupled to the other component, or there can be intervening components.

[0050] As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0051] Unless defined otherwise herein, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art in the field to which this application belongs. Unless otherwise defined herein, terms defined in a general dictionary have meanings matching those in the context of the relevant art and are not to be interpreted ideally or overly formally.

[0052] Examples set forth below can be implemented (1) as a type of software products, (2) as a type of hardware products, and (3) as a type of software products and hardware products combined. Examples can be implemented as any one of the following various types of products: personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, smart vehicles, kiosks, and wearable devices, for example. Examples can be applied to smart phones, mobile devices, smart home systems, smart vehicles, and automated teller machines (ATMs) for user authentication, for example. Hereinafter, examples will be described in detail with reference to the accompanying drawings, in which like reference numerals are used to refer to the same elements.

[0053] Figure 1 An example of a method of performing combined authentication is illustrated. Referring to Figure 1 , a case in which a user 50 performs biometric recognition or authentication using an image sensor 110, an infrared (IR) sensor 120, or a fingerprint sensor 130 of a mobile device 100 is illustrated.

[0054] In one example, biometric recognition or authentication is performed using a plurality of modalities based on various biometrics, such as a face, an iris, a fingerprint, and a vein. The term "modality" denotes unique biometric information of a user, such as a face, a fingerprint, an iris, a vein, a palm print, a signature, a voice, a gait, and a DNA structure of the user, for example, or denotes each aspect of unique information for recognizing or authenticating a user.

[0055] When various biometrics are used, a security level of biometric authentication varies for each modality. For example, an iris modality has a false acceptance rate (FAR) of one in ten million, and a fingerprint modality and a face modality each have a FAR of one in one million. The "FAR" denotes a ratio of misrecognizing biometric information of another person as biometric information of one person. As the FAR of a modality for biometric information decreases, security (i.e., a security level) of the modality increases.

[0056] In a case of using combined authentication that performs authentication through a combination of various modalities, security is improved by taking advantage of a plurality of modalities. Combined authentication through a combination of a plurality of modalities guarantees authentication performance suitable for various situations.

[0057] In one example, it is assumed that a user will be authenticated using an iris image and a face image. The iris image and the face image are captured through an IR camera, a color camera, a monochrome camera, or a three-dimensional (3D) camera. The 3D camera is implemented as various types of cameras such as a time-of-flight (ToF) camera and a structured light camera. However, those types are merely exemplary cases, and the 3D camera is not limited thereto. In one example, the iris image is captured through an IR camera, and the face image is captured through a color camera or a 3D camera. The iris image and the face image are captured through various combinations of the above-described cameras.

[0058] The iris modality has higher security than the face modality. It is assumed that a user A and a user B are twins. For the iris modality, the user A and the user B have different feature vectors at the level of different persons. However, for the face modality, the user A and the user B have similar feature vectors at the level of the same person. Thus, it is difficult to distinguish the twins based on the face modality, but it is relatively easy to distinguish the twins based on the iris modality.

[0059] However, depending on a captured environment, the performance of the iris modality is lower than that of the face modality. For example, the iris image is captured using an IR sensor after emitting light from a light-emitting diode (LED) included in a mobile device. Under strong outdoor light, the quality of the iris image decreases. If the image quality decreases, the recognition performance decreases. Thus, in the case of using only the iris modality, the recognition performance decreases under a certain environment (e.g., in the presence of strong outdoor light).

[0060] The quality of the face image does not decrease despite the strong outdoor light. Thus, even in the case where the recognition performance of the iris modality decreases, the recognition performance of the face modality does not decrease. However, unlike the iris image, the face image is vulnerable to a change in the type of external light, a low-illumination situation, or a change in a face pose.

[0061] Thus, the iris image and the face image are used in combination to maintain the recognition performance under various situations.

[0062] Examples set forth below consider both security and user convenience when performing combined authentication through a combination of multiple modalities. For example, the examples provide a technique to improve user convenience while maintaining security, or a technique to improve security while maintaining user convenience.

[0063] The examples perform single authentication first, and then perform combined authentication.

[0064] In one example, for a payment application, security needs to be considered important, and thus a scheme for performing single authentication and combined authentication is set to improve user convenience while maintaining security. The example performs single authentication first using a modality having relatively high security (e.g., an iris modality), and performs combined authentication through a combination of multiple modalities (e.g., an iris modality and a face modality) in response to a failure of the first authentication.

[0065] In another example, for a lock function of a smart phone, user convenience needs to be considered important, and thus a scheme for performing single authentication and combined authentication is set to improve security while maintaining user convenience. The example performs single authentication first using a modality having relatively high convenience (e.g., a face modality), and performs single authentication secondly using a modality having different characteristics from the modality used for the first authentication (e.g., an iris modality) in response to a failure of the first authentication. In response to a failure of the second authentication as well, the example performs combined authentication through a combination of multiple modalities (e.g., a face modality and an iris modality).

[0066] The example provides a technique of satisfying security and user convenience together by determining an entry condition for determining whether to perform combined authentication before performing combined authentication. The entry condition is determined in advance based on a combination of multiple modalities for combined authentication or a type of an application to which combined authentication is to be applied.

[0067] The entry condition for determining whether to perform combined authentication is a condition related to at least one of the multiple modalities for combined authentication. For example, the entry condition includes a first condition related to a face modality, a second condition related to an iris modality, and a combination of the first condition and the second condition. The entry condition can be a condition different from a reference condition for determining whether single authentication using a separate modality is successful.

[0068] In one example, the entry condition is a condition different from a reference condition for determining whether single authentication using a separate modality is successful. For example, in a case where the entry condition includes a second condition related to an iris modality, the second condition is a condition requiring a lower FAR than single authentication using the iris modality.

[0069] In one example, combined authentication is applied to a payment application. In this example, an entry condition considering that security is important and improving user convenience is set. The example determines an entry condition for determining whether to perform combined authentication using a modality having relatively high security (e.g., an iris modality), and then performs combined authentication through a combination of multiple modalities (e.g., an iris modality and a face modality).

[0070] In another example, the combined authentication is applied to a lock-unlocking function of a smart phone. In this example, an entry condition in which user convenience is important and security is improved is set. The example determines an entry condition for determining whether to perform the combined authentication using a single modality (e.g., an iris modality, a face modality, or a fingerprint modality), and then performs the combined authentication through a combination of a plurality of modalities (e.g., a face modality, a fingerprint modality, and an iris modality).

[0071] As described below, the entry condition is set to use a combination of a plurality of single modalities, rather than to use only one single modality. Furthermore, when the combined authentication is performed, a scheme of combining a plurality of modalities is set in various ways.

[0072] For example, a score of each modality is determined in the form of a matching score or a distance score. The matching score corresponds to a score indicating how similar the enrollment frame and the input frame are to each other (i.e., similarity between the frames). A low matching score indicates a low similarity between the frames, and a high matching score indicates a high similarity between the frames. The higher the matching score, the higher the probability that the mobile device 100 accepts the authentication of the user 50. Furthermore, the distance score corresponds to a score indicating a feature distance (e.g., a Euclidean distance) between the enrollment frame and the input frame. A low distance score indicates a short feature distance between the frames in a feature vector space, and a high distance score indicates a long feature distance between the frames. The lower the distance score, the higher the probability that the mobile device 100 accepts the authentication of the user 50.

[0073] As described above, when compared to the user authentication using a single modality, the example improves user convenience in accessing a device and guarantees relatively high security via the combined authentication through a combination of a plurality of modalities. Furthermore, the example improves user convenience by maintaining user recognition and authentication performance in various environments using each modality.

[0074] Figure 2 is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 2 In operation 210, a device for performing combined authentication (hereinafter, referred to as an "authentication device") performs single authentication based on a first modality among a plurality of modalities. The authentication device performs single authentication of a separate modality, and performs combined authentication in response to a failure of the single authentication. In detail, the authentication device performs single authentication by determining whether a first feature of the first modality satisfies a first condition. For example, the first feature is a Euclidean distance or a similarity score determined based on a feature vector extracted from the first modality. Furthermore, for example, the first condition corresponds to a threshold distance for single authentication or a threshold score for single authentication. The first condition for single authentication is differently determined based on a type of the modality. In response to the single authentication being accepted in operation 210, the authentication device determines that the authentication is successful in operation 240.

[0075] In response to the single authentication failure in operation 210, in operation 220, the authentication device determines whether an entry condition for determining whether to perform a combined authentication by a combination of a plurality of modalities is satisfied based on a second condition different from the first condition for the single authentication. Here, the "combined authentication" is explained as performing authentication by various combinations of a plurality of modalities (e.g., (fingerprint, iris), (fingerprint, face), (iris, face), and (fingerprint, iris, face) among a plurality of modalities such as (fingerprint, iris, face)).

[0076] In operation 220, the authentication device determines whether to perform a combined authentication by a corresponding combination for a combination of a plurality of modalities. The authentication device determines whether to perform a combined authentication by a combination of a plurality of modalities based on whether the second condition is satisfied for the combination of a plurality of modalities. In this example, the second condition is determined differently for each combination of a plurality of modalities.

[0077] For example, the second condition is determined based on a FAR of a modality having the highest security among the combination of a plurality of modalities. For example, in a case where the combination of a plurality of modalities includes an iris modality and a face modality, the second condition is determined based on one in a hundred million, which is a FAR of the iris modality having higher security than the face modality. The second condition requires a FAR different from the FAR of one in a hundred million. In another example, in a case where the combination of a plurality of modalities includes a face modality and a fingerprint modality, the second condition is determined based on one in ten million, which is a FAR of the fingerprint modality having higher security than the face modality. The second condition requires a FAR lower than the FAR of one in ten million.

[0078] For example, the second condition is determined based on a FAR of a modality having the highest convenience among the combination of a plurality of modalities. For example, in a case where the combination of a plurality of modalities includes a face modality and a signature modality, the second condition is determined based on two in a thousand, which is a FAR of the face modality having higher convenience than the signature modality. The second condition requires a FAR different from the FAR of two in a thousand. It will be described with reference to Figure 3A and Figure 3B The method in which the authentication device determines the first condition and the second condition will be further described.

[0079] For example, the combination of a plurality of modalities includes a first modality and a second modality different from the first modality. In another example, the combination of a plurality of modalities can include three modalities (such as a first modality, a second modality, and a third modality) or more. The combination of a plurality of modalities is determined based on security or convenience of the combined authentication. For example, the number and / or category of a plurality of modalities included in the combination of a plurality of modalities are determined based on the security or convenience of the combined authentication.

[0080] In operation 220, the authentication device determines whether any one or any combination of the first feature of the first modality and the second feature of the second modality satisfies a second condition. In response to the second condition being satisfied (e.g., accepted), the authentication device determines that the combined authentication is performed by the combination of the plurality of modalities. In response to the second condition not being satisfied (e.g., failed) in operation 220, in operation 250, the authentication device determines that the authentication fails.

[0081] In response to determining to perform the combined authentication in operation 220, in operation 230, the authentication device performs the combined authentication. In operation 230, the authentication device performs the combined authentication by determining a condition that combines a score based on the first feature and a score based on the second feature. For example, the authentication device determines whether a third condition that combines a condition for the score based on the first feature and a condition for the score based on the second feature is satisfied by a logical operation.

[0082] In another example, the authentication device generates a third feature by fusing the first feature and the second feature, and performs the combined authentication based on the third feature. For example, the authentication device performs the combined authentication based on whether the third feature satisfies a third condition for the combined authentication. The third condition is determined based on a target FAR for the combined authentication.

[0083] In operation 230, the authentication device determines whether the combination of the first feature and the second feature or the third feature satisfies the third condition, for example, using a pre-trained classifier. This will be described with reference to Figure 5 A method in which the authentication device performs the combined authentication based on the third feature is further described.

[0084] Figure 3A and Figure 3B An example of determining the first condition and the second condition is illustrated. For example, the first condition corresponds to a threshold distance for performing the single authentication, and the second condition corresponds to a threshold distance for determining whether to perform the combined authentication. The threshold distance is determined based on a verification rate (VR), a FAR, a false rejection rate (FRR), or a combination thereof.

[0085] Hereinafter, for ease of description, a method of determining the first condition and the second condition using a FAR will be described. However, the method of determining the first condition and the second condition is not limited thereto, and the first condition and the second condition are determined based on various performance indicators. Furthermore, although the performance indicators for determining the first condition and the second condition are the same, target scores for determining respective threshold values corresponding to the first condition and the second condition are different from each other. For example, the threshold value corresponding to the first condition is determined to satisfy a predetermined FAR for a raw (RAW) score. The threshold value corresponding to the second condition is determined to satisfy a predetermined FAR for a score obtained by applying additional processing (e.g., filtering) to the RAW score.

[0086] Referring toFigure 3A The histogram 310 represents a feature distance of the first feature of the first modality between items of data corresponding to the user (the same person). In addition, the histogram 330 shows a feature distance of the first feature of the first modality between data corresponding to the user and data corresponding to another person. Here, the feature distance corresponds to a Euclidean distance representing a level of difference between a face image of the user and a face image of another person. The feature distance has a relatively small value when the similarity between data to be compared is relatively high, and has a relatively large value when the similarity between data to be compared is relatively low.

[0087] In one example, in addition to the feature distance, the authentication device determines the first condition and the second condition based on various performance indicators such as, for example, a normalized cross correlation (NCC) between feature vectors, a match score, or a similarity score.

[0088] In Figure 3A In the graph, the horizontal axis represents a feature distance of a feature of the first modality, and the vertical axis represents the number of samples corresponding to the feature distance.

[0089] The authentication device determines the first condition based on whether security or convenience of application is emphasized. For example, in a case where the user is to execute a financial institution application such as a bank or a stock trading application, the authentication device determines the first condition by emphasizing security so that strict authentication is performed. In a case where the user is to unlock the mobile device to execute a simple function of the mobile device such as a camera or a memo, the authentication device determines the first condition by emphasizing convenience so that authentication is omitted or authentication is performed based on new criteria.

[0090] For example, in a case where security is emphasized, the authentication device sets a feature distance of a boundary line 350 as the first condition, where the boundary line 350 distinguishes a region in which a false acceptance rate (FAR) is allowed in the iris modality (e.g., below one in ten million) from the entire region of the histogram 330 showing the feature distance of the first feature of the iris modality of another person. In this example, in response to the feature distance of the iris modality not satisfying the FAR of one in ten million, the single authentication for the user fails.

[0091] However, the iris modality does not work well in an outdoor environment having strong ultraviolet rays, and thus, although authentication by the iris modality fails, if the iris modality satisfies a predetermined condition, the iris modality is used to perform combined authentication in combination with a face modality that works well even in an outdoor environment to alleviate inconvenience of the user. If the iris modality does not satisfy a feature distance threshold of the boundary line 350 for single authentication but satisfies a feature distance threshold of another boundary line 370, the authentication device determines to perform combined authentication by combination of the iris modality and the face modality. The iris modality satisfying the feature distance threshold indicates that a feature distance of the iris modality is less than (or equal to) the feature distance threshold.

[0092] In this example, the boundary line for the first condition and the boundary line for the second condition are determined by different score histograms. For example, referring to Figure 3B , the boundary line 350 for the first condition is set based on original modality scores (e.g., the histogram 310 and the histogram 330), and the boundary line 370 for the second condition is set based on scores obtained by applying additional processing (e.g., filtering) to the respective scores (e.g., the histogram 320 and the histogram 340).

[0093] In one example, although single authentication by the iris modality fails, if the iris modality satisfies the second condition, the authentication device configures the iris modality as an element of combined authentication (i.e., combination of multiple modalities), thereby improving user convenience while maintaining security. In more detail, in response to single authentication failure, by setting an entry condition (e.g., the second condition), rather than directly performing combined authentication, a rate of misidentifying another person as one person is reduced. That is, combined authentication is performed only in response to the entry condition (e.g., the second condition) being satisfied, thereby maintaining a relatively high security when compared to a case where the entry condition (e.g., the second condition) is not used. Further, a rate of excluding one person as another person in error is reduced when compared to a case where only the first condition is used, and thus, user convenience is improved.

[0094] Figure 4 is a block diagram illustrating a device for performing combined authentication. Referring to Figure 4 , the authentication device includes an enrollment database (DB) 410, a matcher 420, an entry condition determiner 430, and an authenticator 440. Operations of the matcher 420, the entry condition determiner 430, and the authenticator 440 are performed by the processor 1510 of Figure 15 , which will be described later.

[0095] The enrollment DB 410 includes an enrollment feature vector for each modality. For example, the enrollment DB 410 is provided for each of the multiple modalities, or the enrollment DB 410 is provided as a single unified DB including all modalities.

[0096] The matcher 420 calls the enrolled feature vector of each of the first modality (Input1) and the second modality (Input2) being inputted in real time from the enrollment DB 410.

[0097] The matcher 420 extracts the feature vector of the first modality and / or the feature vector of the second modality and calculates a feature distance or a similarity score by matching the feature vector of the first modality and / or the feature vector of the second modality with the enrolled feature vectors of the plurality of modalities stored in the enrollment DB 410. The matcher 420 extracts the feature vector of the first modality and then extracts the feature vector of the second modality as needed based on the authentication situation of the authenticator 440, rather than extracting the feature vector of the first modality and the feature vector of the second modality at the same time.

[0098] The matcher 420 transmits the feature distance or the similarity score corresponding to the matching result to the entry condition determiner 430 and the authenticator 440.

[0099] For example, in response to the feature distance between the feature vector of the first modality and the enrolled feature vector received from the matcher 420 satisfying a first condition, the authenticator 440 determines that the single authentication is successful and outputs "single authentication success". In response to the single authentication success, the authentication device outputs "authentication acceptance". In response to the feature distance between the feature vector of the first modality and the enrolled feature vector not satisfying the first condition, the authenticator 440 determines that the single authentication is failed and outputs "single authentication failure". The authenticator 440 transmits the authentication result to the entry condition determiner 430.

[0100] The entry condition determiner 430 determines whether the entry condition for the combined authentication is satisfied based on the matching result (e.g., single authentication success or single authentication failure) received from the matcher 420. In response to receiving the single authentication failure from the matcher 420, the entry condition determiner 430 determines whether the entry condition for the combined authentication is satisfied.

[0101] The entry condition determiner 430 determines whether to perform the combined authentication through the combination of the plurality of modalities based on a second condition different from the first condition for the single authentication. For example, in response to receiving information indicating that the feature distance between at least one of the first feature vector of the first modality and the second feature vector of the second modality and the enrollment feature vector corresponding to the at least one feature vector among the plurality of modalities' enrollment feature vectors stored in the DB satisfies the second condition from the matcher 420, the entry condition determiner 430 determines that the entry condition for the combined authentication is satisfied. In response to determining that the entry condition for the combined authentication is satisfied, the entry condition determiner 430 requests the authenticator 440 to perform the combined authentication through the combination of the plurality of modalities. In response to determining that the entry condition for the combined authentication is not satisfied, the authentication device outputs "authentication failure".

[0102] The authenticator 440 performs the combined authentication based on a condition in which the score based on the first feature and the score based on the second feature are combined. In another example, the authenticator 440 generates a third feature by fusing the first feature and the second feature, and performs the combined authentication based on the third feature. The authenticator 440 outputs a result of performing the combined authentication (e.g., combined authentication success or combined authentication failure). The authentication device outputs authentication success in response to the combined authentication success, and outputs authentication failure in response to the combined authentication failure. For example, the authenticator 440 performs the combined authentication by the classifier 500 described later with reference to Figure 5 The operation of the classifier 500 performing the combined authentication will be described with reference to Figure 5 The process in which the authenticator 440 performs the combined authentication based on the third feature will be further described.

[0103] Figure 5 An example of the operation of the classifier performing the combined authentication is shown. Referring to Figure 5 , the classifier 500 performs the combined authentication based on a third feature generated by fusing a first feature of a first modality and a second feature of a second modality.

[0104] For example, it is assumed that a 16-dimensional (16D) feature vector of an iris image and a 3D feature vector of a face image are input into the classifier 500.

[0105] The classifier 500 generates a 19-dimensional (19D) feature vector by fusing the 16D feature vector of the iris modality and the 3D feature vector of the face modality. In this example, the 16D feature vector of the iris modality is a feature vector corresponding to features such as a Hamming distance of the iris, a bit count of the iris, a radius of the iris, a shape and color of the iris, and a morpheme of retinal capillary, etc. In addition, the 3D feature vector of the face modality is a feature vector corresponding to features of the entire face or a partial face.

[0106] The classifier 500 determines whether the combined authentication is accepted or failed by comparing the 19D feature vector to a boundary line or reference value that distinguishes one person from another.

[0107] Figure 5 The lower left graph illustrates the learning phase of the classifier 500. In the graph illustrating the learning phase, the horizontal axis represents the 16D feature space of the iris image, and the vertical axis represents the 3D feature space of the face image. For ease of description, the 3D feature space will be described as an example. However, in one example, the feature space is extended to a 20-dimensional (20D) feature space in which there is a separate axis for each dimension.

[0108] The points marked in the learning phase correspond to the 19D vectors generated by fusing the 16D feature vector and the 3D feature vector. The boundary line 510 corresponds to the parameters that the classifier 500 learns to determine whether the combined authentication through the 19D vector is accepted or failed. For example, the classifier 500 uses a support vector machine (SVM) to learn the parameters or weights for the feature vector. In this example, the parameters that the classifier 500 learns are determined based on the level of the target FAR of the authentication device.

[0109] Figure 5 The lower right graph illustrates the authentication phase of the classifier 500. In the authentication phase, the boundary line 530 corresponds to the parameters of the classifier 500. The line 530 corresponds to a simplified two-dimensional (2D) representation of the 19D feature vector. In response to the generation of the fused 19D feature vector, the authentication device performs the combined authentication for the 19D input based on the boundary line 530. For example, the authentication device determines a first input (I1, F1) on the left side of the boundary line 530 as authentication accepted, and a second input (I2, F2) on the right side of the boundary line 530 as authentication failed.

[0110] Figure 6A is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 6A , a process of performing combined authentication for iris data corresponding to a first modality and face data corresponding to a second modality is illustrated.

[0111] The authentication device performs combined authentication through the operation 610 of extracting feature information of the modalities and the operation 650. The authentication device also performs the operation 630 of determining whether to perform combined authentication based on a security level. In one example, the operation 630 and the operation 650 are performed in sequence or not in sequence.

[0112] In operation 610, in response to the iris data and the face data being input, the authentication device extracts a feature vector for each of the plurality of modalities. The authentication device calculates an iris score or a face score by matching the extracted feature vector with the enrollment feature vector of each modality stored in the enrollment DB 605. In this example, the iris score or the face score corresponds to a feature distance or a similarity score between the feature vector extracted from each of the plurality of modalities and the enrollment feature vector.

[0113] In operation 650, the authentication device determines whether the iris score or the face score satisfies a single authentication condition. In response to the single authentication condition being satisfied, the authentication device accepts the authentication.

[0114] In response to the iris score or the face score not satisfying the single authentication condition, in operation 630, the authentication device determines whether the face score and / or the iris score satisfies a predetermined FAR based on the security level. In detail, whether the face score satisfies the predetermined FAR alone, whether the iris score satisfies the predetermined FAR alone, or whether both the face score and the iris score satisfy the predetermined FAR is used to determine whether the predetermined FAR is satisfied.

[0115] The predetermined FAR is determined as a FAR for authentication of the face data or a FAR for authentication of the iris data based on the security level. In response to (the face score, the iris score) not satisfying the predetermined FAR in operation 630, the authentication device acquires new iris data and / or face data and performs the combined authentication process of Figure 6A again.

[0116] In response to (the face score, the iris score) satisfying the predetermined FAR in operation 630, in operation 650, the authentication device determines whether a combination of the iris score and the face score or a fusion score calculated by fusing the iris score and the face score satisfies a combined authentication condition. In response to the combined authentication condition being satisfied, the authentication device accepts the authentication.

[0117] In response to the combined authentication condition not being satisfied, the authentication device acquires new iris data and face data and performs the combined authentication process of Figure 6A again.

[0118] Although not shown in the drawings, in response to the condition not being satisfied in operation 630 or operation 650, the authentication device determines that the authentication fails by counting a predetermined number of times and / or a predetermined time. For example, as a non-limiting example, refer to Figure 6Athe predetermined number of times is t+1, where t is an integer greater than or equal to 0; when the authentication is performed a plurality of times within the predetermined period, the number of times is decreased by 1 each time the authentication is performed; and when the number of times is decreased to 0, it is determined that the authentication has failed. For example, in response to the condition not being satisfied in operation 630 or operation 650, the authentication device accumulates and stores the number of times of authentication failure, and performs the combined authentication process again. In response to the accumulated number of times of failure exceeding a predetermined threshold, the authentication device determines that the authentication has failed. In another example, the authentication device accumulates and stores the time for the combined authentication process, and in response to the accumulated time exceeding a predetermined threshold, determines that the authentication has failed. In response to the condition not being satisfied in operation 630 or operation 650, the authentication device determines whether the accumulated time exceeds a predetermined threshold. In response to determining that the authentication has failed, the authentication device provides feedback indicating that the authentication has failed to the user.

[0119] Figure 6B is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 6B In operation 610, the authentication device extracts feature information of the modal. In operation 690, the authentication device sequentially performs single authentication and combined authentication. For example, the authentication device performs face single authentication first, iris single authentication second, and face+iris combined authentication third. In response to the single authentication or the combined authentication being successful in each operation, the authentication device finally determines that the authentication is successful without performing subsequent authentication. Also, operation 670 can be similar to operation 630 of Figure 6A , and thus, for the sake of simplicity, repeated descriptions are omitted.

[0120] In response to the face single authentication and the iris single authentication failing, the authentication device determines an entry condition related to whether to perform combined authentication before performing the combined authentication.

[0121] In response to the entry condition or the condition for the combined authentication not being satisfied, the authentication device repeats the above-described authentication process based on new input data. In the case where the authentication is not successful until a predetermined number of times is exceeded or a predetermined time elapses, the authentication device finally determines that the authentication has failed.

[0122] Figure 7 is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 7 A combined authentication process that improves user convenience while enhancing security is illustrated.

[0123] Figure 6A Operations 610 to 650 of Figure 7The process is different in that operations 710 through 750 determine whether the combi condition and the f(combi) condition are satisfied for various combinations of the plurality of modalities. Hereinafter, for convenience of description, the condition for entering the combination authentication will be denoted as the "combi condition", and the condition for accepting the combination authentication of the fusion feature vector will be denoted as the "f(combi) condition". The combi condition and / or the f(combi) condition are determined differently for each combination of the corresponding modalities.

[0124] In Figure 7 In a case where the security level through the combination authentication by each combination of the plurality of modalities is very high, or the security level required for the combination authentication is not high, the condition determination for various authentication methods is added or omitted for each combination of the plurality of modalities.

[0125] In response to the data of the plurality of modalities being input, the authentication device performs authentication through various combinations of the plurality of modalities. For example, the authentication device performs combination authentication through M kinds of authentication methods including a combination of at least one of the N modalities. M denotes the number of authentication methods through single authentication or combination authentication.

[0126] However, before performing the combination authentication through each combination of the plurality of modalities, the authentication device determines whether to perform the combination authentication based on the scores S1, S2,..., SN of the plurality of modalities corresponding to each combination of the plurality of modalities. N Whether the f(combi) condition and / or the combi condition is satisfied determines whether to perform the combination authentication. For example, the f(combi) condition and / or the combi condition is in the form of being greater than or equal to, less than or equal to, less than, greater than a predetermined score, or a combination thereof. In this example, the scores S1, S2,..., SN of the plurality of modalities are compared with the f(combi) condition and / or the combi condition. N The score for the plurality of modalities corresponds to a feature distance or a similarity score between a feature vector corresponding to the plurality of modalities and an enrolled feature vector of the plurality of modalities stored in the enrollment DB 705. The condition for the score of the plurality of modalities is used as an entry condition for determining whether to enter the combination authentication.

[0127] The authentication device performs the combination authentication for each combination of the corresponding modalities in response to each of the scores of the plurality of modalities being input, rather than performing the authentication when all of the scores of the N modalities for the combination authentication are completely input. The authentication device determines the criteria for determining whether to enter the combination authentication based on the modalities having excellent security.

[0128] In operation 750, the authentication device determines whether to perform the combination authentication based on the scores S1, S2,..., SN of the plurality of modalities. NIt is determined whether the fusion score of the combination of the first modalities satisfies the f1(combi1) condition. In this example, the combination of the first modalities corresponds to a single modality or a combination of at least two modalities. Further, the f1(combi1) condition corresponds to a single authentication condition in case the combination of the first modalities includes a single modality and a combined authentication condition in case the combination of the first modalities includes at least two modalities.

[0129] In response to the fusion score of the combination of the first modalities not satisfying the f1(combi1) condition, the authentication device determines whether a score corresponding to a combination of the second modalities satisfies the combi2 condition and / or a fusion feature vector corresponding to the combination of the second modalities satisfies the f2(combi2) condition. The combination of the second modalities is different from the combination of the first modalities, and the combination of the second modalities corresponds to a single modality or a combination of at least two modalities. Further, the f2(combi2) condition corresponds to a single authentication condition in case the combination of the second modalities includes a single modality and a combined authentication condition in case the combination of the second modalities includes at least two modalities. As described above, the process of determining the combi condition and the f(combi) condition for various authentication methods is added for each combination of the plurality of modalities in case the security level of the combined authentication is low or the requirement for convenience is high. Further, the process of determining the combi condition and the f(combi) condition for various authentication methods is added or omitted for each combination of the plurality of modalities.

[0130] Figure 8 is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 8 In operation 810, the authentication device determines whether to perform combined authentication based on any one or any combination of a first feature of a first modality and a second feature of a second modality. For example, the authentication device determines whether the first feature satisfies a first condition for single authentication. In response to the first feature not satisfying the first condition, the authentication device determines whether any one or any combination of the first feature and the second feature satisfies a second condition different from the first condition. In response to the second condition being satisfied, the authentication device determines to perform combined authentication by a combination of the first modality and the second modality. For example, the second condition is determined based on the FAR of the first modality, the FAR of the second modality, or a combination of the FAR of the first modality and the FAR of the second modality.

[0131] In operation 820, the authentication device generates a third feature by fusing the first feature and the second feature in response to determining to perform combined authentication.

[0132] In operation 830, the authentication device performs combined authentication based on the third feature. In response to the third feature satisfying a third condition for combined authentication, the authentication device determines that the authentication is successful.

[0133] Figure 9 is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 9 , in operation 910, the authentication device determines whether a first entry condition corresponding to a first combination of a plurality of modalities is satisfied. For example, the authentication device determines whether the first entry condition is satisfied based on whether the first combination of the plurality of modalities satisfies a second condition different from a first condition for single authentication. The first combination of the plurality of modalities includes a first modality and a second modality different from the first modality. The authentication device determines whether any one or any combination of a first feature of the first modality and a second feature of the second modality satisfies the second condition. In response to any one or any combination of the first feature and the second feature satisfying the second condition, the authentication device determines that the first entry condition is satisfied.

[0134] In operation 920, the authentication device performs authentication through the first combination in response to the first entry condition being satisfied.

[0135] In operation 930, the authentication device determines whether a second entry condition corresponding to a second combination of the plurality of modalities is satisfied in response to the authentication through the first combination failing. In this example, the first entry condition and the second entry condition are differently determined for each combination of the plurality of modalities.

[0136] In operation 940, the authentication device performs authentication through the second combination in response to the second entry condition being satisfied.

[0137] Figure 10 is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 10 , a combined authentication process is illustrated. Figure 10 Operations 1010 to 1050 of Figure 7 are similar to operations 710 to 750 of Figure 7 , only operations different from those of

[0138] Unlike Figure 7 , in response to data of many modalities being input, in operation 1030, the authentication device determines whether a combination of the first modality satisfies a combil condition based on scores S1, S2, …, S N In response to the combination of the first modality not satisfying the combil condition, the authentication device determines whether a combination of the second modality satisfies a combi2 condition. In response to each of the combinations of the plurality of modalities not satisfying the combi condition, the authentication device continues comparison of a new combination of the plurality of modalities with the combi condition until the combi condition is satisfied.

[0139] For example, in the case of combil condition = (M1, M2, M3) in operation 1030, the entry condition corresponding to the combil combination is determined by, for example, the M1 condition, the M2 condition, the M3 condition, or various combinations thereof.

[0140] In response to the combination of the first modalities satisfying the combil condition, in operation 1050, the authentication device determines whether the fusion score of the combination of the first modalities satisfies the fl(combil) condition. In this example, the score of each modality is used to determine whether to enter the combined authentication. In response to the fusion score of the combination of the first modalities not satisfying the fl(combil) condition, the authentication device determines whether the combination of the second modalities satisfies the combi2 condition. In response to the fusion score of the combination of the first modalities satisfying the fl(combil) condition, the authentication device accepts the combined authentication.

[0141] Figure 11 is a flowchart illustrating an example of a method of performing combined authentication. Referring to Figure 11 , a process of performing combined authentication using an IR image and a color face image in response to the IR image and the color face image being input is illustrated. For ease of description, an application using a color image will be described. However, the example is not limited to such a scheme, and is also substantially equally applicable to an application using a monochrome image or a depth image or a 3D image.

[0142] The authentication device matches the plurality of features or feature vectors of the iris modality obtained from the IR image through the feature extraction process of operation 1110 and the plurality of features or feature vectors corresponding to the face modality obtained from the color face image with the enrollment feature vectors of the plurality of modalities enrolled in the enrollment DB. The authentication device calculates the scores i1, i2, …, i N and the scores f1, f2, …, f O of the plurality of features corresponding to the iris modality and the face modality through operation 1110. In this example, the plurality of features of the iris modality include a Hamming distance of the iris, a bit count of the iris, a radius of the iris, a shape and color of the iris, and a shape of retinal capillary blood vessels. Further, the plurality of features corresponding to the face modality include features corresponding to a partial face and features corresponding to the entire face.

[0143] In operation 1130, the authentication device determines whether the fusion score of the combination of the plurality of modalities satisfies the fl(combi) condition based on the scores i1, i2, …, i N and f1, f2, …, f OIt is determined whether the combination of the first modalities satisfies a combi1 condition. For example, the combination of the first modalities is a combination of a Hamming distance of an iris and a partial face. In response to the combination of the first modalities not satisfying the combi1 condition, the authentication device determines whether a new combination of second modalities satisfies a combi2 condition. For example, the combination of the second modalities is a combination of a morphology of retinal capillaries and an entire face.

[0144] In response to the combination of the second modalities satisfying the combi2 condition, the authentication device determines that a fusion score of the combination of the second modalities satisfies a f2(combi2) condition. In response to the fusion score of the combination of the second modalities satisfying the f2(combi2) condition, the authentication device accepts the combined authentication by the combination of the second modalities.

[0145] In one example, the authentication device configures the combi condition to be always true or always false. For example, the authentication device configures the combi1 condition corresponding to the face score f1 among a plurality of scores corresponding to face modalities to be always true, so that the single authentication of the face score is always performed regardless of whether the combi1 condition is satisfied. In another example, the authentication device configures the combi1 condition corresponding to the face score f1 to be always false, so that the single authentication of the face score is always skipped regardless of whether the combi1 condition is satisfied.

[0146] In addition, in addition to the above-described examples, the authentication device performs authentication considering both user convenience and security by combining the combi condition and the f(combi) condition in various ways. For example, the f(combi) condition is set to perform one or more single authentications based on individual features and one or more combined authentications based on combinations of a plurality of features in a predetermined order. The combi condition corresponding to each f(combi) condition is set based on a condition different from a single authentication condition used for the f(combi) condition. In some cases, the combi condition is configured to be always true or always false.

[0147] Figure 12 Examples of a user interface are illustrated. Referring to Figure 12 When a user uses biometric authentication, the type of modalities used to unlock the smart phone is displayed on the lock screen of the smart phone. For example, as shown in a first screen 1210, in the case where the modality used to unlock the smart phone is a face modality, a face icon is displayed on the lock screen. As shown in a second screen 1230, in the case where the modality used to unlock the smart phone is an iris modality, an iris icon is displayed on the lock screen. In the case where the modality used to unlock the smart phone is a combination of a face modality and an iris modality, the face icon of the first screen 1210 and the iris icon of the second screen 1230 are alternately displayed on the lock screen.

[0148] In one example, a first preview screen for inputting a face image and a second preview screen for inputting an iris image are also selectively displayed. Whether the first preview screen is displayed and whether the second preview screen is displayed are set in various ways. For example, in the case of single authentication using the face modality, the first preview screen is not displayed. However, in the case of single authentication using the iris modality, the second preview screen is displayed. Further, in the case of combined authentication using the face modality and the iris modality, both the first preview screen and the second preview screen are not displayed.

[0149] Although Figure 12 The face modality and the iris modality are shown, but examples can be modified or extended to the case where another modality (such as a fingerprint modality) is used.

[0150] Figure 13A And Figure 13B Examples of a biometric registration process are shown. Referring to Figure 13A In operation 1310, the user registers a face image using an image sensor attached to the smartphone. In operation 1330, the user registers an iris image using an IR sensor attached to the smartphone. If the user wears glasses, the user takes off the glasses to register the iris image.

[0151] In one example, the user selects the type of modality to be used for authentication for a predetermined application or function. For example, the user changes the type of user modality to be used for unlocking the smartphone in the settings of the smartphone.

[0152] Referring to Figure 13B , a screen for selecting the type of modality to be used for biometric authentication is shown. For example, the options to be selected include a first option 1370 corresponding to single authentication using the face modality, a second option 1390 corresponding to single authentication using the iris modality, and a third option 1350 corresponding to combined authentication using a combination of the face modality and the iris modality. In response to the first option 1370 being selected, the face modality is used for biometric authentication. In response to the second option 1390 being selected, the iris modality is used for biometric authentication. In response to the third option 1350 being selected, both the face modality and the iris modality are used for biometric authentication. The first option 1370, the second option 1390, and the third option 1350 are selected exclusively from each other. For example, in response to any one option being selected, the previously selected option is automatically canceled.

[0153] In one example, the third option 1350 corresponding to the combined authentication is not displayed alone. In this example, the first option 1370 and the second option 1390 are selected simultaneously. In a case where the first option 1370 and the second option 1390 are selected simultaneously, the combined authentication using the combination of the face modality and the iris modality is performed for the biometric authentication.

[0154] Although the case where the face modality and the iris modality are used for the biometric authentication is described in Figure 13B , the examples are not limited thereto. The examples are modified to the case where other modalities or at least three modalities are used.

[0155] In response to the type of the modality being set, the authentication device verifies whether the enrollment data corresponding to the type of the modality is stored. In response to it being verified that the enrollment data corresponding to the type of the modality is stored, the authentication device immediately changes the type of the modality used for the authentication without performing an additional registration process. If the enrollment data corresponding to the type of the modality is not stored, the authentication device additionally receives an input of the enrollment data corresponding to the type of the modality.

[0156] In one example, the combined authentication of the face modality and the iris modality is set in a case where the pre-stored enrollment data does not exist. In this example, the authentication device acquires a face image and an iris image, and stores the enrollment data of the face modality and the enrollment data of the iris modality.

[0157] In another example, the combined authentication of the face modality and the iris modality is set in a case where only the enrollment data of the face modality has been stored. In this example, the authentication device acquires only the iris image, and additionally stores the enrollment data of the iris modality.

[0158] In a case where the user changes the setting from the combined authentication to the single authentication of the iris modality, the authentication device determines whether the enrollment data of the iris modality is stored. In this example, since the enrollment data of the iris modality has been stored, the authentication device changes the setting to use the single authentication of the iris modality without performing an additional registration process.

[0159] In one example, the enrollment data is managed for each modality. For example, the user selectively deletes the enrollment data of a predetermined modality among the pre-stored enrollment data. In this example, the authentication device reconsiders the authentication type based on the currently set authentication type and the remaining enrollment data. For example, the enrollment data of the face modality and the enrollment data of the iris modality have been stored, and the currently set authentication type is the combined authentication. If the enrollment data of the iris modality is deleted, the authentication device changes the authentication type to the single authentication of the face modality. In another example, all of the enrollment data of the biometric modalities is deleted, and the authentication type is changed to the type that performs the authentication through other means such as a pattern or a password.

[0160] The above examples are applicable to devices other than smartphones, and are also applicable to modalities other than the face modality and / or the iris modality.

[0161] Figure 14 An example of performing combined authentication by exploiting a correlation between different modalities is illustrated. The authentication device exploits a correlation between a first image for a first modality and a second image for a second modality in a process of acquiring the first image and the second image.

[0162] For example, in a case where the first modality is the face modality and the second modality is the iris modality, an eye region in the first image is expected to have a relatively high correlation with the second image. In this example, in response to a single authentication using the first image failing, the authentication device determines whether the eye region is detected in the first image before performing single authentication or combined authentication using the second image. In detail, a field of view of an image sensor for the first image is larger than a field of view of an IR sensor for the second image. In this example, if the eye region is not detected from the first image, a probability that the second image does not include iris information is relatively high. In a case where the iris information is not included in the second image, it is not possible to perform single authentication or combined authentication using the second image. Therefore, the authentication device acquires the first image again instead of acquiring the second image, thereby retrying single authentication using the first image.

[0163] Although a case where the first image is a color image 1410 and the second image is an IR image 1420 is described in Figure 14 , examples are not limited thereto. In one example, the first image is an IR image, the second image is a color image, or both the first image and the second image are IR images or color images. In another example, the first image is a depth image, and the second image is a color image. That is, the first image is one of a color image, an IR image, and a depth image, and the second image is also one of a color image, an IR image, and a depth image.

[0164] The authentication device acquires a color image 1410 as the first image. As shown in Figure 14 , a first image sensor of the authentication device generates the color image 1410 by capturing a face of a person as an object.

[0165] In response to single authentication using the color image 1410 failing, the authentication device identifies a landmark point of the object with respect to the color image 1410. For example, the authentication device extracts a feature point of the face of the person from the color image 1410 based on an object model. In one example, the authentication device checks whether the landmark point is identified within a predetermined region 1411. The landmark point is represented as a circle in the color image 1410. Figure 14The predetermined region 1411 of the color image 1410 is a region corresponding to the angle of view of the IR image 1420. In a case where no key point is detected in the predetermined region 1411 of the color image 1410, the authentication device predicts that the iris information is not included in the IR image 1420.

[0166] Figure 15 is a block diagram illustrating an example of a device for performing combined authentication. Referring to Figure 15 , the authentication device 1500 includes a processor 1510. The authentication device 1500 further includes a memory 1530, a communication interface 1550, and a sensor 1570. The processor 1510, the memory 1530, the communication interface 1550, and the sensor 1570 communicate with each other through a communication bus 1505.

[0167] The processor 1510 performs single authentication based on a first modality among a plurality of modalities. In response to a failure of the single authentication, the processor 1510 determines whether to perform combined authentication by a combination of the plurality of modalities based on a second condition different from a first condition for the single authentication. The processor 1510 performs the combined authentication in response to a determination to perform the combined authentication.

[0168] The memory 1530 includes an enrollment DB including feature vectors of the plurality of modalities. For example, the enrollment DB corresponds to the enrollment DB 410 of Figure 4 . The memory 1530 is a volatile memory or a non-volatile memory.

[0169] The communication interface 1550 outputs a single authentication result and / or a combined authentication result to a display device (not shown) of the authentication device 1500 or an outside of the authentication device 1500. The communication interface 1550 receives at least one modality from the outside of the authentication device 1500, or receives information related to an environment in which the modalities are collected from the user.

[0170] For example, the sensor 1570 includes an image sensor, an IR sensor, a fingerprint recognition sensor, and a voice recognition sensor. The sensor collects various modalities.

[0171] In one example, the processor 1510 determines whether to perform the combined authentication based on any one or any combination of a first feature of the first modality and a second feature of the second modality. The processor 1510 generates a third feature by fusing the first feature and the second feature in response to a determination to perform the combined authentication, and performs the combined authentication based on the third feature.

[0172] In another example, the processor 1510 determines whether a first entry condition corresponding to a first combination of a plurality of modalities is satisfied, and in response to the first entry condition being satisfied, performs authentication through the first combination. In response to the authentication through the first combination failing, the processor 1510 determines whether a second entry condition corresponding to a second combination of the plurality of modalities is satisfied, and in response to the second entry condition being satisfied, performs authentication through the second combination.

[0173] Further, the processor 1510 performs at least one method described with reference to Figures 1 to 11 The processor 1510 executes a program and controls the authentication device 1500. Program codes executed by the processor 1510 are stored in the memory 1530. The authentication device 1500 is connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown) and exchanges data with the external device. The authentication device 1500 includes a smart television, a smart phone, a smart vehicle, and various electronic systems.

[0174] The authentication device 1500 and herein for Figure 15Other devices, units, modules, apparatuses, and other components described herein are implemented by hardware components. Examples of hardware components that can be employed in implementing appropriate means for performing operations described in this disclosure include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this disclosure. In other examples, one or more of the hardware components utilized in performing the operations described in this disclosure are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as logic arrays, controllers, and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and perform instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this disclosure. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this disclosure, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. The one or more processors, or a processor and a controller, can implement a single hardware component, or two or more hardware components. The hardware components can have any one or more of various processing configurations, examples of which include a single-processor, independent-processor, parallel-processor, single-instruction single-data (SISD) multiprocessor, single-instruction multiple-data (SIMD) multiprocessor, multiple-instruction single-data (MISD) multiprocessor, and multiple-instruction multiple-data (MIMD) multiprocessor.

[0175] In Figure 2 and Figures 6A to 11The methods of performing the operations described in this application, as illustrated in the block diagrams, are performed by computing hardware (e.g., by one or more processors or computers implemented to execute instructions or software as described above to perform the operations described in this application as performed by the methods). For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, can perform a single operation or two or more operations.

[0176] The instructions or software for controlling the processor or computer-implemented hardware components and performing the methods as described above can be written in any of a number of computer programming languages, including object code, assembly code, machine code, or any combination thereof, to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code generated by a compiler. In another example, the instructions or software include higher-level code, which is executed by a processor or computer using an interpreter. As should be stated, an ordinary skill in programming is capable of readily translating software and instructions into this form based on the flow diagrams and block diagrams illustrated in the drawings, and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0177] The instructions or software for controlling the processor or computer-implemented hardware components and performing the methods as described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in one or more non-transitory computer-readable storage media, or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), card-type memory such as a multimedia card micro or card (e.g., a secure digital (SD) or extreme digital (XD)), a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions to a processor or computer so that the processor or computer can execute the instructions.

[0178] While the present disclosure includes certain examples, those skilled in the art will appreciate that many modifications are possible in the examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each example should be considered to apply to other examples as described herein. Suitable results can be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the present disclosure is not intended to be limited to the specific embodiments disclosed herein, but will be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents will be construed as being included in the present disclosure.

Claims

1. A method for performing combined authentication, the method comprising: Single authentication is performed based on the first modality among multiple modalities; In response to a single authentication failure, determine whether to perform combined authentication through a combination of the multiple modalities; In response to determining to perform combined authentication, perform combined authentication. The determined steps include: In response to a single authentication failure and a first feature of a first modality satisfying a second condition for determining whether to perform combined authentication through a combination of the multiple modalities, combined authentication is determined to be performed through a combination of the multiple modalities, and the second condition is different from the first condition for single authentication; In response to a single authentication failure and the first feature not meeting the second condition, it is determined that combined authentication will not be performed. The steps for performing single authentication include: determining whether a first feature of a first modality satisfies a first condition for single authentication. The steps involved in performing combined authentication include: In response to determining to perform combined authentication, a second feature of the second modality included in the plurality of modalities is extracted; The third feature is generated by fusing the first feature of the first modality and the second feature of the second modality; Perform combined authentication based on the third feature. The first condition corresponds to a first threshold distance for performing single authentication, and the second condition corresponds to a second threshold distance for determining whether to perform combined authentication, wherein the first threshold distance is different from the second threshold distance.

2. The method according to claim 1, wherein, The multiple modes include: a first mode and a second mode different from the first mode. The steps include determining whether a first feature of a first modality, a second feature of a second modality, or a combination of the first and second features satisfies a second condition that is different from the first condition used for single authentication.

3. The method according to claim 1, wherein, The steps for performing combined authentication also include: determining whether the third feature meets the third condition for combined authentication.

4. The method according to claim 1, wherein, The second condition is determined differently for each combination of the multiple modes.

5. The method according to claim 1, wherein, The second condition is determined based on the error acceptance rate of the mode with the highest security among the multiple modes.

6. The method according to claim 1, wherein, The second condition is determined based on the error acceptance rate of the mode with the highest convenience among the multiple modes.

7. The method according to claim 1, wherein, The combination of the multiple modalities is determined based on the security or convenience of combined authentication.

8. The method according to claim 1, wherein, The multiple modalities include any one or any combination of the following: a user's facial image, fingerprint image, iris image, vein image, palm print image, signature, voice, gait, and DNA structure.

9. A method for performing combined authentication, the method comprising: Determine whether to perform combined authentication based on any one or any combination of the first feature of the first modality and the second feature of the second modality; In response to determining to perform combined authentication, combined authentication is performed based on the first feature and the second feature. The determined steps include: In response to a single authentication failure performed based on the first modality, it is determined whether the first feature, the second feature, or a combination of the first and second features satisfies a second condition that is different from the first condition used to determine whether to perform combined authentication; In response to the second condition being met, it is determined that combined authentication shall be performed through a combination of the first and second modalities; In response to the second condition not being met, it is determined that combined authentication will not be performed. The steps involved include: The third feature is generated by fusing the first and second features; Perform combined authentication based on the third feature. The first condition corresponds to a first threshold distance for performing single authentication, and the second condition corresponds to a second threshold distance for determining whether to perform combined authentication, wherein the first threshold distance is different from the second threshold distance.

10. The method according to claim 9, wherein, The second condition is determined based on the first false acceptance rate of the first mode, the second false acceptance rate of the second mode, or a combination of the first false acceptance rate and the second false acceptance rate.

11. The method according to claim 9, wherein, The steps for performing combined authentication based on a third feature include: determining whether the third feature satisfies the third condition used for combined authentication.

12. A method for performing combined authentication, the method comprising: Single authentication is performed based on the first modality among multiple modalities; In response to a single authentication failure performed based on the first modality, it is determined whether the first entry condition corresponding to the first combination of multiple modalities is satisfied; In response to the first entry condition being met, authentication is performed through the first combination; In response to the first entry condition not being met, it is determined that authentication through the first combination will not be performed; In response to the failure of authentication through the first combination, it is determined whether the second entry condition corresponding to the second combination of the plurality of modalities is satisfied; In response to the second entry condition being met, authentication is performed via the second combination. The first combination of the plurality of modes includes: a first mode and a second mode different from the first mode. The step of determining whether the first entry condition is met includes: determining whether the first entry condition is met based on whether any one or any combination of the first features of the first modality and the second features of the second modality satisfies a second condition that is different from the first condition used for single authentication to determine whether to perform authentication through the first combination. The steps for performing single authentication include: determining whether a first feature of a first modality satisfies a first condition for single authentication. The steps for performing authentication through the first combination include: Extract the second feature of the second modality; The third feature is generated by fusing the first feature of the first modality and the second feature of the second modality; Authentication is performed based on the third feature, using the first combination of features. Wherein, the first condition corresponds to a first threshold distance for performing a single authentication, and the first entry condition corresponds to a second threshold distance for determining whether to perform authentication through the first combination, wherein the first threshold distance is different from the second threshold distance.

13. The method according to claim 12, wherein, The first entry condition and the second entry condition are determined differently for each combination of the plurality of modes.

14. A method for performing combined authentication, the method comprising: Single authentication is performed based on the first modality among multiple modalities; In response to a single authentication failure, another single authentication is performed based on the second of the multiple modalities; In response to the failure of the other single authentication, determine whether the entry conditions for determining whether to perform combined authentication are met; In response to the determination that the entry conditions are met, combined authentication is performed through a combination of the multiple modalities. In response to the determination that the entry conditions are not met, it is determined that combined authentication will not be performed. The steps for performing single authentication include: determining whether a first feature of a first modality satisfies a first condition for single authentication. The steps involved in performing combined authentication include: A third feature is generated by fusing a first feature of a first modality with a second feature of a second modality from among the multiple modalities; Perform combined authentication based on the third feature. The first condition corresponds to a first threshold distance for performing single authentication, and the entry condition corresponds to a second threshold distance for determining whether to perform combined authentication, wherein the first threshold distance is different from the second threshold distance.

15. A biometric authentication method for authenticating a user using a first biometric modality and a second biometric modality that are different from each other, the biometric authentication method comprising: Determine whether a user's biometric information meets one of the following conditions: a first condition reflecting a first biometric modality and a second condition reflecting a second biometric modality; In response to the user's biometric information not meeting one of the conditions and the entry condition for determining whether to perform combined authentication is met, it is determined whether the user's biometric information meets the combined condition of features reflecting the first biometric modality and features reflecting the second biometric modality. Upon determining that the user's biometric information meets the combination conditions, the combined authentication is deemed successful. If the user's biometric information does not meet one of the conditions and the entry condition is not met, it is determined that combined authentication will not be performed. The steps for determining whether a user's biometric information meets the combination conditions include: A third feature is generated by fusing features from the first biometric modality and features from the second biometric modality. Determine whether the combination conditions reflecting the third characteristic are satisfied. The condition corresponds to a first threshold distance for performing single authentication based on a single biometric modality, and the entry condition corresponds to a second threshold distance for determining whether to perform combined authentication, wherein the first threshold distance is different from the second threshold distance.

16. The biometric authentication method according to claim 15, wherein, Combination conditions are the conditions that reflect the features of the first biometric modality and the features of the second biometric modality fused together to form the features.

17. The biometric authentication method according to claim 15, wherein, The combination condition is a condition that combines the score calculated based on the features of the first biometric modality and the score calculated based on the features of the second biometric modality.

18. The biometric authentication method according to claim 15, wherein, The steps to determine whether a user's biometric information meets the combination criteria include: A third score is generated by combining a first score calculated based on features of a first biometric modality and a second score calculated based on features of a second biometric modality. Based on the third score, determine whether the combination conditions are satisfied.

19. The biometric authentication method according to claim 15, wherein, The first biometric modality is the iris modality, and the second biometric modality is the facial modality.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 19.

21. An electronic device comprising: processor; Memory, which stores computer programs. The computer program, when executed by a processor, causes the processor to perform the method described in any one of claims 1 to 19.

Images