Method and system for verifying liveness detection, system training method and medium

Abstract

A method and system for verifying liveness detection of a user is provided. The method includes the following steps: obtaining a video of a user speaking a phrase in response to a question presented to the user or a randomly generated phrase; inputting video data and audio data of the obtained video into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; inputting a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether the user spoke the predetermined phrase; and determining whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user spoke the predetermined phrase.

Description

Technical Field

[0001] The present disclosure relates to user identity authentication, and more particularly, to liveness detection using audio-visual consistency checks to authenticate user identity. Background Art

[0002] Electronic Know Your Customer (eKYC) procedures are used in various industries such as banking, insurance, hospitality, financial services, payment-based services, etc. to identify and verify the identity of customers. Existing eKYC solutions usually use biometric verification systems. Figure 1 As shown in , the biometric recognition and verification system includes two stages: an enrollment stage, in which the user's biometric features are mapped to users in a database; and a verification stage, in which the user's biometric features are extracted by sensors and compared with the features previously registered in the database in order to authenticate the user.

[0003] However, prior art biometric recognition systems are vulnerable to various types of presentation attacks, i.e., fraudulent presentations of a biometric capture subsystem (e.g., a biometric recognition sensor) intended to deceive or otherwise interfere with the operation of the biometric recognition system. For example, an artifact artificially representing a replica of a biometric characteristic may be presented to the biometric recognition system to impersonate a registered user and authenticate the attacker.

[0004] Presentation attacks include two-dimensional (2D) attacks, three-dimensional (3D) attacks, and voice attacks. 2D attacks include print attacks, display attacks, and video attacks. In print attacks, a printed photo of the registered user (e.g., a facial image) or an artifact of the registered user's biometrics (e.g., a fake fingerprint) is presented. In display attacks, an image of the registered user is presented via a display device such as a mobile phone, and in video attacks, a video of the registered user is presented. Furthermore, 3D attacks include mask attacks, in which the attacker wears a wearable printed or silicone 3D mask, or wears or otherwise presents a 3D artifact of the user's biometrics (e.g., a fake iris) to impersonate the registered user.

[0005] Existing eKYC solutions employ various presentation attack detection mechanisms to automatically identify and prevent presentation attacks. These mechanisms include facial detection through depth map analysis, which creates a 3D facial depth map for user authentication; remote photoplethysmography (rPPG) methods, which authenticate users based on heart rate or heart monitoring; and thermal imaging-based facial liveness detection, which authenticates users based on real-time thermal images. However, these mechanisms have various drawbacks. For example, both 3D facial depth analysis and thermal imaging-based facial liveness detection incur additional cost and complexity due to the additional sensors required (e.g., thermal imagers). Furthermore, rPPG methods require significant time and have relatively low accuracy. Summary of the Invention

[0006] Provided are a liveness detection verification method and system for verifying a user's audiovisual similarity check based on a random phrase challenge.

[0007] Also provided is a liveness detection verification method and system that includes a speech recognition machine learning model to verify the detection results of an audio-visual similarity detection machine learning model based on a random phrase test.

[0008] Also provided are methods and systems for training an audio-visual similarity detection machine learning model and a speech recognition machine learning model for authenticating users in eKYC applications.

[0009] According to one aspect of the embodiment, a method for verifying liveness detection by at least one processing device is provided, the method comprising the following steps: obtaining a video of a user speaking a phrase in response to a question presented to the user or a randomly generated phrase; processing the video to obtain video data and audio data of the user; inputting the obtained video data and the obtained audio data into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; inputting a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether the phrase spoken by the user corresponds to a predetermined phrase; and determining whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase to verify the liveness detection of the user.

[0010] The method may further comprise the steps of randomly generating the randomly generated phrase presented to the user from a pre-stored vocabulary set, wherein obtaining the video may comprise obtaining a video of the user speaking a phrase in response to the presentation of the randomly generated phrase, wherein the predetermined phrase may be the randomly generated phrase, wherein the pre-stored vocabulary set may comprise predetermined terms for each of a predetermined number of categories, and wherein the randomly generated phrase may comprise a combination of one term randomly selected from each of the predetermined number of categories.

[0011] Obtaining the video may include: sending one of the question and the randomly generated phrase to a user device; and receiving from the user device a video of a user speaking a phrase in response to one of the question and the randomly generated phrase presented to the user via the user device.

[0012] Processing the video may include: obtaining an audio stream and a video stream from the video; obtaining a second predetermined number of frames of video data in a first predetermined number of batches, the frames of the video data including a cropped mouth region generated from the video stream; and obtaining the second predetermined number of frames of audio data in the first predetermined number of batches, the frames of the audio data including a mel-scaled spectrogram generated from the audio stream.

[0013] Inputting the obtained video data and the obtained audio data into the first determination model may include the following steps: inputting the obtained video data into a first machine learning neural network, and inputting the obtained audio data into a second machine learning neural network; and obtaining a contrast loss value corresponding to the distance between a feature from the output of the first machine learning neural network and a feature from the output of the second machine learning neural network as the first determination.

[0014] Inputting into the second determination model may include: inputting the first input and the second input into a third machine learning neural network, the first input being features extracted from the output of the second machine learning neural network, and the second input being character alignment or word alignment of the predetermined phrase; and obtaining a connectionist temporal classification (CTC) loss value as the second determination.

[0015] Determining whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase may include: determining that the user did not speak the predetermined phrase correctly based on the contrast loss value being less than a predetermined threshold; determining that the user did not speak the predetermined phrase correctly based on the predetermined phrase not corresponding to a predicted label obtained from the CTC loss value; and determining that the user spoke the predetermined phrase correctly based on both the contrast loss value being greater than the predetermined threshold and the predicted label corresponding to the predetermined phrase.

[0016] According to an aspect of another embodiment, a liveness detection verification system is provided, which includes: a memory configured to store instructions; and at least one processor configured to execute the instructions to: obtain a video of a user speaking a phrase in response to a question presented to the user or a randomly generated phrase; process the video to obtain video data and audio data of the user; input the obtained video data and the obtained audio data into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; input a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether the phrase spoken by the user corresponds to the predetermined phrase; and determine whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase to verify the liveness detection of the user.

[0017] The at least one processor may also be configured to execute instructions to randomly generate the randomly generated phrase presented to the user from a pre-stored vocabulary set; the obtained video may be a video of the user speaking the phrase in response to the presented randomly generated phrase; the predetermined phrase may be the randomly generated phrase; the pre-stored vocabulary set may include predetermined terms for each of a predetermined number of categories; and the randomly generated phrase may include a combination of one term randomly selected from each of the predetermined number of categories.

[0018] The liveness detection verification system may further include: a communication interface, wherein the at least one processor is further configured to execute instructions to: control the communication interface to send the question and one of the randomly generated phrases to a user device; and receive from the user device via the communication interface a video of the user speaking a phrase in response to presentation of the question and one of the randomly generated phrases to the user via the user device.

[0019] The at least one processor may also be configured to execute instructions to: obtain an audio stream and a video stream from the video; obtain a second predetermined number of frames of video data in a first predetermined number of batches, the frames of the video data including a cropped mouth region generated from the video stream; and obtain the second predetermined number of frames of audio data in the first predetermined number of batches, the frames of the audio data including a Mel-scale spectrogram generated from the audio stream.

[0020] The at least one processor may also be configured to execute instructions to: input the obtained video data into a first machine learning neural network, and input the obtained audio data into a second machine learning neural network; and obtain a contrast loss value corresponding to a distance between a feature from an output of the first machine learning neural network and a feature from an output of the second machine learning neural network as the first determination.

[0021] The at least one processor may also be configured to execute instructions to: input the first input and the second input into a third machine learning neural network, the first input being features extracted from an output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; and obtain a connectionist temporal classification (CTC) loss value as the second determination.

[0022] The at least one processor may also be configured to execute instructions to: determine that the liveness detection fails verification based on the contrast loss value being less than a predetermined threshold; determine that the liveness detection fails verification based on the predetermined phrase not corresponding to the predicted label obtained from the CTC loss value; and determine that the user correctly said the predetermined phrase based on both the contrast loss value being greater than the predetermined threshold and the predicted label corresponding to the predetermined phrase.

[0023] According to an aspect of another embodiment, a non-transitory computer-readable medium is provided, which stores instructions that, when executed by one or more processors, cause the one or more processors to: obtain a video of a user speaking a phrase in response to a question presented to the user or a randomly generated phrase; process the video to obtain video data and audio data of the user; input the obtained video data and the obtained audio data into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; input a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether the phrase spoken by the user corresponds to the predetermined phrase; and determine whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase to verify liveness detection of the user.

[0024] The instructions may also cause the one or more processors, when executed by the one or more processors, to randomly generate the randomly generated phrase presented to the user from a pre-stored vocabulary set; the obtained video may be a video of the user saying a phrase in response to the presented randomly generated phrase; the predetermined phrase may be the randomly generated phrase; the pre-stored vocabulary set may include predetermined terms for each of a predetermined number of categories; and the randomly generated phrase may include a combination of one term randomly selected from each of the predetermined number of categories.

[0025] When executed by the one or more processors, the instructions may further cause the one or more processors to: obtain an audio stream and a video stream from the video; obtain a second predetermined number of frames of video data in a first predetermined number of batches, the frames of the video data including a cropped mouth region generated from the video stream; and obtain the second predetermined number of frames of audio data in the first predetermined number of batches, the frames of the audio data including a Mel-scale spectrogram generated from the audio stream.

[0026] When the instructions are executed by the one or more processors, the one or more processors may also be caused to: input the obtained video data into a first machine learning neural network, and input the obtained audio data into a second machine learning neural network; and obtain a contrast loss value corresponding to the distance between the features from the output of the first machine learning neural network and the features from the output of the second machine learning neural network as the first determination.

[0027] When the one or more processors execute the instructions, the one or more processors may also: input the first input and the second input to the third machine learning neural network, the first input being features extracted from the output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; and obtain a connectionist temporal classification CTC loss value as the second determination.

[0028] When the one or more processors execute the instructions, the one or more processors may also be caused to: determine that the liveness detection fails verification based on that the contrast loss value is less than a predetermined threshold; determine that the liveness detection fails verification based on that the predetermined phrase does not correspond to the predicted label obtained from the CTC loss value; and determine that the liveness detection passes verification based on both that the contrast loss value is greater than the predetermined threshold and that the predicted label corresponds to the predetermined phrase.

[0029] According to one aspect of another embodiment, a method for training a liveness detection verification system is provided, the method comprising the following steps: obtaining a video of a user speaking a phrase; processing the video to obtain video data and audio data of the user; and training a first machine learning model for detecting audio-visual similarity and a second machine learning model for speech recognition, wherein the first machine learning model is trained based on the video data and the audio data, and the second machine learning model is trained based on a first input corresponding to the audio data and a predetermined phrase, the predetermined phrase corresponding to one of a randomly generated phrase and an answer to a predetermined question.

[0030] The training may include training the first machine learning model and the second machine learning model simultaneously.

[0031] The first input corresponding to the audio data may include features extracted from a fully connected layer of the first machine learning model

[0032] The training may include: inputting the video data and the audio data into the first machine learning model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; inputting the first input corresponding to the audio data and the second input corresponding to the predetermined phrase into the second machine learning model to obtain a second determination indicating whether the phrase spoken by the user is the predetermined phrase.

[0033] Inputting the obtained video data and the obtained audio data into the first machine learning model may include: inputting the obtained video data into a first machine learning neural network, and inputting the obtained audio data into a second machine learning neural network; and obtaining a contrast loss value corresponding to the distance between a feature from the output of the first machine learning neural network and a feature from the output of the second machine learning neural network as the first determination.

[0034] Inputting the first input and the second input into the second machine learning model may include the following steps: inputting the first input and the second input into a third machine learning neural network, the first input being features extracted from an output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; and obtaining a connectionist temporal classification (CTC) loss value as the second determination.

[0035] The training may include: inputting the video data and the audio data into the first machine learning model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; inputting the first input corresponding to the audio data and the second input corresponding to the predetermined phrase into the second machine learning model to obtain a second determination indicating whether the phrase spoken by the user is the predetermined phrase; aggregating the first determination and the second determination; and training the first machine learning model and the second machine learning model based on the aggregated determination.

[0036] Additional aspects will be at least partly apparent or elucidated in the description which follows, or may be learned by practice of the presented embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] These and / or other aspects will become more readily apparent and understood from the following description taken in conjunction with the accompanying drawings, in which:

[0038] Figure 1 It is a flow chart of the biometric identification verification system;

[0039] Figure 2 is a flowchart of a training method for training a first machine learning model and a second machine learning model according to an embodiment;

[0040] Figure 3 is a flowchart of an example of an implementation of a training method according to an embodiment;

[0041] Figure 4 is a flow chart of a liveness detection and verification method according to an embodiment;

[0042] Figure 5is a flowchart of an example of an implementation method of a liveness detection and verification method according to an embodiment;

[0043] Figure 6 is a flowchart of an example of an implementation method of a liveness detection and verification method according to an embodiment;

[0044] Figure 7 is a flow chart of a data preprocessing method of a liveness detection and verification system according to an embodiment;

[0045] Figure 8 is a block diagram of a liveness detection and verification system according to an embodiment;

[0046] Figure 9 is a block diagram of a data preprocessor of a liveness detection and verification system according to an embodiment; and

[0047] Figure 10 is a block diagram of an example environment in which systems and / or methods in accordance with one or more embodiments may be implemented. DETAILED DESCRIPTION

[0048] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, in which like reference numerals always indicate like elements. However, it is to be understood that the present disclosure is not limited to the embodiments described herein, and features and components in one embodiment may be included in another embodiment or may be omitted.

[0049] In addition, it is to be understood that, as used herein, phrases such as "at least one of" preceding a list of elements modify the entire list of elements and do not modify the individual elements within the list. For example, the phrase "at least one of [A], [B], and [C]" or "at least one of [A], [B], or [C]" means only A, only B, only C, A and B, B and C, A and C, or A, B, and C.

[0050] It will also be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms (e.g., should not be construed as specifying relative order or importance). These terms are only used to distinguish one element from another.

[0051] In addition, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless clearly indicated otherwise or indicated by the surrounding context.

[0052] One or more embodiments of the present disclosure provide a liveness detection verification system and method for eKYC applications that can more reliably and effectively ensure biometric authentication and prevent demonstration attacks. Specifically, the liveness detection verification method according to one or more embodiments includes a first determination that captures and compares the user's audio-visual information to determine the consistency of mouth movements with the audio (i.e., audio-visual similarity detection). In order to further prevent demonstration attacks (such as video attacks), the liveness detection verification method utilizes a random phrase (e.g., a random phrase is generated at the time of authentication) and includes an additional consistency check (a second determination) to confirm that the random phrase has been spoken, thereby verifying the liveness detection. By using a random phrase and combining voice recognition to confirm that the phrase has been spoken, the user is required to speak in real time, thereby preventing the use of previously recorded or tampered videos.

[0053] According to one or more embodiments, the first determination (i.e., audiovisual similarity detection) can use a first machine learning model (e.g., a neural network, a deep learning model, a convolutional neural network (CNN), etc.) to determine a first probability that the mouth movement is synchronized with the audio (e.g., a probability of a first degree of fit between the mouth movement and the audio). In addition, the second determination (i.e., speech recognition) can use a second machine learning model (e.g., a neural network, a deep learning model, an artificial recurrent neural network, a long short-term memory (LSTM) network, etc.) to determine a second probability that the user said a random phrase (e.g., a probability of a second degree of fit between the audio and the random phrase). Therefore, when both the first probability and the second probability exceed a threshold (e.g., if the combined loss is lower than a predetermined value or satisfies one or more predetermined conditions), the user's liveness detection is verified.

[0054] An eKYC application (e.g., an online authentication system) that incorporates a liveness detection verification system and method according to one or more embodiments may also include additional measures for authenticating users, such as facial recognition. Here, facial recognition may be performed separately, for example, by capturing additional images before or after verifying liveness detection, or may be performed using images obtained from a video captured for liveness detection. It should be understood that facial recognition is only one example of biometric authentication, and one or more other embodiments may utilize other biometric authentication schemes, such as fingerprint detection, iris scanning, etc., to authenticate users.

[0055] Figure 2 is a flowchart of a training method for training a first machine learning model and a second machine learning model according to an embodiment, Figure 3 is a flowchart of an example of an implementation of a training method according to an embodiment.

[0056] Reference Figure 2In operation 210, data preprocessing is performed on the captured video of the user speaking the randomly generated phrase. The data preprocessing is performed to obtain video data (e.g., video frames) and audio data (e.g., audio frames) corresponding to the randomly generated phrase for input to a first machine learning model. For example, the first machine learning model may include a 3D CNN for video data and a 3D CNN for audio data, such as Figure 3 Below, refer to Figure 7 To provide a detailed description of data preprocessing.

[0057] Although the present embodiment is described with reference to randomly generated phrases, it is understood that one or more other embodiments are not limited thereto. For example, according to another embodiment, data pre-processing is performed on a captured video of a user speaking a phrase corresponding to an answer to a predetermined question.

[0058] In operation 220, the randomly generated phrase is input to a speech recognizer (e.g., a speech recognition engine, a speech recognition processor, etc.) that includes a second machine learning model (e.g., a recurrent neural network). For example, character or word alignments of the randomly generated phrase can be input to the speech recognizer as labels for determining a loss (e.g., a connectionist temporal classification (CTC) loss). It is understood that operations 210 and 220 can be performed simultaneously, substantially simultaneously, or sequentially (e.g., operation 210 is performed at least partially or completely before or after operation 220).

[0059] As explained above, although the present embodiment is described with reference to randomly generated phrases, it should be understood that one or more other embodiments are not limited thereto. For example, according to another embodiment, a predetermined phrase corresponding to an answer to a predetermined question is input to a speech recognizer. In this case, the character or word alignment of the predetermined phrase can be input to the speech recognizer as a label for determining a loss (e.g., CTC loss).

[0060] In operation 230, a first machine learning model processes the video data and the audio data to obtain a first probability that the mouth movement is synchronized with the audio (e.g., a probability of a first degree of fit between the mouth movement and the audio). For example, the first probability may be a contrastive loss value based on a measured distance between features from a fully connected layer of a 3D CNN for the video data and a fully connected layer of a 3D CNN for the audio data. The contrastive loss function (L1(Y, X)) according to an embodiment may be according to the following formula 1.

[0061] [Formula 1]

[0062]

[0063] in,

[0064] and

[0065]

[0066] Where: L1 corresponds to the contrast loss; X vid Corresponding to the input to network 1 (video network), video frame data, and (n×W×H) is the shape of the input data; X aud Corresponding to the input of network 2 (audio network), audio frame data, and (n×W×H) is the shape of the input data; (X vid ,X aud ) i Corresponding to the i-th input pair; Y i is the corresponding label (1-true pair, i.e., when the frame (X vid ,X aud ) from the same identity, and 0-impersonated pairs, i.e., when the frame (X vid ,X aud ) from different identities); N is the number of training samples; D C is a function with input X vid and X aud is the Euclidean distance between the outputs of the FC layers of the two networks; λ is the regularization parameter; M is the predetermined margin; n corresponds to the number of frames; W corresponds to the width of the frame; and H corresponds to the height of the frame.

[0067] In operation 240, the second machine learning model processes the audio data based on the randomly generated phrase input in operation 220 to perform speech recognition, thereby obtaining a second probability that the user said the random phrase. The second probability can be a CTC loss value. According to an embodiment, the second machine learning model can receive the audio data output from the first machine learning model to perform speech recognition. For example, the second machine learning model can receive a fully connected layer (or its features) of the audio data for the first machine learning model and a character or word aligned label for the randomly generated phrase from a 3D CNN. Based on these inputs, the second machine learning model can perform speech recognition to obtain a CTC loss value.

[0068] As explained above, although the present embodiment is described with reference to randomly generated phrases, it is to be understood that one or more other embodiments are not limited thereto. For example, according to another embodiment, a predetermined phrase corresponding to an answer to a predetermined question is input to a speech recognizer to determine whether a phrase spoken by the user (i.e., audio data) corresponds to the predetermined phrase. That is, according to another embodiment, the second machine learning model may receive a fully connected layer (or features thereof) of the audio data for the first machine learning model and character or word aligned labels for the predetermined phrase from a 3D CNN. Based on these inputs, the second machine learning model may perform speech recognition to obtain a CTC loss value.

[0069] In operation 250 , the first probability (eg, the contrastive loss value) and the second probability (eg, the CTC loss value) are aggregated.

[0070] In operation 260, the probability of aggregation (e.g., aggregation loss) is propagated through the back-propagation algorithm, and the weights of the first machine learning model and the second machine learning model (e.g., 3D CNN for video data, 3D CNN for audio data, and recurrent neural network) are updated. The training method can then be repeatedly performed until, for example, a stopping criterion is reached, such as Figure 3 As shown in .

[0071] According to an embodiment, as described above, both the first machine learning model and the second machine learning model are trained from the same training data set and are related to each other. Therefore, the first machine learning model and the second machine learning model can be trained simultaneously, thereby compressing and saving training time compared to the case where the machine learning models are trained separately. The trained models can then be used to perform the liveness detection verification method, as described below with reference to Figures 4 to 6 Descriptive.

[0072] Figure 4 is a flowchart of a liveness detection and verification method according to an embodiment, Figure 5 is a flowchart of an example of an implementation method of a liveness detection and verification method according to an embodiment. Figure 6 1 is a flowchart illustrating an example of an implementation of a liveness detection and verification method according to an embodiment. For example, the liveness detection and verification method may be performed in conjunction with an eKYC application by a server for performing online authentication of a user or customer.

[0073] Reference Figure 4 In operation 410, data pre-processing is performed to obtain video data (e.g., video frames) and audio data (e.g., Figure 6) for input to a first machine learning model of an audio-visual similarity detector (e.g., an audio-visual similarity detection engine, an audio-visual similarity detection processor, etc.). As described above and in Figure 5 and Figure 6 As shown in the example of , the first machine learning model may include a video network (e.g., 3D CNN) for video data and an audio network (e.g., 3D CNN for audio data). In addition, the data preprocessing in operation 410 is consistent with the data preprocessing performed for training the machine learning model (i.e., Figure 2 The operation 210 in FIG. 2 is the same as or substantially similar to the operation 210 in FIG. 2

[0074] During data preprocessing, a randomly generated phrase is provided for the user to read. For example, the first server may generate or otherwise obtain the phrase and send it to a user device (e.g., a mobile phone, a personal computer, etc.) for display on a display of the user device (e.g., via a web page, an application, a user interface, etc.). Subsequently, video data and audio data of the user reading the randomly generated phrase may be obtained. For example, the first server or the second server may receive a video (including video data and audio data) of the user reading the randomly generated phrase from the user device. The video may be captured by a camera or image sensor and a microphone of the user device. The received video is then preprocessed to obtain video data (e.g., video frames) and audio data (e.g., as Figure 6 Next, refer to Figure 7 To provide a detailed description of data preprocessing.

[0075] Although the present embodiment is described with reference to randomly generated phrases, it is understood that one or more other embodiments are not limited thereto, and any other predetermined phrases may be applied. For example, according to another embodiment, a predetermined question is presented to the user for display. Subsequently, video data and audio data of the user providing an answer to the question may be obtained. Thus, the first server or the second server may receive a video (including video data and audio data) of a user speaking an answer to the presented question from the user device. The received video is then pre-processed to obtain video data (e.g., video frames) and audio data (e.g., Figure 6 ).

[0076] In operation 420, the randomly generated phrase is input to a speech recognizer (e.g., a speech recognition engine, a speech recognition processor, etc.) that includes a second machine learning model (e.g., a recurrent neural network). For example, character or word alignments of the randomly generated phrase can be input to the speech recognizer as labels for determining a loss (e.g., a connectionist temporal classification (CTC) loss). It is understood that operations 410 and 420 can be performed simultaneously, substantially simultaneously, or sequentially (e.g., operation 410 is performed at least partially or completely before or after operation 420).

[0077] As explained above, although the present embodiment is described with reference to randomly generated phrases, it should be understood that one or more other embodiments are not limited thereto. For example, according to another embodiment, a predetermined phrase corresponding to an answer to a predetermined question is input to a speech recognizer to determine whether a phrase spoken by a user (i.e., audio data) corresponds to the predetermined phrase. In this case, the character or word alignment of the predetermined phrase can be input to the speech recognizer as a label for determining the loss (e.g., CTC loss).

[0078] In operation 430, the first machine learning model processes the video data and the audio data to obtain a first probability that the mouth movement is synchronized with the audio (e.g., a probability of a first degree of fit between the mouth movement and the audio). For example, the first probability may be based on a contrastive loss value of a measured distance between features from a fully connected layer of a 3D CNN for video data and a fully connected layer of a 3D CNN for audio data, such as Figure 5 As shown in . According to the embodiment, the contrast loss function can be as described above. Figure 2 Described in Eq. 1.

[0079] In operation 440, the speech recognizer processes the audio data based on the randomly generated phrase input in operation 420 to perform speech recognition, thereby obtaining a second probability that the user said the random phrase. The second probability may be a CTC loss value. According to an embodiment, the second machine learning model may receive the audio data output from the first machine learning model to perform speech recognition. For example, the second machine learning model may receive a fully connected layer of a 3D CNN for the audio data of the first machine learning model (or features extracted therefrom) and character or word aligned labels for the randomly generated phrase. Based on these inputs, the second machine learning model may perform speech recognition to obtain a CTC loss value, such as Figure 5 and Figure 6 As shown in .

[0080] As explained above, although the present embodiment is described with reference to randomly generated phrases, it is to be understood that one or more other embodiments are not limited thereto. For example, according to another embodiment, a predetermined phrase corresponding to an answer to a predetermined question is input to a speech recognizer to determine whether a phrase spoken by the user (i.e., audio data) corresponds to the predetermined phrase. That is, according to another embodiment, the second machine learning model can receive a fully connected layer (or features thereof) of a 3D CNN for audio data of the first machine learning model and character or word aligned labels for the predetermined phrase. Based on these inputs, the second machine learning model can perform speech recognition to obtain a CTC loss value.

[0081] In operation 450, liveness detection is verified based on whether the first prediction value and the second prediction value satisfy one or more conditions. Figure 6 As illustrated in , a combined loss determination based on a first prediction value (e.g., a contrast loss value) and a second prediction value (e.g., a CTC loss value) can be performed to verify liveness detection.

[0082] Figure 5 An example of combined loss determination is shown in . Specifically, the contrast loss value (L1) obtained from the first machine learning model is compared with a threshold value (e.g., a predetermined margin value (M)). If the contrast loss is less than the threshold value, it is determined that the liveness detection has not passed the verification. In addition, the CTC loss value (L2) obtained from the second machine learning model is used to predict the label of the recognized speech. If the predicted label does not correspond to the randomly generated phrase, it is determined that the liveness detection has not passed the verification. In addition, if the contrast loss value is greater than (or greater than or equal to) the threshold value and the predicted label corresponds to the randomly generated phrase, it is determined that the liveness detection has passed the verification.

[0083] Repeat for each batch of video and audio data obtained from the data pre-processing of operation 410 Figure 4 For example, Figure 5 and Figure 6 As illustrated in , three batches of 25 frames of video and audio data, each corresponding to 1 second of the captured video, are successively input to the first machine learning model.

[0084] The liveness detection verification method according to the embodiment described above can more reliably ensure biometric authentication and prevent demonstration attacks by implementing voice recognition to verify audio-visual similarity detection based on a randomly generated phrase (or a question randomly selected from a plurality of predetermined (or frequently / periodically updated) questions). For example, by utilizing a random phrase (or question) and combining it with voice recognition to confirm that the phrase (or answer to the question) is spoken, the user is required to speak in real time, thereby preventing the use of previously recorded or tampered videos.

[0085] Figure 7Flowchart of the data preprocessing method of the liveness detection and verification system according to the embodiment. Figure 7 to train the first machine learning model and / or the second machine learning model (i.e., Figure 2 210), or performing liveness detection verification using the previously trained first and second machine learning models (i.e., Figure 4 Operation 410).

[0086] Reference Figure 7 In operation 710, a random phrase challenge is generated and provided to the user. For example, the phrases may be randomly generated from a pre-stored dictionary, vocabulary set, or database and displayed to the user via a display. In one exemplary implementation, the vocabulary set may include a predetermined number of entries or candidate terms (e.g., words, letters, numbers, etc.) for each of a predetermined number of categories. An example of such a vocabulary set is shown in Table 1 below, which includes six categories: commands, colors, prepositions, letters, numbers, and adverbs:

[0087] Table 1

[0088] Order color preposition letter number adverb Bin blue at AZ* 0-9 again Lay green by *Excluding W now Place red in please Set White with soon

[0089] Using the above vocabulary set as an example, a phrase such as "Place blue at F9 now" or "Lay green by W 3 again" can be generated and provided (e.g., to a user device or customer terminal) for display on a screen. However, it is to be understood that this is merely an example, and one or more other embodiments are not limited to the vocabulary set and / or six candidate term categories as provided above, and any number of candidate terms and categories can be implemented. Ultimately, by using a random phrase challenge with a large number of possible outputs, an attacker is prevented from impersonating a registered user via a previously recorded or doctored video.

[0090] While this embodiment has been described with reference to a random phrase, it is understood that one or more other embodiments are not limited thereto and may utilize another predetermined phrase. For example, a predetermined phrase corresponding to the answer to a question may be employed. In this regard, the question may be among a plurality of predetermined questions, which may be updated or changed from time to time to enhance security (by reducing the likelihood that a pre-recorded answer could be successfully used by an attacker). According to another embodiment, the question (or the corresponding predetermined phrase / answer) may be presented to the user in order to obtain a video of the user speaking the answer.

[0091] Next, a video of the user speaking the presented random phrase challenge (or the answer to the predetermined question) is obtained in operation 720. For example, once the random phrase challenge is presented (i.e., when the user speaks the random word challenge), the camera or image sensor and microphone capture a video of a predetermined length (e.g., 3 seconds).

[0092] The obtained video is divided into a video stream and an audio stream in operation 730. For example, a demultiplexer or an image processor demultiplexes the captured video into elementary video and audio streams.

[0093] In operation 740, the video stream is processed for input to a first machine learning model. The video stream may be processed for input to train the model (i.e., Figure 2 methods) and / or determining the consistency of mouth movements with audio (i.e., Figure 4 Here, the processing may include at least one of converting the video stream frame rate to a predetermined frame rate (e.g., 25 frames per second (fps)) and extracting a predetermined number of video frames (e.g., 75 video frames corresponding to 3 seconds) such that each frame corresponds to a predetermined duration (e.g., 40 milliseconds of video). The processing may also include at least one of detecting mouth coordinates for each video frame, cropping the mouth region according to the detected coordinates (or bounding box), and converting the cropped mouth region into a grayscale or single-channel image and resizing it to a predetermined size (e.g., 60×100 pixels).

[0094] In operation 750, the audio stream obtained in operation 730 is processed for input to a first machine learning model for determining consistency of mouth movements with audio. Here, the processing may include converting the audio into a single channel stream, changing the sample frequency to a predetermined frequency (e.g., 22.05 kHz), and extracting a predetermined number of non-overlapping frames (e.g., 75 non-overlapping frames corresponding to 3 seconds) such that each frame corresponds to at least one of a predetermined duration (e.g., 40 milliseconds of audio). In addition, Figure 7 For the data preprocessing used to train the liveness detection model, the processing of operation 750 may also include tampering the input audio frame into the negative class of the training data set (or not tampering the input audio frame into the positive class). Tampering may include modifying at least one of the phonemes and pitch of the audio frame. Of course, if Figure 7 If the data preprocessing is not used to train the liveness detection model, tampering is omitted.

[0095] The processing of operation 750 may also include extracting a mel-scaled spectrogram for each audio frame. Like video frames, mel-scaled spectrograms may also be converted into grayscale or single-channel images.

[0096] It is to be understood that operations 740 and 750 may be performed simultaneously, substantially simultaneously, or sequentially (with operation 740 being performed at least partially or completely before or after operation 750).

[0097] In operation 760, a plurality of batches of a predetermined number of frames (e.g., 25 frame batches corresponding to 1 second of data) may be created for each of the processed video frames and the processed audio frames. For example, three batches of 25 frames of video and audio may be generated in operation 760 and may be input to the first machine learning model.

[0098] The video and audio data obtained in operation 760 (e.g., Figure 3 、 Figure 5 and Figure 6 The video frames and spectrograms shown in are input to the first machine learning model as described above with reference to Figure 2 and Figure 3 Train the model as described above, or as referenced above Figures 4 to 6 Audiovisual similarity detection and verification are performed descriptively.

[0099] Figure 8 8 is a block diagram of a liveness detection and verification system 800 according to an embodiment. The liveness detection and verification system 800 may be implemented in one or more servers (e.g., for online authentication systems), user terminals (e.g., customer terminals), or distributed across user terminals and one or more servers.

[0100] Reference Figure 8 The liveness detection and verification system 800 includes a data preprocessor 810 , an audio-visual similarity detector 820 , a speech recognizer 830 and a liveness detection and verification unit 840 .

[0101] The data preprocessor 810 obtains video data (eg, video frames) and audio data (eg, Figure 6 exemplified in ). Figure 9 A detailed block diagram of a data pre-processor according to an embodiment is illustrated in FIG.

[0102] Reference Figure 9 , the data pre-processor 810 includes a memory 910 , a random phrase generator 920 , a communication unit 930 , a demultiplexer 940 , a video stream processor 950 , and an audio stream processor 960 .

[0103] The memory 910 stores the above-mentioned Figure 7The vocabulary set described. For example, the vocabulary set may include a predetermined number of entries or candidate terms (e.g., words, letters, numbers, etc.) for each category in a predetermined number of categories as exemplified in Table 1 above. The memory 910 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette tape, a magnetic tape, and / or another type of non-transitory computer-readable medium together with a corresponding drive. In addition, it is understood that the memory 910 may be provided in different or separate devices or servers.

[0104] The random phrase generator 920 randomly generates phrases from the vocabulary. For example, the random phrase generator 920 may randomly combine one term from each category of the vocabulary to generate a phrase.

[0105] The communication unit 930 (e.g., a communicator, a communication interface, etc.) transmits the randomly generated phrase to a user terminal (e.g., a mobile phone, a personal computer, etc.) via at least one of a wired or wireless network (e.g., a cellular network, an Internet network, a fiber optic network, a Wi-Fi network, etc.). Furthermore, the communication unit 930 receives a video of the user speaking the randomly generated phrase from the user terminal. The video may have a predetermined length (e.g., 3 seconds) and may be captured by the user terminal's camera, image sensor, and microphone once the random phrase challenge is presented (i.e., when the user speaks the random phrase challenge).

[0106] It is understood that the communication unit 930 can provide the user with a randomly generated phrase and / or receive the captured video through a web server of a website hosting an online authentication service or a customer portal (e.g., a financial institution's website). In addition, it is understood that in the case where the data preprocessor 810 is used to train the first machine learning model and the second machine learning model, and / or in the case where the data preprocessor 810 obtains a video of the user locally reading the randomly generated phrase (e.g., in the case where the liveness detection verification system 800 is included in a user terminal or a customer terminal), the communication unit 930 can be omitted.

[0107] The demultiplexer 940 demultiplexes the obtained video into elementary video and audio streams.

[0108] The video stream processor 950 processes the video stream for input to the audiovisual similarity detector 820 to train the model and / or obtain a decision from the model. Here, the video processing may include at least one of the following: converting the video stream frame rate to a predetermined frame rate (e.g., 25 frames per second (fps)), extracting a predetermined number of video frames (e.g., 75 video frames corresponding to 3 seconds) so that each frame corresponds to a predetermined duration (e.g., 40 milliseconds of video), detecting mouth coordinates for each video frame, cropping the mouth area according to the detected coordinates (or bounding box), converting the cropped mouth area into a grayscale or single-channel image, and resizing the grayscale or single-channel image to a predetermined size (e.g., 60×100 pixels).

[0109] In addition, the video stream processor 950 obtains and outputs a predetermined number of grayscale or single-channel images of a predetermined number of frames (e.g., 25 frame batches corresponding to 1 second of data). For example, the video stream processor 950 can output three batches of 25 frames of video.

[0110] The audio stream processor 960 processes the audio stream for input to the audio-visual similarity detector 820 to train the model and / or obtain a decision from the model. Here, the audio processing may include converting the audio into a single-channel stream, changing the sample frequency to a predetermined frequency (e.g., 22.05KHz), and extracting a predetermined number of non-overlapping frames (e.g., 75 non-overlapping frames corresponding to 3 seconds) so that each frame corresponds to at least one of a predetermined duration (e.g., 40 milliseconds of audio). In addition, insofar as audio processing is used to train a liveness detection model, the processing may also include tampering with the audio frame for input into the inverse class of the training data set (or not tampering with the audio frame for input into the positive class). The tampering may include modifying at least one of the tempo and pitch of the audio frame.

[0111] The audio processing may further include extracting a Mel-scale spectrogram for each audio frame and converting the spectrogram into at least one of a grayscale or a mono-channel spectrogram.

[0112] In addition, the audio stream processor 960 obtains and outputs a predetermined number of grayscale or single-channel spectrograms of a predetermined number of frames in a predetermined number of batches (e.g., 25 frame batches corresponding to 1 second of data). For example, three batches of 25 frames can be output by the audio stream processor 960.

[0113] It is understood that one or more of the aforementioned components of the data preprocessor 810 may be implemented as circuitry, hardware, software, or a combination of hardware and software. Furthermore, it is understood that one or more of the aforementioned components of the data preprocessor 810 may be implemented as software executable by at least one processor. Furthermore, it is understood that one or more of the aforementioned components of the data preprocessor 810 may be distributed across multiple devices.

[0114] Refer to Figure 8 The audiovisual similarity detector 820 obtains the video data (e.g., multiple batches of video frames) and audio data (e.g., multiple batches of audio frames) output by the data preprocessor 810 (i.e., the video stream processor 950 and the audio stream processor 960), and outputs a first determination value indicating whether the video data (e.g., mouth movements in the video data) are synchronized with the audio. Specifically, the audiovisual similarity detector 820 includes a first machine learning model to obtain a first probability of mouth movement synchronization with the audio (e.g., a probability of a first degree of consistency between the mouth movement and the audio) for each corresponding batch of video data and audio data.

[0115] The first machine learning model may include a first neural network (e.g., 3D CNN) for video data, a second neural network (e.g., 3D CNN) for audio data, and a contrastive loss calculator. Here, the first neural network may output a fully connected layer corresponding to the video data to the contrastive loss calculator, and the second neural network may output a fully connected layer corresponding to the audio data to the contrastive loss calculator. The contrastive loss calculator may measure the distance (e.g., distances) between features extracted from the fully connected layers of the first neural network and the second neural network. Figure 5 ). The contrast loss function according to the embodiment can be as shown in the above reference Figure 2 Described in Eq. 1.

[0116] In addition, the audiovisual similarity detector 820 may output the audio data to the speech recognizer 830. For example, the audiovisual similarity detector 820 may output a fully connected layer of the audio data (or features thereof) to the speech recognizer 830.

[0117] The speech recognizer 830 receives the randomly generated phrase from the data preprocessor 810 and the audio data from the audio-visual similarity detector 820, and performs speech recognition based on the received audio data to output a second determination value. Specifically, the speech recognizer 830 processes the audio data to perform speech recognition and determines whether the user has spoken the random phrase (e.g., obtains a second probability that the user has spoken the random phrase as the second determination value).

[0118] To this end, the speech recognizer 830 may include a second machine learning model (e.g., a recurrent neural network, an LSTM network, etc.). For example, the second machine learning model may receive as input a fully connected layer of a 3D CNN for the audio data of the first machine learning model (or features extracted therefrom) and labels for character or word alignments of randomly generated phrases. Here, the labels for character or word alignments of the randomly generated phrases may be obtained by the speech recognizer 830 to determine a loss (e.g., a connectionist temporal classification (CTC) loss). The speech recognizer 830 may generate labels for character or word alignments, or may obtain labels from the data preprocessor 810. Based on these inputs, the second machine learning model may perform speech recognition to obtain a second determination value (e.g., a CTC loss value, such as Figure 5 and Figure 6 ).

[0119] To understand, you can refer to the above Figure 2 and Figure 3 Describing training of a first machine learning model and a second machine learning model.

[0120] The liveness detection verifier 840 verifies liveness detection based on the first determination value (e.g., the first probability) and the second determination value (e.g., the second probability). For example, the liveness detection verifier 840 verifies liveness detection based on whether the first prediction value and the second prediction value satisfy one or more conditions. Figure 6 As illustrated in , the liveness detection verifier 840 may perform a combined loss determination based on a first prediction value (eg, a contrast loss value) and a second prediction value (eg, a CTC loss value) to verify liveness detection.

[0121] As mentioned above, Figure 5 An example of combined loss determination is shown in . Specifically, the contrast loss value (L1) obtained from the first machine learning model is compared with a threshold value (e.g., a predetermined margin value (M)). If the contrast loss is less than the threshold value, the liveness detection verifier 840 determines that the liveness detection has not passed the verification. In addition, the CTC loss value (L2) obtained from the second machine learning model is used to predict the label of the recognized speech. If the predicted label does not correspond to the randomly generated phrase, the liveness detection verifier 840 determines that the liveness detection has not passed the verification. In addition, if the contrast loss value is greater than (or greater than or equal to) the threshold value and the predicted label corresponds to the randomly generated phrase, the liveness detection verifier 840 determines that the liveness detection has passed the verification.

[0122] The operations of the audiovisual similarity detector 820 , the speech recognizer 830 , and the liveness detection verifier 840 are repeated for each batch of video data and audio data obtained from the data preprocessor 810 .

[0123] It is understood that one or more of the aforementioned components of the liveness detection verification system 800 can be implemented as circuitry, hardware, software, or a combination of hardware and software. Furthermore, it is understood that one or more of the aforementioned components of the liveness detection verification system 800 can be implemented as software executable by at least one processor. Furthermore, it is understood that one or more of the aforementioned components of the liveness detection verification system 800 can be distributed across multiple devices.

[0124] Figure 10 is a block diagram of an example environment 1000 in which systems and / or methods according to one or more embodiments may be implemented. Figure 10 As shown in FIG, environment 1000 may include a user device 1100, a platform 1200, and a network 1300. The devices of environment 1000 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0125] User device 1100 is a device capable of receiving, generating, storing, processing, and / or providing information associated with platform 1200. For example, user device 1100 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a portable device (e.g., a portable multimedia player, a navigation device, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), a terminal device (e.g., a customer terminal, an ATM, a self-service checkout terminal or machine, etc.), or the like. User device 1100 may receive information from platform 1200 and / or send information to platform 1200.

[0126] In more detail, the user device 1100 may include a memory 1110 and at least one processor 1120, the processor 1120 being configured to execute instructions (e.g., instructions stored in a non-transitory computer-readable medium of the memory 1110) to perform various operations as described herein. The memory 1110 includes storage space within a single physical storage device or storage space across multiple physical storage devices. Software instructions may be read into the memory 1110 from another computer-readable medium or from another device via the communication unit 1160. When executed, the software instructions stored in the memory 1110 may cause the at least one processor 1120 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0127] The user device 1100 may also include an image capture device 1130 (eg, a camera), a microphone 1140 , a display 1150 , and a communication unit 1160 (eg, a communicator, a communication interface, a communication circuit, etc.).

[0128] At least one processor 1120 may control the display 1150 to output the random phrase received from the platform 1200 via the communication unit 1160. According to another embodiment, the at least one processor 1120 may implement at least a portion of the data pre-processor 810, as described above with reference to Figure 9 For example, at least one processor 1120 may implement the random phrase generator 920 and generate random phrases from a vocabulary set as described above (eg, a vocabulary set stored in the memory 1110 or stored externally).

[0129] At least one processor 1120 may control the image capture device 1130 and the microphone 1140 to capture video (combined or corresponding video data and audio data). For example, based on the output of a random phrase on the display 1150, the at least one processor 1120 may control the image capture device 1130 and the microphone 1140 to capture a video of a predetermined length (e.g., 3 seconds). Here, the image capture device 1130 and the microphone 1140 may be controlled to capture video in response to the generated random phrase at the same time as the random phrase is first output on the display 1150 or a predetermined time after the random phrase is first output on the display 1150 (e.g., 1 second after the random phrase begins to be output on the display 1150).

[0130] In addition, the at least one processor 1120 can control the display 1150 to output an indicator to inform the user when video is being captured (e.g., a countdown as to when video capture will begin, a countdown as to when video capture will stop, a symbol, an icon, or a graphical user interface (GUI) entry indicating that video is being captured, etc.). In addition, the display 1150 can stop displaying the random phrase based on the capture of the video (e.g., once video capture stops or a predetermined amount of time after video capture stops).

[0131] The communication unit 1160 communicates with the platform 1200 via the network 1300. According to an embodiment, the communication unit 1160 transmits the captured video (i.e., the combined or corresponding audio and video data) to the platform 1200. According to another embodiment, the at least one processor 1120 may implement at least one of the demultiplexer 940, the video stream processor 950, and the audio stream processor 960 of the data preprocessor 810. In this case, as described above, the communication unit 1160 may transmit at least one of the processed video stream and the processed audio stream, or at least one of the batch of video frames and the batch of audio frames or the spectrogram as described above, to the platform 1200.

[0132] In addition, the communication unit 1160 may transmit the randomly generated phrase to the platform 1200 (for example, when the random phrase is generated by the user device 1100). Of course, when the randomly generated phrase is generated by the platform 1200 and received by the user device 1100 via the communication unit 1160, the communication unit 1160 may not transmit the randomly generated phrase back to the platform 1200. According to another embodiment, when the platform 1200 includes multiple devices (for example, a server) and the user device 1100 receives the randomly generated phrase from a first device among the multiple devices, the communication unit 1160 may transmit the randomly generated phrase (and the captured video) to a second device among the multiple devices.

[0133] The platform 1200 includes one or more devices that implement all or at least a portion of the liveness detection and verification system 800 described above. In some implementations, the platform 1200 may include a server (e.g., a cloud server, a network server, an authentication server, etc.) or a server group. In some embodiments, the platform 1200 may be designed to be modular so that certain software components can be swapped out or in according to specific needs.

[0134] like Figure 10 As shown in , platform 1200 includes memory 1210, at least one processor 1220 and communication unit 1230 (e.g., communicator, communication interface, communication circuit, etc.). Memory 1210 includes storage space within a single physical storage device or storage space distributed across multiple physical storage devices or servers. Software instructions can be read into memory 1210 from another computer-readable medium or from another device via communication unit 1230. When executed, the software instructions stored in memory 1210 can cause at least one processor 1220 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuits can be used to replace software instructions or in combination with software instructions to perform one or more processes described herein. Therefore, the implementation described herein is not limited to any specific combination of hardware circuits and software.

[0135] Depending on the embodiment, at least one processor 1220 may be controlled to implement all or at least a portion of the liveness detection and verification system 800 described above. For example, at least one processor 1220 may be controlled to implement at least a portion of the data preprocessor 810. To this end, the memory 1210 may store a vocabulary, and the at least one processor 1220 may randomly generate a phrase from the vocabulary. In addition, the at least one processor 1220 may control the communication unit 1230 to transmit the randomly generated phrase to the user device 1100 via the network 1300, and may receive a captured video of the user speaking the randomly generated phrase from the user device 1100 via the network 1300.

[0136] At least one processor 1220 may demultiplex the video into elementary video and audio streams. Furthermore, at least one processor 1220 may be controlled to perform at least one of the following: converting the video stream frame rate to a predetermined frame rate (e.g., 25 frames per second (fps)), extracting a predetermined number of video frames (e.g., 75 video frames corresponding to 3 seconds) such that each frame corresponds to a predetermined duration (e.g., 40 milliseconds of video), detecting mouth coordinates for each video frame, cropping the mouth region according to the detected coordinates (or bounding box), converting the cropped mouth region into a grayscale or single-channel image, and resizing the grayscale or single-channel image to a predetermined size (e.g., 60×100 pixels).

[0137] In addition, at least one processor 1220 can obtain a predetermined number of grayscale or single-channel images of a predetermined number of frames in a predetermined number of batches (e.g., 25 frame batches corresponding to 1 second of data). For example, three batches of 25 frames of video can be output by the video stream processor 950.

[0138] At least one processor 1220 can also process an audio stream. Here, audio processing can include converting the audio into a single-channel stream, changing the sample frequency to a predetermined frequency (e.g., 22.05KHz), and extracting a predetermined number of non-overlapping frames (e.g., 75 non-overlapping frames corresponding to 3 seconds) so that each frame corresponds to at least one of a predetermined duration (e.g., 40 milliseconds of audio). In addition, insofar as audio processing is used to train a machine learning model, processing can also include tampering with the input audio frame to the inverse class of the training data set (or not tampering with the input audio frame to the positive class). Tampering can include modifying at least one of the tempo and pitch of the audio frame.

[0139] The audio processing may further include extracting a Mel-scale spectrogram for each audio frame and converting the spectrogram into a grayscale or mono-channel spectrogram.

[0140] In addition, at least one processor 1220 can obtain and output a predetermined number of grayscale or single-channel images of a predetermined number of frames in a predetermined number of batches (e.g., 25 frame batches corresponding to 1 second of data). For example, three batches of 25 frames can be output by the audio stream processor 960.

[0141] At least one processor 1220 may also implement the audiovisual similarity detector 820 and the speech recognizer 830 as described above. To this end, the at least one processor 1220 may process video data (e.g., multiple batches of video frames) and audio data (e.g., multiple batches of audio frames) and output a first determination value indicating whether the video data (e.g., mouth movements in the video data) are synchronized with the audio. Specifically, the at least one processor 1220 may input the video data and audio data into a first machine learning model to obtain a first probability that the mouth movement is synchronized with the audio (e.g., a probability of a first degree of consistency between the mouth movement and the audio) for each corresponding batch of video data and audio data.

[0142] In addition, at least one processor may perform speech recognition on the audio data and determine whether the user utters the random phrase (e.g., obtain a second probability that the user utters the random phrase as a second determination value). The fully connected layer of the audio data (or its features) is sent to the speech recognizer 830.

[0143] To this end, at least one processor 1220 can input the fully connected layer of the 3DCNN of the audio data for the first machine learning model (or features extracted therefrom) into a second machine learning model (e.g., a recurrent neural network, an LSTM network, etc.). The at least one processor 1220 can also be controlled to input labels for character or word alignment of randomly generated phrases into the second machine learning model. Here, the labels for character or word alignment of the randomly generated phrases can be obtained by at least one processor 1220 to determine the loss (e.g., connectionist temporal classification (CTC) loss). The at least one processor 1220 can generate labels for character or word alignment, or can obtain labels from another device. Based on these inputs, the second machine learning model can perform speech recognition to obtain a second determination value (e.g., a CTC loss value, such as Figure 5 and Figure 6 ).

[0144] In addition, at least one processor 1220 can be controlled to train the first machine learning model and / or the second machine learning model, as described above with reference to Figure 2 and Figure 3 described.

[0145] The at least one processor 1220 may further implement the liveness detection verifier 840 as described above to verify liveness detection based on a first determination value (e.g., a first probability) and a second determination value (e.g., a second probability). For example, the at least one processor 1220 may control to verify liveness detection based on whether the first prediction value and the second prediction value satisfy one or more conditions. In this case, the at least one processor 1220 may perform a combined loss determination based on the first prediction value (e.g., a contrast loss value) and the second prediction value (e.g., a CTC loss value) to verify liveness detection.

[0146] The network 1300 includes one or more wired and / or wireless networks. For example, the network 1300 may include at least one of a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a fourth generation (4G) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, etc., and / or a combination of these or other types of networks.

[0147] supply Figure 10 The number and arrangement of devices and networks shown in FIG are examples. Figure 10 There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks compared to the devices and / or networks shown in FIG. 1 . Furthermore, the present invention may be implemented within a single device. Figure 10 two or more devices shown in, or Figure 10 A single device shown in FIG. 1000 may be implemented as multiple distributed devices. Additionally or alternatively, one or more devices of environment 1000 may perform one or more functions described as being performed by another set of devices of environment 1000.

[0148] As described above, the liveness detection verification system and method according to the embodiment can verify audiovisual liveness detection by utilizing a random phrase and a second consistency check (voice recognition), thereby more reliably and effectively ensuring biometric authentication and preventing presentation attacks. For example, by using a random phrase and combining it with voice recognition to confirm that the phrase was spoken, the user is required to speak in real time, thereby preventing the use of previously recorded or tampered video.

[0149] The exemplary embodiments may be implemented as computer-readable code on a computer-readable recording medium, but are not limited thereto. A computer-readable recording medium is any data storage device that can store data that can be subsequently read by a computer system. Examples of computer-readable recording media include read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices. The computer-readable recording medium may also be distributed to networked computer systems so that the computer-readable code is stored and executed in a distributed manner. In addition, the exemplary embodiments may be written as a computer program transmitted via a computer-readable transmission medium such as a carrier wave, and received and implemented in a general-purpose or special-purpose digital computer that executes the program.

[0150] although Figures 2 to 7 Example blocks of the methods described with reference thereto are shown, but in some implementations, the corresponding methods may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted. Additionally or alternatively, two or more of the blocks of the corresponding methods may be executed in parallel.

[0151] supply Figures 8 to 10 The number and arrangement of components shown in the figures are provided as examples. In practice, the liveness detection verification system 800, the data preprocessor 900, the environment 1000, the user device 1100, and the platform 1200 may include additional components, fewer components, different components, or components arranged differently than those shown in the figures. Additionally or alternatively, a group of components (e.g., one or more components) of various devices may perform one or more functions described as being performed by another group of components.

[0152] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired through practice of the implementations.

[0153] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0154] While the above embodiments refer to phrases, it will be appreciated that one or more other embodiments may be applicable to words, for example, obtaining a video of a user speaking a word and determining whether the user's spoken word corresponds to a predetermined word.

[0155] It will be clear that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, with the understanding that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0156] Even if particular combinations of features are set forth in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, some of these features may be combined in ways not specifically set forth in the claims and / or disclosed in this specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0157] Although one or more exemplary embodiments have been described above with reference to the accompanying drawings, it will be understood that those skilled in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope as defined, at least in part, by the following claims.

Claims

1. A method for verifying liveness detection by at least one processing device, the method comprising the following steps: obtaining a video of a user speaking a phrase in response to a question or randomly generated phrase presented to the user; Processing the video to obtain video data and audio data of the user; inputting the obtained video data and the obtained audio data into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; Inputting the obtained video data into a first machine learning neural network, and inputting the obtained audio data into a second machine learning neural network; obtaining, as the first determination, a contrastive loss value corresponding to a distance between a feature from an output of the first machine learning neural network and a feature from an output of the second machine learning neural network; inputting a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether the phrase spoken by the user corresponds to the predetermined phrase; as well as It is determined whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase to verify liveness detection of the user.

2. The method according to claim 1, further comprising the steps of: randomly generating said randomly generated phrases presented to said user from a pre-stored vocabulary, wherein obtaining the video comprises obtaining a video of the user speaking a phrase in response to being presented with the randomly generated phrase, wherein the predetermined phrase is the randomly generated phrase, wherein the pre-stored vocabulary includes predetermined terms for each of a predetermined number of categories, and The randomly generated phrase includes a combination of one term randomly selected from each of a predetermined number of categories.

3. The method according to claim 1, wherein Obtaining the video includes: sending one of the question and the randomly generated phrase to a user device; and A video of the user speaking a phrase in response to one of the question and the randomly generated phrase presented to the user via the user device is received from the user device.

4. The method according to claim 1, wherein Processing the video includes the following steps: Obtaining an audio stream and a video stream from the video; obtaining a first predetermined number of batches of a second predetermined number of frames of video data, the frames of video data including a cropped mouth region generated from the video stream; and The second predetermined number of frames of audio data of the first predetermined number of batches are obtained, the frames of audio data comprising mel-scale spectrograms generated from the audio stream.

5. The method according to claim 1, wherein Inputting into the second determination model comprises the following steps: inputting the first input and the second input into a third machine learning neural network, the first input being features extracted from an output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; as well as The connectionist temporal classification (CTC) loss value is obtained as the second determination.

6. The method according to claim 5, wherein: Determining whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user spoke the predetermined phrase comprises the following steps: determining, based on the contrast loss value being less than a predetermined threshold, that the user did not correctly speak the predetermined phrase; determining that the user did not correctly say the predetermined phrase based on that the predetermined phrase does not correspond to the predicted label obtained from the CTC loss value; and Based on the fact that the contrast loss value is greater than the predetermined threshold and the predicted label corresponds to the predetermined phrase, it is determined that the user has correctly spoken the predetermined phrase.

7. A liveness detection and verification system, comprising: a memory configured to store instructions; as well as at least one processor configured to execute the instructions to: obtaining a video of a user speaking a phrase in response to a question or randomly generated phrase presented to the user; Processing the video to obtain video data and audio data of the user; inputting the obtained video data and the obtained audio data into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; Inputting the obtained video data into a first machine learning neural network, and inputting the obtained audio data into a second machine learning neural network; obtaining, as the first determination, a contrastive loss value corresponding to a distance between a feature from an output of the first machine learning neural network and a feature from an output of the second machine learning neural network; inputting a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether a phrase spoken by the user corresponds to the predetermined phrase; as well as It is determined whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase to verify liveness detection of the user.

8. The liveness detection and verification system according to claim 7, wherein: The at least one processor is further configured to execute instructions to randomly generate the randomly generated phrase presented to the user from a pre-stored vocabulary set; The obtained video is a video of the user speaking the phrase in response to being presented with the randomly generated phrase; The predetermined phrase is the randomly generated phrase; The pre-stored vocabulary includes predetermined terms for each of a predetermined number of categories; and The randomly generated phrase includes a combination of one term randomly selected from each of the predetermined number of categories.

9. The liveness detection and verification system according to claim 7, further comprising: Communication interface, The at least one processor is further configured to execute instructions to: controlling the communication interface to transmit one of the question and the randomly generated phrase to a user device; and A video of the user speaking a phrase in response to one of the question and the randomly generated phrase presented to the user via the user device is received from the user device via the communication interface.

10. The liveness detection and verification system according to claim 7, wherein: The at least one processor is further configured to execute the instructions to: Obtaining an audio stream and a video stream from the video; obtaining a second predetermined number of frames of video data from a first predetermined number of batches, the frames of video data including a cropped mouth region generated from the video stream; as well as The second predetermined number of frames of audio data of the first predetermined number of batches are obtained, the frames of audio data comprising mel-scale spectrograms generated from the audio stream.

11. The liveness detection and verification system according to claim 7, wherein: The at least one processor is further configured to execute the instructions to: inputting the first input and the second input into a third machine learning neural network, the first input being features extracted from an output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; as well as The connectionist temporal classification (CTC) loss value is obtained as the second determination.

12. The liveness detection and verification system according to claim 11, wherein: The at least one processor is further configured to execute instructions to: Determining that the liveness detection fails verification based on the contrast loss value being less than a predetermined threshold; determining that the liveness detection fails verification based on that the predetermined phrase does not correspond to the predicted label obtained from the CTC loss value; as well as Based on the fact that the contrast loss value is greater than the predetermined threshold and the predicted label corresponds to the predetermined phrase, it is determined that the user has correctly spoken the predetermined phrase.

13. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtaining a video of a user speaking a phrase in response to a question or randomly generated phrase presented to the user; Processing the video to obtain video data and audio data of the user; inputting the obtained video data and the obtained audio data into a first determination model to obtain a first determination indicating whether the user's mouth movement is synchronized with the audio data; Inputting the obtained video data into a first machine learning neural network, and inputting the obtained audio data into a second machine learning neural network; obtaining, as the first determination, a contrastive loss value corresponding to a distance between a feature from an output of the first machine learning neural network and a feature from an output of the second machine learning neural network; inputting a first input corresponding to the audio data and a second input corresponding to a predetermined phrase into a second determination model to obtain a second determination indicating whether the phrase spoken by the user corresponds to the predetermined phrase; as well as It is determined whether the first determination indicates that the mouth movement is synchronized with the audio data and whether the second determination indicates that the user speaks the predetermined phrase to verify liveness detection of the user.

14. The non-transitory computer-readable medium of claim 13, wherein: the instructions, when executed by the one or more processors, further cause the one or more processors to randomly generate the randomly generated phrase for presentation to the user from a pre-stored vocabulary; The obtained video is a video of the user speaking a phrase in response to being presented with the randomly generated phrase; The predetermined phrase is the randomly generated phrase; The pre-stored vocabulary includes predetermined terms for each of a predetermined number of categories; and The randomly generated phrase includes a combination of one term randomly selected from each of the predetermined number of categories.

15. The non-transitory computer readable medium of claim 13, wherein: The instructions, when executed by the one or more processors, further cause the one or more processors to: Obtaining an audio stream and a video stream from the video; obtaining a second predetermined number of frames of video data from a first predetermined number of batches, the frames of video data including a cropped mouth region generated from the video stream; as well as The second predetermined number of frames of audio data of the first predetermined number of batches are obtained, the frames of audio data comprising mel-scale spectrograms generated from the audio stream.

16. The non-transitory computer readable medium of claim 13, wherein: The instructions, when executed by the one or more processors, further cause the one or more processors to: inputting the first input and the second input into a third machine learning neural network, the first input being features extracted from an output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; as well as The connectionist temporal classification (CTC) loss value is obtained as the second determination.

17. The non-transitory computer readable medium of claim 16, wherein: The instructions, when executed by the one or more processors, further cause the one or more processors to: Determining that the liveness detection fails verification based on the contrast loss value being less than a predetermined threshold; determining that the liveness detection fails verification based on that the predetermined phrase does not correspond to the predicted label obtained from the CTC loss value; as well as Based on the fact that the contrast loss value is greater than the predetermined threshold and the predicted label corresponds to the predetermined phrase, it is determined that the living body detection has passed the verification.

18. A method for training a liveness detection and verification system, the method comprising the following steps: Get a video of the user saying the phrase; Processing the video to obtain video data and audio data of the user; training a first machine learning model for detecting audiovisual similarity and a second machine learning model for speech recognition, wherein the first machine learning model is trained based on the video data and the audio data, and the second machine learning model is trained based on a first input corresponding to the audio data and a predetermined phrase, the predetermined phrase corresponding to one of a randomly generated phrase and an answer to a predetermined question; Inputting the obtained video data into a first machine learning neural network, and inputting the obtained audio data into a second machine learning neural network; as well as A contrastive loss value corresponding to a distance between a feature from the output of the first machine learning neural network and a feature from the output of the second machine learning neural network is obtained as a first determination.

19. The method according to claim 18, wherein The training includes simultaneously training the first machine learning model and the second machine learning model.

20. The method according to claim 18, wherein The first input corresponding to the audio data includes features extracted from a fully connected layer of the first machine learning model.

21. The method according to claim 20, wherein The training includes: inputting the video data and the audio data into the first machine learning model to obtain a first determination indicating whether the user's mouth movements are synchronized with the audio data; The first input corresponding to the audio data and the second input corresponding to the predetermined phrase are input to the second machine learning model to obtain a second determination indicating whether the phrase spoken by the user is the predetermined phrase.

22. The method according to claim 21, wherein Inputting the first input and the second input to the second machine learning model comprises the following steps: inputting the first input and the second input into a third machine learning neural network, the first input being features extracted from an output of the second machine learning neural network, and the second input being a character alignment or a word alignment of the predetermined phrase; and The connectionist temporal classification (CTC) loss value is obtained as the second determination.

23. The method according to claim 18, wherein The training includes: inputting the video data and the audio data into the first machine learning model to obtain a first determination indicating whether the user's mouth movements are synchronized with the audio data; inputting the first input corresponding to the audio data and the second input corresponding to the predetermined phrase into the second machine learning model to obtain a second determination indicating whether the phrase spoken by the user is the predetermined phrase; aggregating the first determination and the second determination; and The first machine learning model and the second machine learning model are trained based on the aggregated determination.

Images