Risk identification method and device based on multiple modes, equipment and storage medium

Abstract

The invention discloses a risk identification method and device based on multiple modes, equipment and a storage medium, and the method comprises the steps: splitting a video stream and an audio stream based on obtained video data information, respectively extracting a corresponding lip feature sequence and an audio spectrum feature sequence, calculating a sound-lip comparison synchronization rate based on the two sequences, and obtaining a risk identification result; the audio content existing in the video is ensured to be generated by the corresponding user, and further, the risk value of the video is obtained by combining the text content converted by the two sequences and calculating the character-level similarity value between the two text content, so that the situation that the lip movement of the user is correct but the speaking content is different can be avoided, and the user experience is improved. The technical problem that at present, user identity risk identification is carried out only by adopting a face identification technology, and accurate early warning and substituted operation cannot be carried out is solved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a multimodal risk identification method, device, equipment and storage medium. Background Art

[0002] The online transformation of financial audit services is a major trend in the current financial industry, especially in the context of the novel coronavirus infection. Onlineization not only improves the convenience of financial services, but also places higher demands on the technical reserves and rapid response capabilities of financial institutions. In this process, how to effectively identify customers and prevent fraud and credit risks has become a key challenge facing financial institutions.

[0003] Take credit card approval as an example. The traditional approval process has many risk points. For example, account managers may perform operations on behalf of customers in pursuit of speed. This may not only lead to approval errors, but also cause a credit card crisis. It also does not meet the requirements of regulators. In order to improve risk control capabilities, some financial institutions have begun to explore the application of new technologies.

[0004] Currently, facial recognition technology is mostly used for supervision, but current facial recognition technology cannot accurately warn of the risks of proxy operations. Therefore, a more accurate information identification and risk monitoring method is urgently needed. Summary of the invention

[0005] The present application provides a multimodal risk identification method, device, equipment and storage medium, which solves the technical problem that only face recognition technology is currently used to identify user identity risks, and there is an inability to accurately warn of alternative operations.

[0006] In view of this, the first aspect of the present application provides a multi-modal risk identification method, the method comprising:

[0007] Acquire video data information containing at least one user;

[0008] Separating and processing the video data information to obtain audio stream data and video stream data respectively;

[0009] Extracting a lip feature sequence of a lip feature area from the video stream data based on a preset facial feature regression recognition model;

[0010] Extracting an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model;

[0011] According to the lip feature sequence and the audio spectrum feature sequence, the lip contrast synchronization rate is calculated by a pre-trained lip recognition model;

[0012] If the lip-matching synchronization rate is lower than a preset synchronization rate threshold, the video data information is marked as risky.

[0013] Optionally, it also includes:

[0014] If the lip-to-voice comparison synchronization rate is not lower than the preset synchronization threshold, converting the lip feature sequence into a text sequence to obtain a corresponding first text content;

[0015] Convert the audio stream data into text to obtain corresponding second text content;

[0016] Calculating a character-level similarity value between the first text content and the second text content;

[0017] Calculating a risk value of the video data information according to the lip-contrast synchronization rate and the character-level similarity value;

[0018] If the risk value is greater than a preset risk threshold, the video data information is marked as having risk, otherwise it is not marked.

[0019] Optionally, the lip contrast synchronization rate calculated by a pre-trained lip recognition model according to the lip feature sequence and the audio spectrum feature sequence is specifically:

[0020] The lip feature sequence and the audio spectrum feature sequence are respectively associated and identified according to the smallest identifiable segment by using synchronization tags;

[0021] The minimum segment segments after the association mark are sequentially input into a pre-trained lip recognition model, and the lip recognition model outputs a probability value for indicating synchronization confidence based on the synchronization label;

[0022] Calculate the number of synchronized frames and the total number of frames of the lip feature sequence and the audio spectrum feature sequence according to the minimum segment segment;

[0023] The lip-contrast synchronization rate of the lip feature sequence and the audio spectrum feature sequence is calculated according to the ratio of the synchronization frame number to the total frame number in combination with the probability value.

[0024] Optionally, converting the lip feature sequence into a text sequence to obtain the corresponding first text content specifically includes:

[0025] Using the lip feature sequence as an input of a preset sequence-to-sequence model, obtaining a numerical text sequence containing a plurality of first timestamps output by the preset sequence-to-sequence model, wherein the preset sequence-to-sequence model is specifically a Transformer model, an Encoder-Decoder architecture model, an RNN model, an LSTM model, or a GRU model;

[0026] After converting the numerical text sequence into words one by one, the words are concatenated to generate first text content, wherein the first text content includes the first timestamp.

[0027] Optionally, the converting the audio stream data into text to obtain the corresponding second text content specifically includes:

[0028] Using the audio stream data as input of a preset ASR speech recognition model, obtaining second text content output by the preset ASR speech recognition model and including a plurality of second timestamps;

[0029] Based on vocabulary search and grammar check, correction processing is performed on the second text content.

[0030] Optionally, the calculating of the character-level similarity value between the first text content and the second text content is specifically:

[0031] Create a distance matrix according to the first length of the first text content and the second length of the second text content, wherein the rows of the distance matrix are all the first substrings of the first text content, the columns are all the second substrings of the second text content, and the elements of the i-th row and j-th column in the distance matrix are the edit distances between the first i characters of the first text content and the first j characters of the second text content;

[0032] Performing insertion, deletion and replacement operations on each of the elements in the distance matrix respectively, and taking the minimum value of the insertion, deletion and replacement operations calculated as the value of the element;

[0033] The character-level similarity value between the first text content and the second text content is calculated based on the ratio of the value of the last element in the distance matrix to the maximum value of the first length and the second length.

[0034] Optionally, if the risk value is greater than a preset risk threshold, marking the video data information as risky further includes:

[0035] Performing user behavior analysis on the video data information;

[0036] Dynamically adjust question difficulty, question type, or prompt question re-answers based on user behavior analysis results.

[0037] A second aspect of the present application provides a multi-modal risk identification device, the device comprising:

[0038] An acquisition unit, configured to acquire video data information containing at least one user;

[0039] A video processing unit, used for separating and processing the video data information to obtain audio stream data and video stream data respectively;

[0040] A first feature sequence extraction unit, configured to extract a lip feature sequence of a lip feature area from the video stream data based on a preset facial feature regression recognition model;

[0041] A second feature sequence extraction unit, configured to extract an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model;

[0042] A synchronization rate calculation unit, configured to calculate the lip contrast synchronization rate through a pre-trained lip recognition model according to the lip feature sequence and the audio spectrum feature sequence;

[0043] The first processing unit is configured to mark the video data information as risky if the lip-matching synchronization rate is lower than a preset synchronization rate threshold.

[0044] Optionally, it also includes:

[0045] A second processing unit is configured to convert the lip feature sequence into a text sequence to obtain a corresponding first text content if the lip-to-voice comparison synchronization rate is not lower than the preset synchronization threshold;

[0046] A third processing unit, configured to convert the audio stream data into text to obtain corresponding second text content;

[0047] A similarity calculation unit, used to calculate a character-level similarity value between the first text content and the second text content;

[0048] A risk calculation unit, configured to calculate a risk value of the video data information according to the lip-contrast synchronization rate and the character-level similarity value;

[0049] The fourth processing unit is configured to mark the video data information as risky if the risk value is greater than a preset risk threshold, and not mark it otherwise.

[0050] Optionally, the synchronization rate calculation unit is specifically used for:

[0051] The lip feature sequence and the audio spectrum feature sequence are respectively associated and identified according to the smallest identifiable segment by using synchronization tags;

[0052] The minimum segment segments after the association mark are sequentially input into a pre-trained lip recognition model, and the lip recognition model outputs a probability value for indicating synchronization confidence based on the synchronization label;

[0053] Calculate the number of synchronized frames and the total number of frames of the lip feature sequence and the audio spectrum feature sequence according to the minimum segment segment;

[0054] The lip-contrast synchronization rate of the lip feature sequence and the audio spectrum feature sequence is calculated according to the ratio of the synchronization frame number to the total frame number in combination with the probability value.

[0055] Optionally, the second processing unit is specifically configured to:

[0056] Using the lip feature sequence as an input of a preset sequence-to-sequence model, obtaining a numerical text sequence containing a plurality of first timestamps output by the preset sequence-to-sequence model, wherein the preset sequence-to-sequence model is specifically a Transformer model, an Encoder-Decoder architecture model, an RNN model, an LSTM model, or a GRU model;

[0057] After converting the numerical text sequence into words one by one, the words are concatenated to generate first text content, wherein the first text content includes the first timestamp.

[0058] Optionally, the third processing unit specifically includes:

[0059] Using the audio stream data as input of a preset ASR speech recognition model, obtaining second text content output by the preset ASR speech recognition model and including a plurality of second timestamps;

[0060] Based on vocabulary search and grammar check, correction processing is performed on the second text content.

[0061] Optionally, the similarity calculation unit is specifically used to:

[0062] Create a distance matrix according to the first length of the first text content and the second length of the second text content, wherein the rows of the distance matrix are all the first substrings of the first text content, the columns are all the second substrings of the second text content, and the elements of the i-th row and j-th column in the distance matrix are the edit distances between the first i characters of the first text content and the first j characters of the second text content;

[0063] Performing insertion, deletion and replacement operations on each of the elements in the distance matrix respectively, and taking the minimum value of the insertion, deletion and replacement operations calculated as the value of the element;

[0064] The character-level similarity value between the first text content and the second text content is calculated based on the ratio of the value of the last element in the distance matrix to the maximum value of the first length and the second length.

[0065] Optionally, a behavior analysis response unit is further included, for:

[0066] Performing user behavior analysis on the video data information;

[0067] Dynamically adjust question difficulty, question type, or prompt question re-answers based on user behavior analysis results.

[0068] A third aspect of the present application provides a multi-modal risk identification device, the device comprising a processor and a memory:

[0069] The memory is used to store program code and transmit the program code to the processor;

[0070] The processor is used to execute the steps of the multi-modal risk identification method as described in the first aspect according to the instructions in the program code.

[0071] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0072] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0073] In the present application, a multimodal risk identification method, apparatus, device and storage medium are provided. Based on the acquired video data information, the video stream and audio stream are split, and the corresponding lip feature sequence and audio spectrum feature sequence are extracted respectively. The lip-to-sound comparison synchronization rate is calculated based on the two sequences to ensure that the audio content in the video is generated by the corresponding user. Furthermore, the risk value of the video is obtained by calculating the character-level similarity value between the two text contents in combination with the text content converted from the two sequences. This can avoid the situation where the user's lip movements are correct but the speech content is different, and solves the technical problem that only face recognition technology is currently used for user identity risk identification, and there is no accurate warning of alternative operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A method flow chart of a multi-modal risk identification method in an embodiment of the present application;

[0075] Figure 2 Schematic diagram of the structure of a multi-modal risk identification device in an embodiment of the present application;

[0076] Figure 3 Schematic diagram of the structure of a multi-modal risk identification device in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] The present application designs a multimodal risk identification method, device, equipment and storage medium, which solves the technical problem that only face recognition technology is currently used for user identity risk identification, and there is no accurate warning and substitution operation.

[0079] For easier understanding, see Figure 1 , Figure 1 is a method flow chart of a multi-modal risk identification method in an embodiment of the present application, such as Figure 1 As shown, specifically:

[0080] S101, obtaining video data information containing at least one user;

[0081] It should be noted that the system needs to obtain video data containing at least one user, which can be achieved by capturing with a camera or by a user uploading a video file.

[0082] Example: In a bank's remote customer service system, customers record videos through mobile phone applications, and the system receives and stores these video data in the background.

[0083] S102, separating and processing the video data information to obtain audio stream data and video stream data respectively;

[0084] It should be noted that the video data is separated into two parts, an audio stream and a video stream, so as to be processed separately.

[0085] Embodiment: Use an audio / video separation tool to separate the audio and video tracks in a video file and save them as independent files.

[0086] S103, extracting a lip feature sequence of a lip feature area from the video stream data based on a preset facial feature regression recognition model;

[0087] It should be noted that the facial recognition technology is used to locate and extract the user's lip features from the video stream to obtain a lip feature sequence.

[0088] Embodiment: A facial feature detector is used to identify and track lip movements in a video. Further, an AAM (Active Appearance Models) statistical model is used to capture changes in lip shape or a DNN (Deep Neural Networks) deep learning model, such as a convolutional neural network (CNN), is used to extract a high-level lip feature sequence.

[0089] S104, extracting an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model;

[0090] It should be noted that key spectral features are extracted from the audio stream, and these features can represent the characteristics of the speech signal.

[0091] Common methods for implementing this step include:

[0092] 1. Preprocessing. Preprocessing operations may include denoising and normalization. Specifically:

[0093] Denoising: Use filters to remove background noise and improve signal quality.

[0094] Normalize: Adjust the volume to an appropriate level to reduce the dynamic range of subsequent processing.

[0095] 2. Window function: Use window function to segment the audio stream data. Specifically:

[0096] The continuous audio signal is divided into short-time frames, and a window function such as a Hamming window or a Hanning window is usually used to reduce the discontinuity at the frame boundary.

[0097] 3. Short-time Fourier transform (STFT):

[0098] A Fast Fourier Transform (FFT) is applied to the data within each window to convert the time domain signal into a frequency domain signal.

[0099] STFT can provide a time-frequency representation of the signal, showing the frequency components at different time points.

[0100] 4. Spectral feature extraction:

[0101] Calculate Power Spectral Density (PSD): Represent the spectrum of the signal by calculating the power of each frequency component.

[0102] Mel-Frequency Cepstral Coefficients (MFCCs): A widely used spectral feature that emphasizes the characteristic that the human ear is more sensitive to certain frequencies through a Mel filter bank, and then calculates the discrete cosine transform (DCT) of the logarithmic power spectrum.

[0103] 5. Feature sequence construction:

[0104] Features extracted from methods such as STFT or MFCC can form feature sequences for subsequent pattern recognition or machine learning tasks.

[0105] S105, calculating the lip contrast synchronization rate through a pre-trained lip recognition model according to the lip feature sequence and the audio spectrum feature sequence;

[0106] It should be noted that the lip recognition model is used to combine the lip feature sequence and the audio spectrum feature sequence to calculate the lip contrast synchronization rate (G). The specific steps will be described in detail in the subsequent embodiments.

[0107] S106: If the lip-matching synchronization rate is lower than a preset synchronization rate threshold, the video data information is marked as risky.

[0108] It should be noted that by setting the synchronization threshold, when the lip synchronization rate is lower than the synchronization threshold, the system marks it as a potential risk and triggers manual review.

[0109] Further, based on step S106, the following steps may be performed:

[0110] S107: if the lip-to-voice comparison synchronization rate is not lower than a preset synchronization threshold, converting the lip feature sequence into a text sequence to obtain a corresponding first text content;

[0111] S108, converting the audio stream data into text to obtain corresponding second text content;

[0112] S109, calculating the character-level similarity value between the first text content and the second text content;

[0113] S110, calculating a risk value of the video data information according to the lip-contrast synchronization rate and the character-level similarity value;

[0114] S111. If the risk value is greater than a preset risk threshold, the video data information is marked as having risk, otherwise it is not marked.

[0115] It should be noted that if the lip synchronization rate is not lower than the threshold, the system converts the lip feature sequence into a text sequence and converts the audio stream data into text content.

[0116] To calculate the character-level similarity between two text contents, Levenshtein distance can be used as a measure.

[0117] The risk value of the video data is calculated by combining the lip synchronization rate and text similarity.

[0118] Perform user behavior analysis on videos marked as risky to determine if there is fraudulent behavior.

[0119] Furthermore, in the above embodiment, step S105 specifically includes:

[0120] S1051, associate and identify the lip feature sequence and the audio spectrum feature sequence according to the smallest identifiable segment by using synchronization tags;

[0121] It should be noted that in order to improve processing efficiency and accuracy, the continuous lip feature sequence and audio spectrum feature sequence are divided into smaller segments. These segments should be the smallest units that the model can process, for example, the length of a sentence or a phrase.

[0122] Example: In a 10-second video, if the average length of each sentence is 2 seconds, the video is divided into 5 segments. Each segment corresponds to a set of lip features and audio spectrum features.

[0123] S1052, inputting the minimum segment segments after the association mark into the pre-trained lip recognition model in order, and the lip recognition model outputs a probability value for indicating synchronization confidence based on the synchronization label;

[0124] It should be noted that in order to ensure that the lip movements in the video clips and the corresponding audio can be correctly matched, each clip will be marked with a synchronization tag. This tag is used to maintain the consistency of audio and video data in subsequent processing.

[0125] Embodiment: Use a unique identifier (such as a timestamp or sequence number) to mark each audio and video segment pair to ensure that they can be accurately re-associated during analysis.

[0126] The audio and video clips with synchronization labels are fed into a pre-trained lip recognition model, which can understand the association between lip movements and sounds and evaluate their synchronization.

[0127] Embodiment: A deep learning model is used, which is trained on a large amount of audio and video data to learn the correspondence between lip movements and sounds.

[0128] The lip recognition model outputs a probability value, which indicates the confidence level of synchronization between the lip movements and the sound in a given segment. This value is usually between 0 and 1, with higher values ​​indicating higher synchronization.

[0129] Example: The model may output a probability value of 0.9, indicating that for a particular segment, the model is 90% confident that the lip movement and sound are synchronized.

[0130] S1053, calculating the number of synchronized frames and the total number of frames of the lip feature sequence and the audio spectrum feature sequence according to the minimum segment segment;

[0131] It should be noted that for each segment, the number of frames judged by the model to be synchronized is calculated, as well as the total number of frames in the segment. This can be achieved by comparing the probability value output by the model with a threshold, and frames above the threshold are considered to be synchronized.

[0132] Example: If a 2-second clip contains 60 frames, and the model determines that 54 of them are synchronized, then the number of synchronized frames is 54 and the total number of frames is 60.

[0133] S1054. Calculate the lip-contrast synchronization rate of the lip feature sequence and the audio spectrum feature sequence according to the ratio of the number of synchronized frames to the total number of frames in combination with the probability value.

[0134] It should be noted that the lip-sync ratio is calculated by dividing the number of synchronized frames by the total number of frames and then multiplying it by the synchronization confidence probability value output by the model. This ratio provides a comprehensive measure of the degree of lip sync of the entire clip.

[0135] Example: If the number of synchronized frames is 54, the total number of frames is 60, and the synchronization confidence probability value output by the model is 0.9, then the lip contrast synchronization rate is (54 / 60)*0.9=0.81.

[0136] Suppose an online banking system needs to verify whether the user's lip movements and voice in the video are synchronized to prevent fraud. The system performs the following steps:

[0137] Video data acquisition: Users are asked to record a video of reading a specific sentence when applying for a loan.

[0138] Data preprocessing: The system automatically separates video and audio data, and performs denoising and normalization.

[0139] Feature extraction: Use the preset facial recognition model to extract lip features, and use the audio processing tool to extract audio spectrum features.

[0140] Lip recognition model: The extracted features are fed into a pre-trained deep learning model that is able to recognize the synchronization between lip movements and sounds.

[0141] Risk Assessment: If the lip-sync rate calculated by the model is lower than a threshold set by the system (e.g. 85%), the system marks the video as high risk and may trigger additional security review steps.

[0142] In this way, banks can effectively leverage multimodal data to improve their risk management capabilities while protecting the security and privacy of their users.

[0143] Furthermore, in the above embodiment, converting the lip feature sequence into a text sequence in step S107 to obtain the corresponding first text content specifically includes:

[0144] S1071. Using the lip feature sequence as an input of a preset sequence-to-sequence model, obtaining a numerical text sequence including a plurality of first timestamps output by the preset sequence-to-sequence model, wherein the preset sequence-to-sequence model is specifically a Transformer model, an Encoder-Decoder architecture model, an RNN model, an LSTM model, or a GRU model;

[0145] It should be noted that the lip feature sequence extracted from the video is input into a preset sequence-to-sequence model, which can convert visual features into text.

[0146] Example: Use a Transformer-based model that is trained to understand how lip movements correspond to specific words or syllables.

[0147] The model outputs a series of numeric text sequences, which contain the first timestamp indicating the start time of each word or syllable.

[0148] Example: The model outputs a series of words and corresponding timestamps, such as “[“hello”, 1.2], [“world”, 2.5]”.

[0149] The preset sequence-to-sequence model is specifically a Transformer model, an Encoder-Decoder architecture model, an RNN model, an LSTM model, or a GRU model. The differences and advantages of each model are as follows:

[0150] 1. Transformer Model

[0151] Difference: The Transformer model is completely based on the attention mechanism, does not rely on the recurrent neural network structure, and can process sequence data in parallel, thereby greatly improving training efficiency.

[0152] advantage:

[0153] Parallel processing: Since it is independent of the order of the sequence, the entire sequence can be processed in parallel, accelerating the training and inference process.

[0154] Long-distance dependencies: Through the self-attention mechanism, Transformer can effectively capture long-distance dependencies in the sequence.

[0155] Flexibility: The model structure is flexible, easy to adjust and expand, and suitable for various NLP tasks.

[0156] 2. Encoder-Decoder Architecture Model

[0157] Difference: The Encoder-Decoder model usually consists of two parts: Encoder and Decoder. The encoder processes the input sequence and generates a context vector, and the decoder generates an output sequence based on this context vector.

[0158] advantage:

[0159] Modularity: The structures of the encoder and decoder can be designed and optimized independently, which improves the flexibility of the model.

[0160] Wide applicability: This architecture is widely used in various sequence-to-sequence tasks such as machine translation, text summarization, and question-answering systems.

[0161] Effective context processing: Through the encoder-decoder structure, the model is able to better process the context information of the input sequence.

[0162] 3.RNN model (recurrent neural network)

[0163] Difference: The RNN model processes sequence data through a cyclic structure, and the output of each time step depends not only on the current input, but also on the output of the previous time step.

[0164] advantage:

[0165] Sequence processing ability: RNN is particularly suitable for processing sequence data because it can capture the dynamic characteristics in time series.

[0166] Simplicity: The model has a simple structure and is easy to understand and implement.

[0167] Applicability: Although it may be limited in processing long sequences, it works well when processing shorter sequence data.

[0168] 4. LSTM Model (Long Short-Term Memory Network)

[0169] Difference: LSTM is a variant of RNN, which solves the long-term dependency problem of standard RNN by introducing a gating mechanism (input gate, forget gate, output gate).

[0170] advantage:

[0171] Long sequence processing: LSTM can effectively process long sequence data and capture long-term dependencies.

[0172] Parameter sharing: Parameters are shared throughout the entire sequence processing process, reducing model complexity.

[0173] Stability: Compared with standard RNN, LSTM is more stable during training and is less prone to gradient vanishing or gradient exploding.

[0174] 5. GRU Model (Gated Recurrent Unit)

[0175] Difference: GRU is a simplified version of LSTM, which merges the three gates in LSTM into two (update gate and reset gate), reducing model parameters.

[0176] advantage:

[0177] Parameter efficiency: Due to the small number of parameters, GRU is usually faster to train and easier to optimize.

[0178] The performance is comparable to LSTM: In many tasks, the performance of GRU is comparable to LSTM, or even better in some cases.

[0179] Easy to train: Due to its simple structure, GRU is easier to converge during training.

[0180] The choice of model depends on the requirements of the specific task, the characteristics of the data, and the computing resources. For example, for tasks that require parallel processing and capturing long-distance dependencies, Transformer may be a better choice. For shorter sequence data, traditional RNN or GRU may be sufficient. In practical applications, you may need to try multiple models and determine which model performs best on a specific task through experiments.

[0181] S1072: Convert the numerical text sequence into words one by one, and then concatenate them to generate first text content, where the first text content includes a first timestamp.

[0182] It should be noted that the numerical text sequence is converted into natural text to form a complete first text content.

[0183] Example: The above output words and timestamps are concatenated into the text "Hello world", where "hello" appears at 1.2 seconds and "world" appears at 2.5 seconds.

[0184] Furthermore, in the above embodiment, step S108 specifically includes:

[0185] S1081, using the audio stream data as an input of a preset ASR speech recognition model, and obtaining second text content output by the preset ASR speech recognition model including a plurality of second timestamps;

[0186] It should be noted that the audio stream data is input into the automatic speech recognition (ASR) model, and the model converts the speech into text.

[0187] Example: Use Google's Cloud Speech-to-Text API to convert recorded audio into text.

[0188] S1082: Based on vocabulary search and grammar check, correct the second text content.

[0189] It should be noted that the text output by the ASR model is corrected in vocabulary and grammar to improve accuracy.

[0190] Embodiment: Use a language model and a grammar checking tool to correct the converted text, such as correcting "hellow" to "hello".

[0191] Furthermore, in the above embodiment, step S109 specifically includes:

[0192] S1091, creating a distance matrix according to the first length of the first text content and the second length of the second text content, wherein the rows of the distance matrix are all first substrings of the first text content, the columns are all second substrings of the second text content, and the elements of the i-th row and j-th column in the distance matrix are the edit distances between the first i characters of the first text content and the first j characters of the second text content;

[0193] It should be noted that an edit distance matrix is ​​created based on the lengths of the first text content and the second text content.

[0194] Example: If the first text is "hello world" and the second text is "hello there", create a matrix to compare the two sequences.

[0195] S1092, perform insertion, deletion and replacement operations on each element in the distance matrix, and use the minimum value of the insertion, deletion and replacement operations calculated as the value of the element;

[0196] It should be noted that each element in the calculation matrix represents the minimum number of editing operations required to convert one string into another string.

[0197] Example: For "world" and "there", calculate the minimum number of editing operations, such as replacing "w" with "t", deleting "r" and "l", and inserting "e".

[0198] S1093. Calculate the character-level similarity value between the first text content and the second text content according to the ratio of the value of the last element in the distance matrix to the maximum value of the first length and the second length.

[0199] It should be noted that the character-level similarity value is calculated using the edit distance and string length.

[0200] Example: If the edit distance is 3 and the longest string length is 5, the similarity value is 1-3 / 5=0.4.

[0201] Furthermore, in the above embodiment, after step S111, the following steps are further included:

[0202] S112, performing user behavior analysis on the video data information;

[0203] It should be noted that user behaviors in video data are analyzed to identify possible risk patterns.

[0204] Example: Analyze abnormal behaviors of users in videos, such as frequent blinking or unnatural head movements.

[0205] S113. Dynamically adjust the difficulty of questions, dynamically adjust the question type, or prompt to re-answer questions based on the results of the user behavior analysis.

[0206] It should be noted that the difficulty or type of questions is dynamically adjusted based on the results of user behavior analysis to improve the accuracy of verification.

[0207] Example: If it is detected that the user may be trying to cheat the system, the system will automatically increase the difficulty of the question or change the type of question.

[0208] In an online banking identity verification system, users need to verify their identity by recording a video. The system performs the following steps:

[0209] Application scenario: Online banking identity verification system

[0210] background

[0211] With the popularity of online financial services, banks need to ensure the authenticity of user identities to prevent fraud and protect user account security. In this scenario, an online bank uses the multimodal risk identification method described in the embodiment of the present application to enhance its identity authentication process.

[0212] 1. User registration / login process:

[0213] Users choose to use video authentication when registering or logging in.

[0214] The system prompts the user to record a video, in which the user needs to read aloud a set of numbers and letters randomly generated by the system.

[0215] 2. Video data acquisition:

[0216] Users record videos using their phone or computer's camera.

[0217] The user clicks the "Upload Video" button and the video file is sent to the bank's server.

[0218] 3. Video data preprocessing:

[0219] The server receives the video file and performs pre-processing, including video stabilization, resolution adjustment, and noise reduction.

[0220] 4. Separate audio and video streams:

[0221] The system automatically separates the audio and video streams in the video file.

[0222] The audio stream is used for speech recognition and the video stream is used for lip feature extraction.

[0223] 5. Lip feature extraction:

[0224] The video stream uses facial recognition technology to locate the user's lip area.

[0225] The motion features of the lip area are extracted to form a lip feature sequence.

[0226] 6. Audio spectrum feature extraction:

[0227] The audio stream is processed by a preset audio processing algorithm to extract the audio spectrum feature sequence.

[0228] 7. Lip synchronization rate calculation:

[0229] The lip feature sequence and the audio spectrum feature sequence are input into the pre-trained lip recognition model.

[0230] The model calculates the lip synchronization rate, and if it is lower than the preset threshold, it proceeds to the next step of risk assessment.

[0231] 8. Text content conversion:

[0232] If the lip synchronization rate is qualified, the system converts the lip feature sequence into text (first text content) through a sequence-to-sequence model.

[0233] At the same time, the audio stream is converted into text (second text content) using the ASR model.

[0234] 9. Similarity calculation:

[0235] Calculate the character-level similarity value between the first text content and the second text content.

[0236] If the similarity falls below a preset threshold, the system will flag the video as high risk.

[0237] 10. Risk Assessment:

[0238] If a video is flagged as high risk, the system will ask the user to take additional identity verification steps, such as answering security questions or performing a secondary verification.

[0239] 11. User behavior analysis:

[0240] For videos marked as high-risk, the system also analyzes the user's behavioral patterns, such as eye blinks, head movements, etc.

[0241] 12. Dynamic adjustment verification issues:

[0242] Based on the results of user behavior analysis, the system may dynamically adjust verification questions, increase question difficulty, or change question types.

[0243] result

[0244] If all verification steps are passed, the user's identity is confirmed and they can continue to register or log in.

[0245] If any step fails, the system denies access and notifies the user and the bank's security team.

[0246] Furthermore, corresponding safety measures can also be provided, such as:

[0247] All video data is encrypted during transmission and storage.

[0248] The user's facial and voice data is used only for identity verification purposes and is deleted immediately after verification.

[0249] Furthermore, the user experience can be improved through relevant prompts, such as:

[0250] The system provides clear instructions on how to record videos correctly.

[0251] The verification process is made as fast as possible to reduce user waiting time.

[0252] If a user encounters difficulty during the verification process, the system provides help options, such as re-recording the video or contacting customer service.

[0253] This example demonstrates how to improve the security and user experience of online banking services by combining lip reading technology and automatic speech recognition technology.

[0254] See also Figure 2, Figure 2 Schematic diagram of the structure of the multi-modal risk identification device in the embodiment of the present application. Figure 2 As shown, specifically:

[0255] An acquisition unit 201 is used to acquire video data information containing at least one user;

[0256] The video processing unit 202 is used to separate and process the video data information to obtain audio stream data and video stream data respectively;

[0257] A first feature sequence extraction unit 203 is used to extract a lip feature sequence of a lip feature area from the video stream data based on a preset facial feature regression recognition model;

[0258] A second feature sequence extraction unit 204 is used to extract an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model;

[0259] The synchronization rate calculation unit 205 is used to calculate the lip contrast synchronization rate through a pre-trained lip recognition model according to the lip feature sequence and the audio spectrum feature sequence;

[0260] The first processing unit 206 is configured to mark the video data information as risky if the lip-matching synchronization rate is lower than a preset synchronization rate threshold.

[0261] Another embodiment of the present invention provides a multi-modal risk identification device, such as Figure 3 As shown, the device 10 includes:

[0262] One or more processors 110 and memory 120, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0263] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0264] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for constructing a multilingual phoneme representation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for constructing a multilingual phoneme representation model in the above method embodiment.

[0265] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0266] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0267] Acquire video data information containing at least one user;

[0268] Separate and process the video data information to obtain audio stream data and video stream data respectively;

[0269] Extracting a lip feature sequence of a lip feature area from video stream data based on a preset facial feature regression recognition model;

[0270] Extracting an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model;

[0271] According to the lip feature sequence and the audio spectrum feature sequence, the lip contrast synchronization rate is calculated by using the pre-trained lip recognition model;

[0272] If the lip-matching synchronization rate is lower than a preset synchronization rate threshold, the video data information is marked as risky.

[0273] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, any one of the multi-modal risk identification methods described in the above embodiments is implemented.

[0274] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0275] In an embodiment of the present application, a multimodal risk identification method, apparatus, device and storage medium are provided. Based on the acquired video data information, the video stream and the audio stream are split, and the corresponding lip feature sequence and the audio spectrum feature sequence are extracted respectively. The lip-to-sound comparison synchronization rate is calculated based on the two sequences to ensure that the audio content in the video is generated by the corresponding user. Furthermore, the text content converted from the two sequences is combined, and the character-level similarity value between the two text contents is calculated to obtain the risk value of the video. This can avoid the situation where the user's lip movements are correct but the speech content is different, and solves the technical problem that only face recognition technology is currently used for user identity risk identification, and there is no accurate warning of alternative operations.

[0276] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0277] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0278] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0279] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0280] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0282] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0283] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0284] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A multimodal risk identification method, characterized in that: include: Acquire video data information containing at least one user; Separating and processing the video data information to obtain audio stream data and video stream data respectively; Extracting a lip feature sequence of a lip feature area from the video stream data based on a preset facial feature regression recognition model; Extracting an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model; According to the lip feature sequence and the audio spectrum feature sequence, the lip contrast synchronization rate is calculated by a pre-trained lip recognition model; If the lip-matching synchronization rate is lower than a preset synchronization rate threshold, the video data information is marked as risky.

2. The multimodal risk identification method according to claim 1, characterized in that: Also includes: If the lip-to-voice comparison synchronization rate is not lower than the preset synchronization threshold, converting the lip feature sequence into a text sequence to obtain a corresponding first text content; Convert the audio stream data into text to obtain corresponding second text content; Calculating a character-level similarity value between the first text content and the second text content; Calculating a risk value of the video data information according to the lip-contrast synchronization rate and the character-level similarity value; If the risk value is greater than a preset risk threshold, the video data information is marked as having risk, otherwise it is not marked.

3. The multimodal risk identification method according to claim 1, characterized in that: The lip contrast synchronization rate calculated by the pre-trained lip recognition model according to the lip feature sequence and the audio spectrum feature sequence is specifically: The lip feature sequence and the audio spectrum feature sequence are respectively associated and identified according to the smallest identifiable segment by using synchronization tags; The minimum segment segments after the association mark are sequentially input into a pre-trained lip recognition model, and the lip recognition model outputs a probability value for indicating synchronization confidence based on the synchronization label; Calculate the number of synchronized frames and the total number of frames of the lip feature sequence and the audio spectrum feature sequence according to the minimum segment segment; The lip-contrast synchronization rate of the lip feature sequence and the audio spectrum feature sequence is calculated according to the ratio of the synchronization frame number to the total frame number in combination with the probability value.

4. The multimodal risk identification method according to claim 2, characterized in that: The step of converting the lip feature sequence into a text sequence to obtain the corresponding first text content specifically includes: Using the lip feature sequence as an input of a preset sequence-to-sequence model, obtaining a numerical text sequence containing a plurality of first timestamps output by the preset sequence-to-sequence model, wherein the preset sequence-to-sequence model is specifically a Transformer model, an Encoder-Decoder architecture model, an RNN model, an LSTM model, or a GRU model; After converting the numerical text sequence into words one by one, the words are concatenated to generate first text content, wherein the first text content includes the first timestamp.

5. The multimodal risk identification method according to claim 4, characterized in that: The converting the audio stream data into text to obtain the corresponding second text content specifically includes: Using the audio stream data as input of a preset ASR speech recognition model, obtaining second text content output by the preset ASR speech recognition model and including a plurality of second timestamps; Based on vocabulary search and grammar check, correction processing is performed on the second text content.

6. The multimodal risk identification method according to claim 2, characterized in that: The specific step of calculating the character-level similarity value between the first text content and the second text content is: Create a distance matrix according to the first length of the first text content and the second length of the second text content, wherein the rows of the distance matrix are all the first substrings of the first text content, the columns are all the second substrings of the second text content, and the elements of the i-th row and j-th column in the distance matrix are the edit distances between the first i characters of the first text content and the first j characters of the second text content; Performing insertion, deletion and replacement operations on each of the elements in the distance matrix respectively, and taking the minimum value of the insertion, deletion and replacement operations calculated as the value of the element; The character-level similarity value between the first text content and the second text content is calculated based on the ratio of the value of the last element in the distance matrix to the maximum value of the first length and the second length.

7. The multimodal risk identification method according to claim 2, characterized in that: If the risk value is greater than a preset risk threshold, marking the video data information as having a risk further includes: Performing user behavior analysis on the video data information; Dynamically adjust question difficulty, question type, or prompt question re-answers based on user behavior analysis results.

8. A multi-modal risk identification device, characterized in that: include: An acquisition unit, configured to acquire video data information containing at least one user; A video processing unit, used for separating and processing the video data information to obtain audio stream data and video stream data respectively; A first feature sequence extraction unit, configured to extract a lip feature sequence of a lip feature area from the video stream data based on a preset facial feature regression recognition model; A second feature sequence extraction unit, configured to extract an audio spectrum feature sequence from the audio stream data based on a preset audio feature extraction model; A synchronization rate calculation unit, configured to calculate the lip contrast synchronization rate through a pre-trained lip recognition model according to the lip feature sequence and the audio spectrum feature sequence; The first processing unit is configured to mark the video data information as risky if the lip-matching synchronization rate is lower than a preset synchronization rate threshold.

9. A multi-modal risk identification device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the multimodal risk identification method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the multi-modal risk identification method according to any one of claims 1 to 7.

Images