Speech recognition authentication method and system based on multi-modal features and dynamic evaluation

Abstract

The application discloses a voice recognition authentication method and system based on multi-modal features and dynamic evaluation in the technical field of voice recognition authentication, which comprises the following steps: collecting original voice signals of a user by using a microphone, pre-processing the original voice signals to obtain pre-processed voice data; extracting feature data of the pre-processed voice data by using multi-dimensional feature hierarchical extraction technology; in an initial training stage based on a voiceprint feature model, injecting multi-source noise samples into an acoustic feature space of the voiceprint feature model to construct an anti-noise mixed voiceprint atlas. By integrating voiceprint, semantic, behavior features and environment adaptation mechanisms, the application can adjust the authentication threshold in real time according to dynamic risk evaluation, calculate a comprehensive risk value by using a risk scoring model, and enable different levels of authentication modes according to different risk scenarios, and can force the implementation of two-factor authentication for high-risk scenarios, so that the authentication security and reliability can be significantly enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition authentication, in particular to a speech recognition authentication method and system based on multi-modal features and dynamic evaluation. BACKGROUND

[0002] Speech recognition authentication is a technology that verifies identity by analyzing user speech features. In the prior art, a voiceprint recognition method is disclosed in Chinese patent application No. CN201910281641.1. The method receives an unknown user's to-be-identified speech signal, extracts the frame voiceprint features of each frame, calculates the posterior probability, classifies the frame voiceprint features based on the posterior probability, generates a to-be-identified model and a voiceprint recognition model, and determines the user identity through model similarity. Although this method can improve the accuracy of text-independent speech signal recognition, especially in the recognition efficiency of short text-independent speech signals, it mainly focuses on single acoustic feature processing and model similarity comparison, and does not fully consider comprehensive factors such as multi-modal feature fusion, dynamic risk evaluation, and attack resistance.

[0003] The existing voiceprint authentication has the following disadvantages:

[0004] (1) The existing voiceprint authentication is vulnerable, with an equal error rate (EER) of about 8.2% based on MFCC features in a pure laboratory environment, but the EER deteriorates to 15.7% in an actual multi-noise scene (signal-to-noise ratio ≤ 20 dB), and the false acceptance rate for synthetic speech attacks (such as WaveGAN generation) is as high as 23.5%.

[0005] (2) The fixed threshold of the existing static speech authentication strategy cannot adapt to dynamic risk scenarios such as remote login, resulting in an increase in false rejection rate or security vulnerabilities.

[0006] (3) The existing single-factor speech authentication method has low security, is easy to crack, has high privacy leakage risk, and is difficult to resist attacks, resulting in high security risks. SUMMARY

[0007] The present application aims to provide a speech recognition authentication method and system based on multi-modal features and dynamic evaluation, which integrates voiceprint, semantic, behavioral features, and environmental adaptation mechanisms, combines a quantitative risk evaluation model for dynamic risk evaluation, and adjusts the authentication threshold in real time. It can enable different levels of authentication according to different risk scenarios, and for high-risk scenarios, it enforces two-factor authentication, significantly enhancing authentication security and reliability.

[0008] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0009] In a first aspect, a speech recognition authentication method based on multi-modal features and dynamic evaluation is provided, comprising:

[0010] An original speech signal of a user is collected by a microphone, and the original speech signal is preprocessed to obtain preprocessed speech data;

[0011] Multi-dimensional feature hierarchical extraction technology is used to extract feature data of the preprocessed speech data, wherein the feature data includes feature alignment criteria, prosodic behavior, and semantics of the preprocessed speech data;

[0012] In an initial training phase based on a voiceprint feature model, multi-source noise samples are injected into an acoustic feature space of the voiceprint feature model, an anti-noise mixed voiceprint atlas is constructed, an Adv-GAN (adversarial generative adversarial network) discriminator is deployed, a deep fake speech generated by a WaveGAN (waveform generative adversarial network) in the discriminator is deconstructed based on a spectral time domain joint discrimination mechanism, and a dynamic adaptation mechanism of an acoustic environment is simultaneously established;

[0013] Based on a mobile terminal device of the user and a preset scoring rule, fingerprint information, geographic location, and historical behavior logs of the user are collected, a risk scoring model is used to identify a speech signal recognition risk of the user, and a speech recognition risk score is obtained;

[0014] Multi-modal feature fusion and model training are performed based on anti-noise MFCC acoustic features, prosodic behavior, and semantics;

[0015] When the authentication scenario is determined to be a low-risk scenario, only speech authentication is required; when the authentication scenario is determined to be a medium-risk scenario, dual-factor authentication is enabled for enhanced verification; and when the authentication scenario is determined to be a high-risk scenario, second-factor authentication is forcibly enabled, wherein the second-factor authentication is fingerprint or dynamic password authentication;

[0016] In a high-risk scenario, live detection, feature comparison, and second-factor verification tasks are assigned to a multi-core processor for parallel execution;

[0017] When authentication is passed, access is authorized and an encrypted token is generated, and login time, device information, risk score, and authentication results are stored in a secure log.

[0018] As a further scheme of the present application, an original speech signal of a user is collected by a microphone, and the original speech signal is preprocessed to obtain preprocessed speech data, comprising:

[0019] The original speech signal is read frame by frame and repeated or invalid segments are removed to obtain valid speech data, the valid speech data is stored as a standardized data set, and the standardized data set is converted into a WAV format file and output;

[0020] The RNNoise deep learning model is used to eliminate environmental noise in the WAV format file to obtain a de-noised audio, and non-speech audio segments of the de-noised audio are filtered to obtain a frequency domain speech signal, wherein the non-speech audio segments include mechanical noise and / or burst noise.

[0021] The frequency domain speech signal is segmented into a short-time frame speech signal by using a speech segmentation method, speech start and end points of the short-time frame speech signal are detected based on short-time energy and zero-crossing rate, and frame segmentation speech data is obtained, wherein the short-time frame speech signal is a speech signal of no more than 20 ms / frame.

[0022] Speech segments with insufficient speech length or too low signal-to-noise ratio in the frame segmentation speech data are removed to obtain pre-processed speech data, wherein the speech segment with insufficient speech length is a speech segment with a time length of less than 0.5 seconds, and the speech segment with too low signal-to-noise ratio is a speech segment with a signal-to-noise ratio of less than 15 dB.

[0023] As a further scheme of the present application: the multi-dimensional feature hierarchical extraction technology is used to extract feature data of the pre-processed speech data, including:

[0024] A feature alignment reference of the pre-processed speech data is obtained.

[0025] Prosodic behavior of the pre-processed speech data is analyzed.

[0026] The semantics of the pre-processed speech data are understood.

[0027] As a further scheme of the present application: the feature alignment reference is obtained, including:

[0028] An anti-noise MFCC including first-order and / or second-order dynamic difference and fundamental frequency trajectory fluctuation analysis method is used to extract short-time energy envelope prosodic behavior markers of the pre-processed speech data.

[0029] A behavior-enhanced voiceprint graph of the prosodic behavior markers is generated based on time-frequency domain joint coding to obtain the feature alignment reference of the pre-processed speech data, wherein the prosodic behavior markers include syllable start points and / or energy jump gradient.

[0030] The prosodic behavior of the pre-processed speech data is analyzed, including:

[0031] The speech stream of the pre-processed speech data is segmented into independent syllables based on an energy envelope mutation detection algorithm.

[0032] The dynamic coefficient of variation of speech rate of the independent syllables is calculated, and the pronunciation stability is quantified based on the dynamic coefficient of variation of speech rate, wherein the dynamic coefficient of variation of speech rate includes a standard deviation and / or a mean value.

[0033] Adopting an LSTM classifier to analyze distribution rules of abnormal pauses of the independent syllables, and extracting a fundamental frequency second derivative fluctuation energy spectrum (0-20Hz) of the independent syllables to depict naturalness of the fundamental frequency variation, to obtain fundamental frequency variation data, wherein the abnormal pause is a speech segment with a pause duration greater than 300 milliseconds, and the fundamental frequency second derivative fluctuation energy spectrum is an energy spectrum with a frequency of 0-20Hz;

[0034] Based on a self-attention mechanism, features of a speed mutation ±20% region in the fundamental frequency variation data are weighted and fused, and a 128-dimensional prosody feature vector is generated to detect abnormal behaviors of unnatural speed jumps, wherein the abnormal behaviors include robot speech.

[0035] The understanding of the semantic of the preprocessed voice data includes:

[0036] Based on a BERT-wwm (BERT-based natural language processing) model, the voice text of the preprocessed voice data is input into the BERT-wwm model to extract a 768-dimensional [CLS] semantic vector, and a dialogue logic graph is constructed.

[0037] A double-channel strategy is adopted to detect sensitive words.

[0038] As a further scheme of the application, the construction of the dialogue logic graph includes:

[0039] A node in the 768-dimensional [CLS] semantic vector is defined as a named entity, and a BiLSTM-CRF (bidirectional long short-term memory network and conditional random field) model is used to identify and obtain the named entity.

[0040] Based on the co-occurrence frequency of the semantic vector and the correlation degree of BERT (a pre-trained language model), a threshold value of a cosine similarity greater than 0.7 is used as a strong correlation determination standard to calculate edge weights.

[0041] As a further scheme of the application, the double-channel strategy for detecting sensitive words includes:

[0042] The first channel is set as an accurate matching link, and a preset word library is strictly matched in the accurate matching link, wherein the preset word library includes standard sensitive words and variant combinations of the standard sensitive words.

[0043] The second channel is set as a fuzzy matching link, and a Sentence-BERT (an improved model based on BERT) model is used to calculate semantic similarity between the voice text and sensitive concepts.

[0044] As a further scheme of the application, the synchronous establishment of the acoustic environment dynamic adaptation mechanism includes:

[0045] Set the threshold of the spectrum feature KL divergence ≤0.15, compare the spectrum feature KL divergence of the registered voiceprint and the real-time voice scene, train the spectrum correlation decision model, and obtain the target spectrum correlation decision model;

[0046] Adopt the target spectrum correlation decision model to detect the credibility mapping and the fake trace of the acoustic feature, wherein the fake trace includes an artificial harmonic distortion component.

[0047] As a further scheme of the present application: based on the user's mobile terminal equipment and the preset scoring rule, the user's fingerprint information, geographical position and historical behavior log are collected, and a risk scoring model is used to identify the voice signal recognition risk, including:

[0048] The scoring rule is set as:

[0049] When the user uses the device for the first time or the device is not bound, the score of the device strangeness degree is increased;

[0050] When the device logs in from a different place or a high-risk IP triggers, the score of the geographical position abnormality degree is increased;

[0051] When a high-frequency voice collection attempt fails or an irregular operation is detected, the score of the behavior deviation degree is increased, wherein the high-frequency voice collection is a case where the collection frequency per minute is greater than 3 times, and the irregular operation includes accessing sensitive functions from 1am to 3am;

[0052] The weight of each modality is set as:

[0053] The weight proportion of the device strangeness degree is 30%, the weight proportion of the geographical position abnormality degree is 40%, and the weight proportion of the behavior deviation degree is 30%;

[0054] The risk scoring model is adopted:

[0055] , the recognition risk of the voice signal is calculated, wherein, is the comprehensive risk score, is the device strangeness degree score, is the geographical position abnormality degree score, is the behavior deviation degree score, , , all are weights, and are respectively 0.3, 0.4 and 0.3;

[0056] When , it is determined as a low-risk scene;

[0057] When , it is determined as a medium-risk scene;

[0058] When , it is determined as a high-risk scene.

[0059] As a further aspect of the application: the multi-modal feature fusion and model training based on anti-noise MFCC acoustic features, prosody behavior and semantics includes:

[0060] Integrate anti-noise MFCC acoustic features, prosody and semantic features, use multi-modal Transformer network and DTW algorithm in cross-modal attention mechanism to align the acoustic features and prosody timing of the speech, and introduce attention mechanism to dynamically allocate the weight of each modality based on the risk assessment result, while using a learnable parameter matrix to adaptively adjust the weight of each modality to obtain the fusion features;

[0061] Input the fusion features into SVM (kernel switching support vector machine), and use RBF (radial basis function) kernel to process nonlinear data;

[0062] When the scenario of kernel processing nonlinear data is a sparse data set, switch to a linear kernel to effectively prevent overfitting problems;

[0063] Use a multi-task learning framework to integrate the losses of voiceprint recognition, semantic analysis and behavior prediction, use F1-score index to optimize weight distribution, and achieve overall performance balance.

[0064] In a second aspect, the application provides a speech recognition authentication system based on multi-modal features and dynamic assessment, which is applied to the speech recognition authentication method based on multi-modal features and dynamic assessment as described in the above-mentioned scheme. The system includes a data preprocessing module, a feature extraction module, an adaptive mechanism establishment module, a risk scoring module, a model training module, a scene authentication module, an execution module and an authentication passing module:

[0065] The data preprocessing module is configured to use a microphone to collect the original speech signal of a user, preprocess the original speech signal to obtain preprocessed speech data;

[0066] The feature extraction module is configured to use multi-dimensional feature hierarchical extraction technology to extract feature data of the preprocessed speech data, wherein the feature data includes feature alignment benchmarks, prosody behavior and semantics of the preprocessed speech data;

[0067] The adaptive mechanism establishment module is configured to inject multi-source noise samples into the acoustic feature space of the voiceprint feature model based on the initial training stage of the voiceprint feature model, construct an anti-noise mixed voiceprint atlas, deploy an Adv-GAN discriminator, and based on a spectrum-time domain joint discrimination mechanism, deconstruct the deep fake speech generated by WaveGAN in the discriminator, and simultaneously establish an acoustic environment dynamic adaptation mechanism;

[0068] The risk score module is configured to collect fingerprint information, geographical position and historical behavior log of the user based on the mobile terminal equipment of the user and preset scoring rules, identify a voice signal recognition risk of the user, and obtain a voice recognition risk score;

[0069] The model training module is configured to perform multi-modal feature fusion and model training based on anti-noise MFCC acoustic features, prosody behavior and semantics;

[0070] The scene authentication module is configured to only require voice authentication when judging that the authentication scene is a low-risk scene, enable dual-factor authentication for enhanced verification when judging that the authentication scene is a medium-risk scene, and forcibly enable second-factor authentication when judging that the authentication scene is a high-risk scene, wherein the second-factor authentication is fingerprint or dynamic password authentication;

[0071] The execution module is configured to assign the tasks of living body detection, feature comparison and second-factor verification to a multi-core processor for parallel execution in a high-risk scene.

[0072] The authentication passing module is configured to, when authentication is passed, authorize access and generate an encrypted token, and store login time, device information, risk score and authentication result to a security log.

[0073] Compared with the prior art, the present application has the following beneficial effects:

[0074] 1. In the present application, by breaking through the traditional single-modal dependence, a three-dimensional feature system is innovatively constructed based on the acoustic layer, the prosody behavior layer and the semantic layer, wherein the prosody layer divides syllables through energy envelope mutation detection, calculates the coefficient of variation of speech rate, analyzes abnormal pauses and fundamental frequency fluctuation energy spectrum in combination with LSTM, extracts 128-dimensional time sequence features to quantify natural pronunciation characteristics, the semantic layer generates context vectors using BERT-wwm, blocks semantic attacks in combination with precise matching and Sentence-BERT fuzzy detection, further aligns multi-modal time sequences through a dynamic time warping algorithm, realizes semantic-guided acoustic focusing using a cross-modal attention mechanism, and dynamically allocates weights based on risk scores, and finally generates a comprehensive voice representation with significantly enhanced anti-forgery capability.

[0075] 2. In the present application, by integrating voiceprint, semantic, behavior features and environment adaptation mechanism, a formulaic risk score model is first introduced in voice authentication The present application realizes quantitative determination of risk scenes, can ensure the convenience of voice risk identification in a low-risk scene, can strengthen security in a high-risk scene, adjusts the authentication threshold in real time according to dynamic risk assessment, enables different levels of authentication modes according to different degrees of risk scenes, simultaneously, for a high-risk scene, forcibly implements dual-factor authentication, and significantly enhances the authentication security and reliability. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 is a method framework diagram of the present application;

[0077] Figure 2 is a method step diagram of the present application;

[0078] Figure 3 is a system module diagram of the present application.

[0079] In the figure: 1, data preprocessing module; 2, feature extraction module; 3, adaptive mechanism establishment module; 4, risk score module; 5, model training module; 6, scene authentication module; 7, execution module; 8, authentication passing module. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0081] Embodiment:

[0082] Please refer to Figures 1-2 In the embodiments of the present application, a voice recognition authentication method based on multi-modal features and dynamic evaluation is provided, including the following steps:

[0083] S1: The original voice signal of the user is collected by using a microphone, and the original voice signal is preprocessed to obtain preprocessed voice data;

[0084] S2: The feature data of the preprocessed voice data is extracted by using multi-dimensional feature hierarchical extraction technology, wherein the feature data includes feature alignment reference, prosody behavior and semantics of the preprocessed voice data;

[0085] S3: In the initial training stage based on the voiceprint feature model, multi-source noise samples are injected into the acoustic feature space of the voiceprint feature model, an anti-noise mixed voiceprint atlas is constructed, an Adv-GAN discriminator is deployed, the deep fake voice generated in the discriminator WaveGAN is deconstructed based on the spectral time domain joint discrimination mechanism, and a dynamic adaptive mechanism of acoustic environment is established synchronously;

[0086] S4: Based on the mobile terminal device of the user and the preset scoring rules, the fingerprint information, geographic location and historical behavior log of the user are collected, a risk scoring model is used to identify the voice signal recognition risk of the user, and the voice recognition risk score is obtained to identify the voice signal recognition risk;

[0087] S5: Multi-modal feature fusion and model training based on anti-noise MFCC acoustic features, prosody behavior and semantics;

[0088] S6: Determine the scene level according to the risk score result, when When the authentication scene is determined as a low-risk scene, the cosine similarity threshold is set to 0.80, and only voice authentication is required; when When the authentication scene is determined as a medium-risk scene, the cosine similarity threshold is set to 0.85, and dual-factor authentication is enabled for enhanced verification; when When the authentication scene is determined as a high-risk scene, the cosine similarity threshold is set to 0.90, and the second factor authentication is forcibly enabled, wherein the second factor authentication is fingerprint or dynamic password authentication;

[0089] S7: In the high-risk scene, the tasks of live detection, feature comparison and second factor verification are assigned to the multi-core processor for parallel execution;

[0090] S8: When the authentication is passed, the authorized access is generated, and the encrypted token is stored, and the login time, device information, risk score and authentication result are stored to the security log.

[0091] In this embodiment, the risk score model is:

[0092]

[0093] Among them:

[0094] is the comprehensive risk score;

[0095] is the device strangeness score;

[0096] is the geographic location abnormality score;

[0097] is the behavior deviation score;

[0098] , , are weights, respectively 0.3, 0.4, 0.3.

[0099] In this embodiment, the low-risk scene is the scene with a score ≤30, the low-risk scene threshold is set to 0.80 (cosine similarity), the medium-risk scene is the scene with a score >30 and ≤70, the medium-risk scene threshold is set to 0.85 (cosine similarity), and the high-risk scene is the scene with a score >70, and the high-risk scene threshold is set to 0.90;

[0100] In this embodiment, the live detection includes fingerprint detection and iris detection.

[0101] In this embodiment, the signal-to-noise ratio of the multi-source noise sample is 20dB-35dB.

[0102] In this embodiment, the invalid segment includes a silent segment.

[0103] In this embodiment, step S1 is used to complete the standardization processing of the original speech signal.

[0104] In this embodiment, the feature deconstruction is to detect false formants.

[0105] In this embodiment, the multi-modal Transformer network is a deep learning model based on Transformer, and the attention mechanism includes semantic-guided acoustic focusing.

[0106] In this embodiment, the dynamic time warping (DTW) technology is used to align the acoustic features and prosodic timing, ensuring the synchronization of multi-modal data. In cross-modal fusion, attention mechanisms (such as semantic-guided acoustic focusing) are introduced to enhance the interaction between different modalities. Based on the risk assessment results, the weights of each modality are dynamically allocated, and adaptive adjustment is achieved through a learnable parameter matrix, improving the robustness of the model. The kernel switching support vector machine (SVM) after feature fusion defaults to radial basis function (RBF) kernel processing for nonlinear data, but automatically switches to linear kernel in sparse data set scenarios, effectively preventing overfitting problems. The multi-task learning framework integrates voiceprint recognition, semantic analysis, and behavior prediction losses, and uses F1-score indicators to optimize task weight allocation, achieving overall performance balance.

[0107] Preferably, a microphone is used to collect the original speech signal of the user, and the original speech signal is preprocessed to obtain preprocessed speech data, including:

[0108] The original speech signal is read frame by frame and repeated or invalid segments are removed to obtain valid speech data, the valid speech data is stored as a standardized data set, and the standardized data set is converted into a WAV format file and outputted;

[0109] The RNNoise deep learning model is used to eliminate environmental noise in the WAV format file to obtain a de-noised audio, and non-speech audio segments of the de-noised audio are filtered to obtain a frequency domain speech signal, wherein the non-speech audio segments include mechanical noise and / or burst noise.

[0110] The frequency domain speech signal is segmented into short-time frame speech signals using a speech segmentation method, and the speech start and end points of the short-time frame speech signals are detected based on short-time energy and zero-crossing rate to obtain segmented speech data, wherein the short-time frame speech signal is a speech signal of no more than 20ms / frame.

[0111] The speech length insufficient speech segment is a speech segment with a time length less than 0.5 seconds, and the signal-to-noise ratio insufficient speech segment is a speech segment with a signal-to-noise ratio less than 15 dB.

[0112] Preferably, the feature data of the preprocessed speech data is extracted by using a multi-dimensional feature hierarchical extraction technology, including:

[0113] A feature alignment reference of the preprocessed speech data is obtained.

[0114] The prosody behavior of the preprocessed speech data is analyzed.

[0115] The semantics of the preprocessed speech data are understood.

[0116] Preferably, the feature alignment reference is obtained, including:

[0117] The short-time energy envelope prosody behavior markers of the preprocessed speech data are extracted by using an anti-noise MFCC and fundamental frequency trajectory fluctuation analysis method including a second-order dynamic difference;

[0118] The behavior-enhanced voiceprint graph of the prosody behavior markers is generated based on joint time-frequency domain coding to obtain the feature alignment reference of the preprocessed speech data, wherein the prosody behavior markers include syllable starting points and / or energy jump gradient;

[0119] The prosody behavior of the preprocessed speech data is analyzed, including:

[0120] The speech stream of the preprocessed speech data is segmented into independent syllables based on an energy envelope mutation detection algorithm;

[0121] The speech rate dynamic variation coefficient of the independent syllables is calculated, and the pronunciation stability is quantified based on the speech rate dynamic variation coefficient, wherein the speech rate dynamic variation coefficient includes a standard deviation and / or a mean value;

[0122] The distribution rule of the abnormal pause of the independent syllables is analyzed by using an LSTM classifier, and the fundamental frequency second-order derivative fluctuation energy spectrum (0-20 Hz) of the independent syllables is extracted to depict the naturalness of the fundamental frequency change to obtain the fundamental frequency change data, wherein the abnormal pause is a speech segment with a pause duration greater than 300 milliseconds, and the fundamental frequency second-order derivative fluctuation energy spectrum is an energy spectrum with a frequency of 0-20 Hz;

[0123] The features in the speech rate mutation ± 20% region of the fundamental frequency change data are weighted and fused based on a self-attention mechanism, and a 128-dimensional prosody feature vector is generated to detect abnormal behaviors of unnatural speech rate jumps, wherein the abnormal behaviors include robot speech.

[0124] In this embodiment, the abnormal behavior of non-natural speech speed jumps is detected by a 128-dimensional prosody feature vector, which can break through the limitations of traditional methods relying on a single fundamental frequency mean, effectively improving the accuracy and comprehensiveness of prosody behavior analysis.

[0125] Understanding the semantics of preprocessed voice data includes:

[0126] Based on the BERT-wwm (BERT-based natural language processing) model, input the voice text of the preprocessed voice data into the BERT-wwm model to extract a 768-dimensional [CLS] semantic vector, and construct a dialogue logic graph.

[0127] A double-channel strategy is used to detect sensitive words.

[0128] In this embodiment, MFCC is a widely used feature extraction method in the field of audio signal processing and speech recognition. Specifically, MFCC is used for noise reduction processing and combines first-order and / or second-order dynamic differences, which can better describe the time-domain variation characteristics of speech signals (such as changes in pitch or speech rate), thereby improving the robustness of speech recognition or classification systems. MFCC converts audio signals into a set of coefficients that can represent their spectral characteristics by simulating the way the human ear perceives sound. These coefficients can be used to train machine learning models or perform speech analysis.

[0129] Preferably, the dialogue logic graph is constructed, including:

[0130] The nodes in the 768-dimensional [CLS] semantic vector are defined as named entities, and a BiLSTM-CRF (Bidirectional Long Short-Term Memory Network and Conditional Random Field) model is used to identify and obtain named entities.

[0131] Based on the co-occurrence frequency of semantic vectors and the correlation degree of BERT (pre-trained language model), a threshold of cosine similarity greater than 0.7 is used as a strong correlation determination standard to calculate edge weights.

[0132] In this embodiment, by calculating the edge weights based on the co-occurrence frequency of semantic vectors and the correlation degree of BERT (pre-trained language model), the semantic coherence and reasoning ability of the graph can be enhanced.

[0133] Preferably, a double-channel strategy is used to detect sensitive words, including:

[0134] The first channel is set as an accurate matching link, and in the accurate matching link, the preset word library is strictly matched, wherein the preset word library includes standard sensitive words and variant combinations of standard sensitive words.

[0135] Set the second channel as a fuzzy matching link, and use the Sentence-BERT (an improved model based on BERT) model to calculate the semantic similarity between the voice text and the sensitive concept.

[0136] In this embodiment, the precise matching preset word library (including variant combinations such as "identity verification") and the fuzzy matching through semantic similarity (Sentence-BERT) detection to evade expressions (such as "account permission confirmation" mapped to "identity verification") are used. By setting the first channel and the second channel, the first channel can be strictly matched with the preset word library to ensure efficient interception of direct expressions, and at the same time, the second channel can effectively detect evasive expressions to improve the recognition coverage of variants or implicit expressions.

[0137] Preferably, a synchronous acoustic environment dynamic adaptation mechanism is established, including:

[0138] Set the threshold of the spectral feature KL divergence ≤0.15, compare the spectral feature KL divergence of the registered voiceprint and the real-time acoustic scene, train the spectral correlation decision model, and obtain the target spectral correlation decision model;

[0139] The target spectral correlation decision model is used to detect the credibility mapping and counterfeit traces of acoustic features, wherein the counterfeit traces include artificial harmonic distortion components.

[0140] Preferably, based on the user's mobile terminal device and the preset scoring rules, the user's fingerprint information, geographic location and historical behavior logs are collected, and the user's voice signal recognition risk is identified, including:

[0141] The scoring rules are set as:

[0142] When the user uses the device for the first time or the device is not bound, the score of device strangeness is increased;

[0143] When the device logs in from a different place or a high-risk IP triggers, the score of geographic location abnormality is increased;

[0144] When a high-frequency voice collection attempt fails or an irregular operation is detected, the score of behavior deviation is increased, wherein the high-frequency voice collection is a case where the collection frequency per minute is greater than 3 times, and the irregular operation includes accessing sensitive functions at 3 am;

[0145] The weights of each modality are set as:

[0146] The weight proportion of device strangeness is 30%, the weight proportion of geographic location abnormality is 40%, and the weight proportion of behavior deviation is 30%.

[0147] Preferably, multi-modal feature fusion and model training are performed based on anti-noise MFCC acoustic features, prosodic behavior and semantics, including:

[0148] The anti-noise MFCC acoustic feature, the prosody feature and the semantic feature are integrated, the multi-modal Transformer network and the DTW algorithm in the cross-modal attention mechanism are used to align the acoustic feature and the prosody time sequence of the voice, the attention mechanism is introduced to dynamically allocate the weight of each mode based on the risk evaluation result, a learnable parameter matrix is used to adaptively adjust the weight of each mode, and the fusion feature is obtained;

[0149] The fusion feature is input into the SVM (kernel switching support vector machine), and the RBF (radial basis function) kernel is used to process the nonlinear data;

[0150] When the scene of processing nonlinear data by the kernel is a sparse data set, the linear kernel is switched to effectively prevent overfitting problems;

[0151] The multi-task learning framework is used to integrate the loss of voiceprint recognition, semantic analysis and behavior prediction, the F1-score index is used to optimize the weight distribution, and the overall performance balance is realized.

[0152] As shown in Figure 3 , the embodiment provides a voice recognition authentication system based on multi-modal features and dynamic evaluation, which is applied to the voice recognition authentication method based on multi-modal features and dynamic evaluation as described above. The system includes a data preprocessing module 1, a feature extraction module 2, an adaptive mechanism establishment module 3, a risk scoring module 4, a model training module 5, a scene authentication module 6, an execution module 7 and an authentication passing module 8.

[0153] The data preprocessing module 1 is configured to collect the original voice signal of the user by using the microphone, preprocess the original voice signal, and obtain the preprocessed voice data;

[0154] The feature extraction module 2 is configured to extract feature data of the preprocessed voice data by using multi-dimensional feature hierarchical extraction technology, wherein the feature data includes feature alignment benchmarks, prosody behaviors and semantics of the preprocessed voice data.

[0155] The adaptive mechanism establishment module 3 is configured to inject multi-source noise samples into the acoustic feature space of the voiceprint feature model based on the initial training stage of the voiceprint feature model, construct an anti-noise mixed voiceprint atlas, deploy an Adv-GAN discriminator, and based on a spectrum-time domain joint discrimination mechanism, the feature of the deep fake voice generated by the WaveGAN in the discriminator is deconstructed, and a dynamic acoustic environment adaptation mechanism is established at the same time.

[0156] The risk scoring module 4 is configured to collect the fingerprint information, geographic location and historical behavior log of the user based on the mobile terminal device of the user and the preset scoring rule, identify the voice signal recognition risk of the user, and obtain the voice recognition risk score;

[0157] The model training module 5 is configured to perform multi-modal feature fusion and model training based on the anti-noise MFCC acoustic features, prosody behaviors and semantics;

[0158] The scene authentication module 6 is configured to perform voice authentication only when the authentication scene is determined to be a low-risk scene, to perform enhanced verification by enabling dual-factor authentication when the authentication scene is determined to be a medium-risk scene, and to forcibly enable second-factor authentication, which is fingerprint or dynamic password authentication, when the authentication scene is determined to be a high-risk scene.

[0159] The execution module 7 is configured to assign the tasks of live detection, feature comparison and second-factor verification to the multi-core processor for parallel execution in a high-risk scene.

[0160] The authentication passing module 8 is configured to authorize access and generate an encrypted token when authentication is passed, and to store the login time, device information, risk score and authentication result to a security log.

[0161] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application.

Claims

1. A voice recognition authentication method based on multi-modal features and dynamic evaluation, characterized in that, The method comprises the following steps: Collecting the original voice signal of a user by using a microphone, preprocessing the original voice signal to obtain preprocessed voice data; Extracting feature data of the preprocessed voice data by using a multi-dimensional feature hierarchical extraction technology, wherein the feature data includes a feature alignment reference, prosodic behavior, and semantics of the preprocessed voice data; In an initial training stage based on a voiceprint feature model, injecting a multi-source noise sample into an acoustic feature space of the voiceprint feature model, constructing an anti-noise mixed voiceprint graph, deploying an Adv-GAN discriminator, deconstructing the deep fake voice generated by WaveGAN in the discriminator based on a spectral time domain joint discrimination mechanism, and synchronously establishing an acoustic environment dynamic adaptation mechanism; Based on the mobile terminal device of the user and the preset scoring rules, collecting the fingerprint information, geographical location, and historical behavior log of the user, identifying the voice signal recognition risk of the user by using a risk scoring model, and obtaining a voice recognition risk score; Based on the anti-noise MFCC acoustic feature, prosodic behavior, and semantics, performing multi-modal feature fusion and model training; When it is judged that the authentication scene is a low-risk scene, only voice authentication is required; when it is judged that the authentication scene is a medium-risk scene, dual-factor authentication is enabled for enhanced verification; when it is judged that the authentication scene is a high-risk scene, the second factor authentication is forcibly enabled, wherein the second factor authentication is fingerprint or dynamic password authentication; In a high-risk scene, the tasks of live detection, feature comparison, and second factor verification are assigned to a multi-core processor for parallel execution; When the authentication is passed, authorized access is generated, and an encrypted token is stored, and the login time, device information, risk score, and authentication result are stored in a security log. 2.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 1, characterized in that: Collecting the original voice signal of a user by using a microphone, preprocessing the original voice signal to obtain preprocessed voice data, comprising: Reading the original voice signal frame by frame and removing repeated or invalid segments to obtain valid voice data, storing the valid voice data as a standardized data set, converting the standardized data set into a WAV format file and outputting; Using an RNNoise deep learning model to eliminate environmental noise in the WAV format file to obtain a de-noised audio, filtering non-speech audio segments of the de-noised audio to obtain a frequency domain voice signal, wherein the non-speech audio segments include mechanical noise and / or burst noise; Using a voice segmentation method to segment the frequency domain voice signal into short-time frame voice signals, detecting the start and end points of the short-time frame voice signals based on short-time energy and zero-crossing rate to obtain segmented voice data, wherein the short-time frame voice signals are voice signals with a frame length of not more than 20 ms; Removing voice segments with insufficient voice length or excessively low signal-to-noise ratio from the segmented voice data to obtain preprocessed voice data, wherein the voice segments with insufficient voice length are voice segments with a time length of less than 0.5 seconds, and the voice segments with excessively low signal-to-noise ratio are voice segments with a signal-to-noise ratio of less than 15 dB. 3.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 2, characterized in that: The method comprises the following steps: Obtaining the feature alignment reference of the preprocessed voice data; Analyzing the prosodic behavior of the preprocessed voice data; understanding semantics of the preprocessed voice data. 4.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 3, characterized in that: obtaining a feature alignment benchmark, comprising: extracting short-time energy envelope prosody behavior markers of the preprocessed voice data by using an anti-noise MFCC and fundamental frequency trajectory fluctuation analysis method including first-order and / or second-order dynamic difference; generating an enhanced behavior voiceprint graph of the prosody behavior markers based on joint time-frequency domain coding to obtain a feature alignment benchmark of the preprocessed voice data, wherein the prosody behavior markers include syllable starting points and / or energy jump gradient; the analysis of the prosody behavior of the preprocessed voice data, comprising: segmenting the voice stream of the preprocessed voice data into independent syllables based on an energy envelope mutation detection algorithm; calculating the dynamic coefficient of variation of the speech rate of the independent syllables to quantify the pronunciation stability based on the dynamic coefficient of variation of the speech rate, wherein the dynamic coefficient of variation of the speech rate includes the standard deviation and / or the mean; using an LSTM classifier to analyze the distribution of abnormal pauses of the independent syllables, and extracting the fundamental frequency second derivative fluctuation energy spectrum to depict the naturalness of the fundamental frequency change to obtain the fundamental frequency change data, wherein the abnormal pause is a voice segment with a pause duration greater than 300 milliseconds, and the fundamental frequency second derivative fluctuation energy spectrum is an energy spectrum with a frequency of 0-20 Hz; based on the self-attention mechanism, the features in the region of the sudden change of the speech rate ± 20% in the fundamental frequency change data are weighted and fused, and a 128-dimensional prosody feature vector is generated to detect abnormal behavior of unnatural speech rate jumps, wherein the abnormal behavior includes robot voice; the understanding of the semantics of the preprocessed voice data, comprising: inputting the voice text of the preprocessed voice data into the BERT-wwm model based on the BERT-wwm model to extract a 768-dimensional semantic vector and construct a dialogue logic graph; detecting sensitive words using a dual-channel strategy. 5.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 4, characterized in that: the construction of the dialogue logic graph, comprising: defining the nodes in the 768-dimensional semantic vector as named entities, and identifying the named entities using a BiLSTM-CRF model; based on the co-occurrence frequency of the semantic vector and the correlation degree of BERT, using a threshold value of cosine similarity greater than 0.7 as a strong correlation determination standard to calculate the edge weight. 6.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 5, characterized in that: the detection of sensitive words using a dual-channel strategy, comprising: setting the first channel as an accurate matching link, and strictly checking against a preset word library in the accurate matching link, wherein the preset word library includes standard sensitive words and variant combinations of the standard sensitive words; setting the second channel as a fuzzy matching link, and using a Sentence-BERT model to calculate the semantic similarity between the voice text and the sensitive concept. 7.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 6, characterized in that: the synchronous establishment of the acoustic environment dynamic adaptation mechanism, comprising: setting the threshold value of the spectral feature KL divergence ≤0.15, comparing the spectral feature KL divergence of the registered voiceprint and the real-time acoustic scene, training a spectral correlation decision model, and obtaining a target spectral correlation decision model; using the target spectral correlation decision model to detect the credibility mapping and the counterfeit traces of the acoustic features, wherein the counterfeit traces include artificial harmonic distortion components. 8.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 7, characterized in that: The user-based mobile terminal device and the preset scoring rule collect fingerprint information, geographical position and historical behavior log of the user, and adopt a risk scoring model to identify the voice signal recognition risk of the user, comprising: The scoring rule is set as: When the user uses the device for the first time or does not bind the device, the score of device strangeness is increased; When the device logs in from a different place or a high-risk IP triggers, the score of geographical position abnormality is increased; When high-frequency voice collection attempt failure or irregular operation is detected, the score of behavior deviation is increased, wherein the high-frequency voice collection is a case where the collection frequency per minute is greater than 3 times, and the irregular operation includes accessing sensitive functions from 1 am to 3 am; The weight of each mode is set as: The weight proportion of device strangeness is 30%, the weight proportion of geographical position abnormality is 40%, and the weight proportion of behavior deviation is 30%; The risk scoring model is adopted: calculating a recognition risk of the voice signal, wherein is a comprehensive risk score, is a device strangeness score, is a geographic location abnormality score, is a behavior deviation score, , , are weights, respectively 0.3, 0.4, 0.

3. When a low risk scenario is determined; When a medium risk scenario is determined; When a high-risk scenario is determined. 9.The voice recognition authentication method based on multi-modal features and dynamic evaluation according to claim 8, characterized in that: The multi-modal feature fusion and model training based on anti-noise MFCC acoustic features, prosody behavior and semantics, comprising: Integrating anti-noise MFCC acoustic features, prosody and semantic features, using multi-modal Transformer network and DTW algorithm in cross-modal attention mechanism to align the acoustic features and prosody timing of the voice, and introducing attention mechanism to dynamically allocate the weight of each mode based on the risk assessment result, while using a learnable parameter matrix to adaptively adjust the weight of each mode to obtain the fusion features; The fusion features are input into the SVM, and RBF kernel is used to process nonlinear data; When the scene of kernel processing nonlinear data is a sparse data set, switch to linear kernel; Using a multi-task learning framework to integrate the losses of voiceprint recognition, semantic analysis and behavior prediction, and using F1-score index to optimize weight distribution. 10.A voice recognition authentication system based on multi-modal features and dynamic evaluation, characterized in that: The system is applied to the voice recognition authentication method based on multi-modal features and dynamic evaluation according to any one of claims 1-9, comprising: A data preprocessing module configured to collect the original voice signal of the user using a microphone, preprocess the original voice signal to obtain preprocessed voice data; A feature extraction module configured to extract feature data of the preprocessed voice data using multi-dimensional feature hierarchical extraction technology, wherein the feature data includes feature alignment criteria, prosody behavior and semantics of the preprocessed voice data; An adaptive mechanism establishment module configured to, based on an initial training stage of a voiceprint feature model, inject multi-source noise samples into the acoustic feature space of the voiceprint feature model, construct an anti-noise mixed voiceprint atlas, deploy an Adv-GAN discriminator, perform feature deconstruction on deep fake voice generated by WaveGAN in the discriminator based on a spectrum-time domain joint discrimination mechanism, and simultaneously establish an acoustic environment dynamic adaptation mechanism; A risk scoring module configured to collect fingerprint information, geographical position and historical behavior log of the user based on a user-based mobile terminal device and a preset scoring rule, identify the voice signal recognition risk of the user, and obtain a voice recognition risk score. The model training module is configured to perform multi-modal feature fusion and model training based on anti-noise MFCC acoustic features, prosodic behaviors and semantics. The scene authentication module is configured to perform voice authentication only when the authentication scene is determined to be a low-risk scene, to enable dual-factor authentication for enhanced verification when the authentication scene is determined to be a medium-risk scene, and to force the use of second-factor authentication when the authentication scene is determined to be a high-risk scene, wherein the second-factor authentication is fingerprint or dynamic password authentication. The execution module is configured to assign the tasks of live detection, feature comparison and second-factor verification to the multi-core processor for parallel execution in a high-risk scene. The authentication passing module is configured to authorize access and generate an encrypted token when authentication is passed, and to store the login time, device information, risk score and authentication result to a security log.

Images