ASV system risk assessment method and system based on multi-dimensional pronunciation characterization decoupling and fusion

Abstract

The invention provides an ASV system risk assessment method and system based on multi-dimensional pronunciation characterization decoupling and fusion, and the method comprises the steps: obtaining a reference audio of a target user, and carrying out the decoupling extraction of a volume feature vector, a pitch feature vector, and a speaking style feature vector from the reference audio through a multi-dimensional pronunciation feature extraction network; a text to be verified is converted into a phoneme sequence, the phoneme sequence is input into a pronunciation feature prediction network for predicting a frame-level dynamic pronunciation feature sequence based on the phoneme sequence, and three pronunciation feature vectors are injected into the network through an adaptive instance normalization mechanism to dynamically modulate a prediction process. Generating a frame-level dynamic pronunciation feature sequence containing time sequence rhythm change; inputting a VAE-GAN synthesis trunk, re-injecting the three pronunciation feature vectors through an adaptive instance normalization mechanism, and generating a test voice sample consistent with the voiceprint characteristics of the target user; and initiating an identity verification query for the ASV system, and calculating safety indexes of different user groups according to a verification result so as to evaluate the risk of the ASV system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of speech signal processing, and particularly relates to an ASV system risk assessment method and system based on multi-dimensional pronunciation representation decoupling and fusion. BACKGROUND

[0002] Automatic Speaker Verification (ASV) technology, as a biometric means of determining the identity of a speaker using speech waveforms, has been widely used in smart home access control, judicial evidence, and financial payment fields with high security requirements. This technology analyzes the voiceprint features in the input speech to determine whether the input speech comes from the claimed user. Since these features can be collected by microphones, digitized and modeled, deep learning models can also quickly and accurately generalize these features, making ASV systems vulnerable to voice synthesis attacks. Such attacks train a model using a small number of target speaker speech samples, and then generate speech that is similar to the target speaker's voice with arbitrary text content. In order to ensure the reliability of ASV systems in real open environments, it is urgent to build high-quality and high-fidelity adversarial test samples and test methods to conduct comprehensive risk assessment and security boundary detection.

[0003] Although existing ASV system risk assessment research has made some progress, especially in the field of voice synthesis, it can partially simulate the timbre characteristics of system users, effectively interfering and testing early ASV systems. The latest research has implemented zero-shot test audio generation based on global speaker embedding, further providing empirical evidence for quantifying its robustness. However, most of these test sample generation methods only focus on the similarity of timbre, resulting in dull and lackluster test samples. Moreover, the extraction of only one embedding vector representing identity causes the volume, pitch, and speaking style to be highly entangled. Due to the lack of explicit decoupling of pronunciation physical characteristics, the test samples generated by existing evaluation methods are difficult to accurately reproduce microscopic prosody and non-linear vocal cord texture features such as pitch jitter while maintaining timbre similarity. There are still obvious artificial traces in terms of auditory perception and spectral details, making it difficult to evaluate the robustness of the new generation of ASV systems integrated with fake speech detection when facing high-fidelity voice synthesis.

[0004] In addition, existing ASV systems are usually well-trained on standard adult datasets and perform well, but when faced with long-tail populations such as children (whose voices have higher fundamental frequencies and sparse spectral characteristics) or the elderly (whose voices are often accompanied by frequency fluctuations and breathiness characteristics caused by vocal cord aging), they often perform weakly due to unclear discrimination boundaries. Such groups are more likely to be targeted by voice synthesis attacks due to their acoustic characteristics deviating from the mainstream data distribution, exposing higher risks. At the same time, within the same age group, different gender groups have different vocal tract changes, and their acoustic characteristics are also different. However, existing voice generation models generally have difficulty accurately reproducing the acoustic characteristics of these groups, such as high-frequency vocalization characteristics or hoarse voiceprints. The generated test samples often exhibit a tendency to "average" or "over-smooth", failing to truly reveal the potential security vulnerabilities of ASV systems when dealing with long-tail populations, resulting in blind spots in evaluation results. SUMMARY

[0005] In view of the defects and deficiencies of the prior art, the present application provides an ASV system risk assessment method, system and device based on multi-dimensional pronunciation representation decoupling and fusion. The core of this scheme is that first, from the reference audio of the target user, through a specially designed multi-dimensional pronunciation feature extraction network, three independent pronunciation feature vectors respectively representing sound energy distribution, fundamental frequency change trajectory and vocalization texture characteristics are explicitly decoupled and extracted, thereby breaking the limitation of the high coupling of these key attributes in existing methods. Subsequently, the text to be verified is converted into a phoneme sequence and input into a pronunciation feature prediction network; through an adaptive instance normalization mechanism, the aforementioned three decoupled pronunciation feature vectors are dynamically injected into the internal layers of the network to modulate its prediction process, thereby generating a frame-level dynamic pronunciation feature sequence that contains both text content information and the target user's personalized timing prosody changes. Then, the sequence is input into a VAE-GAN synthesis backbone, and the same three pronunciation feature vectors are injected again through the adaptive instance normalization mechanism to drive the synthesis process, ultimately generating test speech samples that are highly consistent with the target user's voiceprint characteristics and have high fidelity. Using these samples, identity verification queries are initiated to the target ASV system, and based on the verification results, the false acceptance rate and other security indicators are calculated for user groups divided according to different demographic dimensions (such as age and gender), thereby systematically evaluating the robustness differences and potential security vulnerabilities of ASV systems for different groups. By decoupling and dynamically fusing multi-dimensional pronunciation representations, the present application can generate test speech that is expressive and realistic, especially by introducing a balanced acoustic fine-tuning dataset across age and gender and a low-rank adaptive fine-tuning technique, effectively improving the model's ability to model and reproduce the acoustic characteristics of long-tail populations such as children and the elderly, and achieving a more comprehensive and fair risk assessment of ASV systems.

[0006] The technical scheme specifically adopted by the present application to solve its technical problems is:

[0007] An ASV system risk assessment method based on multi-dimensional pronunciation feature decoupling and fusion, comprising:

[0008] Obtaining a reference audio of a target user, and decoupling and extracting a volume feature vector representing sound energy distribution, a pitch feature vector representing a base frequency change trajectory, and a speaking style feature vector representing a voice texture characteristic from the reference audio through a multi-dimensional pronunciation feature extraction network;

[0009] Converting a to-be-verified text into a phoneme sequence, inputting a pronunciation feature prediction network for predicting a frame-level dynamic pronunciation feature sequence based on the phoneme sequence, injecting the three pronunciation feature vectors into the network through an adaptive instance normalization mechanism to dynamically modulate the prediction process, and generating a frame-level dynamic pronunciation feature sequence containing timing rhythm changes;

[0010] Inputting the frame-level dynamic pronunciation feature sequence into a VAE-GAN synthesis backbone, injecting the three pronunciation feature vectors again through an adaptive instance normalization mechanism, and generating a test speech sample consistent with the voiceprint characteristics of the target user;

[0011] Using the test speech sample to initiate an identity verification query of an ASV system, calculating security indicators of different user groups according to the verification result, and evaluating the risk of the ASV system.

[0012] Further, the process of decoupling and extracting the three pronunciation feature vectors through the multi-dimensional pronunciation feature extraction network comprises:

[0013] First, the reference audio is preprocessed, sampled, and framed and windowed, linear frequency spectrum is obtained through short-time Fourier transform, and then the linear frequency spectrum is converted into a mel spectrum graph through mel filtering and logarithmic transformation;

[0014] The multi-dimensional pronunciation feature extraction network comprises:

[0015] The volume feature extraction submodule: calculates the energy of each frame in the mel spectrum graph, takes the average value of the energy of all frames, and obtains the volume feature vector;

[0016] The pitch feature extraction submodule: adopts a convolution-residual-bidirectional recurrent network for pitch extraction to extract time-frequency information from the mel spectrum graph layer by layer, and outputs the pitch feature vector;

[0017] The speaking style feature extraction submodule: is composed of six convolution-enhanced sequence transformation modules for speaking style extraction and a recurrent module, inputs the volume feature vector, the pitch feature vector, and the mel spectrum graph, and outputs the speaking style feature vector.

[0018] Furthermore, the pronunciation feature prediction network includes a text encoder, a duration predictor, an aligner module, and a sequence transformation module;

[0019] The process of converting the text to be verified into a phoneme sequence and inputting it into the pronunciation feature prediction network includes:

[0020] The text to be verified is converted into a phoneme sequence using the CMU pronunciation dictionary. The phoneme sequence is then input into a text encoder to obtain a hidden feature sequence. The text encoder includes an input embedding layer, a Transformer encoder backbone, and an output projection layer.

[0021] The duration of each phoneme is predicted by the duration predictor, generating a monotonic alignment relationship. The hidden feature sequence is then mapped to frame-level features by the aligner module.

[0022] The three types of pronunciation feature vectors are injected into the internal layer of the sequence conversion module through an adaptive instance normalization mechanism. The frame-level features are dynamically modulated to adapt to the pronunciation characteristics of the target user, and finally a frame-level dynamic pronunciation feature sequence containing temporal prosodic changes is generated.

[0023] Furthermore, the VAE-GAN synthesis backbone includes a priori encoder, an adversarial generator, and a HiFi-GAN vocoder;

[0024] The process of inputting frame-level dynamic pronunciation feature sequences into VAE-GAN to synthesize the backbone and injecting pronunciation feature vectors includes:

[0025] The prior encoder injects three types of pronunciation feature vectors through an adaptive instance normalization mechanism to predict latent variables from frame-level dynamic pronunciation feature sequences;

[0026] The adversarial generator injects three types of pronunciation feature vectors through an adaptive instance normalization mechanism to decode latent variables and generate Mel spectrograms.

[0027] The HiFi-GAN vocoder converts Mel spectrograms into test speech samples in audio format.

[0028] Furthermore, the security indicator is the proportion of test voice samples that were incorrectly identified as legitimate users by the ASV system to the total number of test samples;

[0029] When calculating security metrics for different user groups, target users are divided into multiple subgroups based on demographic dimensions, and the independent error acceptance rate for each group is calculated separately.

[0030] When assessing the risks of an ASV system, if the false acceptance rate of a certain group is significantly higher than that of a preset baseline group, the ASV system is determined to have a security vulnerability in the dimension corresponding to that group. The risk assessment results are output in the form of a report that includes an overall security score, a comparison of the robustness of each group, and vulnerability defense recommendations.

[0031] Furthermore, before decoupling and extracting the pronunciation feature vector, a step of domain-adaptive fine-tuning of the test sample generation model is included, specifically:

[0032] Construct a multi-source heterogeneous acoustic fine-tuning dataset, with data sources including at least: speech from young and middle-aged adults, speech from children, and speech from the elderly;

[0033] Stratified sampling was performed on samples from each age group in the dataset to ensure statistical balance in terms of quantity and duration of samples from different genders, thus forming a balanced acoustic feature distribution across age and gender.

[0034] The model is fine-tuned using the low-rank adaptive LoRA technique: most of the backbone parameters of the model generated from test samples are frozen, and only trainable low-rank matrices are injected into the attention mechanism projection layer and convolution kernel of the multidimensional pronunciation feature extraction network and the VAE-GAN synthesized backbone. The parameters of the low-rank matrices are updated using the fine-tuning dataset, so that the model can quickly adapt to the acoustic representation of long-tailed groups.

[0035] Furthermore, the reference audio is obtained from publicly available audio resources on the Internet, including social media, and the target users are equally divided into three age groups: children, young and middle-aged adults, and the elderly, based on both age and gender.

[0036] At least two audio recordings are obtained for each target user: one for registering with the tested ASV system to simulate the real registration process, and the other as a reference audio for extracting pronunciation feature vectors. The duration of each reference audio recording is no less than a preset value.

[0037] Furthermore, the VAE-GAN synthesis backbone also includes a discriminator for distinguishing real audio from synthesized test samples;

[0038] The prior encoder consists of a Transformer encoder and a normalization stream, and is used to predict latent variables from the frame-level dynamic pronunciation feature sequence after injecting three pronunciation feature vectors through an adaptive instance normalization mechanism.

[0039] The adversarial generator consists of a variational autoencoder decoder and a GAN-based generator, which is used to decode the latent variables and generate a Mel spectrogram after injecting three types of pronunciation feature vectors through an adaptive instance normalization mechanism.

[0040] The discriminator is a HiFi-GAN multi-cycle discriminator, which consists of multiple sub-discriminators and distinguishes between real audio and synthetic test samples during the model training phase.

[0041] And, an ASV system risk assessment system based on multidimensional pronunciation representation decoupling and fusion, comprising:

[0042] The reference audio processing module is used to acquire the reference audio of the target user and extract the volume feature vector representing the sound energy distribution, the pitch feature vector representing the fundamental frequency change trajectory, and the speech style feature vector representing the vocal texture characteristics from it through a multi-dimensional pronunciation feature extraction network.

[0043] The dynamic feature prediction module is used to convert the text to be verified into a phoneme sequence, input to the pronunciation feature prediction network for predicting frame-level dynamic pronunciation feature sequences based on the phoneme sequence, and inject three pronunciation feature vectors into the network through an adaptive instance normalization mechanism to dynamically modulate the prediction process, generating a frame-level dynamic pronunciation feature sequence containing temporal prosodic changes.

[0044] The test sample generation module is used to input the frame-level dynamic pronunciation feature sequence into VAE-GAN to synthesize the backbone, and then inject three kinds of pronunciation feature vectors again through an adaptive instance normalization mechanism to generate test speech samples that are consistent with the voiceprint characteristics of the target user.

[0045] The risk assessment module is used to initiate an authentication query against the ASV system using the test voice samples, and calculate security indicators for different user groups based on the verification results in order to assess the risk of the ASV system.

[0046] And an electronic device including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method described above.

[0047] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0048] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0049] First, through a core design of multi-dimensional pronunciation feature decoupling and fusion, this invention effectively overcomes the deficiency of highly entangled features in existing ASV risk assessment methods. Specifically, it explicitly decouples three static feature vectors—volume, pitch, and speaking style—from the reference audio, avoiding the prosodic monotony caused by traditional single-timbre embedding. This allows the generated test samples to maintain timbre similarity while possessing richer temporal prosodic variations and acoustic expressiveness, thereby significantly improving the naturalness and realism of the samples.

[0050] Secondly, thanks to the dual injection of the Adaptive Instance Normalization (AdaIN) mechanism into the pronunciation feature prediction network and the speech synthesis backbone, this invention achieves dynamic fusion and modulation of pronunciation features. This mechanism ensures that the generated test samples can accurately replicate the microscopic acoustic texture of the target user (such as fundamental frequency jitter and vocal cord aging features), which not only enhances the sample's ability to reproduce nonlinear vocal cord changes, but also makes it easier for the next-generation integrated forgery detection ASV system to expose the true security boundary.

[0051] Furthermore, to address the blind spots in long-tail group assessment, this invention improves the model's generalization ability to groups such as children and the elderly whose acoustic characteristics deviate from the mainstream distribution by constructing multi-source heterogeneous datasets and using low-rank adaptive (LoRA) fine-tuning technology. This design enables the generated test samples to accurately simulate special features such as high-frequency vocalization and sparse spectrum, thereby covering a more comprehensive demographic dimension in risk assessment and eliminating fairness loopholes.

[0052] Finally, the risk quantification framework based on a fine-grained age-gender cross-assessment matrix can accurately reveal the differential robustness of the ASV system across different groups. For example, experiments show that the system is more vulnerable to groups such as elderly women, but identifying this difference provides a targeted direction for optimization, enhancing the overall practicality and guiding value of risk assessment.

[0053] Through the synergistic effect of the above-mentioned innovations, this invention provides a more reliable and comprehensive assessment tool for the security of ASV systems in real open environments. Attached Figure Description

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0055] Figure 1 This is a schematic diagram illustrating the working mechanism and module structure of an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the process of an embodiment of the present invention;

[0057] Figure 3 This is a basic framework diagram of the system in an embodiment of the present invention. Detailed Implementation

[0058] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0059] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0060] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0061] The development of speech synthesis technology has posed a serious challenge to the security of ASV systems. However, most existing ASV system risk assessment systems rely on single timbre embedding to generate adversarial test samples, resulting in highly entangled features and difficulty in replicating the unique acoustic textures of long-tail groups such as children and the elderly, leading to significant blind spots in the assessment results in terms of demographic dimensions.

[0062] Therefore, the technical problem to be solved by the present invention is:

[0063] 1. Existing methods only focus on the similarity of timbre in the generation of test samples, which lacks expressiveness.

[0064] 2. Existing methods generate test samples with severe feature entanglement, making it difficult to achieve fine-grained control over the sample speech.

[0065] 3. Existing methods have difficulty replicating the pronunciation characteristics of long-tail groups, resulting in blind spots in risk assessment systems for ASV systems.

[0066] To address the aforementioned problems, this invention proposes a risk assessment method and system for ASV systems based on the decoupling and fusion of multidimensional pronunciation representations. It provides a formal definition and effectiveness analysis of the assessment process, and offers a concrete instance implementation. Specifically, to improve the similarity and expressiveness of the generated speech test samples, such as... Figure 3 As shown, this invention constructs a multi-dimensional pronunciation feature extraction network and a pronunciation feature prediction network to extract and fuse the volume, pitch, and speaking style of reference speech. To improve the comprehensiveness of the evaluation results, this invention constructs a dedicated fine-tuning dataset MADS (Multi-Age Diverse Speech) based on multi-source heterogeneous acoustic data, enabling the evaluation model's ability to generate high-fidelity test samples to generalize to groups of different ages and genders. Experimental and theoretical analyses show that this system can accurately quantify the defense effectiveness and potential blind spots of the ASV system when facing highly realistic speech synthesis interference covering all age and gender groups, providing an important reference for evaluating and improving the robustness and security of the ASV system in real-world environments.

[0067] This scheme first constructs a dedicated fine-tuning dataset, MADS, containing multi-source heterogeneous acoustic data, and combines LoRA technology to enhance the model's generalization ability across age and gender features. Secondly, a multi-dimensional articulation feature extraction network explicitly decouples volume, pitch, and speech style vectors from reference audio. Subsequently, the AdaIN mechanism is used to dynamically inject the feature vectors into the articulation feature prediction network and the VAE-GAN synthesis backbone to generate high-fidelity adversarial test samples. This invention provides a general system architecture, a formal definition of the evaluation process, and a detailed instantiation implementation for this risk assessment method. Experimental analysis shows that existing ASV systems exhibit the worst robustness against speech synthesis attacks targeting elderly female users, with their FAR (Failure Rate of Arrival) being on average 18.4% higher than that of the most robust middle-aged and young male users. Compared with existing schemes, this approach can accurately quantify the security vulnerabilities of ASV systems when facing long-tail groups, providing an important reference for assessing and improving the risks of ASV systems in real-world environments.

[0068] The main innovations of this invention lie in addressing two core shortcomings of existing risk assessment methods for ASV systems: first, the feature extraction method based on a single static timbre vector results in monotonous prosody and severe feature entanglement in the generated test samples, making it difficult to detect the true security boundaries of next-generation ASV systems integrating forged speech detection; second, most existing assessment models are only optimized for young and middle-aged groups, lacking sufficient generalization ability on long-tail groups, leading to blind spots in the assessment results. This invention proposes an audio generation architecture based on the decoupling and fusion of multi-dimensional pronunciation representations, constructs a multi-source heterogeneous fine-tuning dataset MADS, and combines LoRA fine-tuning technology to propose the first ASV system risk assessment method covering multiple age and gender groups. Experiments show that this invention only requires inputting reference audio from system users during the inference stage to generate high-fidelity, high-expressive adversarial test samples. It not only achieves a quantitative assessment of the robustness of ASV systems but also accurately reveals the significant security vulnerability of existing systems when facing elderly women, where the risk index is 18.4% higher than that of young and middle-aged men, significantly improving the breadth and depth of ASV system risk assessment.

[0069] Please refer to Figure 1The ASV system risk assessment system based on the decoupling and fusion of multi-dimensional pronunciation representations provided in this embodiment mainly includes an automated testing interface, the ASV system under test, and a core adversarial test sample generation module. The evaluation process begins with the automated testing interface acquiring audio used to simulate the target identity through publicly available channels such as the internet, and preparing verification text for the ASV system under test. Subsequently, the system inputs the reference audio into the model's multi-dimensional pronunciation feature extraction network, which decouples three pronunciation feature vectors representing the user's voice: volume, pitch, and speaking style. Simultaneously, the verification text is converted into phonemes and input into the pronunciation feature prediction network. The three pronunciation feature vectors are injected into this network through the AdaIN (Adaptive Instance Normalization) mechanism, generating a frame-level dynamic pronunciation feature sequence. Next, the feature sequence is input into the VAE-GAN synthesis backbone. Both the prior encoder and the adversarial generator of the synthesis backbone inject the three pronunciation feature vectors through the AdaIN mechanism, and the vocoder ultimately generates high-fidelity test samples. Finally, the automated testing interface submits the generated test samples to the ASV system under test for identity verification testing. The system performs statistical analysis based on the feedback results from the ASV system and the system verification password, calculates security indicators, and finally outputs a quantitative risk assessment report.

[0070] More specifically, Figure 1 The presented ASV system risk assessment system architecture can be divided into three core modules:

[0071] 1. Automated Testing Interface Module

[0072] This module mimics common methods used in speech synthesis attacks. It is responsible for collecting reference audio simulating the target user's identity and user identity audio used to register the ASV system under test from the Internet, and drafting the ASV system verification text. After the test samples are generated, the module will submit the test samples in batches to the ASV system under test for identity verification and receive feedback results, which are ultimately used to generate a risk assessment report.

[0073] 2. Adversarial Test Sample Generation Module

[0074] Preprocessing stage: The reference audio is converted into a Mel spectrogram, and the verification text is converted into a phoneme sequence;

[0075] Multidimensional pronunciation feature extraction network: Receives the Mel spectrogram of the reference audio and decouples it into three pronunciation feature vectors representing the user's identity: volume, pitch, and speaking style. Pitch extraction uses the JDC (Joint Detection and Classification Network) network, while speaking style extraction is implemented by a convolutional enhanced Transformer module and an LSTM module.

[0076] The pronunciation feature prediction network receives the phoneme sequence of the verification text, generates a hidden feature sequence through the text encoder, predicts the phoneme duration through the duration predictor, establishes the alignment relationship between the phonemes and frame-level features through the duration predictor (Aligner), and then injects the above three pronunciation feature vectors through the Conformer Blocks stack and the AdaIN mechanism, finally outputting a frame-level dynamic pronunciation feature sequence.

[0077] VAE-GAN synthesis backbone: The prior encoder predicts latent variables from the frame-level dynamic pronunciation feature sequence. The adversarial generator combines three pronunciation feature vectors injected by AdaIN. The adversarial decoder first generates Mel spectrograms based on the latent variables, and then the HiFi-GAN vocoder generates high-fidelity test samples.

[0078] 3. Tested ASV system module

[0079] Receive test samples submitted by the automated testing interface and complete identity verification, and provide feedback (pass / reject) results and confidence scores to support subsequent calculation of error acceptance rate (FAR) and generation of risk assessment reports.

[0080] Please refer to Figure 2 Corresponding to the above system implementation schemes, this embodiment provides a risk assessment method for ASV systems based on decoupling and fusion of multidimensional pronunciation representations. Its implementation process includes the following steps:

[0081] S101. Construct a test sample generation architecture based on multi-dimensional pronunciation features, which includes a multi-dimensional pronunciation feature extraction network, a pronunciation feature prediction network, and a VAE-GAN synthesis backbone. Then, pre-train it on the LibriTTS dataset to obtain a test sample generation model.

[0082] S102. Construct a dedicated dataset MADS for fine-tuning based on multi-source heterogeneous speech, and use LoRA technology to fine-tune the pre-trained model, and prepare a set of reference audio from users of the ASV system under test.

[0083] S103. Convert the reference audio into a Mel spectrogram and input it into the multidimensional pronunciation feature extraction network of the fine-tuning model to decouple out three pronunciation feature vectors that represent the user's identity.

[0084] S104. The verification text of the tested ASV system is converted into a phoneme sequence and input into the pronunciation feature prediction network of the fine-tuning model to predict the frame-level dynamic pronunciation feature sequence.

[0085] S105. Input the three pronunciation feature vectors and the frame-level dynamic pronunciation feature sequence into the VAE-GAN architecture to synthesize the backbone and generate test samples that are highly similar to the user's voice and have rich expressiveness.

[0086] S106. Using synthetic test samples, initiate authentication queries to the ASV system, and finally generate a risk assessment report based on the authentication query results of a large number of test samples.

[0087] Furthermore, the detailed process of the above steps is as follows:

[0088] In step S101, a test sample generation model based on multi-dimensional pronunciation features is constructed and trained, specifically including:

[0089] First, a large-scale, low-noise, multi-speaker speech dataset with high naturalness is selected for pre-training. This approach uses the LibriTTS dataset, a mainstream dataset in the current speech synthesis field. The training set of this dataset has three sizes; in this embodiment, the train-clean-360 version is selected, containing 360 hours of labeled clean audio from 482 male speakers and 439 female speakers. The model consists of a multi-dimensional pronunciation feature extraction network, a pronunciation feature prediction network, and a VAE-GAN synthesis backbone. The model is then trained for 300 epochs with a batch size of 16. The AdamW algorithm is chosen as the optimizer, supporting multi-GPU collaborative training. The learning rate is... The initial value was 0.005, and it was reduced by 10% after every 30 training rounds. The sampling rate for audio was uniformly set to 16kHz, and the mapping between text and phonemes was implemented using a CMU pronunciation dictionary. The loss function for model training was:

[0090] Total loss Loss due to VAE GAN loss Pronunciation feature prediction loss and feature extractor regularization loss The loss term is composed of various components, each assigned a corresponding weight. (Different indices correspond to different loss types), achieving a balance constraint for multi-objective optimization. Finally, after 300 rounds of training, a pre-trained test sample generation model is obtained.

[0091] In step S102, the dedicated dataset MADS is constructed and fine-tuned, the test sample generation model is fine-tuned, and evaluation materials are prepared, specifically including:

[0092] To address the issue of insufficient generalization ability of general pre-trained models on long-tailed populations, this invention constructs a multi-source heterogeneous acoustic fine-tuning dataset, named MADS. The dataset's data sources consist of three parts: speech samples from the LibriTTS dataset (train-clean-100 version) representing standard young and middle-aged populations; speech samples from the MyST dataset representing children with high fundamental frequency and sparse spectral features; and speech samples from the DementiaBank dataset representing healthy control groups with vocal cord aging texture and frequency perturbation features, representing the elderly population. During dataset construction, stratified sampling was performed within each age segment to ensure statistical balance in both the quantity and duration of male and female samples, thus forming a balanced acoustic feature distribution across multiple age groups and genders.

[0093] After the MADS dataset was constructed, the pre-trained test sample generation model obtained from S101 was fine-tuned for domain adaptation using the low-rank adaptive LoRA technique. Specifically, while keeping most of the backbone parameters of the pre-trained model frozen, trainable low-rank matrices were injected only into the attention mechanism projection layer and convolutional kernel of the multidimensional pronunciation feature extraction network and the VAE-GAN synthesis backbone. The parameters of these low-rank matrices were updated using the constructed MADS dataset, enabling the model to quickly adapt to and capture the fine-grained acoustic representations of long-tailed groups without forgetting general pronunciation knowledge.

[0094] Finally, based on the actual application scenario of the ASV system to be evaluated, a set of reference audio recordings of registered users of the tested system is obtained. To simulate a speech synthesis attack risk environment as realistically as possible, this invention selects user audio from internet channels such as social media. Users are equally divided into three age groups: children, young adults, and the elderly, with a roughly equal gender composition. At least two audio recordings are obtained for each user: one as identity audio used to register with the tested ASV system, and the other as reference audio used to generate test samples. Each reference audio recording is typically no less than 10 seconds long, serving as the sole identity acoustic benchmark for subsequently generating adversarial test samples.

[0095] In step S103, the reference audio is converted into a Mel spectrogram and then input into the model's multidimensional pronunciation feature extraction network to decouple out three pronunciation feature vectors representing the speech identity, specifically including:

[0096] Because Mel spectrograms can more effectively represent audio features and make them easier for neural networks to learn, audio needs to be preprocessed into Mel spectrograms before being input into the multidimensional pronunciation feature extraction network. In the preprocessing stage, the audio is first sampled; in this embodiment, the sampling frequency is 16kHz. Sampling sequence After reducing boundary effects through framing and windowing, a short-time Fourier transform is performed to obtain the linear spectrum. The formalized form of this process is:

[0097]

[0098] in Indicates frame shift, The window length is represented by m, the frame index is represented by k, and the window function is represented by w.

[0099] After performing Mel filtering and logarithmic transformation, the Mel spectrum is obtained. .

[0100] In the multidimensional pronunciation feature extraction network, the volume extraction module first calculates the volume of each frame. The energy, i.e., the L1 norm of the frame vector, is calculated, and then the average energy E of all frames is taken to approximate the audio volume. The formula is:

[0101] in, Indicates the total number of time frames. This refers to the number of Mel frequency channels, typically 80.

[0102] In this embodiment, the pitch extraction module employs the JDC network, which consists of multiple convolutional layers, residual connection layers, and bidirectional LSTM layers. This network extracts time-frequency information layer by layer from the input acoustic features. The convolutional layers capture local spectral patterns and spatial correlations, the residual connection layers alleviate the vanishing gradient problem in deep networks and enhance feature propagation, while the bidirectional LSTM layers model the sequential dependencies of the time series, thereby capturing the dynamic changes in pitch and melody more accurately in the time dimension.

[0103] The speech style extractor consists of six convolutional enhanced Transformer modules and one LSTM module, requiring the results of the first two steps and a Mel spectrogram as input. During training, the network incorporates a feature extractor regularization loss. Constraints are imposed on the training process. The formula is:

[0104]

[0105] in, and This represents the pitch and speaking style vectors extracted by the initial pre-trained model. and This indicates a new prediction result in a certain step. This is the L1 regularization computation symbol.

[0106] In step S104, the prepared ASV system verification text is converted into a phoneme sequence and input into the pronunciation feature prediction network of the fine-tuning model to predict the frame-level dynamic pronunciation feature sequence, specifically including:

[0107] First, the prepared verification text of the ASV system under test is converted into the corresponding phoneme sequence using the CMU pronunciation dictionary. After this preprocessing step, it is input into the text encoder and duration predictor of the pronunciation feature prediction network. The text encoder will... Convert into a hidden feature sequence The encoder consists of an input embedding layer, an encoder backbone composed of a Transformer encoder, and an output projection layer. The duration predictor, composed of two 1D convolutional blocks and an output projection layer, predicts the duration of each phoneme, generating a hard, monotonic alignment. Used in the encoded feature sequence and frame-level features Alignment is established between frames. This alignment functionality is implemented by the aligner module, which is a monotonically surjective function that sets the frame index... Mapping to phoneme feature index and through Upsampling is achieved. This alignment's monotonicity and surjectivity ensure that the text is pronounced in the correct order, without skipping any phonemes. During this process, the synthesized speech rate can be controlled by multiplying the prediction duration by a factor.

[0108] Frame-level features The three pronunciation features, along with those from the multidimensional pronunciation feature extraction network, are fed into the pronunciation predictor, a sequence-to-sequence transformation module whose core architecture consists of a stack of Conformer blocks. The three pronunciation features are then injected into the inner layers of the Conformer stack via the AdaIN mechanism. The AdaIN mechanism uses the three feature vectors to generate adaptive scaling and offset parameters, modulating the frame-level content features being processed by the Conformer block. Finally, an output projection layer transforms the high-dimensional hidden state into a specific number of channels, generating the frame-level dynamic pronunciation feature sequence required by the VAE-GAN network. During training, this pronunciation feature prediction network incorporates a pronunciation feature prediction loss. Constraints are imposed on the training process. The formula is:

[0109] in, This represents the actual duration (dur), volume, pitch, and speaking style sequence extracted from real audio. This indicates the prediction result of the network.

[0110] In step S105, the pronunciation feature vector and frame-level dynamic pronunciation feature sequence are input into the VAE-GAN synthesis backbone to generate test samples that are similar to the user's voice and have rich expressiveness, specifically including:

[0111] During the inference phase, the synthetic backbone consists only of a prior encoder and an adversarial generator, both of which inject three types of articulation feature vectors via the AdaIN mechanism during execution. The prior encoder, derived from the conditional VAE, predicts latent variables from the input frame-level dynamic articulation feature sequence. Adversarial generators, on the other hand, use latent variables. Generate the final test sample speech.

[0112] During the training phase, due to the need for adversarial training, the network consists of four modules: a prior encoder, a posterior encoder, an adversarial generator, and a discriminator. The prior encoder's role remains the same: predicting latent variables. It consists of a Transformer encoder and a normalized stream. The posterior encoder in a VAE reads the real reference audio and encodes it into an ideal latent variable. This module consists of a stack containing 16 non-causal WaveNet residual blocks. The adversarial generator primarily comprises a VAE decoder and a GAN-based generator. This module first processes the latent variables through the adversarial decoder... The decoder generates a Mel spectrogram, which is then used by a HiFi-GAN vocoder to generate the final synthesized audio. The discriminator distinguishes real audio from test samples synthesized by the adversarial generator during training. This module's architecture consists of a HiFi-GAN multi-cycle discriminator composed of a set of sub-discriminators. During training, this loss consists of two parts: VAE loss and GAN loss, with the following formulas:

[0113]

[0114] in, and These are the weighting coefficients of their respective loss formulas. In VAE loss, the first term is the KL divergence loss, and the second term is the Mel-spectrum reconstruction loss. For the divergence loss, This represents the dynamic sequence of pronunciation features output by the pronunciation feature prediction network; This represents the ideal latent variable extracted from real audio; Indicates the prior encoder; This represents the posterior encoder. For the reconstruction loss, This represents a Mel spectrogram extracted from real audio. This represents the generated Mel spectrogram. In the GAN loss, the first term is the adversarial loss, and the second term is the feature matching loss. For the adversarial loss, it uses mean squared error to evaluate the realism of the synthesized audio. Specifically, This represents the discriminator's score for the synthesized audio, and this loss term forces the generator... Generate a discriminator Audio with a score as low as possible of 1. For the feature matching loss, this loss is calculated by plotting the true audio waveform. Audio synthesized with adversarial generator In feature discriminator The L1 distance between the internal feature maps is used to stabilize the training process.

[0115] In step S106, the generated test sample is used to initiate an authentication query to the ASV system to be evaluated, in order to quantitatively evaluate the robustness of the ASV system and generate a risk assessment report, specifically including:

[0116] First, a standardized automated testing interface is constructed to batch input the adversarial test sample set generated in step S105 into the ASV system to be evaluated. For each test sample, the system records the ASV system's feedback result (pass / reject) and confidence score. Second, the core security indicator FAR is calculated based on the test results, which is the proportion of test samples incorrectly identified as legitimate users by the ASV system out of the total number of test samples. Furthermore, to achieve multi-dimensional risk quantification, this invention introduces a six-dimensional cross-evaluation matrix of "age-gender," separately calculating the independent FAR for six subgroups: childhood males, childhood females, middle-aged males, middle-aged females, elderly males, and elderly females. If the FAR of a certain long-tail group is significantly higher than the FAR of the middle-aged male baseline group, the ASV system is determined to have a fairness security vulnerability in that demographic dimension. Finally, a comprehensive risk assessment report is generated by combining the ASV system verification text. The report includes the overall security score of the ASV system, robustness radar charts for each subgroup, spectroscopic analysis of typical successful intrusion samples, and defense recommendations for the discovered vulnerabilities.

[0117] As described above, this invention proposes an adversarial test sample generation architecture based on the decoupling and fusion of multi-dimensional pronunciation representations. Combining LoRA fine-tuning technology with the dedicated dataset MADS, it constructs the first risk assessment method and system for ASV systems across multiple age groups. This invention explicitly decouples volume, pitch, and speaking style features from reference speech to guide VAE-GAN synthesis of high-fidelity test samples, overcoming the shortcomings of existing assessment methods in terms of insufficient generalization ability and feature entanglement in long-tailed populations. This invention provides a formal definition and concrete instantiation of the risk assessment process. Experimental and theoretical analyses show that the test samples generated by this method not only possess high fidelity and expressiveness but also accurately assess the safety boundaries of ASV systems. In particular, fine-grained matrix analysis reveals the objective fact that ASV systems exhibit the worst robustness and significant fairness vulnerabilities when facing elderly women. Compared with existing solutions, this invention significantly improves the breadth and depth of ASV system risk assessment, providing an important reference for assessing and improving the robustness of ASV systems in real-world complex environments.

[0118] Given the significant limitations of existing risk assessment methods for ASV systems—traditional assessment models focus solely on extracting static timbre vectors, resulting in test samples that, while timbre-similar, suffer from monotonous rhythms and severe feature entanglement—these samples are unable to effectively probe the true security boundaries of next-generation ASV systems that integrate spoofing detection. Furthermore, existing assessment methods primarily target young and middle-aged users, making it difficult for system models to replicate the pronunciation characteristics of long-tail groups such as children and the elderly, leading to significant blind spots in the assessment results.

[0119] The embodiments provided above address the aforementioned problems by proposing the first ASV system risk assessment method and system based on the decoupling and fusion of multi-dimensional pronunciation representations, based on the representation and deep fusion of physical pronunciation characteristics. This invention extracts three pronunciation feature vectors from the reference audio and uses these vectors to guide the audio synthesis of the VAE-GAN synthesis backbone. This ensures that the generated test sample speech has similar pronunciation characteristics and natural rhythm to the reference audio, overcoming the shortcomings of existing assessment methods that generate test sample speech with a single style and lack of expressiveness. Simultaneously, the system model is fine-tuned using the MADS dataset based on multi-source heterogeneous speech, improving its generalization ability when dealing with long-tailed groups and addressing the blind spot in risk assessment systems. Therefore, this invention only requires inputting speech segments from the ASV system user reference audio set one by one during the inference stage and testing the ASV system with the output test sample set to effectively assess the risk of the ASV system. Evaluation experiments on four mainstream open-source ASV systems and four closed-source ASV systems show that existing ASV systems exhibit the worst robustness and highest risk when facing speech synthesis attacks targeting elderly female users. Their False Acceptance Rate (FAR) is on average 18.4% higher than that of the most robust young and middle-aged male user group, with a minimum increase of 10.9%. The next highest is the FAR for elderly male users, averaging 16.5% higher. Furthermore, robustness against children is significantly lower than that against young and middle-aged users, averaging 11.7% higher. This invention proposes a general framework for ASV system risk assessment, providing a formal definition and concrete instantiation of its assessment process. Experimental and theoretical analyses demonstrate that this method can generate high-expressiveness and high-fidelity test samples across different age and gender groups. Compared to existing solutions, this significantly improves the breadth and depth of ASV system risk assessment, providing an important reference for evaluating and improving the robustness of ASV systems in real-world complex environments.

[0120] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0121] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0122] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

[0124] This invention is not limited to the above-described preferred embodiments. Anyone inspired by this invention can derive other forms of risk assessment methods and systems for ASV systems based on the decoupling and fusion of multidimensional pronunciation representations. All equivalent changes and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A method for ASV system risk assessment based on multi-dimensional pronunciation representation decoupling and fusion, characterized in that, Comprise: Obtaining the reference audio of the target user, and extracting a volume feature vector representing sound energy distribution, a pitch feature vector representing a base frequency change trajectory, and a speaking style feature vector representing speaking texture characteristics from the reference audio by a multi-dimensional pronunciation feature extraction network; Converting the text to be verified into a phoneme sequence, inputting the phoneme sequence into a pronunciation feature prediction network for predicting a frame-level dynamic pronunciation feature sequence based on the phoneme sequence, and injecting the three pronunciation feature vectors into the network by an adaptive instance normalization mechanism to dynamically modulate the prediction process, so as to generate a frame-level dynamic pronunciation feature sequence containing timing prosody changes; Inputting the frame-level dynamic pronunciation feature sequence into a VAE-GAN synthesis backbone, and again injecting the three pronunciation feature vectors by the adaptive instance normalization mechanism to generate a test speech sample consistent with the voiceprint characteristics of the target user; Using the test speech sample to initiate an identity verification query of an ASV system, and calculating a security index of different user groups according to a verification result to evaluate the risk of the ASV system.

2. The ASV system risk evaluation method based on multi-dimensional pronunciation feature decoupling and fusion according to claim 1, characterized in that: The process of extracting the three pronunciation feature vectors by the multi-dimensional pronunciation feature extraction network comprises: First, pre-processing the reference audio, sampling and framing, windowing, obtaining a linear spectrum by short-time Fourier transform, and converting the linear spectrum into a mel spectrum graph by mel filtering and logarithmic transformation; The multi-dimensional pronunciation feature extraction network comprises: A volume feature extraction submodule: calculating the energy of each frame in the mel spectrum graph, and obtaining a volume feature vector by averaging the energy of all frames; A pitch feature extraction submodule: a convolution-residual-bidirectional recurrent network for pitch extraction is used to extract time-frequency information from the mel spectrum graph layer by layer, and output a pitch feature vector; A speaking style feature extraction submodule: composed of six convolution-enhanced sequence transformation modules for speaking style extraction and a recurrent module, inputting the volume feature vector, the pitch feature vector and the mel spectrum graph, and outputting a speaking style feature vector.

3. The ASV system risk evaluation method based on multi-dimensional pronunciation feature decoupling and fusion according to claim 1, characterized in that: The pronunciation feature prediction network comprises a text encoder, a duration predictor, an aligner module and a sequence transformation module; The process of converting the text to be verified into a phoneme sequence and inputting the phoneme sequence into the pronunciation feature prediction network comprises: Converting the text to be verified into a phoneme sequence by a CMU pronunciation dictionary, inputting the phoneme sequence into a text encoder to obtain a hidden feature sequence, and the text encoder comprises an input embedding layer, a Transformer encoder backbone and an output projection layer; Predicting the duration of each phoneme by a duration predictor to generate a monotonic alignment relationship, and mapping the hidden feature sequence into a frame-level feature by an aligner module; Injecting the three pronunciation feature vectors into the internal layers of the sequence transformation module by an adaptive instance normalization mechanism to dynamically modulate the frame-level feature to adapt to the pronunciation characteristics of the target user, and finally generating a frame-level dynamic pronunciation feature sequence containing timing prosody changes.

4. The ASV system risk assessment method based on multi-dimensional pronunciation feature decoupling and fusion according to claim 1, characterized in that: the VAE-GAN synthesis backbone comprises a prior encoder, an adversarial generator, and a HiFi-GAN vocoder; the process of inputting the frame-level dynamic pronunciation feature sequence into the VAE-GAN synthesis backbone and injecting the pronunciation feature vector comprises: the prior encoder injects three kinds of pronunciation feature vectors through an adaptive instance normalization mechanism to predict latent variables from the frame-level dynamic pronunciation feature sequence; the adversarial generator injects the three kinds of pronunciation feature vectors through an adaptive instance normalization mechanism to decode the latent variables to generate a mel-spectrogram; the HiFi-GAN vocoder converts the mel-spectrogram into a test speech sample in audio form.

5. The ASV system risk assessment method based on multi-dimensional pronunciation feature decoupling and fusion according to claim 1, characterized in that: the security indicator is the proportion of the number of test speech samples incorrectly identified as legitimate users by the ASV system to the total number of test samples; when calculating the security indicators of different user groups, the target users are divided into multiple subgroups according to demographic dimensions, and the independent false acceptance rates of each group are calculated respectively; when assessing the risk of the ASV system, if the false acceptance rate of a certain group is significantly higher than that of a pre-set reference group, it is determined that the ASV system has a security vulnerability in the dimension corresponding to that group, and the risk assessment result is output in the form of a report containing the overall security score, the robustness comparison of each group, and the vulnerability defense suggestions.

6. The ASV system risk assessment method based on multi-dimensional pronunciation feature decoupling and fusion according to claim 1, characterized in that: before extracting the pronunciation feature vector, the method further comprises the step of fine-tuning the test sample generation model for domain adaptation, specifically: construct a multi-source heterogeneous acoustic fine-tuning dataset, which includes at least: young adult group speech, child group speech, and elderly group speech; perform stratified sampling on the samples of each age segment in the dataset to ensure statistical balance in the number and duration of samples of different genders, forming a balanced acoustic feature distribution across age and gender; use low-rank adaptive LoRA technology for model fine-tuning: keep most of the backbone parameters of the test sample generation model frozen, and only inject trainable low-rank matrices into the attention mechanism projection layer and convolution kernel of the multi-dimensional pronunciation feature extraction network and the VAE-GAN synthesis backbone, and update the parameters of the low-rank matrices using the fine-tuning dataset, so that the model can quickly adapt to the acoustic representation of long-tail groups.

7. The ASV system risk assessment method based on multi-dimensional pronunciation feature decoupling and fusion according to claim 1, characterized in that: the reference audio is obtained from Internet public audio resources including social media, and the target users are equally divided into three age groups: children, young adults, and the elderly; each target user obtains at least two audios: one is used to register the ASV system being tested to simulate the real registration process, and the other is used as a reference audio for extracting pronunciation feature vectors, and the duration of each reference audio is not less than a pre-set value.

8. The ASV system risk assessment method based on multi-dimensional pronunciation representation decoupling and fusion according to claim 4, characterized in that: the VAE-GAN synthesis backbone further comprises a discriminator for distinguishing real audio from synthesized test samples; the prior encoder is composed of a Transformer encoder and a normalization flow, and is used to predict latent variables from the frame-level dynamic pronunciation feature sequence after injecting three pronunciation feature vectors through an adaptive instance normalization mechanism; the adversarial generator is composed of a variational autoencoder decoder and a GAN-based generator, and is used to decode the latent variables to generate mel spectrograms after injecting three pronunciation feature vectors through an adaptive instance normalization mechanism; the discriminator is a HiFi-GAN multi-cycle discriminator composed of multiple sub-discriminators, which distinguishes real audio from synthesized test samples in the model training stage.

9. A multi-dimensional pronunciation representation based decoupling and fusion ASV system risk assessment system, characterized in that, comprising: a reference audio processing module for obtaining reference audio of a target user and extracting volume feature vectors representing sound energy distribution, pitch feature vectors representing fundamental frequency change trajectories, and speaking style feature vectors representing voice texture characteristics from the reference audio through a multi-dimensional pronunciation feature extraction network; a dynamic feature prediction module for converting the text to be verified into a phoneme sequence, inputting a pronunciation feature prediction network for predicting a frame-level dynamic pronunciation feature sequence based on the phoneme sequence, and injecting three pronunciation feature vectors into the network through an adaptive instance normalization mechanism to dynamically modulate the prediction process, thereby generating a frame-level dynamic pronunciation feature sequence containing timing and rhythm changes; a test sample generation module for inputting the frame-level dynamic pronunciation feature sequence into the VAE-GAN synthesis backbone and injecting three pronunciation feature vectors again through an adaptive instance normalization mechanism to generate test speech samples consistent with the voiceprint characteristics of the target user; a risk assessment module for initiating an identity verification query of the ASV system using the test speech samples, calculating security indicators for different user groups based on the verification results, and evaluating the risk of the ASV system.

10. A computer device, comprising: comprising a processor and a memory, the memory storing a computer program, the computer program being executed by the processor to implement the ASV system risk assessment method based on multi-dimensional pronunciation representation decoupling and fusion according to any one of claims 1-8.

Images