A deep fake audio identification system based on shanghai dialect

Abstract

The application discloses a deep counterfeit audio identification system based on Shanghai dialect, and belongs to the technical field of audio information processing and security detection. The system first performs data enhancement of the Shanghai dialect specialization on the to-be-tested voice through a multi-source dialect feature extraction module, and extracts multi-source dialect features containing acoustic, prosodic and phonological levels. Then, the multi-source dialect features are input into a self-supervised pre-training model trained by a Shanghai dialect corpus to generate dialect perception embedding features. Next, fusion features are generated through a fusion encoder, and a multi-task adversarial training mechanism is used to jointly optimize the main task of counterfeit detection and the classification auxiliary task for distinguishing the Shanghai dialect from other dialects. When the auxiliary task gradient is back propagated, it is processed through a gradient reversal layer to suppress the interference of the Shanghai dialect category information on the main task, and a speaker feature vector highlighting the feature deviation of the counterfeit is obtained. Finally, a similarity identification module calculates the cosine similarity of the to-be-tested voice feature vector and the target speaker template vector, compares it with a preset threshold, and determines the authenticity of the audio. Through Shanghai dialect adaptation and multi-task adversarial training, the application effectively realizes the decoupling of Shanghai dialect information and counterfeit traces, and significantly improves the identification accuracy and robustness of deep counterfeit audio of the Shanghai dialect.

Description

Technical Field

[0001] The present invention relates to the technical field of audio information processing and security detection, and in particular to a deepfake audio identification system, especially a system for high-precision identification of deepfake Shanghai dialect speech by extracting acoustic, prosodic and phonological features unique to Shanghai dialect, and combining a Shanghai dialect-adapted self-supervised model and a multi-task adversarial training mechanism. Background Art

[0002] With the rapid development of speech synthesis and speech conversion technologies, deepfake speech has expanded from standard Mandarin to various dialects, especially the Wu dialect region represented by Shanghai dialect, which poses a severe challenge to social information security. Existing deepfake speech detection technologies mainly rely on global acoustic statistical features (such as MFCC, LFCC, Mel-spectrogram) or general deep embedding vectors (such as wav2vec 2.0, HuBERT). The training data of these technologies is dominated by standard languages, and lacks explicit modeling for Shanghai dialect variations. As a representative dialect of Wu Chinese, Shanghai dialect has complex phonological features such as checked syllable coda, nasalized vowels, distinction between sharp and rounded initials, and continuous tone sandhi, and these features present acoustic patterns that are completely different from those of Mandarin in speech signals. Studies have shown that current deepfake models have systematic defects in replicating fine-grained variations related to individuals or dialects, but existing detection methods fail to make full use of this weakness.

[0003] To overcome the above drawbacks, some cross-lingual speech processing methods based on large-scale pre-trained models have been proposed in the prior art. For example, Conneau et al. proposed the XLSR model in their paper "Unsupervised Cross-lingual Representation Learning for Speech Recognition" (arXiv:2006.13979, 2020). This technical solution performs multi-lingual joint pre-training on the wav2vec 2.0 model with multi-lingual data to learn cross-lingual speech representations, which significantly improves the speech recognition performance of the model on low-resource languages. Although this solution proves that cross-lingual pre-training can capture shared speech features of different languages, its technical objective is speech recognition rather than security detection. The general cross-lingual features learned by this solution lack explicit discriminative capability for deepfake traces in Shanghai dialect, and cannot cope with deepfake attacks that deliberately imitate Shanghai dialect.

[0004] For example, Chinese invention patent application CN114360560A (publication date: April 15, 2022) discloses a speech authentication method based on multi-task learning. This scheme jointly optimizes the main task of speech authentication and the auxiliary task of speaker identification, aiming to improve feature discriminativeness by utilizing a multi-task learning mechanism. However, this scheme learns speaker identity as a whole category and does not effectively decouple the language form of "Shanghai dialect," which has unique acoustic pattern variations, from individual voiceprint features. When faced with forgery attacks that deliberately imitate the target speaker's Shanghai dialect accent, this scheme is very likely to misjudge the common features of Shanghai dialect as individual identity features, thus allowing forgery traces to be concealed and resulting in missed detections.

[0005] In summary, existing technologies generally have the following limitations:

[0006] 1. Insufficient feature generalization: The detection model is trained based on standard language or general cross-language data, and lacks specialized modeling of the acoustic characteristics of Shanghai dialect such as continuous tone sandhi, entering tone coda, and nasalization, which leads to a significant decrease in the accuracy of identifying fake Shanghai dialect speech.

[0007] 2. Concealed forgery traces: Forgery attacks targeting Shanghai dialect deliberately mimic the characteristics of the target person's Shanghai dialect, making the forgery traces weaker in the global acoustic features. This requires the model to have the ability to separate the commonalities of Shanghai dialect from individual voiceprints and capture subtle forgery deviations.

[0008] 3. Data scarcity: High-quality Shanghai dialect speech datasets with deepfake labels are extremely scarce, making model training difficult and resulting in poor generalization performance.

[0009] Therefore, there is an urgent need for a high-precision deep forgery detection technology that can specifically target the characteristics of the Shanghai dialect and decouple dialect information from individual voiceprints to focus on forgery traces. Summary of the Invention

[0010] This invention aims to provide a deepfake audio detection system based on Shanghai dialect to overcome the shortcomings of existing technologies. By constructing a multi-source feature set sensitive to Shanghai dialect, introducing a self-supervised learning mechanism adapted to Shanghai dialect, and employing a multi-task adversarial training strategy that decouples Shanghai dialect from forged features, this invention achieves high-precision and robust detection of deepfake Shanghai dialect speech.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] This invention provides a deepfake audio detection system based on Shanghai dialect, comprising:

[0013] The voice acquisition module is used to acquire the registered voice of the target speaker and the voice to be tested;

[0014] A multi-source dialect feature extraction module is used to perform data augmentation processing on the speech to be tested, which is specialized in Shanghai dialect, and to extract multi-source dialect features from the processed speech signal; wherein, the multi-source dialect features include at least acoustic-level features, prosodic-level features, and phonological-level features;

[0015] The dialect adaptation coding module integrates a self-supervised pre-trained model adapted and trained with the Shanghai dialect corpus. The dialect adaptation coding module receives the multi-source dialect features and generates dialect-aware embedding features.

[0016] A multi-task adversarial coding module receives the dialect-aware embedding features and includes a fusion encoder, a forgery detection head, a dialect classification head, and a gradient inversion layer. The fusion encoder generates fusion features based on the dialect-aware embedding features. The forgery detection head performs a forgery detection main task based on the fusion features. The dialect classification head performs a classification auxiliary task to distinguish Shanghainese dialect from other dialects based on the fusion features. The gradient inversion layer is set on the gradient backpropagation path between the dialect classification head and the fusion encoder to invert the gradient from the dialect classification head during training, thereby suppressing the interference of Shanghainese dialect category information on the forgery detection main task. This allows the multi-task adversarial coding module to output a speaker feature vector that highlights the bias of forgery features.

[0017] The similarity identification module is used to calculate the similarity between the speaker feature vector of the speech to be tested and the template vector corresponding to the registered speech of the target speaker, and to determine whether the speech to be tested is deepfake audio based on a preset threshold.

[0018] Furthermore, the multi-source dialect feature extraction module includes:

[0019] The data augmentation unit is used to apply general audio enhancement and Shanghai dialect content enhancement to the input Shanghai dialect speech; the general audio enhancement includes noise addition, dynamic range compression and room reverberation simulation, and the Shanghai dialect content enhancement includes enhancing the inherent short tone patterns and continuous tone sandhi curves of the entering tone characters, as well as enhancing the statistical occurrence probability of typical phonemes in Shanghai dialect;

[0020] The feature extraction unit is used to extract acoustic-level features, prosodic-level features, and phonological-level features from the enhanced speech signal;

[0021] The temporal attention pooling fusion processor is used to fuse acoustic-level features and prosodic-level features through an attention mechanism to generate dialect composite features.

[0022] Furthermore, the acoustic-level features include multi-scale fundamental frequency features optimized for the entering tone coda and continuous tone sandhi patterns in Shanghai dialect; the extraction method of the multi-scale fundamental frequency features is as follows: based on continuous wavelet transform, for the tone system of Shanghai dialect, the scale corresponding to the short-term sudden drop tone pattern and continuous tone sandhi sensitive interval of the entering tone character is selected for extraction, so as to capture the fundamental frequency profile and formant profile information that characterize Shanghai dialect.

[0023] Furthermore, the phonological features are posterior features of Shanghai dialect phonemes; the dialect adaptation coding module calculates the posterior probability of the test speech on a predefined set of phonemes containing Shanghai dialect-specific phonemes through a phoneme recognition model pre-trained on multi-dialect data, wherein the Shanghai dialect-specific phonemes include at least entering tone codas and nasalized codas.

[0024] Furthermore, the fusion encoder includes a 1D convolutional layer, a bidirectional GRU network, and a concatenation layer; the 1D convolutional layer is used to perform convolutional dimensionality reduction on the dialect-aware embedded features, the bidirectional GRU network is used to perform sequence modeling on the convolutional features, and the concatenation layer is used to concatenate the dialect comprehensive features generated by fusing the GRU output with the multi-source dialect features in the feature dimension to generate fusion features.

[0025] Furthermore, the similarity identification module includes:

[0026] A template library is used to store the identity template vector of the target speaker, which is obtained by averaging the feature vectors of all registered voices of the target speaker.

[0027] A cosine similarity calculator is used to calculate the cosine similarity between the feature vector of the speech to be tested and the template vector.

[0028] The threshold determiner compares the calculated similarity with a preset threshold and outputs an identification conclusion.

[0029] Furthermore, the gradient inversion layer in the multi-task adversarial coding module keeps the signal unchanged during forward propagation, but multiplies the gradient from the dialect classification head by a negative coefficient during backward propagation, in order to adversarially prevent the fusion encoder from learning the ability to distinguish Shanghai dialect from other dialect categories.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] (1) This invention enhances the Shanghai dialect by specializing in Shanghai dialect, extracts features from multiple dialect sources and adapts to a self-supervised model for Shanghai dialect, enabling the detection model to have the ability to deeply perceive and model unique phenomena such as continuous tone sandhi and entering tone coda in Shanghai dialect. Compared with general models, it significantly improves the accuracy of identifying deep fake audio in Shanghai dialect.

[0032] (2) This invention achieves effective decoupling of Shanghai dialect category information and forgery features through the gradient inversion layer mechanism in multi-task adversarial training, forcing the model to focus on the subtle traces of forgery behavior itself, thereby effectively resisting attacks from various unknown forgery methods and having good adaptability to noise interference in real environment.

[0033] (3) This invention utilizes data augmentation specializing in Shanghai dialect and a contrastive learning mechanism for adapting to Shanghai dialect to learn highly generalized acoustic feature representations from limited Shanghai dialect speech data, effectively alleviating the problem of scarcity of fake Shanghai dialect data. Attached Figure Description

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] Figure 1 The overall flowchart of the deepfake audio identification method based on Shanghai dialect provided in the embodiments of the present invention is shown.

[0036] Figure 2 The system principle diagram of the deepfake audio identification system based on Shanghai dialect provided in the embodiments of the present invention.

[0037] Figure 3 This is a schematic diagram of a multi-source dialect feature extraction module provided in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of a dialect adaptation encoding module provided in an embodiment of the present invention.

[0039] Figure 5 This is a schematic diagram of a multi-task adversarial coding module provided in an embodiment of the present invention.

[0040] Figure 6 This is a schematic diagram of a similarity identification module provided in an embodiment of the present invention.

[0041] List of reference numerals

[0042] 100-Voice Acquisition Module

[0043] 200 - Multi-source dialect feature extraction module; 210 - Data augmentation unit; 220 - Feature extraction unit; 221 - Acoustic-level feature extraction subunit; 222 - Prosodic-level feature extraction subunit; 223 - Phonological-level feature extraction subunit; 230 - Temporal attention pooling fusion unit.

[0044] 300 - Dialect adaptation encoding module, 310 - Self-supervised pre-trained model, 320 - Dialect prototype contrastive loss trainer

[0045] 400-Multi-task Adversarial Encoding Module, 410-Fusion Encoder, 411-1D Convolutional Layer, 412-Bidirectional GRU Network, 413-Concatentation Layer, 420-Spoofing Detection Head, 430-Dialect Classification Head, 440-Gradient Reversal Layer

[0046] 500-Similarity Identification Module, 510-Template Library, 520-Cosine Similarity Calculator, 530-Threshold Decisor Detailed Description of Embodiments

[0047] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further elaborated below with reference to specific drawings.

[0048] Aiming at the technical problems in the prior art such as "insufficient feature generalization", "hidden spoofing traces" and "lack of training data" when detecting deepfake audio of Shanghai dialect, the present invention provides a deepfake audio identification scheme based on Shanghai dialect, so as to systematically solve these technical problems.

[0049] In the solution of the present invention, the "Shanghai dialect-specific data augmentation" strategy is adopted to expand the diversity of samples, and multi-level "multi-source dialect features" from acoustics, prosody to phonology are constructed to fully characterize the characteristics of Shanghai dialect. In particular, in acoustic-level feature extraction; a multi-scale Shanghai dialect fundamental frequency feature based on continuous wavelet transform is innovatively introduced, and targeted extraction is performed on the scale of the tone-sensitive interval of Shanghai dialect. For example, key extraction is performed on the short-time sudden drop tone pattern of checked syllables in Shanghai dialect (such as "one, six, hundred, ten") and the continuous tone sandhi mode of two-character and three-character groups, so that the model can capture much finer tone contour information than traditional methods.

[0050] In the solution of the present invention, the self-supervised pre-trained model is also adapted to Shanghai dialect by introducing "dialect prototype contrast loss", so that the model can learn the acoustic patterns unique to Shanghai dialect and generate more discriminative "dialect-aware embedded features".

[0051] In the solution of the present invention, a "multi-task adversarial training" framework is also innovatively introduced, and the gradient reversal layer is used to force the encoding network to learn the feature representation that is "insensitive to Shanghai dialect categories but highly sensitive to spoofing traces" during the optimization of the main task of spoofing detection and the auxiliary classification task for distinguishing Shanghai dialect from other dialects.

[0052] Through the organic combination of the above technologies, the present solution can accurately capture the subtle deviations introduced by the deepfake process that are independent of the common features of Shanghai dialect in individual voiceprint comparison, thereby achieving high-precision identification.

[0053] Referring to Figure 1The diagram shows the overall flowchart of the deepfake audio identification method based on Shanghai dialect, which is based on the above-mentioned scheme of the present invention.

[0054] This deepfake audio authentication method achieves a complete process from original voice input to final authenticity verification through the coordinated interaction of the following steps S1 to S5.

[0055] Step S1: Voice acquisition step.

[0056] This step provides the analysis object and comparison benchmark for the entire identification process. When conducting audio identification, it is first necessary to obtain the test speech to be identified, as well as several pre-registered, confirmed authentic Shanghai dialect speech recordings of the target speaker to whom the speech claims to belong, as its registered speech. These registered speech recordings are used to construct a benchmark template representing the speaker's Shanghai dialect and voiceprint identity characteristics, serving as the basis for subsequent similarity comparisons.

[0057] Step S2: Dialect feature extraction and enhancement.

[0058] This step first applies data augmentation processing to the test speech, simulating real-world environmental variations and Shanghai dialect-specific variants, thereby expanding the effective distribution space of the samples. Then, multi-source dialect features, encompassing acoustic, prosodic, and phonological levels, are extracted from the augmented signal. Based on these features, the unique properties of the Shanghai dialect can be fully captured from multiple dimensions, including physical attributes, rhythmic patterns, and phoneme systems, providing information-rich input for subsequent network processing. This step addresses the problems of scarce fabricated Shanghai dialect data and insufficient feature representation capabilities.

[0059] Step S3: Dialect perception coding step.

[0060] This step involves a second training of a self-supervised model, pre-trained on a massive amount of general speech data, on a Shanghai dialect dataset—a process known as Shanghai dialect adaptation. By introducing a specially designed contrastive learning objective, this step forces the model to bring closer speech features belonging to the Shanghai dialect and push away features from other dialects within its internal embedding space. As a result, the model's output feature vector (i.e., Shanghai dialect-perceptual embedding features) is exceptionally sensitive to deviations from the Shanghai dialect. This step enables a general acoustic model to acquire a deep perception capability of variations in the Shanghai dialect.

[0061] Step S4: Multi-task adversarial coding steps.

[0062] This step, on top of the encoder that generates the final speaker feature vector, sets up two tasks simultaneously: a primary task (binary classification of true and false information) and an auxiliary task (classification of Shanghainese dialect). Then, through a gradient inversion layer, the learning of the auxiliary task has a "reverse effect" on the encoder during backpropagation; that is, the encoder is trained to perform well in the primary task but not in the auxiliary task. This adversarial mechanism forces the encoder to output a feature "invariant" to the Shanghainese dialect category, thus successfully decoupling the commonalities of Shanghainese dialect from individual forgery traces. This step effectively removes the commonalities of Shanghainese dialect from the features, allowing the model to focus on the traces introduced by the forgery behavior itself, thus addressing the existing problem of "concealed forgery traces."

[0063] Step S5: Similarity identification step.

[0064] This step first utilizes all registered voice recordings of the target speaker and, through steps S1 to S4, generates a template vector representing their identity. Then, it calculates the cosine similarity between the feature vector of the test voice and the template vector. Finally, it compares the calculated similarity with a preset threshold; if the similarity is higher than the threshold, it is identified as the same speaker (not a forgery); otherwise, it is identified as a deepfake. This step provides a quantitative identification conclusion based on the spatial distance between speaker feature vectors.

[0065] The following section elaborates on the specific implementation details of the deepfake audio identification method provided by this invention.

[0066] Step S1: Obtain the registered voice of the target speaker and the voice to be tested.

[0067] During the identification phase, the test speech to be identified is first received via an interface or read from a database, along with several pre-registered, verified authentic Shanghai dialect speech recordings of the target speaker to whom the speech claims to belong, as its registered speech. These registered speech recordings are used in step S5 to construct a baseline template representing the speaker's Shanghai dialect and voiceprint identity characteristics.

[0068] Step S2: Perform data augmentation processing on the speech to be tested to specialize in Shanghai dialect, and extract multi-source dialect features from the processed speech signal.

[0069] This step is specifically used to expand the sample diversity from the limited Shanghai dialect speech and to provide a feature base that can comprehensively characterize the features of the Shanghai dialect for subsequent analysis. Its implementation is detailed below through sub-steps S21 and S22.

[0070] Sub-step S21: Data augmentation for Shanghai dialect specialization.

[0071] The data enhancement implemented in the present invention is different from general noise addition and speed variation enhancement, and is specifically designed based on the characteristics of Shanghai dialect. This sub-step performs the following enhancement operations on the input Shanghai dialect speech:

[0072] General audio enhancement: Noise addition, dynamic range compression and room reverberation simulation are applied to the input Shanghai dialect speech to enhance the robustness of the model against environmental noise. By way of example, during implementation, the parameters can be set as follows: the speed perturbation range is 0.9 times to 1.1 times, the signal-to-noise ratio (SNR) of the added noise is between 5dB and 20dB, and convolution reverberation is performed using a variety of room impulse responses.

[0073] Shanghai dialect content enhancement: For Shanghai dialect, the inherent pitch curve is enhanced, for example, minor perturbation and reinforcement are performed on the short-time sudden falling tone pattern of typical entering tone characters in Shanghai dialect (such as "yi (one), liu (six), bai (hundred), shi (ten)"), as well as the continuous tone sandhi modes of two-character and three-character groups; and the statistical occurrence probability of typical Shanghai dialect phonemes is enhanced. In one implementation, non-speaker-specific speech synthesis or voice conversion algorithms are used to specifically generate training samples that contain more typical Shanghai dialect phoneme variants and have more prominent characteristics of the Shanghai dialect. This enhancement method not only expands the data volume, but also strengthens the model's learning of the key features of Shanghai dialect.

[0074] The operation of this sub-step is completed in the data enhancement unit 210 in the multi-source dialect feature extraction module 200.

[0075] Sub-step S22: Extract multi-source dialect features.

[0076] This sub-step extracts three levels of features from the enhanced speech signal to form multi-source dialect features. In this step, multi-source acoustic features containing Shanghai dialect information as well as prosodic, rhythmic, and phonological features are specifically extracted from the speech signal; meanwhile, the formed multi-source dialect features at least include: original waveform features, multi-scale fundamental frequency features of Shanghai dialect, multi-scale formant features of Shanghai dialect, phoneme posterior features of Shanghai dialect, and general deep features.

[0077] Specifically, extraction of acoustic-level features: After pre-emphasis and framing (window length 25ms, frame shift 10ms) are performed on the input original waveform x(t), this step specifically extracts the following series of fine acoustic features, including:

[0078] Fundamental frequency features: Parselmouth or Praat software is used to extract the fundamental frequency (F0) trajectory, and statistics such as the average fundamental frequency of Shanghai dialect, the variance of the fundamental frequency of Shanghai dialect, and slope difference are calculated.

[0079] Formant features: The first to fourth formants (F1-F4) are extracted using the LPC spectrogram of Soundprint or iFlytek software, so as to obtain the spatial distribution differences of Shanghai dialect vowels.

[0080] Mel-frequency cepstral coefficients: the Librosa MFCC extraction tool is used to extract MFCC features reflecting the regional variation characteristics of vocal tract shape of Shanghainese.

[0081] Mel spectrogram: STFT and Mel filter banks are used to extract the unique spectral texture pattern of Shanghainese.

[0082] Extralinguistic features: the openSMILE tool is used to extract features reflecting differences in emotion, rhythm and speech rate of Shanghainese.

[0083] In this step, the extracted acoustic hierarchical features are spliced according to the time dimension to obtain the Shanghainese acoustic feature tensor F_acoustic, which belongs to the real number space , wherein T is the number of time frames, and D_acoustic is the feature dimension. When 13-dimensional static MFCC features and their first-order and second-order differences are extracted, D_acoustic can reach 105 dimensions.

[0084] Among the above processes, this step also adopts a multi-scale extraction method for fundamental frequency features of Shanghainese. The method of this step does not directly use the original fundamental frequency curve, but on the basis of continuous wavelet transform, focusing on the tone system of Shanghainese, selects scales corresponding to the short-time sudden-falling tone pattern of checked syllables and the sensitive interval of continuous tone sandhi for key extraction. For example, for typical checked syllables in Shanghainese (such as "一, 六, 百, 十"), small scales sensitive to short-time sudden-falling fundamental frequency changes are selected; while for the unique two-syllable and three-syllable continuous tone sandhi patterns of Shanghainese, large scales that can capture longer time-domain fundamental frequency fluctuations are selected. This method can accurately capture the unique fundamental frequency contour and formant contour information of Shanghainese, and its fineness is much higher than that of traditional methods.

[0085] Extraction of prosody-rhythm-phonology level features: for specific Shanghainese, this step specifically extracts the following prosody-rhythm-phonology (PRA) features:

[0086] Tone pattern histogram: each frame of fundamental frequency (F0) is quantized, and a 5-bin tone distribution histogram is constructed to capture tone variation of Shanghainese.

[0087] Syllable duration distribution: syllable boundaries are obtained through forced alignment, and statistics such as mean, standard deviation and skewness of syllable duration are calculated.

[0088] Pause pattern: silent segments (less than 300ms) are detected, and the frequency and position distribution of pauses are counted.

[0089] Phoneme substitution entropy: by comparing the pronunciation dictionaries of Shanghainese and standard language, the entropy of substitution probability is calculated to reflect phonological variation.

[0090] These PRA features constitute the phonological feature vector f_pra of the Shanghai dialect, which belongs to the real number space. Where D_pra is the feature dimension, for example, 32 dimensions.

[0091] Posterior feature extraction of Shanghai dialect phonemes: This step uses a phoneme recognition model pre-trained on multiple Shanghai dialect datasets to calculate the posterior probability of the test speech on a predefined set of phonemes specifically for the target Shanghai dialect, and forms a probability vector sequence. For example, unique phonemes in Shanghai dialect, such as entering tone codas and nasalized finals, are included in this phoneme set. This feature directly reflects whether the pronunciation action conforms to the norms of the target Shanghai dialect at the phonological level, and is particularly effective in detecting forged speech that imitates the overall timbre but deviates in phoneme details.

[0092] Sub-step S23: Feature fusion to form comprehensive features of Shanghai dialect.

[0093] To effectively integrate features from different dimensions, this step employs a temporal attention pooling mechanism. This mechanism allows the model to dynamically focus on segments of speech that are more critical to the identification task, such as syllables carrying complex tone changes. The frame-level acoustic feature F_acoustic is fused with the sentence-level PRA feature f_pra to obtain a fixed-dimensional dialect comprehensive feature Z_dialect. Specifically, this calculation process can be implemented using the following formula:

[0094] ;

[0095] ;

[0096] Where W_a and W_h are weight matrices, b_h and b_a are bias vectors, and F_acoustic, t is the feature vector of F_acoustic in frame t. Through this step, the model can adaptively focus on the speech segments most relevant to Shanghai dialect identification and output a compact 256-dimensional feature vector.

[0097] Step S3: This step inputs the multi-source dialect features into a self-supervised pre-trained model adapted to Shanghai dialect for encoding, generating dialect-aware embedding features.

[0098] This step enables the general pre-trained model to be sensitive to variations in the Shanghai dialect, thereby improving identification accuracy. This step can be implemented through the following sub-steps:

[0099] Sub-step S31: Construct a self-supervised pre-trained model adapted to Shanghai dialect.

[0100] This approach is based on a wav2vec 2.0 Large model pre-trained on large-scale general speech data. This model has 24 Transformer layers with a hidden layer dimension of 1024. To adapt it to the Shanghai dialect scenario, this approach involves continuous pre-training on a Shanghai dialect dataset, i.e., Shanghai dialect adaptation.

[0101] This adaptation process specifically introduces dialect prototype contrast loss for model fine-tuning. Specifically, this scheme initializes a learnable prototype vector for Shanghainese and other major dialects (such as Mandarin, Cantonese, and Sichuanese). During training, the model is optimized so that its extracted quantized features are closer to the prototype vector of their respective dialect in the embedding space, while being farther away from the prototype vectors of other dialects. The complete formula for calculating this loss function is as follows:

[0102] ;

[0103] Where q_i represents the quantized feature of the i-th sample, c_{d(i)} represents the prototype vector corresponding to the Shanghai dialect category d(i) to which the sample belongs, c_k represents the prototype vector of the k-th dialect category, sim(·, ·) represents the similarity calculation function (such as cosine similarity), and tau represents the temperature coefficient, with a default value of 0.1. K is the total number of dialect categories. The prototype vectors are dynamically updated during training using methods such as exponential moving average.

[0104] Sub-step S32: Generate dialect-aware embedding features.

[0105] After the above Shanghai dialect adaptation training, the parameters inside the pre-trained model have formed a specific preference for Shanghai dialect features.

[0106] This scheme extracts the output of a certain intermediate layer (e.g., the 12th layer) of the Transformer as the dialect-aware embedding feature h_dae, which belongs to the real number space R^(T × 1024), i.e. This feature not only contains general acoustic information, but more importantly, its spatial structure is aligned with the specific acoustic patterns of the Shanghai dialect, making it extremely sensitive to subtle deviations in the Shanghai dialect. This characteristic lays the foundation for the accurate capture of forgery traces in the subsequent step S4.

[0107] Step S4: Based on the dialect-aware embedding features, generate fusion features through a fusion encoder, and process them using a multi-task adversarial training mechanism to obtain speaker feature vectors that highlight forgery feature bias.

[0108] This step, by constructing an innovative adversarial training mechanism, forces the model to decouple Shanghai dialect from forgery features, thereby solving the problem of "concealed forgery traces".

[0109] Specifically, this step can be achieved through the following sub-steps:

[0110] Sub-step S41: Feature fusion.

[0111] This scheme fuses the dialect synthesis feature Z_dialect obtained in step S2 with the dialect-aware embedding feature h_dae obtained in step S3. This fusion task is completed by the fusion encoder 410. Figure 5 As shown, the fusion encoder 410 contains a 1D convolutional layer 411 (with a kernel size of 5, a stride of 2, and 512 output channels), a bidirectional GRU network 412 (with 256 hidden units), and a splicing layer 413.

[0112] Accordingly, the fusion encoding process is as follows: the encoder first performs dimensionality reduction on the dialect-aware embedding feature h_dae through a 1D convolutional layer 411, and then feeds it into a bidirectional GRU network 412 for sequence modeling; finally, the encoder concatenates the output of the GRU with the dialect comprehensive feature Z_dialect in the feature dimension through a concatenation layer 413, and obtains the fusion feature h_fusion through a fully connected layer, with a feature dimension of 512. This structural design allows the fusion feature to include both the deep acoustic representation from the self-supervised model and the explicit physical meaning of the manually designed features.

[0113] Sub-step S42: Multi-task adversarial training.

[0114] Building upon the fusion encoder 410, this solution further sets up two task branches in parallel: a main task branch and an auxiliary task branch, such as... Figure 5 As shown:

[0115] The main task branch includes a fake detection head 420, whose input is the fusion feature h_fusion, which is passed through a multilayer perceptron (MLP) and a sigmoid activation function, and outputs the binary classification probability p_fake of "true / false".

[0116] The auxiliary task branch contains a dialect classification head 430, whose input is the fusion feature h_fusion, which is passed through another MLP and Softmax activation function and outputs the probability p_dialect of its dialect category (Shanghai dialect and other dialects).

[0117] The adversarial training mechanism constructed in this step specifically includes inserting a gradient inversion layer 440 before the gradient backpropagation of the auxiliary task (Shanghai dialect classification) to the shared fusion encoder 410. During forward propagation, the gradient inversion layer 440 is equivalent to an identity mapping, and the signal passes through without change; during backpropagation, the gradient inversion layer 440 multiplies the gradient from the dialect classification head 430 by a negative coefficient (-lambda) before passing it to the fusion encoder 410.

[0118] Specifically, the complete multi-task loss function of this mechanism is expressed as follows:

[0119] l_total=l_fake+∝l_dialect+β l_adv ;

[0120] Where L_fake is the binary cross-entropy loss for forgery detection, L_dialect is the cross-entropy loss for dialect classification, and L_adv represents the adversarial loss term after passing through the gradient reversal layer 440. alpha and beta are hyperparameter weights that balance the contributions of each loss term.

[0121] Furthermore, during end-to-end training, this adversarial mechanism forces the fusion encoder 410 to learn a special feature representation. This representation must accurately distinguish between "true" and "false" on the primary task (ensuring L_fake is minimized), while simultaneously failing to effectively distinguish between "Shanghai dialect categories" on the auxiliary task (leading to the maximization of L_dialect, i.e., minimizing the L_adv term in L_total). The ultimate effect is that the model successfully extracts common information about the Shanghai dialect from its features, resulting in a final speaker feature vector that focuses on subtle deviations related to the forgery behavior itself, independent of the Shanghai dialect category—achieving "Shanghai dialect invariance." This mechanism significantly enhances the model's ability to capture hidden forgery traces.

[0122] Step S5: Calculate the similarity between the speaker feature vector of the speech to be tested and the template vector corresponding to the registered speech of the target speaker, and determine whether the speech to be tested is deepfake audio based on a preset threshold.

[0123] This step makes the final determination based on the speaker feature vector extracted in the previous step, which has been stripped of the common features of the Shanghai dialect.

[0124] The specific implementation plan for this step is as follows:

[0125] First, this invention constructs a feature library. All M registered voice recordings from the target speaker's registration database are obtained. Through the aforementioned steps S1 to S4, M feature vectors are extracted, and their average value is calculated, serving as the identity template vector for the target speaker. .

[0126] Subsequently, the feature vector of the speech to be tested is extracted. Template vector of the target speaker Then, cosine similarity is calculated, which is achieved through the following formula:

[0127] ;

[0128] The cosine similarity value ranges from [-1, 1]. The closer the value is to 1, the more consistent the two vectors are in direction, meaning the more similar the speech features are.

[0129] Finally, an identification judgment is made based on a preset high-confidence threshold. For example, based on experimental data, a preset high-confidence threshold is set, such as 0.90. If the calculated similarity is greater than or equal to 0.90, it is identified as the same speaker, and the test speech is not forged; if the similarity is less than 0.90, it is considered that the test speech has significant differences in features, and it is identified as deepfake audio.

[0130] The deepfake audio identification method based on Shanghai dialect proposed in this invention can be configured into a corresponding software program to form a deepfake audio identification system based on Shanghai dialect. When running, this software program will execute the aforementioned deepfake audio identification method based on Shanghai dialect and store it in a corresponding storage medium for the processor to retrieve and execute.

[0131] See Figure 2 The deepfake audio identification system based on Shanghai dialect provided in this invention consists of five components working together: a voice acquisition module 100, a multi-source dialect feature extraction module 200, a dialect adaptation encoding module 300, a multi-task adversarial encoding module 400, and a similarity identification module 500.

[0132] The voice acquisition module 100 runs at the system front end and is configured to collect or receive voice data input from external sources, providing raw materials for the entire identification process. This module is equipped with an input interface to acquire the registered voice and the voice to be tested of the target speaker and transmit the data to downstream modules.

[0133] The multi-source dialect feature extraction module 200 is connected after the speech acquisition module 100, such as... Figure 3 As shown, this module is configured to perform two operations:

[0134] First, data augmentation for Shanghai dialect specialization is performed through data augmentation unit 210 to simulate real environment and fake variations, thereby expanding the sample distribution;

[0135] Secondly, the feature extraction unit 220 extracts multi-level features at the acoustic, prosodic, and phonological levels from the processed signal, and then fuses them through the temporal attention pooling fusion unit 230, providing a rich information foundation for subsequent Shanghai dialect modeling.

[0136] The multi-source dialect feature extraction module 200 can effectively solve the problems of scarce Shanghai dialect data and insufficient feature representation based on these two operations.

[0137] The dialect adaptation encoding module 300 receives the output of the multi-source dialect feature extraction module 200, enabling the system to possess a deep perception capability for variations in the Shanghai dialect. This module integrates a specially trained pre-trained model that transforms the input acoustic features into a "dialect-aware embedding feature." This embedding feature is, in principle, extremely sensitive to deviations from the target Shanghai dialect, creating conditions for capturing subtle forgery traces.

[0138] The multi-task adversarial coding module 400 is the core of the system. It is configured to interact with the dialect adaptation coding module 300, receiving dialect-perceived embedded features and extracting common Shanghai dialect information from these features to highlight traces introduced by the forgery behavior itself. Internally, the multi-task adversarial coding module 400 constructs an adversarial game mechanism. Above the encoder that generates the final speaker feature vector, two tasks are simultaneously set: a primary task (true / false binary classification) and an auxiliary task (classification of Shanghai dialect with other dialects). Then, through a gradient inversion layer, the learning of the auxiliary task has a "reverse effect" on the encoder during backpropagation. That is, the encoder is trained to perform the primary task well but cannot perform the auxiliary task. This causes the encoder to output a feature that is "invariant" to the Shanghai dialect category, capable of distinguishing between true and false but not between different Shanghai dialect categories, thus successfully decoupling the commonalities of Shanghai dialect from individual forgery traces.

[0139] The similarity identification module 500 is located at the end of the system and is used to output the final quantitative identification conclusion. This module is configured to manage and calculate the identity template vector of the target speaker, and perform spatial distance measurement (cosine similarity) with the feature vector of the speech to be tested. Finally, by comparing with a preset threshold, it outputs the judgment result of "real" or "forged".

[0140] The implementation schemes of each module in the system solution will be further explained below, using specific technical means.

[0141] In some embodiments, the voice acquisition module 100 in the system is technically implemented as a configurable data interface. In the registration phase, this module is responsible for collecting multiple pieces of real Shanghainese speech of a specific target speaker, formatting them and storing them in a registered speech database. In the verification phase, this module receives the speech stream or file to be verified that is claimed to originate from the target speaker, and sends it to subsequent processing modules without damage.

[0142] In some embodiments, see Figure 3 , the multi-source dialect feature extraction module 200 in the system is specifically composed of a data enhancement unit 210, a feature extraction unit 220, and a temporal attention pooling fusion module 230.

[0143] Wherein, the data enhancement unit 210 is configured to perform Shanghainese-specific enhancement processing on the speech to be tested, so as to enhance the robustness and generalization ability of the model. The unit is internally configured to perform two enhancement methods:

[0144] The first is general audio enhancement, which includes applying speed perturbation to the input waveform (the perturbation range is 0.9 times to 1.1 times the original speed), adding background noise (the signal-to-noise ratio SNR is set in the range from 5dB to 20dB), and reverberation simulation (implemented by convolution of room impulse response);

[0145] The second is Shanghainese content enhancement, which includes targeted perturbation to the checked tone patterns (such as the short-time abrupt drop tone patterns of "一, 六, 百, 十") and continuous tone sandhi modes of Shanghainese, and uses a non-specific speaker speech synthesis algorithm to generate samples containing typical Shanghainese phonemes (such as checked tone codas and nasalized rhymes) to enrich Shanghainese expressions.

[0146] The feature extraction unit 220 is configured to extract three levels of multi-source features from the enhanced signal. The feature extraction unit 220 specifically includes three sub-units: an acoustic-level feature extraction sub-unit 221, a prosodic-level feature extraction sub-unit 222, and a phonological-level feature extraction sub-unit 223, which extract three levels of multi-source features from the enhanced signal through these three sub-units.

[0147] Specifically, first, the acoustic-level feature extraction sub-unit 221 extracts acoustic-level features, including but not limited to: fundamental frequency trajectories and statistics extracted by the Parselmouth tool, the first four formants obtained from LPC spectrograms, MFCC and Mel spectrograms extracted by the Librosa tool, and out-of-context features extracted by the openSMILE tool. On this basis, all acoustic features are concatenated along the time axis to form an acoustic feature tensor with a dimension of D_acoustic Where T is the number of frames, and D_acoustic can reach 105. In particular, the extraction of fundamental frequency features adopts continuous wavelet transform, and the scale corresponding to the short-term sudden drop tone pattern and continuous tone sandhi sensitive interval of the Shanghai dialect is selected for fine analysis to obtain multi-scale Shanghai dialect fundamental frequency features.

[0148] Secondly, prosodic-rhythmic-phonological features are extracted by the prosodic-level feature extraction subunit 222, including a tone pattern histogram with five levels of quantization of the fundamental frequency, syllable duration statistics (mean, standard deviation, skewness) based on forced alignment, frequency and position distribution of pause patterns (detecting silent segments with a duration of less than 300ms), and phoneme substitution entropy calculated using a Shanghai dialect pronunciation dictionary. These features constitute a fixed-dimensional PRA feature vector. ,in .

[0149] Next, the phonological feature extraction subunit 223 extracts phonological features, that is, it uses a phoneme recognizer pre-trained across Shanghai dialects to calculate the posterior probability vector sequence of the speech to be tested on a predefined set of Shanghai dialect phonemes (which at least includes entering tone codas and nasalized codas).

[0150] The temporal attention pooling fusion unit 230 is configured to fuse frame-level acoustic features with sentence-level PRA features. Its computation of attention weights a_t and dialect synthesis features Z_dialect is implemented through the following computational model:

[0151] ;

[0152] .

[0153] Therefore, the module outputs a fixed-dimensional dialect comprehensive feature Z_dialect, such as a 256-dimensional feature vector. This design allows the model to dynamically focus on speech segments that carry important Shanghai dialect information, improving the efficiency of feature representation.

[0154] In some implementations, see Figure 4 The dialect adaptation coding module 300 in this system is mainly composed of a self-supervised pre-trained model 310 and a dialect prototype contrastive loss trainer 320.

[0155] Among them, the self-supervised pre-trained model 310 is a wav2vec 2.0 Large model adapted for Shanghai dialect. The Shanghai dialect adaptation training of this model is fine-tuned by the dialect prototype contrastive loss trainer 320 using dialect prototype contrastive loss, making it sensitive to variations in the Shanghai dialect. The loss function used during training is:

[0156] ;

[0157] The meanings of the symbols are the same as those mentioned above.

[0158] After this training, the model's forward inference process can transform the input acoustic features or waveforms into dialect-aware embedded features h_dae, with a dimension of T × 1024. The technical advantage of this embedded feature is that it is highly sensitive to the acoustic and prosodic norms of the target Shanghai dialect, and can effectively amplify subtle acoustic traces generated during the forgery process.

[0159] The multi-task adversarial coding module 400 in this system receives h_dae from the dialect adaptation coding module 300 and Z_dialect from the multi-source dialect feature extraction module 200, performs feature fusion based on these, and processes them using a multi-task adversarial training mechanism to obtain speaker feature vectors that highlight forgery feature bias.

[0160] See Figure 5 In some implementations, the multi-task adversarial coding module 400 is mainly composed of a fusion encoder 410, a forgery detection head 420, a dialect classification head 430, and a gradient inversion layer 440.

[0161] The fusion encoder 410 is configured to generate fused features based on dialect-aware embedding features. Specifically, the fusion encoder 410 internally includes a 1D convolutional layer 411 (kernel size 5, stride 2, output channels 512), a bidirectional GRU network 412 (hidden units 256), and a concatenation layer 413. When performing feature fusion, the fusion encoder 410 first performs convolutional dimensionality reduction and sequence modeling on the received h_dae using the 1D convolutional layer 411. Then, the output of the bidirectional GRU network 412 is concatenated with Z_dialect along the feature dimension of the concatenation layer 413. Finally, a fully connected layer is used to generate the fused feature h_fusion (512 dimensions).

[0162] The forgery detection head 420 serves as the main task branch, configured to perform the main forgery detection task based on the fused features. Specifically, the forgery detection head 420 consists of a multilayer perceptron (MLP) and a sigmoid function, and interacts with the fusion encoder 410. It can receive the fusion features h_fusion generated by the fusion encoder 410, process them, and output a binary classification probability p_fake for "true / false".

[0163] The dialect classification head 430 serves as an auxiliary task branch, performing a classification auxiliary task to distinguish Shanghainese dialect from other dialects based on the fusion features. Specifically, the dialect classification head 430 is composed of another MLP and a Softmax function (different from the multilayer perceptron (MLP) and sigmoid function in the forgery detection head 420), and interacts with the fusion encoder 410, receiving the fusion features h_fusion generated by the fusion encoder 410, processing them, and outputting the dialect category probability p_dialect.

[0164] The gradient inversion layer 440, a key component of the multi-task adversarial coding module 400, is positioned on the gradient backpropagation path between the dialect classification head 430 and the fusion encoder 410. It inverts the gradients from the dialect classification head during training to suppress interference from Shanghai dialect category information on the forgery detection main task, thereby enabling the multi-task adversarial coding module to output speaker feature vectors that highlight forgery feature bias.

[0165] The gradient inversion layer 440 is configured such that the signal remains unchanged during forward propagation, and during backward propagation, the gradient of the dialect classification loss transmitted from the dialect classification head 430 is multiplied by a negative coefficient -lambda and then passed to the fusion encoder 410, thereby adversarially preventing the fusion encoder 410 from learning the ability to distinguish Shanghai dialect from other dialect categories.

[0166] The entire module minimizes the total loss function during training:

[0167] l_total=l_fake+∝l_dialect+β l_adv .

[0168] Through this mechanism, the fusion encoder 410 is forced to output a speaker feature vector that has been stripped of the commonalities of Shanghainese dialect and only represents traces of forgery. This significantly improves the system's robustness in detecting forgery attacks that imitate Shanghainese dialect.

[0169] Furthermore, during the inference phase, the gradient inversion layer 440 does not operate, and the encoder directly outputs the trained, clean feature vector.

[0170] In some implementations, see Figure 6 The similarity identification module 500 in this system is configured to receive the speech feature vector V_test output by the multi-task adversarial coding module 400. Internally, it maintains a template library 510, a cosine similarity calculator 520, and a threshold determiner 530.

[0171] The template library 510 in this module stores the identity template vector V_template of each target speaker. This template vector is obtained by averaging the feature vectors of all registered speech of the speaker.

[0172] During authentication, the cosine similarity calculator 520 is configured to calculate the cosine similarity between the received voice feature vector to be tested V_test and the corresponding identity template vector V_template:

[0173] ;

[0174] and send the result to the threshold determiner 530, which compares the received result with a preset threshold (e.g., 0.90):

[0175] if the similarity is greater than or equal to the threshold, output the authentication conclusion as "authentic, not deepfake";

[0176] otherwise, output "deepfake". The threshold setting of this module is based on a large amount of experimental data, ensuring high reliability of the authentication conclusion.

[0177] When the Shanghai dialect-based deepfake audio authentication system formed based on the above solution operates, first, the voice acquisition module 100 sends the voice to be tested to the multi-source dialect feature extraction module 200. The data enhancement unit 210 inside the multi-source dialect feature extraction module 200 performs Shanghai dialect-specific enhancement on the voice, then the feature extraction unit 220 extracts multi-source features such as acoustic features, prosodic features and phonological features, and finally the temporal attention pooling fuser 230 fuses them into a comprehensive dialect feature vector.

[0178] Subsequently, the comprehensive feature is sent to the dialect adaptation encoding module 300, which encodes it into dialect-aware embedding features that are highly sensitive to deviations from Shanghai dialect via the self-supervised pre-trained model 310.

[0179] This embedding feature is then sent to the multi-task adversarial encoding module 400. The fusion encoder 410 and the gradient reversal layer 440 inside the module work collaboratively to strip the common Shanghai dialect information from the embedding feature, and output a speaker feature vector that only reflects forgery traces.

[0180] Finally, the similarity authentication module 500 calculates the cosine similarity between the feature vector and the pre-registered target speaker template vector in the template library 510, and the threshold determiner 530 gives the final authentication result according to the preset threshold. The entire process does not require manual intervention, and can automatically complete high-precision detection of deepfake audio in Shanghai dialect.

[0181] The following further illustrates the application and implementation process of the present solution through specific application examples.

[0182] This example takes Shanghai dialect (Shanghainese) as the target, and builds a prototype system for deepfake audio authentication based on the present invention to verify its effect and demonstrate the implementation process in detail.

[0183] (a) System Deployment:

[0184] The hardware platform uses a server equipped with an Intel Xeon Gold 6330 CPU, an NVIDIA A100 40G GPU, and 64GB of DDR4 memory.

[0185] The software environment is based on Python 3.9 and PyTorch 2.0.1 frameworks, and integrates audio processing libraries such as Librosa 0.10.1, Parselmouth 0.4.3, and OpenSMILE 3.0.

[0186] The core model is based on Facebook's open-source wav2vec 2.0 Large pre-trained model, with minor adjustments for Shanghai dialect adaptation.

[0187] (II) Dataset Construction:

[0188] (2.1) Shanghainese sample dataset;

[0189] The experiment selected a Shanghai dialect speech corpus (containing 200 native Shanghainese speakers, 100 men and 100 women, aged 20-60). Each speaker recorded 50 everyday Shanghainese sentences (such as "Where did you go today?" and "Let's go eat xiaolongbao together"), with a sampling rate of 16kHz, 16-bit mono, and a total duration of about 80 hours, as real speech samples.

[0190] (2.2) A dataset of fake Shanghainese speech;

[0191] Based on the aforementioned authentic Shanghainese speech, three mainstream deepfake techniques were used to generate fake Shanghainese speech. Each technique generated the same number of fake samples as the real samples, with the total number of fake samples equal to the number of real samples in a 1:1 ratio.

[0192] VC speech conversion: Using the VITS-VC model, the speech of non-native Shanghainese speakers is converted into Shanghainese speech, imitating the Shanghainese style of the target speaker;

[0193] TTS speech synthesis: The FastSpeech 2 model is used to synthesize the target speaker's Shanghainese speech based on a Shanghainese pronunciation dictionary;

[0194] GPT-SoVITS Zero-Shot Synthesis: Given only 1 minute of Shanghainese speech from the target speaker, zero-shot multi-sentence Shanghainese speech of that speaker is generated.

[0195] (2.3) Dataset partitioning;

[0196] The real + fake Shanghainese speech dataset was divided into training set, validation set and test set in a 7:2:1 ratio. The training set was expanded using the Shanghainese dialect-specific data augmentation method of this scheme (speed perturbation 0.9×-1.1×, noise addition SNR 5-20dB, reverberation simulation, tone enhancement of entering tone characters, and replacement of Shanghainese phonemes). After expansion, the sample size of the training set was 5 times that of the original sample size.

[0197] (III) Key Parameter Settings:

[0198] The core parameters of the Shanghai dialect deepfake audio detection model built based on this solution in this embodiment are set as follows:

[0199] Feature extraction parameters: pre-emphasis coefficient 0.97, frame window length 25ms, frame shift 10ms; MFCC extracts 13-dimensional static features + 1st-order difference + 2nd-order difference, for a total of 39 dimensions; Shanghai dialect PRA features have 32 dimensions, and the total dimensions of acoustic features are 105.

[0200] wav2vec 2.0 Shanghai dialect adaptation parameters: 24-layer Transformer, hidden layer dimension 1024, dialect contrast loss temperature coefficient 0.1, learning rate 5e-5, training epochs 30, batch size 32;

[0201] CNN-GRU fusion encoder parameters: 1D convolutional kernel size 5, stride 2, output channels 512; bidirectional GRU units 256, dropout rate 0.2; fusion feature dimension 512;

[0202] Multi-task adversarial training parameters: gradient inversion layer coefficient λ=1.0, forgery detection loss weight 1.0, dialect classification loss weight α=0.3, adversarial loss weight β=0.2, optimizer AdamW, weight decay 1e-4, number of training rounds 50, batch size 16;

[0203] Identity comparison parameters: each target speaker has at least 5 registered voice recordings in the target speaker's registration database, and the cosine similarity threshold is 0.90 (i.e., 90%).

[0204] (iv) Experimental steps:

[0205] The entire process of this scheme, which includes Shanghai dialect specialization data augmentation, multi-source feature extraction, self-supervised Shanghai dialect adaptation, multi-task adversarial training, attention encoding, feature library construction, cosine similarity comparison, and identification judgment, is executed as follows:

[0206] The training set was augmented with Shanghai dialect-specific data to generate diverse Shanghai dialect speech samples.

[0207] Multi-source feature extraction was performed on all samples to obtain acoustic, prosodic, and phonological features specific to Shanghainese.

[0208] Based on the wav2vec 2.0 Large model, we performed self-supervised Shanghai dialect adaptation on the Shanghai dialect corpus, introduced dialect contrast loss, and extracted dialect-aware embeddings (DAE).

[0209] The DAE is input into the CNN-GRU fusion encoder to build a multi-task adversarial training framework, which is used to jointly train the forgery detection and Shanghainese classification tasks, and suppresses the leakage of dialect information through the gradient inversion layer.

[0210] An attention-enhanced encoding network based on Shanghai dialect is used to encode the fused features, and a fixed-dimensional speaker feature vector is generated through a global pooling layer.

[0211] A speaker registration database of 200 native Shanghainese speakers was constructed. Feature vectors of each person's registered speech were extracted and the average value was calculated to form an individual "identity template".

[0212] Input the Shanghainese speech samples (real + fake) from the test set into the trained model, extract the feature vectors, and calculate the cosine similarity with the "identity template" of the corresponding target speaker in the registration database;

[0213] The identification results are output based on the similarity threshold (≥0.90 indicates non-forgery, <0.90 indicates forgery), and the identification indicators of the statistical model are also calculated.

[0214] (V) Analysis of Experimental Data and Results

[0215] (5.1) Experimental evaluation indicators

[0216] The mainstream evaluation metrics for deepfake audio detection include accuracy, precision, recall, and F1 score. The F1 score is a comprehensive evaluation metric, with the value closer to 1 indicating better model performance.

[0217] Accuracy: (Number of correctly identified samples / Total number of samples) × 100%;

[0218] Accuracy: Number of genuinely fabricated samples / Number of samples identified as fabricated × 100%;

[0219] Recall rate: (Number of genuinely fabricated samples / Number of actually fabricated samples) × 100%;

[0220] F1 score: 2 × (precision × recall) / (precision + recall).

[0221] (5.2) Experimental Data Results

[0222] The following table shows the comparison data of the identification metrics of this model and the two existing mainstream models on the Shanghai dialect deepfake speech test set (unit: %, F1 value is a unitless coefficient):

[0223] This solution model (Shanghai dialect feature-driven) 98.6 98.2 98.9 0.985 Based on the wav2vec 2.0 general model 75.3 74.8 75.9 0.753 Based on MFCC+CNN general model 68.5 67.9 69.2 0.685

[0224] Authentication index data of this scheme model under different counterfeiting techniques (unit: %, F1 value is a unitless coefficient):

[0225] VC speech conversion 99.1 98.8 99.3 0.990 TTS speech synthesis 98.7 98.3 98.9 0.986 GPT-SoVITS Zero-Shot Synthesis 97.8 97.5 98.1 0.978

[0226] Accuracy data of the proposed model under different noise environments (unit: %, SNR is signal-to-noise ratio):

[0227] Accuracy of the model in this solution 98.6 98.2 95.7 92.3 wav2vec 2.0 general model accuracy 75.3 70.1 62.5 55.8

[0228] Comparison with the general model:

[0229] Model parameter count 89M 317M 24M Inference time per voice message (10s, A100) 0.32s 0.45s 0.18s Single voice inference time (10s, RTX3060) 1.25s 2.18s 0.65s GPU memory usage (during inference) 896MB 1980MB 320MB CPU utilization (during inference) 15-20% 25-30% 10-15% Note: This solution model is designed with Shanghai dialect features in a lightweight manner, with only 28% of the parameters of wav2vec 2.0, resulting in higher inference efficiency, lower hardware requirements, and suitability for engineering deployment.

[0230] (5.3) Results Analysis

[0231] Compared with existing models: The accuracy, precision, recall, and F1 score of the proposed model are all significantly higher than those of the general model based on wav2vec 2.0 and MFCC+CNN. The accuracy is improved by more than 20%, which shows that the specific design of the proposed model for Shanghai dialect features effectively solves the problem of insufficient generalization of Shanghai dialect features in existing models and can accurately identify fake speech in Shanghai dialect.

[0232] Adaptability to different forgery techniques: The model of this scheme has an F1 score of over 0.978 for the identification of three mainstream forgery techniques, with the best identification effect for VC speech conversion (F1=0.990). Even for zero-sample synthesized GPT-SoVITS forged speech, it still maintains an accuracy of 97.8%, indicating that the model has strong robustness to different Shanghai dialect forgery methods and can identify the subtle feature deviations brought about by various forgery techniques.

[0233] Noise resistance performance: In complex noise environments with low signal-to-noise ratio (5dB), the model of this scheme still maintains an accuracy of 92.3%, while the accuracy of the general model is only 55.8%. This shows that the Shanghai dialect specialization data enhancement and multi-source feature fusion enable the model to adapt to speech distortion in real-world scenarios and significantly improve its anti-interference ability.

[0234] Threshold effectiveness: Using a 90% cosine similarity threshold, the false positive rate in the test set was only 1.2% (identifying real speech as fake) and the false negative rate was only 1.0% (identifying fake speech as real). This indicates that the threshold setting is reasonable and can ensure the reliability of the identification results while maintaining high accuracy.

[0235] As demonstrated by the above examples, this invention is the first to deeply integrate the acoustic, prosodic, and phonological features of Shanghai dialect into the entire process of deepfake audio detection. Through the organic combination of "Shanghai dialect-specific data enhancement, multi-level Shanghai dialect feature extraction, Shanghai dialect self-supervised adaptation, and multi-task adversarial training," it systematically solves the three major technical problems of existing technologies in Shanghai dialect deepfake audio detection: data scarcity, insufficient feature generalization, and concealed forgery traces. Simultaneously, through Shanghai dialect-enhanced attention coding and high-confidence cosine similarity comparison, it achieves high-precision, robust, and interpretable detection of deepfake audio in Shanghai dialect scenarios. This system significantly outperforms existing general-purpose models in Shanghai dialect scenarios and possesses excellent noise resistance, filling a technological gap in the field of Shanghai dialect deepfake detection.

[0236] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A deepfake audio detection system based on Shanghai dialect, characterized in that, include: The voice acquisition module is used to acquire the registered voice of the target speaker and the voice to be tested; A multi-source dialect feature extraction module is used to perform data augmentation processing on the speech to be tested, which is specialized in Shanghai dialect, and to extract multi-source dialect features from the processed speech signal; wherein, the multi-source dialect features include at least acoustic-level features, prosodic-level features, and phonological-level features; The dialect adaptation coding module integrates a self-supervised pre-trained model adapted and trained with the Shanghai dialect corpus. The dialect adaptation coding module receives the multi-source dialect features and generates dialect-aware embedding features. A multi-task adversarial coding module receives the dialect-aware embedding features and includes a fusion encoder, a forgery detection head, a dialect classification head, and a gradient inversion layer. The fusion encoder generates fusion features based on the dialect-aware embedding features. The forgery detection head performs a forgery detection main task based on the fusion features. The dialect classification head performs a classification auxiliary task to distinguish Shanghainese dialect from other dialects based on the fusion features. The gradient inversion layer is set on the gradient backpropagation path between the dialect classification head and the fusion encoder to invert the gradient from the dialect classification head during training, thereby suppressing the interference of Shanghainese dialect category information on the forgery detection main task. This allows the multi-task adversarial coding module to output a speaker feature vector that highlights the bias of forgery features. The similarity identification module is used to calculate the similarity between the speaker feature vector of the speech to be tested and the template vector corresponding to the registered speech of the target speaker, and to determine whether the speech to be tested is deepfake audio based on a preset threshold.

2. The system according to claim 1, characterized in that, The multi-source dialect feature extraction module includes: The data augmentation unit is used to apply general audio enhancement and Shanghai dialect content enhancement to the input Shanghai dialect speech; the general audio enhancement includes noise addition, dynamic range compression and room reverberation simulation, and the Shanghai dialect content enhancement includes enhancing the inherent short tone patterns and continuous tone sandhi curves of the entering tone characters, as well as enhancing the statistical occurrence probability of typical phonemes in Shanghai dialect; The feature extraction unit is used to extract acoustic-level features, prosodic-level features, and phonological-level features from the enhanced speech signal; The temporal attention pooling fusion processor is used to fuse acoustic-level features and prosodic-level features through an attention mechanism to generate dialect composite features.

3. The system according to claim 1, characterized in that, The acoustic-level features include multi-scale fundamental frequency features optimized for the entering tone coda and continuous tone sandhi patterns in Shanghai dialect. The extraction method of the multi-scale fundamental frequency features is as follows: based on continuous wavelet transform, for the tone system of Shanghai dialect, the scale corresponding to the short-term sudden drop tone pattern and continuous tone sandhi sensitive interval of the entering tone is selected for extraction, so as to capture the fundamental frequency profile and formant profile information that characterize Shanghai dialect.

4. The system according to claim 1, characterized in that, The phonological features are posterior features of Shanghai dialect phonemes; the dialect adaptation coding module calculates the posterior probability of the test speech on a predefined set of phonemes containing Shanghai dialect-specific phonemes by using a phoneme recognition model pre-trained on multi-dialect data, wherein the Shanghai dialect-specific phonemes include at least entering tone codas and nasalized codas.

5. The system according to claim 1, characterized in that, The fusion encoder includes a 1D convolutional layer, a bidirectional GRU network, and a concatenation layer. The 1D convolutional layer is used to perform convolutional dimensionality reduction on the dialect-aware embedded features. The bidirectional GRU network is used to perform sequence modeling on the convolutional features. The concatenation layer is used to concatenate the dialect comprehensive features generated by fusing the GRU output with the multi-source dialect features in the feature dimension to generate fused features.

6. The system according to claim 1, characterized in that, The similarity identification module includes: A template library is used to store the identity template vector of the target speaker, which is obtained by averaging the feature vectors of all registered voices of the target speaker. A cosine similarity calculator is used to calculate the cosine similarity between the feature vector of the speech to be tested and the template vector. The threshold determiner compares the calculated similarity with a preset threshold and outputs an identification conclusion.

7. The system according to claim 1, characterized in that, The gradient inversion layer in the multi-task adversarial coding module keeps the signal unchanged during forward propagation, but multiplies the gradient from the dialect classification head by a negative coefficient during backward propagation to adversarially prevent the fusion encoder from learning to distinguish Shanghai dialect from other dialect categories.

Images