Training methods for fake speech detection models, fake speech detection methods and equipment
By fusing multimodal features from audio sample files and iteratively training a self-distilled acoustic model, the problem of insufficient utilization of modal feature association information in the fake speech detection model is solved, improving the model's expressive and generalization capabilities and increasing the accuracy of fake speech detection.
Patent Information
- Application Number
- CN202410442526.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-12
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-04-12
AI Technical Summary
In existing technologies, spoofing detection models cannot fully utilize the correlation information between features of different modalities, resulting in weak expressive and generalization abilities of the models and insufficient accuracy in spoofing detection.
By extracting multimodal sample features from audio sample files for feature fusion, and using a self-distillation acoustic model for iterative training, the expressive power and generalization ability of the model are improved by combining multi-view information fusion and self-distillation methods.
It improved the accuracy of the fake speech detection model and enhanced the model's ability to recognize fake speech.
Smart Images

Figure CN118366433B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimedia security technology, and in particular to a training method for a fake speech detection model, a fake speech detection method and device. Background Technology
[0002] Voice forgery typically refers to the use of artificial intelligence and speech synthesis technologies to generate fake voice content that sounds like a real human voice. This technology can be used to create fake phone calls, fake broadcasts, and spread deceptive information. Through voice forgery technology, voice content with specific intonation, timbre, and speaking style can be generated to achieve illicit purposes.
[0003] In recent years, with the development of speech synthesis technology, the level of forged speech has gradually improved, making it more difficult to distinguish between real and forged speech. Related technologies extract feature information from different modalities of audio files and use this feature information for model training. However, the correlation information between features from different modalities cannot be fully utilized, resulting in weak expressive and generalization abilities of the model and insufficient accuracy in forged speech detection. Summary of the Invention
[0004] This application provides a training method, a method for detecting fake speech, and a device for fake speech detection, in order to at least solve the problem that related technologies cannot fully utilize the correlation information between different modal features, resulting in weak expressive power and generalization ability of the model.
[0005] To solve the above-mentioned technical problems, this application is implemented as follows:
[0006] In a first aspect, embodiments of this application provide a training method for a forged speech detection model, comprising: acquiring multiple audio sample files, wherein the audio sample files include text data and audio data; extracting multimodal sample features from the text data and audio data of the multiple audio sample files, performing feature fusion processing on the multimodal sample features to obtain fused features corresponding to each audio sample file; using the fused features corresponding to the multiple audio sample files to iteratively train a pre-constructed self-distillation acoustic model, wherein in each iteration of training, multi-view information fusion is performed on the fused features to obtain target acoustic features; inputting the target acoustic features into a teacher model to obtain a first category label, and inputting the target acoustic features into a student model to obtain a second category label; determining a model detection loss value based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model; updating and iterating the student model based on the model detection loss value until a preset convergence condition is reached, and determining the converged self-distillation acoustic model as a forged speech detection model.
[0007] Secondly, embodiments of this application provide a method for detecting fake speech, comprising: acquiring an input target audio file, the target audio file including target text data and target audio data; extracting multimodal features from the target text data and target audio data, performing feature fusion processing on the multimodal features to obtain target fusion features; obtaining a category label of the target audio file by inputting the target fusion features into a fake speech detection model; wherein, the fake speech detection model is trained according to the training method of the fake speech detection model described in the first aspect; and determining the authenticity of the target audio file based on the category label.
[0008] Thirdly, embodiments of this application provide a training apparatus for a forged speech detection model, comprising: a sample acquisition module for acquiring multiple audio sample files, wherein the audio sample files include text data and audio data; a feature fusion module for extracting multimodal sample features from the text data and audio data of the multiple audio sample files, performing feature fusion processing on the multimodal sample features to obtain fused features corresponding to each audio sample file; and a model training module for iteratively training a pre-constructed self-distillation acoustic model using the fused features corresponding to the multiple audio sample files, wherein in each iteration of training, multi-view information fusion is performed on the fused features to obtain target acoustic features; the target acoustic features are input into a teacher model to obtain a first category label, and the target acoustic features are input into a student model to obtain a second category label; a model detection loss value is determined based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model; the student model is updated and iterated based on the model detection loss value until a preset convergence condition is reached, and the converged self-distillation acoustic model is determined as a forged speech detection model.
[0009] Fourthly, embodiments of this application provide a forged speech detection device, comprising: a file acquisition module for acquiring an input target audio file, the target audio file including target text data and target audio data; a feature extraction module for extracting multimodal features from the target text data and target audio data, performing feature fusion processing on the multimodal features to obtain target fused features; a category determination module for obtaining category labels of the target audio file by inputting the target fused features into a forged speech detection model; wherein the forged speech detection model is trained using the training device for the forged speech detection model described in the third aspect above; and a speech detection module for determining the authenticity of the target audio file based on the category labels.
[0010] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect above.
[0011] In a sixth aspect, embodiments of this application provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect above.
[0012] In a seventh aspect, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the method described in the first aspect above.
[0013] In this embodiment, multiple audio sample files are acquired; multimodal sample features are extracted from the text and audio data of the multiple audio sample files, and feature fusion processing is performed on the multimodal sample features to obtain the multimodal fusion features corresponding to each audio sample file; the multimodal fusion features corresponding to the multiple audio sample files are used to iteratively train a pre-built self-distillation acoustic model. In each iteration, multi-view information fusion is performed on the multimodal sample features to obtain target acoustic features; the target acoustic features are input into the teacher model to obtain a first category label, and the target acoustic features are input into the student model to obtain a second category label; the model detection loss value is determined based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model; the student model is updated and iterated based on the model detection loss value until a preset convergence condition is reached, and the converged self-distillation acoustic model is determined as a fake speech detection model. In this way, by extracting multimodal sample features from audio sample files and fusing these features, and then using the fused features to iteratively train the self-distillation acoustic model, the self-distillation acoustic model can fully explore the correlation between different modal features, improve the model's expressive power and generalization ability, and thus help improve the accuracy of the fake speech detection model in detecting fake speech.
[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0016] Figure 1 A flowchart illustrating the training method of the fake speech detection model provided in an embodiment of this application is shown.
[0017] Figure 2 A flowchart illustrating the text data processing method provided in an embodiment of this application is shown;
[0018] Figure 3 A flowchart illustrating the audio data processing method provided in an embodiment of this application is shown;
[0019] Figure 4 A schematic diagram of the modal information complementarity and fusion method provided in the embodiments of this application is shown;
[0020] Figure 5 It shows Figure 4 Network structure diagram of the mid-modal complementary module;
[0021] Figure 6 The ResNet network skeleton diagram in the related technology is shown;
[0022] Figure 7 The Res2Net network skeleton diagram in the related technology is shown;
[0023] Figure 8 The diagram shows the structure of a deep residual network based on a multi-view information fusion module provided in an embodiment of this application.
[0024] Figure 9 A schematic diagram of the structure of an efficient attention mechanism block in related technologies is shown;
[0025] Figure 10 A schematic flowchart of the self-distillation process provided in an embodiment of this application is shown;
[0026] Figure 11 A flowchart illustrating the forged voice detection method provided in an embodiment of this application is shown;
[0027] Figure 12 A schematic diagram of the structure of the training device for the fake speech detection model provided in an embodiment of this application is shown;
[0028] Figure 13 This paper shows a schematic diagram of the structure of a forged voice detection device provided in an embodiment of this application;
[0029] Figure 14 This paper shows a schematic diagram of another forged voice detection device provided in an embodiment of this application;
[0030] Figure 15 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0032] In the field of multimedia security technology, some solutions propose extracting and combining multimodal features such as speech, semantics, and interaction from call data to form a multimodal speech call feature set, which is then applied to the analysis of numerous downstream tasks. However, the correlation information between different modal features cannot be fully utilized, resulting in weak expressive and generalization abilities of the model, thus affecting the accuracy of forged speech detection.
[0033] To address the problems existing in the above-mentioned fake speech detection process, this application provides a training method for a fake speech detection model. This method extracts multimodal sample features from audio sample files, performs feature fusion on the multimodal sample features, and uses the fused features to iteratively train a self-distillation acoustic model, so that the self-distillation acoustic model can fully explore the correlation between different modal features and improve the model's expressive power and generalization ability.
[0034] Figure 1 This document illustrates a flowchart of a training method for a forged speech detection model provided in this application embodiment. The execution entity of this method can be a terminal device or a server. The terminal device can be a personal computer, a mobile terminal device such as a mobile phone, tablet, or IPTV, or a user-used terminal device. The server can be a standalone server or a server cluster composed of multiple servers. Furthermore, the server can be a backend server for a specific business function or a backend server for a platform or application (e.g., a virtual assistant, social media platform, chatbot, voice security and anti-fraud applications, etc.). This application embodiment uses a server as the execution entity for illustration. For the terminal device scenario, the following related content can be used, and it will not be elaborated further here. As shown in the figure, the training method 100 for this forged speech detection model may include the following steps:
[0035] S101: Obtain multiple audio sample files, wherein the audio sample files include text data and audio data.
[0036] In the process of spoofing speech detection, utilizing different data sources or information formats to extract features can help improve detection performance. In this embodiment, multiple audio sample files are acquired, each containing text data and audio data, to extract multimodal sample features from the text and audio data for training the spoofing speech detection model.
[0037] S102: Extract multimodal sample features from the text data and audio data of multiple audio sample files, perform feature fusion processing on the multimodal sample features, and obtain the fused features corresponding to each audio sample file.
[0038] In this embodiment, since the features that clearly distinguish real speech from fake speech mainly exist in the sub-band features and high-frequency and low-frequency features of the speech signal, and in addition, the text data contains a large amount of sensitive information and speaker emotional features, multimodal sample features are extracted from the text data and audio data of multiple audio sample files. To enable the model to fully exploit the correlation between different modal features, the extracted multimodal sample features are subjected to feature fusion processing to obtain the fused features corresponding to each audio sample file.
[0039] In one possible implementation, step S102 above involves extracting multimodal sample features from the text and audio data of multiple audio sample files, performing feature fusion processing on the multimodal sample features to obtain fused features corresponding to each audio sample file, including:
[0040] For each audio sample file, multimodal sample features are extracted from the text data and audio data of the audio sample file. The multimodal sample features include semantic features corresponding to the text data, subband features corresponding to the audio data, and high-level features extracted based on the spectrogram corresponding to the audio data. A weight matrix is determined based on the semantic features, subband features, and high-level features. The multimodal sample features in the audio sample file are weighted using the weight matrix to obtain the multimodal fusion features corresponding to the audio sample file. The multimodal fusion features corresponding to each audio sample file in the multiple audio sample files are obtained.
[0041] In this embodiment of the application, for each audio sample file, after acquiring audio data and text data, semantic features for representing sensitive semantic information and emotional semantic information are extracted from the text data, sub-band features are extracted from the audio data, and the audio data is converted into a spectrogram. Based on the spectrogram, high-level features such as Mel frequency cepstral coefficients (MFCC), Mel frequency cepstral graph features, high-frequency features, and low-frequency features are extracted, and then the complementary and deep fusion of multimodal sample features are performed.
[0042] The semantic features corresponding to text data can be extracted in the following ways:
[0043] The text data is preprocessed to obtain a text sequence; wherein, the text preprocessing includes data cleaning and Chinese word segmentation;
[0044] Obtain the word embedding vectors corresponding to the text sequence, and input the word embedding vectors into a natural language processing model for feature fusion processing to obtain semantic features.
[0045] In specific implementation, such as Figure 2 As shown, firstly, special characters and punctuation marks are removed from the text data to ensure cleanliness and consistency. The cleaned text is then split into individual words or word units, and the segmented text is presented as a word sequence. Next, each word in the text sequence is replaced with its corresponding word embedding vector. These word embedding vectors are then input into a natural language processing model for feature fusion to obtain semantic features. Specifically, before inputting the word embedding vectors into the natural language processing model for feature fusion, the text data with the word embedding vectors can be input into a word segmenter such as WordPiece or SentencePiece to obtain a tokenized text sequence. The tokenized text sequences may have different lengths. Padding or truncation operations are used to pad the text sequences to the same length or to truncate them. The processed text sequences are then input into a natural language processing model (Bidirectional Encoder Representations from Transformers, BERT) to obtain deep language representations of sentiment and sensitive information. The feature sequences extracted by the BERT model are then processed by feature pooling to merge the representations of multiple tokens into a fixed-dimensional (e.g., 768-dimensional) word vector representation. Max pooling is then used to output the most significant sentiment and sensitive information features to obtain the semantic features corresponding to the text data.
[0046] The sub-band features corresponding to the audio data can be extracted in the following ways:
[0047] The audio data is processed by speech framing to obtain audio data frames; the audio data frames are then subjected to Fourier transform to obtain the sub-band features corresponding to the audio data.
[0048] Furthermore, high-level features can be extracted in the following ways:
[0049] The audio data is converted into a spectrogram; the spectrogram is then input into a convolutional neural network to extract the high-level features corresponding to the audio data.
[0050] In the embodiments of this application, such as Figure 3 As shown, the audio data is first sampled to a fixed value, such as 16kHz, to ensure compatibility in different environments. Noise cancellation is then used to remove background noise or interference, improving signal quality. Speech endpoint detection detects the start and end points of the speech signal for better processing. Next, speech framing divides the processed audio signal into a series of short frames for feature extraction. In practical applications, each frame can be 20ms long with a 10ms frame shift. A Hamming window is used for each frame to reduce discontinuities at frame boundaries, minimizing spectral leakage.
[0051] After performing the above preprocessing operations on the audio data, feature extraction is performed on the obtained audio data segments through two channels. On the one hand, the preprocessed audio data frames are decomposed into multiple sub-bands for feature extraction through short-time Fourier transform. This is because the interference of fake speech exists at the sub-band level of speech, and the decomposition of sub-bands helps the subsequent model to capture the fine features of the mid-frequency band of fake speech. After mapping the features of each sub-band to a fixed-dimensional speech embedding vector through Wave2Vec, the speech embedding vector sequence corresponding to each audio frame is output, with a dimension of 512.
[0052] On the other hand, the preprocessed audio data is converted into spectrograms to extract features. In spectrogram feature extraction, short-time Fourier transform is used to convert the audio signal into a frequency domain representation to generate a spectrogram with a shape of 512*512 and a dimension of 2. Then, the converted spectrogram is input into a convolutional neural network (CNN) model, and its high-frequency and low-frequency features are extracted using convolutional layers. This is because fake speech has obvious differences from real speech in the high and low frequency parts, and extracting this feature is important for the subsequent training and detection of the model. Pooling layers are used to reduce the dimensionality of the output features to a 1*512 feature vector for subsequent fusion of multimodal features.
[0053] After extracting the semantic features corresponding to the text data, the sub-band features corresponding to the audio data, and the high-level features extracted based on the spectrogram of the audio data, feature fusion processing is performed on the multimodal sample features. The specific fusion processing method is as follows:
[0054] Each modality's features (semantic features, subband features, and high-level features) undergo an attention mechanism that focuses each modality's features on important regions, aiding in the extraction of key information within each modality. Then, a cross-attention mechanism allows features from different modalities to mutually attend to each other. Each modality's features can be considered a query, while features from other modalities serve as keys and values. By calculating attention weights, the attention distribution of one modality on other modalities can be determined, thereby capturing the correlation between modalities. The multimodal sample feature fusion processing method and the structure of the modality fusion module are as follows: Figure 4 and Figure 5 As shown in the image.
[0055] In the multimodal feature fusion part, stacked multi-layer Transformer layers can be used for iterative fusion of modalities. Through continuous iterative fusion, the model can capture the relationships between different modalities at different levels. Each layer can be viewed as modeling the association between modalities at different levels; iterative fusion allows information to be passed between different layers, so the model can obtain richer feature representations from the fusion of previous layers; through iterative fusion, the interaction between different modalities can be gradually enhanced, the model can better fuse multimodal information, and a multi-head attention mechanism is introduced to dynamically allocate attention weights according to the needs of the task. Each attention head can focus on different aspects of the data, enabling the model to better fuse multimodal information. The features fused from each layer are used as input to the next layer, thereby increasing the continuous interaction between modalities to achieve deep fusion.
[0056] In the self-attention mechanism, the input X∈R n×d It is linearly transformed into three parts, namely the query. Keywords value Where n is the sequence length, d, d k d v These represent the dimensions of input, query (keyword), and value, respectively. R is the attention distribution matrix. Scaled dot-product attention is applied to Q, K, and V:
[0057]
[0058] Attention mechanisms can treat raw input data as...<Key,Value> In key-value pair format, the similarity coefficient between the key and the query is calculated based on the given task query. This yields the weight coefficient W corresponding to the value. Then, the values are weighted and summed using this weight coefficient to obtain the output.
[0059] Attention(Q,K,V)=W·V=softmax(QK T )V.
[0060] S103: The pre-constructed self-distillation acoustic model is iteratively trained using the fusion features corresponding to multiple audio sample files. In each iteration, the fusion features are fused from multiple perspectives to obtain target acoustic features. The target acoustic features are input into the teacher model to obtain a first category label, and the target acoustic features are input into the student model to obtain a second category label. The model detection loss value is determined based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model. The student model is updated and iterated based on the model detection loss value until a preset convergence condition is reached. The converged self-distillation acoustic model is then identified as a fake speech detection model.
[0061] In this embodiment, after obtaining the fusion features corresponding to multiple audio sample files, they are input as parameters into the self-distillation acoustic model to be trained for a limited number of iterations. In each iteration, the fusion features are first subjected to multi-view information fusion to obtain the target acoustic features. Specifically, the Res2Net network can be improved using the Multi-View Information Fusion (MPIF) module, and the improved Res2Net network is used to perform multi-view information fusion on the fusion feature information to obtain the target acoustic features.
[0062] Previous networks like ResNet had limited multi-scale feature extraction capabilities, such as the ResNet network skeleton. Figure 6 As shown, although stacked convolutional layers in ResNet can obtain certain multi-scale features, the receptive field of each layer is fixed, and the multi-scale features mainly come from different layers, so the expressive power is still limited.
[0063] The embodiments of this application use a Res2Net network skeleton, such as Figure 7 As shown, Res2Net is a more efficient multi-scale feature representation method than the ResNet network skeleton. The Res2Net module breaks down 3x3 convolutions into multiple smaller filter banks, which are connected in a hierarchical, residual-like manner. This allows it to learn features at different scales, and Res2Net can improve multi-scale features at a finer granularity. It uses grouped convolutions to divide features into multiple groups, each group passing through different paths to obtain receptive fields of different sizes. This structural design allows for the acquisition of rich multi-scale features within a single module.
[0064] The formula for multi-scale feature processing in the Res2Net network skeleton is as follows:
[0065]
[0066] Among them, K i p represents the convolution operation. i From X i The obtained input, yi-1 It is the output of the i-1 part.
[0067] Although the Res2Net network processes features at multiple scales, it uses a fixed single convolutional kernel size (e.g., 3x3) for feature learning. A fixed-size convolutional kernel can only perceive local information, easily missing global structural information; using the same kernel size in different convolutional layers can lead to overly similar learned false voice features, generating redundant false voice cues; furthermore, similar false voice information can obscure truly effective discriminative information.
[0068] To address the aforementioned issues, a Multi-View Information Fusion (MPIF) module is used to improve the Res2Net network. Its network structure is as follows: Figure 8 As shown in the diagram, the MPIF module uses dilated convolutions with different dilation rates to extract multi-scale information, increasing the receptive field and capturing more global information. It fuses the features extracted by dilated convolutions with different dilation rates to obtain feature representations that encompass multi-scale information, making the information richer and more diverse. The fusion of multi-view information avoids repetitive falsetto learning and helps suppress redundant parts in falsetto cues, thus making the truly discriminative information more prominent. The formula for Res2Net feature processing based on MPIF is as follows:
[0069]
[0070] Among them, MPIF i This indicates multi-perspective information fusion computation.
[0071] like Figure 8 The processing procedure shown on the right side illustrates how the MPIF module in the current channel group i processes p. i or p i +y i-1 To perform different convolution operations, firstly, at the beginning of the MPIF module, p i The results are fed into convolution operations with different dilation parameters j, where j∈[1,2]. The results are then recalculated by dilating the convolutions to recalculate the energy distribution of each channel, normalized using the Sigmoid function, and finally obtained from Conv2d using an average pooling layer. As the weight for each channel k.
[0072] c j =(Conv2d j (p i ));
[0073]
[0074] After that and Perform multiplication and add the results. Then you can obtain the MPIF result for the i-th channel group. i (p i ):
[0075]
[0076] Among them, C j This represents the convolution results with different dilation parameters; This represents the convolution result with different dilation parameters in k channels; Indicates all The resulting weight matrix.
[0077] The purpose of using dilated convolutions is to increase the receptive field, ensuring that each convolution output contains information from a larger range, while keeping the parameters and computational cost constant.
[0078] Ordinary CNNs perform convolution operations independently on each channel, without considering the correlation between different channels, and each channel contributes to the final output to a different degree. Generally, the SE-Block attention mechanism is introduced to learn the attention weights of each channel through global pooling and linear transformation, thereby enhancing the response of effective feature channels and suppressing the response of ineffective channels.
[0079] However, dimensionality reduction using the SE block can lead to information loss. Therefore, this embodiment employs an efficient attention mechanism (ECA) block to help the network learn channel dependencies. Figure 9 As shown, H is the feature map height, W is the feature map width, C is the number of channels, X is the batch size, and k is adaptively determined through the mapping of the channel dimension C. ECA captures local cross-channel interactions by considering each channel and its k neighbors. ECA is efficiently implemented using a fast one-dimensional convolution of size k, where the kernel size k represents the coverage of local cross-channel interactions, i.e., how many neighbors participate in the attention prediction of a channel.
[0080] ECA effectively captures local cross-channel interactions through 1D convolutions, avoiding the computationally intensive task of learning dependencies between all channels. By adaptively selecting the convolution kernel size, it better models the coverage of local interactions with different numbers of channels.
[0081] This application employs a model combining a deep residual network (Res2Net) based on a multi-view information fusion module (MPIF) and efficient channel attention (ECA). A base model for the self-distillation method is trained by setting a certain loss threshold. Then, the trained base model is split into a teacher model and a student model using self-distillation. Loss functions are set at three different feature levels, and a forged speech detection model is obtained through a finite number of iterations.
[0082] The deep residual network model used is Res2Net50. Combined with the improved deep residual network and efficient attention mechanism described above, the final model name is ECA-MPIF-Res2Net50. In forged speech detection scenarios, high accuracy and low latency are required. Traditional methods improve accuracy by deepening or widening the network, but increasing the network size leads to an exponential increase in computational and storage costs, thus prolonging response time.
[0083] In order to improve accuracy without increasing network size and computational load, this application adopts a self-distillation method for the original model, which improves performance by compressing the model through knowledge transfer within the network.
[0084] Table 1 shows the model architecture and configuration of the ECA-MPIF-Res2Net network. Dimensions are arranged in order of channel, frequency, and time. BN represents batch normalization, ReLU represents rectified linear unit, MPIF and ECA are the multi-view information module and efficient channel attention block, respectively. The 50-layer network is configured with 3 convolutions within each block, with kernels of 1, MPIF, 1, and the remaining vectors corresponding to the 34-layer network are (3, 4, 6, 3).
[0085] Table 1. Model architecture and configuration of ECA-MPIF-Res2Net network
[0086]
[0087] like Figure 10 As shown, self-distillation enables a single neural network to perform inference at different depths, making it suitable for adaptive accuracy-efficiency tradeoffs in scenarios where response time is sensitive but computational resources are limited. Its main steps are as follows:
[0088] The target convolutional neural network is divided into multiple shallow segments according to depth, and the Res2Net50 is divided into 4 segments, each of which is a ResBlock. A classifier consisting of a bottleneck layer and a fully connected layer is added after each shallow segment. These newly added layers are only used during training. During training, the deepest segment is used as the teacher model, and the other segments are used as student models for knowledge distillation training. Each student model has 3 sources of supervision: (1) the cross-entropy loss of the labels; (2) the KL divergence loss with the softmax output of the teacher model; and (3) the L2 distance loss with the feature map of the teacher model. The deepest segment (teacher model) only uses the cross-entropy loss of the labels as supervision, and the loss function of the entire network is the weighted sum of the loss functions of each segment. During the inference stage, all newly added classifier parts are removed, and only the convolutional network part is retained, so as not to increase the number of parameters and response time of the network during inference.
[0089] In addition to label cross-entropy loss, the student model during training also incorporates soft label distillation loss and feature map transfer loss from the teacher. By performing self-distillation within the network, shallower layers can acquire more information from deeper layers, improving the overall discriminative ability of the network. Furthermore, self-distillation converges to a wider and flatter minimum, enhancing model robustness and ultimately improving accuracy. Self-distillation also provides scalability by adjusting the inference time and accuracy based on network depth, allowing for dynamic selection of different classifiers according to resource constraints. When resources are limited, shallow classifiers ensure sufficient response speed. Although shallow classifiers have lower accuracy, training through self-distillation surpasses independent training, making them suitable for response-sensitive applications such as speech detection.
[0090] In one possible implementation, the loss function includes a first loss function, a second loss function, and a third loss function; the first loss function is used to characterize the matching degree between the first category label, the second category label, and the true category label, respectively; the second loss function is used to characterize the distance between the first probability distribution of the category features included in the first category label and the second probability distribution of the category features included in the second category label; and the third loss function is used to characterize the similarity between the features corresponding to the first category label and the second category label.
[0091] In this embodiment, the multi-level joint loss function method guides the student model to learn knowledge better by starting from the training process. Forged speech recognition models may face the problem of class imbalance, i.e., a significant difference in the number of normal speech samples and forged speech samples. This may cause the model to overemphasize the more numerous classes while ignoring the fewer numerous classes. By adjusting the weights of the loss function at different training stages of the model, knowledge transfer is performed at each stage of network formation to solve the problem caused by data imbalance, achieving better training results and improving the model's classification accuracy.
[0092] (1) First loss function (hard label loss function)
[0093] The hard-label loss function takes the true labels (real, original labels, a one-hot encoded vector where the position corresponding to the true class is 1 and the other positions are 0), the deepest classifier, and the weighted cross-entropy loss of all shallow classifiers as input. It is computed using the labels from the training dataset and the classifier outputs to fully extract the hidden knowledge in the training dataset. The student model is trained by using hard labels and the teacher model's prediction output, ensuring that the student model's predictions are as close as possible to the teacher model's discrete predictions.
[0094] The hard-label loss function uses the weighted cross-entropy loss function to measure the difference between the model's predicted values and the hard labels (i.e., the true labels), thereby guiding the optimization of model parameters. In addressing the class imbalance problem in voice data from the telecommunications fraud field, different weights are assigned to samples of different classes to better handle the imbalanced sample size. The calculation method of the weighted cross-entropy loss function is shown in the following formula:
[0095] L1=-∑w(i)*p(i)*log(q(i));
[0096] Where: w(i) is the weight of the i-th class, p(i) is the probability of the i-th class in the true label, q(i) is the probability of the i-th class predicted by the model, and Σ represents the summation over all classes. The hard-label loss function is used to measure the difference between the predictions of the student model and the teacher model, that is, the student model should try to imitate the output of the teacher model as much as possible. Through the hard-label loss function, the student model can learn the classification decisions and task-critical information of the teacher model, thereby improving model performance.
[0097] (2) Second loss function (soft label loss function)
[0098] The soft-label loss function takes the outputs of the student and teacher model classifiers as input. The soft label refers to the continuous probability distribution for each class, obtained from the teacher model's output. Soft labels contain richer information, not only telling the student model which class is correct but also providing relevance information to other classes. By calculating the difference in distributions between the two outputs, the shallow network can be guided to learn more. The KL divergence loss function measures the difference between the model's predicted output and the teacher model's soft labels and is incorporated into the softmax layer of each shallow classifier. The formula for the KL divergence loss function is as follows:
[0099] L2=∑p_teacher*logp_teacher / p_student;
[0100] Where p_teacher is the prediction distribution of the teacher model and p_student is the prediction distribution of the student model.
[0101] Because of knowledge distillation, a temperature coefficient T needs to be added to the softmax function to extract negative labels, allowing the soft labels to function more effectively. In the self-distillation model, the softmax function after introducing the temperature coefficient T is as follows:
[0102]
[0103] Among them, z i It is the score of input category i, z jThis represents the model's raw output score or rating for the j-th class in a classification problem. The temperature coefficient T is a hyperparameter used to control the "hardness" of the Softmax function's output distribution. A higher T makes the output distribution smoother, reducing the probability differences between classes, while a lower T makes the output distribution sharper. The temperature coefficient T is used to adjust the output distribution of the teacher model, and then this adjusted distribution is passed as soft labels to the student model. The purpose of this is to make the soft labels more "fuzzy," so that the student model can learn the knowledge of the teacher model, rather than just a single highest probability class. By using the temperature scaling technique, the output distribution of the Softmax function becomes smoother, which helps to convey the knowledge of the teacher model, allowing the soft labels to better guide the training of the student model.
[0104] The soft-label loss function measures the difference between the output distribution of the student model and the output distribution of the teacher model, with the teacher model's output being treated as a probability distribution (soft label). The student model gains richer information; the soft labels not only tell the student model the correct answer but also the relative confidence levels between different categories, thus improving the model's generalization ability.
[0105] (3) Third loss function (feature loss function)
[0106] The feature loss function, calculated by the Huber loss between the fourth layer and the first three shallow layers, allows the implicit knowledge from deep features to be incorporated into shallow features, enabling the shallow network to better match the deep network during prediction. The Huber loss function is introduced to adapt the feature mappings of the shallow classifiers to those of the deepest classifier, thereby transferring implicit knowledge and improving model performance. The formula for calculating Huber Loss is as follows:
[0107] L3=∑[0.5*(y p -y t ) 2 ];
[0108] Huber Loss can handle outliers better when the prediction error satisfies |y p -y t When |≤δ, Huber Loss degenerates into mean squared error loss because the squared term performs well in this region. p y represents the model's predicted value, that is, the model's predicted output for a given input. t This represents the actual value, i.e., the true label value in the dataset. When the prediction error is large, Huber Loss degenerates into Mean Absolute Error Loss because the absolute value term is more robust in this region and can better resist the influence of outliers.
[0109] By minimizing the feature loss function, the model learns similar feature representations across different layers, thereby promoting knowledge transfer and improving generalization ability. Feature mapping layers are added to the model architecture to perform non-linear transformations on the input data to extract richer and more meaningful feature representations. These feature mapping layers are added before the classifier in each shallow segment to enhance the feature representation capability of each layer. The feature loss function measures the difference between the feature representation layers of the student model and the teacher model, helping the student model learn more meaningful feature representations. The feature loss function helps the student model learn more informative features, rather than simply mimicking the output of the teacher model, thus improving the model's robustness and generalization ability.
[0110] Hard-label loss measures the difference between the student model's predictions and the teacher model's predictions, while soft-label loss measures the difference between the student model's output distribution and the teacher model's output distribution. Combining these two loss functions can effectively perform knowledge distillation, helping the student model gain more information from the teacher model and improve generalization ability. Additionally, feature loss functions can be used in conjunction with these two loss functions during training to help the model learn more meaningful feature representations.
[0111] In one possible implementation, determining the model detection loss value based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model includes:
[0112] Based on the first category label and the second category label, determine the loss value of each loss function;
[0113] The loss weight corresponding to each loss function is determined based on the loss value of each loss function.
[0114] The loss values of each loss function are weighted according to the loss weights to obtain the model detection loss value.
[0115] In this embodiment, during the training between the student model and the teacher model, it is necessary to use appropriate loss functions and training strategies to make the model's predictions as consistent as possible with the labels in order to minimize the loss. When training a deep learning model, better balancing the loss weights of different samples achieves the following training effects: (1) In the self-distillation model, both hard and soft labels are used to guide the training of the student model. Due to their different properties, their impact on the model's learning is also different. The adaptive weight loss function can adjust the weights of hard and soft labels according to the training progress to balance their influence, thereby better conveying the knowledge of the teacher model; (2) In the early stages of training, the model may rely more on the output (hard labels) of the teacher model because the student model may perform poorly on the task. By setting larger hard label weights, the convergence of the student model can be accelerated, enabling it to learn the knowledge of the teacher model more quickly, thereby improving training efficiency; (3) The adaptive weight loss function can also be used to control overfitting. By adjusting the weights of different samples, more attention can be paid to the relatively difficult samples during model training, thereby reducing overfitting of easy samples.
[0116] We chose to use the Adaptive Mixing of Losses method to jointly train the hard-label loss function, soft-label loss function, and feature loss function. A weighting network is introduced to adaptively mix the different loss functions. This weighting network learns how to dynamically mix different loss functions, so the model can learn how to assign weights to different loss functions during training without manual adjustment. The calculation formula is as follows:
[0117]
[0118] Among them, L t This is a multi-weighted joint loss function, where w1, w2, and w3 are the weights corresponding to each loss function. The weights are dynamically calculated by the weight network based on the input data to adaptively adapt to the needs of different tasks and data. The hard-labeled loss function and the soft-labeled loss function adjust the weights through a power function.
[0119]
[0120] Adjusting the weights of the loss function using the power law allows you to strengthen or weaken the impact of a particular loss function on the total loss as needed. Different parameter values can achieve different weighting effects: In the early stages of training, the student model may have a weak understanding of the task and struggle to accurately mimic the teacher model's output. Increasing the weight of hard labels can help the student model converge faster to a distribution similar to the teacher model's output, thus accelerating the convergence process. Hard labels typically come from the teacher model's output and can be considered relatively "coarse" knowledge. In the early stages of training, the student model can focus more on this coarse knowledge through stronger hard label weights, thus establishing an initial understanding of the task. As training progresses, the student model gradually becomes more accurate and better fits the teacher model's output distribution. At this point, the weight of hard labels can be gradually decreased while the weight of soft labels is increased, allowing the model to focus more on details and finer-grained knowledge. Hard and soft labels each have different advantages. Hard labels are generally more stable and reliable, while soft labels provide more information, including the uncertainty of the teacher model. By adjusting the weights, the advantages of both can be combined to help the student model learn better.
[0121] Figure 11 This document illustrates a flowchart of a forged voice detection method provided in an embodiment of this application. The execution entity of this method can be a terminal device or a server. The terminal device can be a personal computer, a mobile device such as a mobile phone, tablet, or IPTV, or a user-used terminal device. The server can be a standalone server or a server cluster composed of multiple servers. Furthermore, the server can be a backend server for a specific business function or a backend server for a platform or application (e.g., a virtual assistant, social media platform, chatbot, voice security and anti-fraud application, etc.). This embodiment uses a server as the execution entity for illustration. For the terminal device scenario, the following related content can be used, and will not be elaborated further here. As shown in the figure, the training method 1100 for this forged voice detection model may include the following steps:
[0122] S1101: Obtain the input target audio file, which includes target text data and target audio data.
[0123] In this embodiment of the application, an input target audio file is obtained, which includes target text data and target audio data, so as to extract multimodal features through the target text data and target audio data to identify the authenticity of the target audio file.
[0124] S1102: Extract multimodal features from the target text data and target audio data, perform feature fusion processing on the multimodal features, and obtain target fusion features.
[0125] The multimodal features include semantic features corresponding to the target text data, sub-band features corresponding to the target audio data, and high-level features extracted based on the spectrogram corresponding to the target audio data.
[0126] In this embodiment, target text data and target audio data can be preprocessed by a text preprocessing unit and a data preprocessing unit, respectively, to obtain multimodal information. A multimodal feature extraction unit then extracts features from the preprocessed multimodal information to obtain multimodal features. A weight matrix is then determined based on the semantic features, subband features, and high-level features included in the multimodal features. This weight matrix is used to weight the multimodal features in the audio sample file to obtain the target fusion features corresponding to the target audio file.
[0127] S1103: Obtain the category label of the target audio file by inputting the target fusion feature into the fake speech detection model; wherein, the fake speech detection model is trained according to the training method of the fake speech detection model described above.
[0128] In this embodiment of the application, the target fusion features are judged by the trained fake speech detection model and the category label of the target audio file is output.
[0129] S1104: Determine the authenticity of the target audio file based on the category label.
[0130] In this embodiment of the application, the classification label output by the fake speech detection model is used to determine whether the speech in the target audio file is real speech, and the discrimination result of fake speech detection is output.
[0131] This application provides a method for detecting forged speech. The method involves acquiring an input target audio file; extracting multimodal features from target text data and target audio data; performing feature fusion processing on the multimodal features to obtain target fused features; inputting the target fused features into a forged speech detection model to obtain the category label of the target audio file; and determining the authenticity of the target audio file based on the category label. By extracting multimodal features from the target audio file and then performing feature fusion processing before inputting these features into the forged speech detection model, the forged speech detection model can fully explore the correlations between different modal features, thereby obtaining more accurate category labels and improving the accuracy of forged speech detection.
[0132] Figure 12This illustration shows a schematic diagram of the structure of a training device for a fake speech detection model provided in an embodiment of this application. This training device for the fake speech detection model can achieve the following: Figure 1 The training device 1200 for the fake speech detection model, comprising all or part of the contents shown in the embodiment, includes:
[0133] The sample acquisition module 1210 is used to acquire multiple audio sample files, wherein the audio sample files include text data and audio data;
[0134] The feature fusion module 1220 is used to extract multimodal sample features from the text data and audio data of multiple audio sample files, perform feature fusion processing on the multimodal sample features, and obtain the fused features corresponding to each audio sample file.
[0135] The model training module 1230 is used to iteratively train a pre-constructed self-distillation acoustic model using fusion features corresponding to multiple audio sample files. In each iteration, the fusion features are fused from multiple perspectives to obtain target acoustic features. The target acoustic features are input into the teacher model to obtain a first category label, and the target acoustic features are input into the student model to obtain a second category label.
[0136] The model detection loss value is determined based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model.
[0137] The student model is updated iteratively based on the loss value detected by the model until the preset convergence condition is met. The self-distilled acoustic model after convergence is then identified as the fake speech detection model.
[0138] In one possible implementation, the sample acquisition module 1210, when extracting multimodal sample features from the text data and audio data of multiple audio sample files, and performing feature fusion processing on the multimodal sample features to obtain the multimodal fused features corresponding to each audio sample file, is specifically used for:
[0139] For each audio sample file, multimodal sample features are extracted from the text data and audio data of the audio sample file. The multimodal sample features include semantic features corresponding to the text data, subband features corresponding to the audio data, and high-level features extracted based on the spectrogram corresponding to the audio data.
[0140] The weight matrix is determined based on the semantic features, subband features and high-level features. The weight matrix is then used to weight the multimodal sample features in the audio sample file to obtain the multimodal fusion features corresponding to the audio sample file.
[0141] Obtain the multimodal fusion features corresponding to each audio sample file in the multiple audio sample files.
[0142] In one possible implementation, the loss function includes a first loss function, a second loss function, and a third loss function;
[0143] The first loss function is used to characterize the matching degree between the first category label, the second category label and the true category label, respectively;
[0144] The second loss function is used to characterize the distance between the first probability distribution of the category features included in the first category label and the second probability distribution of the category features included in the second category label;
[0145] The third loss function is used to characterize the similarity between the features corresponding to the first category label and the second category label.
[0146] In one possible implementation, the model training module 1230, when determining the model detection loss value based on the first category label, the second category label, and multiple loss functions of the self-distilling acoustic model, is specifically used for:
[0147] Based on the first category label and the second category label, determine the loss value of each loss function;
[0148] The loss weight corresponding to each loss function is determined based on the loss value of each loss function.
[0149] The loss values of each loss function are weighted according to the loss weights to obtain the model detection loss value.
[0150] Figure 13 This illustration shows a structural schematic diagram of a forged voice detection device provided in an embodiment of this application. This forged voice detection device can achieve the following: Figure 11 The forged voice detection device 1300, comprising all or part of the contents shown in the embodiment, includes:
[0151] The file acquisition module 1310 is used to acquire the input target audio file, wherein the target audio file includes target text data and target audio data;
[0152] The feature extraction module 1320 is used to extract multimodal sample features from the target text data and target audio data, and to perform feature fusion processing on the multimodal sample features to obtain target fusion features;
[0153] The category determination module 1330 is used to obtain the category label of the target audio file by inputting the target fusion features into the fake speech detection model; wherein, the fake speech detection model is trained according to the training device 1200 of the fake speech detection model described above;
[0154] The voice detection module 1340 is used to determine the authenticity of the target audio file based on the category label.
[0155] Figure 14 The diagram shows another spoofed speech detection device provided in this application embodiment. The spoofed speech detection device includes: a multimodal feature extraction module, a self-distillation acoustic model training module, a multi-level loss function sub-module, and an inference module.
[0156] The multimodal feature extraction module extracts multimodal features from audio and text data and performs feature fusion processing on these features for the speech detection model to learn. The audio and text data are corresponding. The module's input is the training dataset, which includes both audio and text data. This module includes:
[0157] The text preprocessing unit takes the source text data as input and performs preprocessing operations on the text data, including word segmentation, stop word filtering, text normalization, and data cleaning of unnecessary information such as punctuation marks. The output is the preprocessed text sequence.
[0158] The word embedding vector unit performs word embedding vector operations on the preprocessed text sequence, mapping Chinese words to a vector space for subsequent semantic feature extraction, and outputs a 100-dimensional word embedding vector sequence.
[0159] The semantic feature extraction unit takes a 100-dimensional word embedding vector sequence as input, uses the BERT model for semantic feature extraction, expresses the relevant fraud semantic information as features, assigns weights to relevant semantic and sentiment information for subsequent model training, and outputs a 768-dimensional text feature vector.
[0160] The data preprocessing unit takes raw audio data as input and performs preprocessing operations on the acquired speech data to enhance data separability, reduce noise, and enhance feature discrimination. The output is the preprocessed audio segment.
[0161] The sub-band feature extraction unit takes the preprocessed audio segment as input, uses wavelet transform to decompose each frame of speech signal into sub-bands of different frequency ranges to capture artifacts in the forged speech, and outputs a multi-frame speech segment containing sub-band feature information.
[0162] The speech embedding vector unit takes the audio segment after sub-band feature extraction as input, applies Wav2Vec to convert the processed audio data into a vector representation for subsequent modality fusion operations of the model, and outputs a speech embedding vector sequence corresponding to the audio frame with a dimension of 512.
[0163] The spectrogram feature extraction unit takes the preprocessed audio segment as input, converts it into a spectrogram, converts the audio time-domain signal into a frequency-domain feature representation, and captures high and low frequency features in the signal for subsequent feature extraction. The output is a 512*512 spectrogram with a 2-dimensional dimension.
[0164] The convolutional network feature extraction unit takes the converted spectrogram as input and extracts it using a CNN model. The convolutional layer is mainly used to capture high-frequency and low-frequency features that are significantly different between fake and real speech in the spectrogram. The pooling layer is used to reduce the dimensionality of the feature map and outputs a 1*512 feature vector.
[0165] The modal feature complementation unit inputs the extracted 1*n-dimensional feature vectors into the modal feature complementation unit. The Transformer model is used and a cascaded cross-attention module is introduced to enable cross-complementation of modal information to promote deep modal fusion.
[0166] The multimodal feature fusion unit uses stacked Transformer layers for iterative modal fusion and introduces a multi-head attention mechanism to dynamically allocate attention according to task needs. The features fused from each layer are used as input to the next layer to increase continuous interaction between modalities and achieve deep fusion.
[0167] The self-distillation acoustic model training module includes:
[0168] The deep residual network unit adopts the ECA-MPIF-Res2Net-50 network architecture to receive various parameters from the multimodal feature fusion unit and train the acoustic model.
[0169] The efficient channel attention mechanism unit is used to calculate the correlation between different channels to automatically learn the target acoustic features of the speech signal, thereby improving the model's ability to recognize important details.
[0170] The student model unit is used to connect to the shallow classifier and input the feature map signal of each training block into the shallow classifier.
[0171] The teacher model unit receives feature information from the previous layer of the network, outputs information to the temperature scaling subunit, and outputs feature map information to the deep classifier unit.
[0172] The shallow classifier unit takes the student model feature map information as input, outputs the feature map information to the shallow classifier feature mapping subunit, and outputs the student model prediction information to the classifier fusion unit.
[0173] The deep classifier unit takes the teacher model feature map information as input, outputs the feature map information to the deep classifier feature mapping subunit, and outputs the teacher model prediction information to the classifier fusion unit.
[0174] The classifier fusion unit is used to receive feature map information from deep and shallow classifiers and output it to the classifier prediction subunit, and to receive prediction information from deep and shallow classifiers and output it to the student model prediction subunit.
[0175] The multi-level loss function submodule is used to balance the loss weights of different samples in the model. By jointly calculating the hard loss function, soft loss function and feature loss function, and adjusting the weights of the model at different training stages, it can better take into account the differences between different categories of samples, thereby improving the model's ability to identify minority class data.
[0176] The model evaluation unit assesses the model's training performance and determines whether to continue training. When the model's training loss falls below a set threshold, training stops, and optimal model parameters are generated; otherwise, training continues.
[0177] The self-distillation optimal model generation unit acquires and saves the parameters of the optimal model obtained after training, and uses them during model inference.
[0178] The multi-level loss function submodule is used to measure the accuracy of model predictions, guide model training, help the model better adapt to specific tasks, and better balance the loss weights of different samples. This module includes:
[0179] The classifier prediction input subunit takes the output of the classifier fusion unit as its input and processes the prediction outputs of the deep classifier and all shallow classifiers.
[0180] The weighted cross-entropy loss function computation subunit calculates the loss between the labels of the training dataset and the outputs of all classifiers, used to fully extract the hidden knowledge in the training dataset. It measures the difference between the model classifier's predictions and the true labels, thus guiding the optimization of model parameters; and assigns different weights to samples of different classes to better handle imbalanced sample sizes.
[0181] The student model predicts the output subunit, which takes the outputs of the shallow and deep classifiers as input, enabling the shallow network to learn deeper levels of knowledge.
[0182] The temperature scaling subunit uses a temperature coefficient T to adjust the output distribution of the teacher model, making the soft labels more "fuzzy," meaning the student model can learn the knowledge from the teacher model, rather than just a single maximum probability class. Using temperature scaling in knowledge distillation makes the output distribution of the Softmax function smoother, helping to convey the teacher model's knowledge and allowing the soft labels to better guide the training of the student model.
[0183] The KL divergence loss function calculation sub-unit takes the student model's predicted output and soft labels as input, enabling the student model to better fit the teacher model's output distribution.
[0184] The soft-label input subunit passes the output distribution of the teacher model classifier, which has been adjusted for temperature scaling, to the student model for calculating the KL divergence loss function.
[0185] The deep classifier feature mapping subunit is used to perform non-linear transformations on the input data to extract richer and more meaningful feature representations;
[0186] The feature mapping subunit of the shallow classifier is used to enhance the feature representation capability of each shallow network by adjusting the weights and activation functions to adapt to features at different levels.
[0187] The Huber loss function computation subunit is used to compute the feature mapping between deeper hidden layers and each shallow network. It introduces the non-explicit knowledge of deep features into shallow features, so that the shallow feature network can better match the deep network during prediction, thereby conveying implicit knowledge and improving model performance.
[0188] The multi-weighted joint loss function unit uses an adaptive weight loss function to assign weights to the hard label loss function, soft label loss function, and feature loss function. This better balances the loss weights of different samples when training deep learning models and reduces the impact of abnormal audio samples on training.
[0189] The inference module performs forged speech detection on the input audio data and outputs the final detection result. This module includes:
[0190] The ASR unit is used to convert the acquired audio data into text for subsequent processing.
[0191] The data preprocessing unit is used to preprocess the acquired test speech data to enhance data separability, reduce noise, and enhance feature discrimination.
[0192] The text preprocessing unit is used to preprocess text data, including word segmentation, stop word filtering, text normalization, and data cleaning of unnecessary information such as punctuation marks for Chinese sequences.
[0193] The multimodal feature extraction unit extracts features from the preprocessed audio and text data for subsequent model detection.
[0194] The fake speech detection unit is used to analyze the extracted and fused features through a model to determine whether the input speech is real speech.
[0195] Figure 15 This diagram illustrates the hardware structure of an electronic device implementing the embodiments of this application. Referring to the diagram, at the hardware level, the electronic device includes a processor and optionally, an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.
[0196] The processor, network interface, and memory can be interconnected via an internal bus, which can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus.
[0197] Memory stores programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.
[0198] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, logically forming a device that locates the target user. The processor executes the program stored in memory and specifically performs the following: Figure 1 or Figure 11 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.
[0199] The above is as stated in this application. Figure 1 or Figure 11 The methods disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0200] The computer device can also execute the methods described in the preceding method embodiments and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.
[0201] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0202] This application also proposes a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform... Figure 1 or Figure 11 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.
[0203] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc.
[0204] Furthermore, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, implement the following process: Figure 1 or Figure 11 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.
[0205] In summary, the above description is merely a preferred embodiment of this application and does not limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
[0206] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0207] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can store information accessible to a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0208] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0209] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
Claims
1. A training method for a fake speech detection model, characterized in that, include: Acquire multiple audio sample files, wherein the audio sample files include text data and audio data; Multimodal sample features are extracted from the text data and audio data of multiple audio sample files, and feature fusion processing is performed on the multimodal sample features to obtain the fused features corresponding to each audio sample file; wherein, the multimodal sample features include semantic features corresponding to the text data for representing sensitive semantic information and emotional semantic information, subband features corresponding to the audio data, and high-level features extracted based on the spectrogram corresponding to the audio data; The pre-constructed self-distillation acoustic model is iteratively trained using fusion features corresponding to multiple audio sample files. In each iteration, multi-view information fusion is performed on the fusion features to obtain target acoustic features. The target acoustic features are input into the teacher model to obtain a first category label, and into the student model to obtain a second category label. The target acoustic features are obtained by extracting multi-scale information from the fusion features using dilated convolutions with different dilation rates, and fusing the multi-scale features corresponding to different dilation rates. The model detection loss value is determined based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model. The student model is updated iteratively based on the loss value detected by the model until the preset convergence condition is met. The self-distilled acoustic model after convergence is then identified as the fake speech detection model.
2. The method according to claim 1, characterized in that, The step of extracting multimodal sample features from the text and audio data of multiple audio sample files, and performing feature fusion processing on the multimodal sample features to obtain the multimodal fused features corresponding to each audio sample file includes: For each audio sample file, extract multimodal sample features from the text data and audio data of that audio sample file; The weight matrix is determined based on the semantic features, subband features and high-level features. The weight matrix is then used to weight the multimodal sample features in the audio sample file to obtain the multimodal fusion features corresponding to the audio sample file. Obtain the multimodal fusion features corresponding to each audio sample file in the multiple audio sample files.
3. The method according to claim 1, characterized in that, The loss function includes a first loss function, a second loss function, and a third loss function; The first loss function is used to characterize the matching degree between the first category label, the second category label and the true category label, respectively; The second loss function is used to characterize the distance between the first probability distribution of the category features included in the first category label and the second probability distribution of the category features included in the second category label; The third loss function is used to characterize the similarity between the features corresponding to the first category label and the second category label.
4. The method according to claim 1, characterized in that, The step of determining the model detection loss value based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model includes: Based on the first category label and the second category label, determine the loss value of each loss function; The loss weight corresponding to each loss function is determined based on the loss value of each loss function. The loss values of each loss function are weighted according to the loss weights to obtain the model detection loss value.
5. A method for detecting spoofed voice, characterized in that, include: Obtain the input target audio file, which includes target text data and target audio data; Multimodal features are extracted from the target text data and target audio data, and feature fusion processing is performed on the multimodal features to obtain target fusion features; The category label of the target audio file is obtained by inputting the target fusion feature into the fake speech detection model; wherein, the fake speech detection model is trained by the training method of the fake speech detection model according to any one of claims 1 to 4; The authenticity of the target audio file is determined based on the category label.
6. A training device for a fake speech detection model, characterized in that, include: The sample acquisition module is used to acquire multiple audio sample files, wherein the audio sample files include text data and audio data; The feature fusion module is used to extract multimodal sample features from the text data and audio data of multiple audio sample files, and to perform feature fusion processing on the multimodal sample features to obtain the fused features corresponding to each audio sample file; wherein, the multimodal sample features include semantic features corresponding to the text data for representing sensitive semantic information and emotional semantic information, subband features corresponding to the audio data, and high-level features extracted based on the spectrogram corresponding to the audio data; The model training module is used to iteratively train a pre-built self-distillation acoustic model using fusion features corresponding to multiple audio sample files. In each iteration, multi-view information fusion is performed on the fusion features to obtain target acoustic features. The target acoustic features are input into the teacher model to obtain a first category label, and input into the student model to obtain a second category label. The target acoustic features are obtained by extracting multi-scale information from the fusion features using dilated convolutions with different dilation rates, and fusing the multi-scale features corresponding to different dilation rates. The model detection loss value is determined based on the first category label, the second category label, and multiple loss functions of the self-distillation acoustic model. The student model is updated iteratively based on the loss value detected by the model until the preset convergence condition is met. The self-distilled acoustic model after convergence is then identified as the fake speech detection model.
7. A device for detecting fake voice, characterized in that, include: The file acquisition module is used to acquire the input target audio file, which includes target text data and target audio data. The feature extraction module is used to extract multimodal features from the target text data and target audio data, and to perform feature fusion processing on the multimodal features to obtain target fused features; The category determination module is used to obtain the category label of the target audio file by inputting the target fusion features into the fake speech detection model; wherein, the fake speech detection model is trained by the training device of the fake speech detection model according to claim 6; The voice detection module is used to determine the authenticity of the target audio file based on the category label.
8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 5.
10. A computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the method as described in any one of claims 1 to 5.
