Voice forgery identification method and device based on dynamic multi-view fusion

Abstract

The invention discloses a dynamic multi-view fusion voice forgery identification method and a dynamic multi-view fusion voice forgery identification device. The method comprises the following steps: acquiring multi-view dynamic feature data according to an original voice signal, and constructing a multi-view data set; clustering the data in the multi-view data set by adopting a dynamic weight clustering integration algorithm to generate a pseudo tag; integrating the pseudo tag with the multi-view dynamic feature data to obtain a first enhanced data set, and training a K-nearest neighbor classifier; and using the trained K neighbor classifier to predict unseen data points except the original voice signals in the voice signals needing to be identified. According to the method, the multi-view voice information is mined through unsupervised learning, and the defects of an existing method in generalization, robustness and feature utilization are overcome. The method is excellent in performance on multiple data sets, key indexes are remarkably improved, generalization and robustness are high for different counterfeit data points, and a new path is provided for voice deep counterfeit detection.

Description

Technical Field

[0001] The present invention relates to the technical field of voice deep forgery detection, and in particular to a dynamic multi-view fusion voice forgery identification method and device. Background Art

[0002] With the rapid development of artificial intelligence generated content (AIGC) technology, speech synthesis technology has made significant progress in anthropomorphism, authenticity, and naturalness. In particular, the application of large models has made the boundary between fake and real speech increasingly blurred. Although this technological advancement has promoted the development of the field of speech synthesis, it has also brought serious security risks, such as identity fraud and the spread of false information. Therefore, the development of efficient and robust speech deep fake detection technology is of great practical significance.

[0003] Existing voice deep fake detection technology mainly relies on supervised learning methods, the core of which includes acoustic feature extraction and classification algorithms. For example, Chinese patent document CN119170052A discloses a voice forgery detection method based on a multi-scale GMM-ResNet model, which uses feature fusion to train the ResNet model; Chinese patent document CN117809694A discloses a forged voice detection method and system based on temporal multi-scale feature representation learning, which extracts preliminary features through wav2vec2.0, and then inputs a convolutional network based on a multi-scale time series to extract the feature matrix. The feature matrix is ​​input into the SCG-Res2Net50 model for training. The above technologies are all based on self-supervised learning models, and the extraction of acoustic features is still focused on deep mining methods. Although the existing methods have achieved certain results on specific data sets, they still have the following limitations: 1. Overfitting problem: Existing methods mainly rely on supervised learning, which is prone to overfitting when data is limited, resulting in insufficient generalization ability of the model in unknown forged audio scenarios.

[0004] 2. Limitations of feature mining: Current research mainly focuses on the deep mining and fusion of acoustic features (i.e., “deep analysis”), while the multi-angle study of speech data (i.e., “breadth analysis”) has not been fully explored, which limits the further improvement of detection performance.

[0005] 3. Neglect of unsupervised learning: Most existing methods rely on labeled data. Unsupervised learning methods have significant advantages in utilizing large amounts of unlabeled data and adapting to different data distributions, but their application in the field of deep fake voice detection is still relatively limited. Summary of the invention

[0006] The present invention aims to solve the problems of overfitting, insufficient generalization ability and insufficient feature utilization existing in the existing voice deep fake detection technology, and proposes a voice forgery identification method and device based on dynamic multi-view fusion. By constructing a voice multi-view dataset, a dynamic weight clustering ensemble algorithm is used for clustering and a K nearest neighbor classifier is trained in combination with pseudo-label enhancement, dynamic K value selection and confidence calibration to improve the robustness and accuracy of the K nearest neighbor classifier.

[0007] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0008] A method for identifying voice forgery based on dynamic multi-view fusion, comprising the following steps: The following steps are involved: S1. According to the original speech signal, obtain multi-view feature data including reconstruction error view, acoustic feature view and quality assessment view, and construct a multi-view dataset; S2. clustering the multi-view feature data in the multi-view dataset using a dynamic weight clustering integration algorithm to generate pseudo labels; S3, integrating the pseudo labels with the multi-view feature data to obtain a first enhanced data set for training a K nearest neighbor classifier; S4. Use the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

[0009] Furthermore, in step S1, the speech generation model HiFiGAN is used to calculate the reconstruction error view, and the time-frequency domain residual analysis is introduced to quantify the local anomalies in the speech generation process, as follows:

[0010] in, Represents reconstruction error (Reconstruction Error); is a variable, representing the original speech signal; Represents the original speech signal At the moment The value of Indicates the total duration of the speech signal (the length of the time series), that is, the total number of time points from the start time to the end time; Represents the decoder in the speech generation model HiFiGAN; Represents the encoder in the speech generation model HiFiGAN; It is a hyperparameter used to balance the weights of the two terms in the formula and is a man-made constant. Represents the original speech signal At the moment Total Variation, which is an indicator used to measure the degree of local variation of the signal; The audio coding model Encodec is used to obtain the acoustic feature view, extract the multi-scale acoustic code, combine the convolutional network to perform feature dimensionality reduction, and then calculate the inner product of the reduced dimensionality vector and normalize it, as follows: The original speech signal (such as wav, MP3 format) is input into the audio coding model Encodec to generate multi-scale acoustic codes; these codes are an abstract representation of audio data, and they contain important audio information, such as pitch, timbre, rhythm and other features at different scales. The multi-scale acoustic code is first extracted through the convolution layer to extract local features, and then the global average pooling layer is used to calculate the average of each channel and compress it into a one-dimensional vector. Finally, the inner product of the one-dimensional vector itself is calculated to obtain the acoustic feature view; The UTMOS model, which is specifically used for voice quality assessment, is used to obtain the quality assessment view, as follows: Input the original speech signal (such as WAV or MP3 format) into the UTMOS model, output the overall speech quality assessment score, and normalize the data to obtain the quality assessment view; The structure of the multi-view dataset is: {reconstruction error view, acoustic feature view, quality assessment view}.

[0011] Further, step S2 includes the following steps: S2.1, adjusting the weight of each view in a multi-view dataset based on information entropy; S2.2, construct a hypergraph that integrates the similarity information of each view; S2.3, based on the hypergraph in step S2.2, generate meta-clustering groups using spectral clustering, as follows: First construct the Laplacian matrix of the hypergraph And the Laplace matrix Perform eigendecomposition to obtain eigenvalues; then automatically determine the optimal number of clusters using the eigenvalue gap method , before selecting The eigenvectors corresponding to the smallest eigenvalues ​​form a matrix , the matrix Each row is considered as a new data point and clustered using traditional clustering algorithms such as K-means to obtain meta-cluster groups; S2.4. Calculate the scores of data points in the multi-view dataset to meta-cluster groups and determine pseudo labels as follows: Through the distance measurement method, the average Euclidean distance from the data point to all data points in the meta-clustering group is calculated to obtain the score of the data point to the meta-clustering group. Then, according to the minimum distance method, the meta-clustering group label with the smallest distance is taken to determine the pseudo-label of the data point.

[0012] Furthermore, in step S2.1, the weight of each view in the multi-view dataset is dynamically adjusted according to the information entropy difference between the views; In the multi-view dataset The information entropy of a view is calculated as follows: The probability of each data point appearing in each view is calculated by taking its logarithm and multiplying it by this probability. The results of this calculation for all data points are added together and the negative number is taken at the end. After obtaining the information entropy of each view, first The information entropy of the first view is exponentially calculated as the numerator; then the information entropy of the three views is exponentially calculated and added together as the denominator; the numerator is divided by the denominator to obtain the The weights of the views are as follows:

[0013]

[0014] in, Represents the first The weight of each view; Indicates The information entropy of a view, the larger its value, the richer the information contained in the view and the higher the uncertainty; represents the exponential function; Indicates The number of data points in each view is the same for all views; Indicates In the view Data points Probability of occurrence.

[0015] Furthermore, in step S2.2, a hypergraph integrating similarity information of each view is constructed, as follows: First, multi-view clustering is performed. The multi-view dataset is , Indicates views; the data in the multi-view dataset are respectively After clustering, we get a set of cluster divisions ,in Representative cluster members (i.e., base clusters), each cluster member consists of multiple cluster subsets, denoted as ,in To indicate the The total number of cluster subsets in the sub-clustering partition, Indicates The first of the cluster members Cluster subsets; The data points in the multi-view dataset are uniformly numbered to calculate the similarity between the data points without distinguishing the specific views. Then, the multi-view similarity is fused. The relevant similarity under each view is calculated and summed, and then the cross-view similarity is added. Figure 1 The consistency coefficient is the same as the first data points and The product of the cosine similarity of the feature vectors of the data points is obtained data points and The similarity between data points The details are as follows:

[0016] in, Represents the first data points and The similarity between data points; Represents the first The weight of the view; In the clustering process of views, data points will be classified into a cluster subset, and all the features in this cluster subset together constitute the first The data point in Feature set under view ; Indicated in In the view, The feature set to which the data point belongs With The feature set to which the data point belongs Between Similarity; Indicates cross-view Figure 1 The consistency coefficient is a constant parameter set artificially; Indicates The feature vector of the data point and The feature vector of the data point The cosine similarity between .

[0017] Further, step S3 includes the following steps: S3.1. Concatenate the pseudo-labels and the multi-view feature data to obtain a first enhanced data set. The structure of the first enhanced data set is: {reconstruction error view, acoustic feature view, quality assessment view, pseudo-label}. Divide the first enhanced data set into a validation set and a training set. S3.2, adjust the K value of the K nearest neighbor classifier according to the density of data points in the training set; S3.3, input the training set into the K nearest neighbor classifier. The K nearest neighbor classifier uses the weighted Euclidean distance to calculate the distance between the data points in the training set according to the K value in step S3.2 to find the K nearest neighbors of each data point. The data point category is determined according to the pseudo label (neighbor label) by voting method to obtain the K nearest neighbor classifier that has been preliminarily trained. The performance (accuracy, recall rate, F1 value) of the K nearest neighbor classifier that has been preliminarily trained is evaluated using the validation set. If the evaluation result meets the requirements or reaches the set maximum number of iterations, execute step S3.4. Otherwise, the parameters in the data point density calculation are set to zero. The set step size will be increased or decreased, the number of iterations will be increased by 1, and the process will return to step S3.2; S3.4. Use the Bayesian model to calibrate the output confidence of the initially trained K nearest neighbor classifier to obtain a trained K nearest neighbor classifier.

[0018] Furthermore, in step S3.2, the K value is adaptively adjusted according to the density of the data points in the training set; Data points , traverse all data points in the training set, for the Data points , calculate the Data points and Data points The distance between ; If the distance Less than the preset range , then the indicator function The value of is 1, otherwise it is 0; add the indicator function values ​​corresponding to all data points to get the Data points The number of local data points , which reflects the The number of other data points within the set range centered on the data point reflects the density of data points around the data point. Get the first Data points The number of local data points Afterwards, through Take the square root and round up to get Data points K value after adaptive adjustment , as follows:

[0019]

[0020] in, Indicates that for the first Data points , K value after adaptive adjustment; Indicates Data points The number of local data points, that is, The range of the setting is centered The number of remaining data points in the The density of surrounding data points; Indicates rounding up; represents the total number of data points in the training set; represents the indicator function, when When established, The value of is 1, otherwise it is 0; Indicates Data points and Data points The distance between.

[0021] Furthermore, in step S3.3, if the evaluation result shows that the K value of the K nearest neighbor classifier is too small, it may mean that the local density calculation range of the data point is too small, then the parameter Increase the set step size; if the evaluation results show that the K value of the K nearest neighbor classifier is too large, it may mean that the local density calculation range of the data point is too large, then increase the parameter Decrease the set step size.

[0022] Furthermore, in step S3.4, a Bayesian probability model is used to calibrate the confidence of the output of the K nearest neighbor classifier that has been initially trained; For the feature vector of the data point input to the K nearest neighbor classifier , we need to determine the given eigenvector Under the condition that Equal to one category of the data point Probability ; By indicating the function Determine the first of the K nearest neighbors Neighbors' categories With category Are they the same? When established, The value of is 1, otherwise it is 0; For the categories in the K nearest neighbors The weights of the neighbors are summed up, that is, , then Divide by the sum of the weights of the K nearest neighbors , based on the information of the K nearest neighbors, we get a preliminary estimate of the data point belonging to the category The probability part of Introducing feature vectors Information entropy , which reflects the eigenvector The uncertainty of the information contained. Through a hyperparameter To adjust the influence of information entropy in the entire probability calculation, that is .

[0023] The data point belongs to the category that is initially estimated based on the information of the K nearest neighbors. The probability part of Add together to get the final , is the feature vector at a given data point Under the condition that the data point belongs to the category The probability of the K nearest neighbor classifier is calculated by comprehensively considering the uncertainty of neighbor information and feature vectors, and the confidence calibration of the K nearest neighbor classifier output is realized, which makes the probability estimation of the classification result more accurate and helps to identify voice forgery more reliably, as follows:

[0024] in, Represents the feature vector at a given data point Under the condition that The probability of is a category of data points; represents the indicator function, when When established, The value of is 1, otherwise it is 0; The feature vector representing the data point input into the K nearest neighbor classifier contains various feature information of the data point in the multi-view feature data and is the basis for classification and confidence calculation; Represents the number of nearest neighbors in the K nearest neighbor classifier; Indicates The weight of the nearest neighbor; Indicates The category of the nearest neighbors; Express The category in the nearest neighbor The weights of the nearest neighbors are summed, reflecting that when the weights of the nearest neighbors are considered, The category in the nearest neighbor = 1 nearest neighbor pair feature vector Category = 1 contribution level; It is a hyperparameter set by humans to adjust the feature vector Information entropy of Represents the feature vector Information entropy is an indicator used to measure information uncertainty. Here, information entropy reflects the characteristic vector The uncertainty or information richness of its own features. Feature vector The higher the information entropy, the greater the uncertainty of its features. In confidence calibration, by introducing information entropy, the impact of the characteristics of the data point itself on the confidence of the classification result can be comprehensively considered.

[0025] Further, in step S4, steps S1 to S3 are performed on the speech signal to be identified to obtain a second enhanced data set corresponding to the speech signal to be identified; the first enhanced data set and the second enhanced data set have the same structure; The second enhanced data set is input into the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

[0026] The present invention also provides a dynamic multi-view fusion voice forgery identification device for implementing the above method, comprising the following modules: Data acquisition and preprocessing module: used to obtain the original speech signal and the speech signal to be identified; Multi-view feature extraction module: used to obtain multi-view feature data based on the original speech signal and construct a multi-view dataset; Integrated clustering module: used to cluster the data in the multi-view dataset using a dynamic weight clustering integrated algorithm to generate pseudo labels; Classifier training module: used to integrate pseudo labels with multi-view feature data to obtain a first enhanced data set and train a K-nearest neighbor classifier; Prediction module: used to use the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

[0027] Compared with the prior art, the advantages of the present invention are: Multi-view feature data construction: The present invention extracts multi-view features from three perspectives: speech acoustic features, overall speech quality assessment, and speech generation and reconstruction errors, to ensure the complementarity and consensus of the data.

[0028] Dynamic multi-view weight allocation: Traditional multi-view methods usually use fixed weights to fuse information from different views, which may cause the information of some views to be over-emphasized while the information of other views is ignored, resulting in feature redundancy. The present invention dynamically adjusts the view weights through information entropy, and can adaptively allocate weights according to the information richness of different views, avoiding feature redundancy problems and improving the effectiveness of feature fusion.

[0029] Hypergraph driven clustering integration: Hypergraph can better model the high-order associations between multiple views. By introducing hypergraph theory into speech forgery detection, the present invention can more comprehensively consider the relationship between different views, thereby improving the stability and accuracy of clustering.

[0030] Adaptive hybrid classifier: The performance of traditional K-nearest neighbor classifiers may be affected when facing complex data distributions. This invention combines pseudo-label enhancement, dynamic K value selection and confidence calibration to train K-nearest neighbor classifiers, which can better adapt to different data distributions, improve the accuracy and robustness of the classifier, and reduce the error rate by more than 15% compared with traditional K-nearest neighbor classifiers. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The figure is a schematic diagram of the structure of a dynamic multi-view fusion voice forgery identification device according to an embodiment of the present invention.

[0032] Figure 2 Schematic diagram of the structure of the audio coding model Encodec in an embodiment of the present invention.

[0033] Figure 3 The present invention is a flowchart of a method for identifying voice forgery using dynamic multi-view fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] In one embodiment, a dynamic multi-view fusion voice forgery identification method is provided. Figure 3 As shown, the following steps are included: S1. Obtain multi-view feature data based on the original speech signal and construct a multi-view dataset; The multi-view feature data includes a reconstruction error view, an acoustic feature view, and a quality assessment view; The speech generation model HiFiGAN (see Jungil Kong, Jaehyeon Kim, and JaekyoungBae. 2020. HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20). Curran Associates Inc., Red Hook, NY, USA, Article 1428, 17022–17033.) is used to calculate the reconstruction error view and introduce time-frequency domain residual analysis to quantify local anomalies in the speech generation process, as follows:

[0036] in, Represents reconstruction error (Reconstruction Error); is a variable, representing the original speech signal; Represents the original speech signal At the moment The value of Indicates the total duration of the speech signal (the length of the time series), that is, the total number of time points from the start time to the end time; Represents the decoder in the speech generation model HiFiGAN; Represents the encoder in the speech generation model HiFiGAN; It is a hyperparameter used to balance the weights of the two terms in the formula and is a man-made constant. Represents the original speech signal At the moment Total Variation is an indicator used to measure the degree of local variation of a signal. In one embodiment, .

[0037] like Figure 2 As shown in the figure, the audio coding model Encodec is used to obtain the acoustic feature view, extract the multi-scale acoustic code, combine the convolutional network to perform feature dimensionality reduction, and then calculate the inner product of the reduced dimensionality vector and normalize it, as follows: Input the original speech signal (such as wav, MP3 format) into the audio coding model Encodec (see D'efossez, A., Copet, J., Synnaeve, G., & Adi, Y. (2022). High Fidelity Neural Audio Compression. ArXiv, abs / 2210.13438 .) Generate multi-scale acoustic codes; these codes are an abstract representation of audio data, and they contain important audio information, such as pitch, timbre, rhythm and other features at different scales. The multi-scale acoustic codes are first extracted through a convolutional layer to extract local features, and then the global average pooling layer is used to calculate the average of each channel and compress it into a one-dimensional vector. Finally, the inner product of the one-dimensional vector itself is calculated to obtain the acoustic feature view; The UTMOS model specifically used for voice quality assessment (see Saeki, T., Xin, D., Nakata,W., Koriyama, T., Takamichi, S., Saruwatari, H. (2022) UTMOS: UTokyo-SaruLabSystem for VoiceMOS Challenge 2022. Proc. Interspeech 2022, 4521-4525, doi:10.21437 / Interspeech.2022-439) is used to obtain the quality assessment view as follows: Input the original speech signal (such as WAV or MP3 format) into the UTMOS model, output the overall speech quality assessment score, and normalize the data to obtain the quality assessment view; The structure of the multi-view dataset is: {reconstruction error view, acoustic feature view, quality assessment view}.

[0038] S2. Cluster the data in the multi-view dataset using a dynamic weight clustering ensemble algorithm to generate pseudo labels, including the following steps: S2.1, adjusting the weight of each view in a multi-view dataset based on information entropy; Dynamically adjust the weight of each view in a multi-view dataset according to the information entropy difference between views; In the multi-view dataset The information entropy of a view is calculated as follows: The probability of each data point appearing in each view is calculated by taking its logarithm and multiplying it by this probability. The results of this calculation for all data points are added together and the negative number is taken at the end. After obtaining the information entropy of each view, first The information entropy of the first view is exponentially calculated as the numerator; then the information entropy of the three views is exponentially calculated and added together as the denominator; the numerator is divided by the denominator to obtain the The weights of the views are as follows:

[0039]

[0040] in, Represents the first The weight of each view; Indicates The information entropy of a view, the larger its value, the richer the information contained in the view and the higher the uncertainty; represents the exponential function; Indicates The number of data points in each view is the same for all views; Indicates In the view Data points Probability of occurrence.

[0041] S2.2. Constructing a hypergraph that integrates similarity information of each view , is the vertex set, including all data points in the multi-view dataset, When the similarity of multiple data points in different views meets the set threshold, these data points are regarded as a hyperedge, as follows: First, multi-view clustering is performed. The multi-view dataset is , Indicates views; the data in the multi-view dataset are respectively After clustering, we get a set of cluster divisions ,in Representative cluster members (i.e., base clusters), each cluster member consists of multiple cluster subsets, denoted as ,in To indicate the The total number of cluster subsets in the sub-clustering partition, Indicates The first of the cluster members Cluster subsets; The data points in the multi-view dataset are uniformly numbered to calculate the similarity between the data points without distinguishing the specific views, and then the multi-view similarity is fused as follows:

[0042] in, Represents the first data points and The similarity between data points; Represents the first The weight of the view; In the clustering process of views, data points will be classified into a cluster subset, and all the features in this cluster subset together constitute the first The data point in Feature set under view ; Indicated in In the view, The feature set to which the data point belongs With The feature set to which the data point belongs Between Similarity; Indicates cross-view Figure 1 The consistency coefficient is a constant parameter set artificially. In this embodiment, ; Indicates The feature vector of the data point and The feature vector of the data point The cosine similarity between .

[0043] S2.3. Based on the hypergraph in step S2.2, the normalized cut (Ncut) algorithm is used in combination with the spectral clustering idea to partition the hypergraph, and spectral clustering is used to generate meta-clustering groups (see Bernhard Schölkopf; John Platt; Thomas Hofmann, "Learning with Hypergraphs: Clustering, Classification, and Embedding," in Advances in Neural Information Processing Systems 19:Proceedings of the 2006 Conference , MIT Press, 2007, pp.1601-1608.), as follows: First construct the Laplacian matrix of the hypergraph And the Laplace matrix Perform eigendecomposition to obtain eigenvalues; then automatically determine the optimal number of clusters using the eigenvalue gap method This method avoids the artificial presetting of the number of clusters and improves the adaptive ability of clustering; The eigenvectors corresponding to the smallest non-zero eigenvalues ​​form a matrix , these feature vectors contain key information about the clustering of data points in the hypergraph; the matrix Each row is considered as a new data point. In one embodiment, the matrix is ​​categorized using the k-means algorithm. Perform clustering to obtain meta-cluster groups .in, Represents the total number of meta-clusters, each meta-cluster Contains a set of similar data points.

[0044] S2.4. Calculate the scores of data points in the multi-view dataset to meta-cluster groups and determine pseudo labels as follows: In one embodiment, the average Euclidean distance from a data point to all data points in a meta-clustering group is calculated by a distance metric method to obtain a score of the data point to the meta-clustering group, and then the pseudo-label of the data point is determined by taking the meta-clustering group label with the smallest distance according to the minimum distance method.

[0045] S3, integrating the pseudo labels with the multi-view feature data to obtain a first enhanced data set, and training a K nearest neighbor classifier, including the following steps: S3.1. Concatenate the pseudo-labels and multi-view feature data to obtain the first enhanced data set, which can increase the dimension and information content of the features, provide more information for the classifier, and improve the performance of the classifier. The structure of the first enhanced data set is: {reconstruction error view, acoustic feature view, quality assessment view, pseudo-label}. The first enhanced data set is divided into a validation set and a training set; S3.2. Adjust the K value of the K nearest neighbor classifier according to the density of data points in the training set, as follows:

[0046]

[0047] in, Indicates that for the first Data points , K value after adaptive adjustment; Indicates Data points The number of local data points, that is, The range of the setting is centered The number of remaining data points in the The density of surrounding data points; Indicates rounding up; represents the total number of data points in the training set; represents the indicator function, when When established, The value of is 1, otherwise it is 0; Indicates Data points and Data points The distance between.

[0048] S3.3, input the training set into the K nearest neighbor classifier, the K nearest neighbor classifier uses the weighted Euclidean distance to calculate the distance between the data points in the training set according to the K value in step S3.2 to find the K nearest neighbors of each data point, and determines the data point category according to the pseudo label (neighbor label) by voting method to obtain the K nearest neighbor classifier that has been preliminarily trained, and uses the validation set to evaluate the performance of the K nearest neighbor classifier that has been preliminarily trained. In one embodiment, the performance of the K nearest neighbor classifier that has been preliminarily trained is evaluated using the accuracy, recall rate and F1 value. If the evaluation result meets the requirements or reaches the set maximum number of iterations, execute step S3.4, otherwise, the parameters in the data point density calculation are set. The set step size will be increased or decreased, the number of iterations will be increased by 1, and the process will return to step S3.2; If the evaluation results show that the K value of the K nearest neighbor classifier is too small, it may mean that the local density calculation range of the data point is too small. Increase the set step size; if the evaluation results show that the K value of the K nearest neighbor classifier is too large, it may mean that the local density calculation range of the data point is too large, then increase the parameter Decrease the set step size.

[0049] If the data points are evenly distributed, a smaller step size, such as 0.1, can be selected. If the data points are widely distributed, a relatively large step size, such as 0.5, can be selected.

[0050] S3.4. Use the Bayesian model to calibrate the output confidence of the initially trained K nearest neighbor classifier to obtain a trained K nearest neighbor classifier.

[0051] In one embodiment, a Bayesian probability model is used to calibrate the confidence of the output of the K nearest neighbor classifier that has been initially trained, as follows:

[0052] Among them, it is assumed that the category is binary classification, = 1 indicates one of the categories; Represents the feature vector at a given data point Under the condition that The probability of represents the indicator function, when When established, The value of is 1, otherwise it is 0; The feature vector representing the data point input into the K nearest neighbor classifier contains various feature information of the data point in the multi-view feature data and is the basis for classification and confidence calculation; Represents the number of nearest neighbors in the K nearest neighbor classifier; Indicates The weight of the nearest neighbor; Indicates The category of the nearest neighbors; Express The category in the nearest neighbor The weights of the nearest neighbors are summed, reflecting that when the weights of the nearest neighbors are considered, The category in the nearest neighbor = 1 nearest neighbor pair feature vector Category = 1 contribution level; It is a hyperparameter set by humans to adjust the feature vector The information entropy, in one embodiment, ; Represents the feature vector Information entropy is an indicator used to measure information uncertainty. Here, information entropy reflects the characteristic vector The uncertainty or information richness of its own features. Feature vector The higher the information entropy, the greater the uncertainty of its features. In confidence calibration, by introducing information entropy, the impact of the characteristics of the data point itself on the confidence of the classification result can be comprehensively considered.

[0053] S4, using the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified; Execute steps S1 to S3 on the speech signal to be identified to obtain a second enhanced data set corresponding to the speech signal to be identified; the first enhanced data set and the second enhanced data set have the same structure; The second enhanced data set is input into the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

[0054] In one embodiment, experiments are conducted on the ASVspoof2019 LA (known attack) and In-the-Wild 2021 (unknown attack) datasets.

[0055] Comparison methods: LCNN - BiLSTM (supervised learning), AST (semi-supervised). These comparison methods are selected to comprehensively evaluate the performance of the present invention. LCNN - BiLSTM is a supervised learning method based on deep learning, and AST is a semi-supervised learning method. By comparing with these methods, the advantages of the present invention can be more clearly demonstrated. The experimental results are shown in Table 1.

[0056] Table 1 Experimental results

[0057] Table 1 shows that the proposed method performs well in the equal error rate (EER) indicator and achieves the best balance between performance and robustness.

[0058] In one embodiment, a dynamic multi-view fusion voice forgery identification device, such as Figure 1 As shown, it includes the following modules: Data acquisition and preprocessing module: used to obtain the original speech signal and the speech signal to be identified; Multi-view feature extraction module: used to obtain multi-view feature data based on the original speech signal and construct a multi-view dataset; Integrated clustering module: used to cluster the data in the multi-view dataset using a dynamic weight clustering integrated algorithm to generate pseudo labels; Classifier training module: used to integrate pseudo labels with multi-view feature data to obtain a first enhanced data set and train a K-nearest neighbor classifier; Prediction module: used to use the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

[0059] The present invention proposes a method and device for identifying voice forgery with dynamic multi-view fusion, which uses unsupervised learning to mine multi-view voice information, solving the shortcomings of existing methods in generalization, robustness and feature utilization. In terms of method, multi-view data construction accurately extracts features, dynamic weight clustering integration improves clustering effect, and combines pseudo-label enhancement, dynamic K value selection and confidence calibration to train K nearest neighbor classifiers to enhance classification performance. The device ensures efficient and real-time practical applications. Experiments show that the present invention has excellent performance on multiple data sets, significantly improved key indicators, strong generalization and robustness in the face of different forged data points, and provides a new path for voice deep forgery detection.

[0060] Through the above embodiments, it is demonstrated in detail how to use the technical method of the present invention to identify voice forgery, from multi-view data construction to integrated clustering, and then to classifier training and prediction. Each step is closely integrated with the core technology, giving full play to the advantages of dynamic multi-view fusion, adaptive clustering and hybrid classifiers, and is expected to achieve good detection effects in practical applications.

Claims

1. A dynamic multi-view fusion voice forgery identification method, characterized in that: The following steps are involved: S1. According to the original speech signal, obtain multi-view feature data including reconstruction error view, acoustic feature view and quality assessment view, and construct a multi-view dataset; S2. clustering the multi-view feature data in the multi-view dataset using a dynamic weight clustering integration algorithm to generate pseudo labels; S3, integrating the pseudo labels with the multi-view feature data to obtain a first enhanced data set for training a K nearest neighbor classifier; S4. Use the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

2. According to the method of claim 1, the method is characterized in that: In step S1, the speech generation model HiFiGAN is used to calculate the reconstruction error view; The audio coding model Encodec is used to obtain the acoustic feature view; The UTMOS model is used to obtain quality assessment views.

3. According to the method of claim 1, the method is characterized in that: Step S2 specifically includes: S2.1, adjusting the weight of each view in a multi-view dataset based on information entropy; S2.2, construct a hypergraph that integrates the similarity information of each view; S2.3, based on the hypergraph in step S2.2, generate meta-clustering groups using spectral clustering; S2.

4. Calculate the average Euclidean distance from the data point in the multi-view dataset to all data points in the meta-clustering group, and then use the minimum distance method to take the meta-clustering group label with the smallest distance to determine the pseudo label of the data point.

4. According to the method of claim 3, the method is characterized in that: In step S2.1, the weight of each view in the multi-view dataset is dynamically adjusted according to the information entropy difference between the views; In the multi-view dataset The information entropy of a view is calculated as follows: The probability of each data point appearing in each view is calculated by taking its logarithm and multiplying it by this probability. The results of this calculation for all data points are added together and the negative number is taken at the end. After obtaining the information entropy of each view, first The information entropy of each view is taken as the index, as the numerator; Then, the information entropy of each view is exponentially added together as the denominator; the numerator is divided by the denominator to obtain the The weight of the view.

5. According to the method of claim 3, the method is characterized in that: In step S2.2, a hypergraph integrating similarity information of each view is constructed as follows: First, multi-view clustering is performed. The multi-view dataset is , Indicates Views; The data in the multi-view dataset are After clustering, we get a set of cluster divisions ,in Representative cluster members, each cluster member consists of multiple cluster subsets, denoted as ,in To indicate the The total number of cluster subsets in the sub-clustering partition, Indicates The first of the cluster members Cluster subsets; The data points in the multi-view dataset are uniformly numbered, and the multi-view similarities are integrated. The relevant similarities under each view are first calculated and summed, and then the cross-view consistency coefficient is added with the first data points and The product of the cosine similarity of the feature vectors of the data points is obtained data points and The similarity between data points .

6. The method for identifying voice forgery based on dynamic multi-view fusion according to claim 1, characterized in that: Step S3 includes the following steps: S3.

1. Concatenate the pseudo-labels and the multi-view feature data to obtain a first enhanced data set, wherein the structure of the first enhanced data set includes a reconstruction error view, an acoustic feature view, a quality assessment view, and a pseudo-label; divide the first enhanced data set into a validation set and a training set; S3.2, adjust the K value of the K nearest neighbor classifier according to the density of data points in the training set; S3.3, input the training set into the K nearest neighbor classifier. The K nearest neighbor classifier uses the weighted Euclidean distance to calculate the distance between the data points in the training set according to the K value in step S3.2 to find the K nearest neighbors of each data point. The data point category is determined according to the pseudo-label by voting method to obtain the K nearest neighbor classifier that has been preliminarily trained. The performance of the K nearest neighbor classifier that has been preliminarily trained is evaluated using the validation set. If the evaluation result meets the requirements or reaches the set maximum number of iterations, execute step S3.

4. Otherwise, the parameters in the data point density calculation are set to zero. The set step size will be increased or decreased, the number of iterations will be increased by 1, and the process will return to step S3.2; S3.

4. Use the Bayesian model to calibrate the output confidence of the initially trained K nearest neighbor classifier to obtain a trained K nearest neighbor classifier.

7. The method for identifying voice forgery based on dynamic multi-view fusion according to claim 6, characterized in that: In step S3.2, the K value is adaptively adjusted according to the density of data points in the training set; For the training set Data points , traverse all data points in the training set, for the Data points , calculate the Data points and Data points The distance between ; If the distance Less than the preset range , then the indicator function The value of is 1, otherwise it is 0; add the indicator function values ​​corresponding to all data points to get the Data points The number of local data points ; Get the first Data points The number of local data points Afterwards, through Take the square root and round up to get Data points K value after adaptive adjustment .

8. The method for identifying voice forgery based on dynamic multi-view fusion according to claim 6, characterized in that: In step S3.4, the Bayesian probability model is used to calibrate the confidence of the output of the K nearest neighbor classifier that has been initially trained, as follows: in, Represents the feature vector at a given data point Under the condition that The probability of is a category of data points; represents the indicator function, when When established, The value of is 1, otherwise it is 0; The feature vector representing the data point input into the K nearest neighbor classifier; Represents the number of nearest neighbors in the K nearest neighbor classifier; Indicates The weight of the nearest neighbor; Indicates The category of the nearest neighbors; Express The category in the nearest neighbor Sum the weights of the nearest neighbors; is a hyperparameter; Represents the feature vector Information entropy.

9. The method for identifying voice forgery based on dynamic multi-view fusion according to claim 1, characterized in that: In step S4, steps S1 to S3 are performed on the speech signal to be identified to obtain a second enhanced data set corresponding to the speech signal to be identified; The first augmented dataset and the second augmented dataset have the same structure; The second enhanced data set is input into the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

10. A dynamic multi-view fusion voice forgery identification device implementing the method described in any one of claims 1 to 9, characterized in that: Includes the following modules: Data acquisition and preprocessing module: used to obtain the original speech signal and the speech signal to be identified; Multi-view feature extraction module: used to obtain multi-view feature data based on the original speech signal and construct a multi-view dataset; Integrated clustering module: used to cluster the data in the multi-view dataset using a dynamic weight clustering integrated algorithm to generate pseudo labels; Classifier training module: used to integrate pseudo labels with multi-view feature data to obtain a first enhanced data set and train a K-nearest neighbor classifier; Prediction module: used to use the trained K nearest neighbor classifier to predict the unseen data points other than the original speech signal in the speech signal to be identified.

Images