False voice attribution method and system based on front-end time-frequency attention

Abstract

The invention discloses a faked voice attribution method and system based on front-end time-frequency attention, and the method comprises the steps: inputting the time-frequency characteristics of a voice sample into a trained faked voice attribution model, and outputting the probability that the time-frequency characteristics of the voice sample belong to various voice faking methods, and taking the probability as a faked voice attribution recognition result. The forged voice attribution model comprises a front-end time-frequency attention module which is used for weighting the time-frequency characteristics of a voice sample by adopting front-end time-frequency attention; the feature extraction module is used for extracting depth features of the voice sample from the front-end time-frequency features subjected to time-frequency attention weighting; and the feature classification module is used for performing category judgment on the depth features of the voice samples. When faked voice attribution is carried out, faking method attribution can be carried out on the voice subjected to encoding and decoding conversion and the voice not subjected to encoding and decoding conversion at the same time, and faking method attribution performance of the voice subjected to encoding and decoding conversion and the voice not subjected to encoding and decoding conversion can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of forged speech attribution, and in particular relates to a forged speech attribution method and system based on front-end time-frequency attention (FTFA). Background Art

[0002] The emergence and rapid development of artificial intelligence generated content (AIGC) such as deepfakes have increased public concerns about voice security and privacy issues on social media. In recent years, the technology to combat deepfakes has evolved from distinguishing authenticity from fakes to understanding their generation process, which is considered a key step in building a robust and explainable anti-fraud strategy. Forgery Speech Attribution (FSA) is a typical work that explains the generation process of forged media, where "attribution" refers to identifying possible forgery methods for generating fake speech.

[0003] However, existing FSA research works have at least the following shortcomings: they only train models on raw speech datasets that have not undergone any codec processing. Here, "raw speech" refers to real speech recorded by a device or fake speech directly generated using some forgery methods. However, in the real world, the forged speech used by attackers (such as voice fraud) is transmitted through social media platforms or telephone channels. The codec conversion introduced during the transmission process will cause the speech signal to be distorted, and the forgery traces contained in it will be destroyed. Therefore, the performance of existing FSA methods on codec forged speech is not satisfactory. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a forged speech attribution method and system based on front-end time-frequency attention. When performing forged speech attribution, it is possible to simultaneously attribute the forgery method to both the speech that has been encoded and decoded and the speech that has not been encoded and decoded, and it is beneficial to improve the forgery method attribution performance for the speech that has been encoded and decoded and the speech that has not been encoded and decoded.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: In a first aspect, a method for attributing a forged voice is provided, comprising: obtaining a voice sample; extracting time-frequency features of the voice sample; inputting the time-frequency features of the voice sample into a trained forged voice attribution model, and outputting the probability that the time-frequency features of the voice sample belong to various voice forgery methods as a recognition result of the forged voice attribution; wherein the forged voice attribution model comprises: a front-end time-frequency attention module, which is used to weight the time-frequency features of the voice sample using the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention; a feature extraction module, which is used to extract the deep features of the voice sample from the front-end time-frequency features weighted by the time-frequency attention; and a feature classification module, which is used to classify the deep features of the voice sample and output the probability that the time-frequency features of the voice sample belong to various voice forgery methods.

[0006] Furthermore, the time-frequency features of the speech sample are weighted by the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention, including: constructing a submodule by using the front-end time-frequency features, dividing the two-dimensional time-frequency features into a number of time-frequency feature subbands of the same shape along the frequency dimension by using an overlapping time-frequency feature division method on the two-dimensional time-frequency features; stacking the time-frequency feature subbands into a three-dimensional time-frequency feature block; using the intra-subband attention submodule to extract the intra-subband attention weights reflecting the importance of the areas in different subbands from the three-dimensional time-frequency feature block; using the inter-subband attention submodule to extract the inter-subband attention weights reflecting the importance of different subbands from the three-dimensional time-frequency feature block; calculating the front-end time-frequency attention weights based on the inter-subband attention weights and the intra-subband attention weights; calculating the front-end time-frequency features weighted by the time-frequency attention based on the front-end time-frequency attention weights and the three-dimensional time-frequency feature block.

[0007] Furthermore, the method for calculating the three-dimensional time-frequency feature block includes: for a speech sample , extract its logarithmic Mel spectrum ,in, are the width and height of the logarithmic Mel spectrum respectively; The length is , the sliding step length is The window along the frequency axis Split into time-frequency characteristic subbands of the same shape; Will The time-frequency feature subbands of the same shape are stacked into a three-dimensional time-frequency feature block ,in, For the time-frequency feature subbands.

[0008] Furthermore, the method for calculating the attention weight within the sub-band includes: , respectively use global adaptive average pooling and global adaptive maximum pooling to aggregate the global information of each sub-band and then input it into the shared convolution block , generating the transformed global average information descriptor and the global maximum information descriptor : (1) in, It includes two convolutional layers, a normalization layer and a ReLu layer; the number of input channels and output channels of the two convolutional layers are both C; the convolution kernel size is 1, the sliding step is 1, and the padding is 0; for The local information of each subband is aggregated using local adaptive average pooling and local adaptive maximum pooling and then input into the shared convolution block. , generating the transformed local average information descriptor and local maximum information descriptor : (2) in, Except that the kernel size of the first convolutional layer is consistent with the output size of the local adaptive pooling operation, the rest of the configuration is the same as The same is used to aggregate the information of each sub-band into a scalar to represent the importance of each sub-band; Represents the output size of the local adaptive pooling operation; After obtaining the four sub-band information descriptors at the global and local scales, they are added element by element and Sigmoid activation is performed on them to obtain the final inter-sub-band attention weights: (3) in, Represents the Sigmoid activation function.

[0009] Furthermore, the method for calculating the attention weight between sub-bands includes: , respectively, perform adaptive maximum pooling operations along the time axis and frequency axis to obtain the corresponding feature descriptors and : (4) in, and Respectively represent the number of frequency bands and the number of frames, and They represent the output sizes corresponding to the AMP operations respectively; The result after dimension swap is the same as Concatenate and then input the shared convolutional block , and obtain the transformed time-frequency feature descriptor : (5) in, It includes a convolution layer, a normalization layer, and a Swish activation function; the number of input channels and output channels of the convolution layer are both C, the convolution kernel size is 1, the stride is 1, and the padding is 0; It means exchanging the 2nd and 3rd dimensions. It means splicing along the second dimension; Split along the second dimension , and after Sigmoid activation, the temporal attention in the sub-band is obtained and frequency attention within subband : (6) Will The transpose and Perform matrix multiplication to obtain the final sub-band attention weight for (7).

[0010] Furthermore, the method for calculating the front-end time-frequency attention weight includes: The results after dimension expansion are similar to Perform element-by-element multiplication to obtain the final front-end time-frequency attention weight : (8).

[0011] Furthermore, the method for calculating the front-end time-frequency features weighted by time-frequency attention includes: The time-frequency characteristics of the input Multiply element by element to obtain the three-dimensional time-frequency features weighted by time-frequency attention. ; in accordance with Constructing front-end time-frequency features weighted by time-frequency attention : (9) in, represents the maximum feature map activation at the element level, that is, only the larger value between the elements at the same position in U and V is retained as the activation value; formula (9) is equivalent to retaining the lower half of the first subband, the upper half of the last subband, and the larger value at the same position in the overlapping area of ​​adjacent subbands.

[0012] Furthermore, the training method of the forged speech attribution model includes: inputting the time-frequency features of the speech samples in the training set, extracting the front-end time-frequency features weighted by the time-frequency attention using the front-end time-frequency attention module; taking the front-end time-frequency features weighted by the time-frequency attention as input, using the feature extraction module to extract the deep features; outputting the deep features to the feature classification module, and outputting the predicted logical values ​​of the speech training samples belonging to each category; According to the predicted logical value, the predicted label of the sample is obtained, and the classification loss is calculated; Classification loss is used to supervise the training of the front-end time-frequency attention module, feature extraction module and feature classification module to obtain the weight parameters of the front-end time-frequency attention module, feature extraction module and feature classification module.

[0013] Further, calculating the classification loss includes: using a Softmax cross entropy loss function or an AM-Softmax loss function to calculate the classification loss; Consider a batch of samples , the number is , taking the logical value and label output by the feature classification module as input, when the Softmax cross entropy loss function is used to calculate the classification loss, the formula is: (10) in, Indicates samples, yes The corresponding category logical value vector, represents the classification module, It is the feature extraction module The extracted deep features, yes The front-end time-frequency features weighted by the front-end time-frequency attention; When the AM-Softmax loss function is used to calculate the classification loss, the calculation formula is as follows: (10). The only difference is the calculation method of the logical value. The logical value used by the AM-Softmax loss function is: (10-1) in, They are deep features and the classification module , Class classifier weights The intra-class and inter-class similarities of is the margin imposed on the intra-class similarity.

[0014] In a second aspect, a forged speech attribution system is provided, comprising: a data acquisition module for acquiring speech samples; a data processing module for extracting time-frequency features of speech samples; an attribution module for inputting the time-frequency features of speech samples into a trained forged speech attribution model, and outputting the probability that the time-frequency features of the speech samples belong to various speech forgery methods as the recognition result of the forged speech attribution; wherein the forged speech attribution model comprises: a front-end time-frequency attention module for weighting the time-frequency features of speech samples using the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention; a feature extraction module for extracting the deep features of speech samples from the front-end time-frequency features weighted by the time-frequency attention; a feature classification module for classifying the deep features of speech samples, and outputting the probability that the time-frequency features of the speech samples belong to each category in the speech forgery method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention inputs the time-frequency features of the speech sample into a trained forged speech attribution model, and outputs the probability that the time-frequency features of the speech sample belong to various speech forgery methods as the recognition result of the forged speech attribution; when performing forged speech attribution, the forgery methods can be attributed to both the speech that has been converted by codec and the speech that has not been converted by codec, and it is beneficial to improve the forgery method attribution performance of the speech that has been converted by codec and the speech that has not been converted by codec; (2) Using the front-end time-frequency features, we construct a three-dimensional time-frequency feature block. We fuse the inter-subband and intra-subband attention to extract the time-frequency attention of the front-end features, and guide the attribution model to focus on the feature areas that are important for attributing fake speech. (3) Compared with existing forged speech attribution methods, the method based on front-end time-frequency attention can not only improve the forgery recognition performance of speech that has not been processed by codec, but also significantly improve the forgery recognition performance of speech that has been processed by codec; (4) Compared with existing attention-based methods, the front-end time-frequency attention-based method fully considers the fine-grained time-frequency details and local waveform modifications involved in the speech forgery process, designs global and local information aggregation modules based on features of different scales, and can give greater attention weights to areas that are important for attributing forged speech. (5) When constructing the time-frequency feature blocks, the present invention takes into account the influence of the time-frequency sub-band division method and avoids information loss between adjacent sub-bands by adopting an overlapping time-frequency sub-band division method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a main flow chart of a method for encoding and decoding robust forged speech attribution based on front-end time-frequency attention provided by an embodiment of the present invention; Figure 2 Schematic diagram of feature visualization learned on the evaluation set of the Codec Forged Speech (CFS) dataset, where (a) is a visualization of features learned by the EMA method, and (b) is a visualization of features learned by the method of the present invention; Figure 3 : is a curve diagram of the variation of the mean value of class activation values ​​(CAV) learned by the method of the present invention on the evaluation set of the CFS data set with the frequency band, wherein (a) is the curve of the variation of the mean value of CAV learned by the EMA method with the frequency band, and (b) is the curve of the variation of the mean value of CAV learned by the method of the present invention with the frequency band; Figure 4 It is a correlation matrix diagram of the CAV mean value variation curve with frequency band learned by the method of the present invention on the evaluation set of the CFS data set, wherein (a) is the correlation matrix of the CAV mean value variation curve with frequency band learned by the EMA method, and (b) is the correlation matrix of the CAV mean value variation curve with frequency band learned by the method of the present invention; Figure 5 It is a flow chart of a training method for a forged speech attribution model in a coding and decoding robust forged speech attribution method based on front-end time-frequency attention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0018] Embodiment 1 like Figure 1~Figure 5 As shown, a forged speech attribution method based on front-end time-frequency attention includes: obtaining a speech sample; extracting the time-frequency features of the speech sample; inputting the time-frequency features of the speech sample into a trained forged speech attribution model, and outputting the probability that the time-frequency features of the speech sample belong to various speech forgery methods as the recognition result of the forged speech attribution.

[0019] The forged speech attribution model includes: a front-end time-frequency attention module, a feature extraction module and a feature classification module. The front-end time-frequency attention module is used to weight the time-frequency features of the speech sample using the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention; the feature extraction module is used to extract the deep features of the speech sample from the front-end time-frequency features weighted by the time-frequency attention using a deep convolutional neural network; the feature classification module is used to classify the deep features of the speech sample and output the probability that the time-frequency features of the speech sample belong to various speech forgery methods. The feature classification module is a fully connected layer followed by a Softmax function, and the number of output nodes of the fully connected layer is the same as the total number of categories in the speech forgery method.

[0020] For the input speech sample , using short-time Fourier transform, Mel filtering and logarithmic operation to Convert to logarithmic Mel spectrum , as the input of the front-end time-frequency attention module. Among them, are the width and height of the logarithmic Mel spectrum, respectively.

[0021] The front-end time-frequency attention is used to weight the time-frequency features of the speech sample to obtain the front-end time-frequency features weighted by the time-frequency attention, including: (I) Use the front-end time-frequency features to construct sub-modules. On the two-dimensional time-frequency features, along the frequency dimension, use an overlapping time-frequency feature division method to divide the two-dimensional time-frequency features into several time-frequency feature sub-bands of the same shape; stack the time-frequency feature sub-bands into three-dimensional time-frequency feature blocks.

[0022] The calculation method of the three-dimensional time-frequency feature block includes: for the speech sample , extract its logarithmic Mel spectrum (two-dimensional time-frequency feature) ,in, are the width and height of the logarithmic Mel spectrum respectively; the length is , the sliding step length is The window of the two-dimensional time-frequency feature is Split into time-frequency characteristic subbands of the same shape; The time-frequency feature subbands of the same shape are stacked into a three-dimensional time-frequency feature block ,in, For the time-frequency feature subbands.

[0023] (ii) Using the intra-sub-band attention sub-module, the intra-sub-band attention weights reflecting the importance of different sub-band regions are extracted from the 3D time-frequency feature blocks.

[0024] The calculation method of the attention weight within the sub-band includes: , using Global Adaptive Average Pooling (GAAP) and Global Adaptive Max Pooling (GAMP) to aggregate the global information of each subband. The aggregated information is then fed into the shared convolutional block (like Figure 1 As shown in Figure 2), the converted global average information descriptor is generated and the global maximum information descriptor : (1) in, It includes two convolutional layers, a normalization layer and a ReLu layer; the number of input channels and output channels of the two convolutional layers are both C; the convolution kernel size is 1, the sliding step is 1, and the padding is 0.

[0025] for The local information of each subband is aggregated using Local Adaptive Average Pooling (LAAP) and Local Adaptive Max Pooling (LAMP), and then the summarized information is input into the shared convolutional block. (like Figure 1 As shown in Figure 2), the converted local average information descriptor is generated and local maximum information descriptor : (2) in, Except that the kernel size of the first convolutional layer is consistent with the output size of the local adaptive pooling operation, the rest of the configuration is the same as The same is used to aggregate the information of each sub-band into a scalar to represent the importance of each sub-band; Represents the output size of the local adaptive pooling operation.

[0026] After obtaining the four sub-band information descriptors at the global and local scales, they are added element by element and Sigmoid activation is performed on them to obtain the final inter-sub-band attention weights: (3) in, Represents the Sigmoid activation function.

[0027] (iii) Using the inter-sub-band attention sub-module, the inter-sub-band attention weights reflecting the importance of different sub-bands are extracted from the 3D time-frequency feature blocks.

[0028] The calculation method of the attention weight between sub-bands includes: , Adaptive Max Pooling (AMP) operations are performed along the time axis and frequency axis to obtain the corresponding feature descriptors and : (4) in, and Respectively represent the number of frequency bands and the number of frames, and They represent the output sizes corresponding to the AMP operations respectively.

[0029] The result after dimension swap is the same as Concatenate and then input the shared convolutional block , and obtain the transformed time-frequency feature descriptor : (5) in, It includes a convolution layer, a normalization layer and a Swish activation function; the number of input channels and output channels of the convolution layer are both C, the convolution kernel size is 1, the stride is 1, and the padding is 0. It means exchanging the 2nd and 3rd dimensions. It refers to splicing along the second dimension.

[0030] Split along the second dimension , and after Sigmoid activation, the temporal attention in the sub-band is obtained and frequency attention within subband : (6) Will The transpose and Perform matrix multiplication to obtain the final sub-band attention weight for (7).

[0031] (iv) Calculate the front-end time-frequency attention weight based on the inter-sub-band attention weight and the intra-sub-band attention weight.

[0032] The calculation method of the front-end time-frequency attention weight includes: The results after dimension expansion are similar to Perform element-by-element multiplication to obtain the final front-end time-frequency attention weight : (8).

[0033] (V) Based on the time-frequency attention weights and the three-dimensional time-frequency feature blocks, calculate the front-end time-frequency features weighted by the time-frequency attention.

[0034] The calculation method of the front-end time-frequency features weighted by time-frequency attention includes: The time-frequency characteristics of the input Multiply element by element to obtain the three-dimensional time-frequency features weighted by time-frequency attention. ; in accordance with Constructing front-end time-frequency features weighted by time-frequency attention : (9) in, represents the maximum feature map (MFM) activation at the element level, that is, only the larger value between the elements at the same position in U and V is retained as the activation value; formula (9) is equivalent to retaining the lower half of the first subband, the upper half of the last subband, and the larger value at the same position in the overlapping area of ​​adjacent subbands.

[0035] A deep convolutional neural network is used to extract the deep features of the speech sample from the front-end time-frequency features weighted by time-frequency attention; the specific method is: Optionally, ResNet-50 is used as a feature extraction module to extract the deep features of the speech sample. ResNet-50 has four convolution blocks, and the output of the last convolution block is used as the deep feature.

[0036] The deep features of the speech samples are used to determine the categories, and the time-frequency features of the output speech samples are used to output the probability that they belong to various speech forgery methods.

[0037] The training method for the forged speech attribution model in the present invention comprises: Input the time-frequency features of the speech samples in the training set, and use the front-end time-frequency attention module to extract the front-end time-frequency features weighted by the time-frequency attention; use the front-end time-frequency features weighted by the time-frequency attention as input, and use the feature extraction module to extract the deep features; output the deep features to the feature classification module, and output the predicted logical values ​​of the speech training samples belonging to each category; According to the predicted logical value, the predicted label of the sample is obtained, and the classification loss is calculated; Classification loss is used to supervise the training of the front-end time-frequency attention module, feature extraction module and feature classification module to obtain the weight parameters of the front-end time-frequency attention module, feature extraction module and feature classification module.

[0038] Calculate the logical value of the speech sample belonging to each type of forgery method. The specific method is: use a fully connected layer followed by a Softmax function as a feature classification module. This module takes deep features as input and outputs the logical value of the feature belonging to each category.

[0039] Calculate the classification loss. In a specific embodiment, the classification loss of the sample can be calculated optionally by one of the following methods.

[0040] Solution 1 for calculating the classification loss of samples: Consider a batch of samples , the number is , taking the logical value and label output by the feature classification module (fully connected layer) as input, when the Softmax cross entropy loss function is used to calculate the classification loss, the formula is: (10) in, Indicates samples, yes The corresponding category logical value vector, represents the classification module, It is the feature extraction module The extracted deep features, yes The front-end time-frequency features weighted by the front-end time-frequency attention.

[0041] Solution 2 for calculating the classification loss of samples: When the AM-Softmax loss function is used to calculate the classification loss, the calculation formula is as follows: (10). The only difference is the calculation method of the logical value. The logical value used by the AM-Softmax loss function is: (10-1) in, They are deep features and the classification module , Class classifier weights The intra-class and inter-class similarities of is the margin imposed on the intra-class similarity.

[0042] It should be noted that other loss functions that can be used for classification can be used here to calculate the classification loss.

[0043] By minimizing the classification loss, the model's classification performance for samples can be improved.

[0044] Use classification loss to supervise the training of the front-end time-frequency attention module, feature extraction module, and classification module. Train for T rounds in total. Update the weight parameters of the front-end time-frequency attention module, feature extraction module, and classification module according to the gradient back propagation algorithm in each round, and calculate the classification accuracy on the validation set. After the training is completed, use the model with the highest classification accuracy on the validation set as the final model. Evaluate the performance of the final model on the evaluation set and output the label of the sample belonging to a certain category.

[0045] In the embodiment of the present invention, the encoding and decoding robust speech forgery attribution based on the front-end time-frequency attention adopts an overlapping frequency sub-band division method to divide and stack sub-band features, providing a good feature processing means for extracting forgery method features from different frequency components. At the same time, through the front-end time-frequency attention module, the inter-sub-band attention and the intra-sub-band attention are integrated, which can effectively model the importance relationship between different frequency sub-bands and within the frequency sub-band. This method can simultaneously improve the model's forgery method attribution performance for unencoded and decoded speech and encoded and converted speech.

[0046] In conjunction with a specific implementation of an embodiment of the present invention, a method flow for implementing encoding and decoding robust forged speech attribution using a front-end time-frequency attention method is introduced on the CFS dataset. Figure 5 As shown, this embodiment builds the training set, validation set, and evaluation set of the CFS data set based on the ASVSpoof2019LA data set. The following steps may be included: S501, dividing the speech data into training samples, verification samples and test samples.

[0047] S502, converting the speech data into a logarithmic Mel spectrum by using short-time Fourier transform, Mel filtering and logarithmic operation.

[0048] S503, input the logarithmic Merp into the front-end time-frequency attention module, and output the time-frequency features weighted by the front-end time-frequency attention.

[0049] S504, input the time-frequency features weighted by the front-end time-frequency attention into the feature extraction module ResNet-50, and output the deep features.

[0050] S505, input the deep features into the classification module and output the category logic value.

[0051] S506, using the category logic value and sample label, using Softmax cross entropy loss to calculate the classification loss .

[0052] S507, using Train the front-end time-frequency attention module, feature extraction module and classification module.

[0053] S508, using the trained front-end time-frequency attention module, feature extraction module and classification module, identify the forgery method of the speech after codec conversion and the speech without codec conversion.

[0054] The Codec Forged Speech Attribution (CFS) dataset is created based on the ASVspoof2019LA dataset. The ASVspoof2019 dataset is widely used in forged speech detection tasks for the purpose of anti-spoofing. The dataset has a rich category of forgery methods and a large amount of voice data. In addition, it also includes commonly used forgery methods and their variants, such as WaveNet and Tacotron2. The ASVspoof2019 dataset contains 20 categories of forgery methods, denoted by A00 to A19, where A00 is a special category representing real speech. The remaining 19 categories can be divided into 3 categories, namely Text-To-Speech (TTS), Voice Conversion (VC), and Waveform Concatenation (WC).

[0055] Opus, AAC, Adaptive Multi-Rate-WideBand (AMR-WB) and G.719 are selected to construct the corresponding codec subsets of the CFS dataset, and the encoding bit rates are set to 8kbps, 16kbps, 23.85kbps and 64kbps respectively. These four codec settings take into account the application scenarios of medium-low rate encoding, high rate encoding and typical application scenarios of audio transmission (such as streaming media, mobile communication, Internet voice communication and video conferencing).

[0056] The CFS dataset includes an original forged speech subset , and the encoding and decoding forged speech subsets converted by different encoding and decoding methods are recorded as , , and . Original forged speech subset It is created by randomly selecting 4,914 speech samples from each type of speech in the ASVspoof2019 dataset. Then it is divided into a training set and a training set in a ratio of 3:1:1. , validation set and the evaluation set (corresponding to 2,948, 983, and 983 speech samples, respectively).

[0057] Based on the original fake speech subset, four codecs, Opus, AAC, AMR-WB and G.719, are used to establish the codec subset of the CFS dataset. For any of these four codecs, the corresponding codec fake speech subset is The parallel versions of , , and The combined versions of the four codec pseudo-speech subsets are collectively referred to as codec subsets. , whose training set, validation set and evaluation set are respectively the combination of the training set, validation set and evaluation set of the four encoding and decoding pseudo speech subsets. Similarly, and The combined version of is called the entire CFS dataset, denoted by , whose training set, validation set and evaluation set are and The information of the CFS dataset is shown in Table 1, where real speech is also included as a special category.

[0058] Table 1 CFS dataset

[0059] The Adam optimizer and batch size of 128 were used for training for 30 rounds. The initial learning rate was set to 2e-4 and the weight decay was 5e-4. The learning rate was reduced to 0.1 times the original value every 20 rounds. In the front-end time-frequency attention module, the length of the sliding window Set to 32, step size Set to 16 to divide the frequency sub-bands.

[0060] We perform calculations and compare FTFA with current typical attention methods, and list the results in Tables 2 to 7.

[0061] Table 2 Comparison of FTFA with other methods on the original forged speech subset (%)

[0062] Table 3 Comparison of FTFA and other methods on G.719 codec forged speech dataset (%)

[0063] Table 4 Comparison of FTFA with other methods on AMR-WB codec forged speech dataset (%)

[0064] Table 5 Comparison of FTFA and other methods on AAC codec forged speech dataset (%)

[0065] Table 6 Comparison of FTFA with other methods on Opus codec forged speech dataset (%)

[0066] Table 7 Comparison of FTFA with other methods on Codec encoding and decoding forged speech dataset (%)

[0067] In Tables 2 to 7, EMA is the method recorded in the article "Efficient Multi-Scale Attention Module using Cross-Space Learning" published in the International Conference on Acoustics, Speech, and Signal Processing in 2023; EABN is the method recorded in the article "Efficient Attention Branch Network with Combined Loss Function for Automatic Speaker Verification Fraud Detection" published in the journal Circuits, Systems, and Signal Processing in 2023; CA is the method recorded in the article "Coordinated Attention for Efficient Mobile Network Design" published in the International Conference on Computer Vision and Pattern Recognition in 2021; SE is the method recorded in the article "Compression and Excitation Networks" published in the International Conference on Computer Vision and Pattern Recognition in 2018; CBA is the method recorded in the article "CBAM: Convolutional Block Attention Module" published in the European Conference on Computer Vision in 2018; "No Attention Used" means a method that does not use attention; FTFA is the method of the present invention.

[0068] It can be seen from the indicators listed in Table 2 that the method of the present invention has achieved the best performance on the original forged speech dataset. It can be seen from the indicators listed in Tables 3 to 6 that FTFA has achieved the best performance on the four codec forged speech datasets. It can be seen from Table 7 that on the Codec forged speech dataset containing four codec conversions, compared with the best baseline EMA, FTFA has improved the accuracy, precision, recall and F1 score by 1.06%, 1.01%, 1.58% and 1.43% respectively. Compared with the "no attention" method, FTFA has improved the accuracy, precision, recall and F1 score by 2.62%, 2.95%, 3.18% and 3.23% respectively. These results show that FTFA is not only conducive to improving the attribution performance of the forgery method of speech without codec conversion, but also can better identify the forgery method corresponding to the speech after codec conversion.

[0069] from Figure 3It can be seen that in the embedded feature representations of samples learned by the EMA method, which is the most advanced method among other methods, some categories are close to each other (such as A19 and A00, A19 and A06, A10 and A07, A10 and A12), or even overlap (such as A04 and A16). Compared with the results learned by EMA, FTFA greatly improves the performance of feature representation: speech samples of different categories can be easily distinguished in the feature space. This shows that the forged method features learned by FTFA have higher discrimination between classes and are more compact within the class, so they are less affected by codec conversion and are robust to various codec conversions.

[0070] from Figure 4 It can be seen that compared with other methods, the mean distribution of CAV in each frequency band learned by FTFA on different evaluation subsets is more consistent.

[0071] from Figure 5 It can be seen that the correlation between the CAV curves learned by FTFA on different evaluation subsets is higher than that of other methods. Figure 4 and Figure 5 It shows that the distribution of class activation values ​​learned by FTFA is more consistent on different evaluation subsets. This shows that the consistency of the forged features learned by FTFA is robust under different codec conversions. Therefore, the feature representation of the forged method learned by FTFA is indeed robust to codec conversion, which is why FTFA outperforms the best baseline method EMA.

[0072] Embodiment 2 Based on the forged speech attribution method based on front-end time-frequency attention described in Example 1, this embodiment provides a forged speech attribution system based on front-end time-frequency attention, including: A data acquisition module, used to acquire speech samples; A data processing module, used to extract time-frequency features of speech samples; An attribution module is used to input the time-frequency features of the speech sample into the trained forged speech attribution model, and output the probability that the time-frequency features of the speech sample belong to various speech forgery methods as the identification result of the forged speech attribution; The forged voice attribution model includes: A front-end time-frequency attention module is used to weight the time-frequency features of the speech sample using the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention; A feature extraction module, used to extract deep features of speech samples from front-end time-frequency features weighted by time-frequency attention; The feature classification module is used to classify the deep features of the speech sample and output the probability that the time-frequency features of the speech sample belong to various speech forgery methods.

[0073] Embodiment 3 Based on the forged speech attribution method based on front-end time-frequency attention described in Example 1, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the forged speech attribution method based on front-end time-frequency attention as described in Example 1 is implemented.

[0074] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0078] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for attributing fake speech, characterized in that: include: Get a voice sample; Extracting time-frequency features of speech samples; Input the time-frequency features of the speech sample into the trained forged speech attribution model, and output the probability that the time-frequency features of the speech sample belong to various speech forgery methods as the recognition result of the forged speech attribution; The forged voice attribution model includes: A front-end time-frequency attention module is used to weight the time-frequency features of the speech sample using the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention; A feature extraction module, used to extract deep features of speech samples from front-end time-frequency features weighted by time-frequency attention; The feature classification module is used to classify the deep features of the speech sample and output the probability that the time-frequency features of the speech sample belong to various speech forgery methods.

2. The method for attributing fake speech according to claim 1, characterized in that: The front-end time-frequency attention is used to weight the time-frequency features of the speech sample to obtain the front-end time-frequency features weighted by the time-frequency attention, including: The submodule is constructed by using the front-end time-frequency features. On the two-dimensional time-frequency features, along the frequency dimension, an overlapping time-frequency feature division method is adopted to divide the two-dimensional time-frequency features into several time-frequency feature sub-bands of the same shape; the time-frequency feature sub-bands are stacked into three-dimensional time-frequency feature blocks; Using the intra-sub-band attention sub-module, the intra-sub-band attention weights reflecting the importance of different sub-band regions are extracted from the 3D time-frequency feature blocks; Using the inter-subband attention submodule, the inter-subband attention weights reflecting the importance of different sub-bands are extracted from the 3D time-frequency feature blocks; Calculate the front-end time-frequency attention weight based on the inter-subband attention weight and the intra-subband attention weight; According to the front-end time-frequency attention weight and the three-dimensional time-frequency feature block, the front-end time-frequency features weighted by time-frequency attention are calculated.

3. The method for attributing fake speech according to claim 2, characterized in that: The method for calculating the three-dimensional time-frequency feature block includes: For speech samples , extract its logarithmic Mel spectrum ,in, are the width and height of the logarithmic Mel spectrum; The length is , the sliding step length is The window along the frequency axis Split into time-frequency characteristic subbands of the same shape; Will The time-frequency feature subbands of the same shape are stacked into a three-dimensional time-frequency feature block ,in, For the time-frequency feature subbands.

4. The method for attributing fake speech according to claim 3, characterized in that: The method for calculating the attention weight within the sub-band includes: for , respectively use global adaptive average pooling and global adaptive maximum pooling to aggregate the global information of each sub-band and then input it into the shared convolution block , generating the transformed global average information descriptor and the global maximum information descriptor : （1） in, It includes two convolutional layers, a normalization layer and a ReLu layer; the number of input channels and output channels of the two convolutional layers are both C; the convolution kernel size is 1, the sliding step is 1, and the padding is 0; for The local information of each subband is aggregated using local adaptive average pooling and local adaptive maximum pooling and then input into the shared convolution block. , generating the transformed local average information descriptor and local maximum information descriptor : （2） in, Except that the kernel size of the first convolutional layer is consistent with the output size of the local adaptive pooling operation, the rest of the configuration is the same as The same is used to aggregate the information of each sub-band into a scalar to represent the importance of each sub-band; Represents the output size of the local adaptive pooling operation; After obtaining the four sub-band information descriptors at the global and local scales, they are added element by element and Sigmoid activation is performed on them to obtain the final inter-sub-band attention weights as （3） in, Represents the Sigmoid activation function.

5. The method for attributing fake speech according to claim 4, characterized in that: The method for calculating the attention weight between sub-bands includes: for , respectively, perform adaptive maximum pooling operations along the time axis and frequency axis to obtain the corresponding feature descriptors and : （4） in, and Respectively represent the number of frequency bands and the number of frames, and They represent the output sizes corresponding to the AMP operations respectively; The result after dimension swap is the same as Concatenate and then input the shared convolutional block , and obtain the transformed time-frequency feature descriptor : （5） in, It includes a convolution layer, a normalization layer, and a Swish activation function; the number of input channels and output channels of the convolution layer are both C, the convolution kernel size is 1, the stride is 1, and the padding is 0; It means exchanging the 2nd and 3rd dimensions. It means splicing along the second dimension; Split along the second dimension , and after Sigmoid activation, the temporal attention in the sub-band is obtained and frequency attention within subband : （6） Will The transpose and Perform matrix multiplication to obtain the final sub-band attention weight for （7）。 6. The method for attributing fake speech according to claim 5, characterized in that: The method for calculating the front-end time-frequency attention weight includes: Will The results after dimension expansion are similar to Perform element-by-element multiplication to obtain the final front-end time-frequency attention weight : （8）。 7. The method for attributing fake speech according to claim 6, characterized in that: The method for calculating the front-end time-frequency features weighted by time-frequency attention includes: Will The time-frequency characteristics of the input Multiply element by element to obtain the three-dimensional time-frequency features weighted by time-frequency attention. ; in accordance with Constructing front-end time-frequency features weighted by time-frequency attention : （9） in, represents the maximum feature map activation at the element level, that is, only the larger value between the elements at the same position in U and V is retained as the activation value; formula (9) is equivalent to retaining the lower half of the first subband, the upper half of the last subband, and the larger value at the same position in the overlapping area of ​​adjacent subbands.

8. The method for attributing fake speech according to claim 1, characterized in that: The training method of the forged speech attribution model comprises: Input the time-frequency features of the speech samples in the training set, and use the front-end time-frequency attention module to extract the front-end time-frequency features weighted by the time-frequency attention; use the front-end time-frequency features weighted by the time-frequency attention as input, and use the feature extraction module to extract the deep features; output the deep features to the feature classification module, and output the predicted logical values ​​of the speech training samples belonging to each category; According to the predicted logical value, the predicted label of the sample is obtained, and the classification loss is calculated; Classification loss is used to supervise the training of the front-end time-frequency attention module, feature extraction module and feature classification module to obtain the weight parameters of the front-end time-frequency attention module, feature extraction module and feature classification module.

9. The method for attributing fake speech according to claim 8, characterized in that: Calculate the classification loss, including: Use Softmax cross entropy loss function or AM-Softmax loss function to calculate classification loss; Consider a batch of samples , the number is , taking the logical value and label output by the feature classification module as input, when the Softmax cross entropy loss function is used to calculate the classification loss, the formula is: （10） in, Indicates samples, yes The corresponding category logical value vector, represents the classification module, It is the feature extraction module The extracted deep features, yes The front-end time-frequency features weighted by the front-end time-frequency attention; When the AM-Softmax loss function is used to calculate the classification loss, the calculation formula is as follows: (10). The only difference is the calculation method of the logical value. The logical value used by the AM-Softmax loss function is: （10-1） in, They are deep features and the classification module , Class classifier weights The intra-class and inter-class similarities of is the margin imposed on the intra-class similarity.

10. A forged speech attribution system, characterized in that: include: A data acquisition module, used to acquire speech samples; A data processing module, used to extract time-frequency features of speech samples; An attribution module is used to input the time-frequency features of the speech sample into the trained forged speech attribution model, and output the probability that the time-frequency features of the speech sample belong to various speech forgery methods as the identification result of the forged speech attribution; The forged voice attribution model includes: A front-end time-frequency attention module is used to weight the time-frequency features of the speech sample using the front-end time-frequency attention to obtain the front-end time-frequency features weighted by the time-frequency attention; A feature extraction module, used to extract deep features of speech samples from front-end time-frequency features weighted by time-frequency attention; The feature classification module is used to classify the deep features of the speech sample and output the probability that the time-frequency features of the speech sample belong to each category in the speech forgery method.

Images