A voice trace evidence method and device, electronic equipment and storage medium

Abstract

The application provides a voice tracing forensics method and device, electronic equipment and storage medium, comprising: processing the to-be-detected voice based on an audio band-pass filter to determine the effective audio of the to-be-detected voice; inputting the effective audio into a pre-trained voice tracing forensics model to extract the audio high-level features of the effective audio, performing convolution processing on the audio high-level features based on the convolution pooling network layer of the voice tracing forensics model to determine the audio high-level features after convolution processing; processing the audio high-level features after convolution processing based on the channel attention mechanism and the fully connected network layer of the voice tracing forensics model to determine the target high-level features, calculating the probability distribution of each tracing category corresponding to the target high-level features based on the activation function to predict the voice tracing forensics result of the to-be-detected voice. The voice tracing forensics result corresponding to the voice is quickly and accurately determined through the voice tracing forensics model.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a method, apparatus, electronic device and storage medium for speech source tracing and evidence collection. Background Technology

[0002] Current audio source tracing tasks still rely on raw audio features as input, such as MFCC, Fbank, and LFCC acoustic features. These features are widely used in some audio classification tasks. However, with the development of large-scale audio models, the training data and model parameters have increased exponentially, significantly improving the naturalness and realism of speech synthesis. Previous speech synthesis and speech conversion algorithms often suffered from problems such as a mechanical feel, loss of sound quality, lack of prosody, and lack of emotion. Large-scale audio generation models, through deep learning and complex acoustic modeling, can more accurately simulate the characteristics and patterns of human speech, generating more natural and fluent audio content. They can accurately synthesize or imitate the speaker's timbre, prosody, emotion, and pronunciation habits. With the powerful learning capabilities of large-scale audio models, the difference between audio generated by different methods and real audio is becoming increasingly smaller, making source tracing tasks between different generation methods more difficult. Therefore, improving the accuracy of audio source determination has become a significant technical challenge. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a speech source tracing and evidence collection method, device, electronic device and storage medium. By adding a channel attention mechanism to the speech source tracing and evidence collection model, the learning ability of the model for fine-grained audio features is improved. During the training process, a gradient inversion network layer is introduced to improve the robustness of the speech source tracing and evidence collection model, so as to quickly and accurately determine the speech source tracing and evidence collection result corresponding to the speech.

[0004] This application provides a method for voice source tracing and evidence collection, the method comprising:

[0005] The speech to be detected is acquired, and the speech is processed based on an audio bandpass filter to determine the effective audio of the speech to be detected.

[0006] The valid audio is input into a pre-trained speech tracing and evidence collection model to extract the high-level audio features of the valid audio. The high-level audio features are then convolved based on the convolutional pooling network layer of the speech tracing and evidence collection model to determine the high-level audio features after convolution. The speech tracing and evidence collection model is determined by simulating adversarial training by adding a channel attention mechanism and a gradient inversion network layer to the neural network model.

[0007] Based on the channel attention mechanism and the fully connected network layer of the speech tracing and evidence collection model, the high-level audio features after convolution are processed to determine the target high-level features. Based on the activation function, the probability distribution of each tracing category corresponding to the target high-level features is calculated to predict the speech tracing and evidence collection result of the speech to be detected. Here, tracing and evidence collection refers to verifying the source or method of generating the speech.

[0008] In one possible implementation, the channel attention mechanism and fully connected network layer of the speech source tracing and evidence collection model process the high-level audio features after convolution to determine the target high-level features. Based on the activation function, the probability distribution of each source tracing category corresponding to the target high-level features is calculated to predict the speech source tracing and evidence collection result of the speech to be detected, including:

[0009] The high-level audio features after convolution are subjected to attention processing based on the channel attention mechanism to determine the high-level audio features after attention processing.

[0010] The high-level audio features after attention processing are processed based on the fully connected network layer to determine the target high-level features;

[0011] The probability distribution of each source tracing category corresponding to the target high-level feature is calculated based on the activation function, and multiple probability values ​​are determined. The source tracing category corresponding to the largest probability value is determined as the speech source tracing and evidence collection result of the speech to be detected.

[0012] In one possible implementation, the voice source tracing and evidence collection model is trained through the following steps:

[0013] Sample audio data is selected from the augmented training data, and the sample audio data is filtered to determine the valid audio samples.

[0014] The valid audio samples are input into a neural network model. High-level features of the valid audio samples are extracted based on self-supervised pre-training. Adversarial training is then performed on the high-level features of the audio samples until the neural network model converges, at which point the training of the neural network model is stopped.

[0015] The parameters of each layer of the neural network model with the best performance during the model training process are averaged, and the averaged parameters of each layer are used to construct the speech source tracing and evidence collection model.

[0016] In one possible implementation, the augmented training data is determined in the following way:

[0017] Add noise with different signal-to-noise ratios to the training data to determine the training data after data augmentation;

[0018] Alternatively, gradient-based adversarial example generation algorithms can generate different adversarial examples to determine the augmented training data.

[0019] In one possible implementation, the step of inputting the sample valid audio into a neural network model, extracting the sample audio high-level features of the sample valid audio based on self-supervised pre-training, and performing adversarial training on the sample audio high-level features until the neural network model converges and training of the neural network model is stopped includes:

[0020] The high-level features of the sample audio are subjected to convolutional pooling and channel attention mechanisms to determine the high-level features of the sample audio after channel attention mechanism processing.

[0021] The sample audio high-level features after the channel attention mechanism are processed based on the first fully connected network layer, and the first sample audio high-level features and the predicted source tag corresponding to the first sample audio high-level features are output.

[0022] A gradient inversion network layer is added after the first fully connected network layer. The high-level features of the first sample audio are input into the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are input into the second fully connected network layer after the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are input into the third fully connected network layer after the second fully connected network layer to output the high-level features of the second sample audio and the adversarial label of the high-level features of the second sample audio.

[0023] Based on the high-level features of the first sample audio, the high-level features of the second sample audio, the predicted source label, the adversarial label, and the true source label of the sample audio data, the total loss value of the neural network model is determined, and training of the neural network model is stopped when the neural network model converges.

[0024] In one possible implementation, the step of performing convolutional pooling and channel attention processing on the high-level features of the sample audio to determine the high-level features of the sample audio after channel attention processing includes:

[0025] The high-level features of the sample audio are subjected to global max pooling and global average pooling in the spatial dimension to determine the global average pooling features and max pooling features.

[0026] The global average pooling feature and the max pooling feature are respectively input into the multilayer perceptron network layer for channel dimension feature learning, and the global average pooling feature and the max pooling feature processed by the multilayer perceptron network layer are output.

[0027] The global average pooling feature processed by the multilayer perceptron network layer and the max pooling feature processed by the multilayer perceptron network layer are concatenated to determine the concatenated feature.

[0028] The concatenated features are mapped based on the activation function to determine the channel attention weight matrix;

[0029] Based on the channel attention weight matrix, the global average pooling feature, and the max pooling feature, the high-level features of the sample audio after processing by the channel attention mechanism are determined.

[0030] In one possible implementation, determining the total loss value of the neural network model based on the first sample audio high-level features, the second sample audio high-level features, the predicted source label, the adversarial label, and the true source label of the sample audio data, and stopping the training of the neural network model when the neural network model converges, includes:

[0031] The classification loss value is determined by calculating the high-level features of the first sample audio, the predicted source label, and the real source label based on the weighted cross-entropy loss function.

[0032] The adversarial loss value is determined by calculating the high-level features of the second sample audio and the adversarial label based on the binary cross-entropy loss function.

[0033] Based on the classification loss value and the adversarial loss value, the total loss value of the neural network model is determined, and training of the neural network model is stopped when the neural network model converges.

[0034] This application embodiment also provides a voice source tracing and evidence collection device, the voice source tracing and evidence collection device comprising:

[0035] The filtering module is used to acquire the speech to be detected, process the speech to be detected based on the audio bandpass filter, and determine the effective audio of the speech to be detected.

[0036] The first processing module is used to input the valid audio into a pre-trained speech tracing and evidence collection model, extract the high-level audio features of the valid audio, and perform convolution processing on the high-level audio features based on the convolutional pooling network layer of the speech tracing and evidence collection model to determine the high-level audio features after convolution processing; wherein, the speech tracing and evidence collection model is determined by adding a channel attention mechanism and a gradient inversion network layer to the neural network model to simulate adversarial training;

[0037] The second processing module is used to process the high-level audio features after convolution based on the channel attention mechanism and fully connected network layer of the speech tracing and evidence collection model, determine the target high-level features, calculate the probability distribution of each tracing category corresponding to the target high-level features based on the activation function, and predict the speech tracing and evidence collection result of the speech to be detected; wherein, tracing and evidence collection refers to verifying the source or method of generating the speech.

[0038] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the voice tracing and evidence collection method described above are performed.

[0039] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the voice tracing and evidence collection method described above.

[0040] This application provides a method, apparatus, electronic device, and storage medium for voice source tracing and evidence collection. The voice source tracing and evidence collection method includes: acquiring a speech to be detected; processing the speech to be detected based on an audio bandpass filter to determine the effective audio of the speech to be detected; inputting the effective audio into a pre-trained voice source tracing and evidence collection model to extract the high-level audio features of the effective audio; performing convolution processing on the high-level audio features based on the convolution pooling network layer of the voice source tracing and evidence collection model to determine the high-level audio features after convolution processing; wherein, the voice source tracing and evidence collection model is determined by simulating generative adversarial training by adding a channel attention mechanism and a gradient inversion network layer to a neural network model; processing the high-level audio features after convolution processing based on the channel attention mechanism and fully connected network layer of the voice source tracing and evidence collection model to determine the target high-level features; calculating the probability distribution of each source tracing category corresponding to the target high-level features based on an activation function to predict the voice source tracing and evidence collection result of the speech to be detected; wherein, source tracing and evidence collection refers to verifying the source or method of generating the speech. By incorporating a channel attention mechanism into the speech tracing and evidence collection model, the model's ability to learn fine-grained audio features is improved. During training, a gradient inversion network layer is introduced to enhance the robustness of the speech tracing and evidence collection model, enabling the model to quickly and accurately determine the speech tracing and evidence collection results corresponding to the speech.

[0041] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a voice source tracing and evidence collection method provided in this application embodiment;

[0044] Figure 2 This is one of the structural schematic diagrams of a voice source tracing and evidence collection device provided in the embodiments of this application;

[0045] Figure 3 This is a second schematic diagram of the structure of a voice source tracing and evidence collection device provided in an embodiment of this application;

[0046] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0048] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of speech recognition technology.

[0049] Research has revealed that existing audio source tracing tasks still rely on raw audio features as input, such as acoustic features like MFCC, Fbank, and LFCC. While these features are widely used in some audio classification tasks, the development of large-scale audio models has led to an exponential increase in training data and model parameters, significantly improving the naturalness and realism of speech synthesis. Previous speech synthesis and conversion algorithms often suffered from issues like a mechanical feel, loss of sound quality, lack of prosody, and lack of emotion. Large-scale audio generation models, through deep learning and complex acoustic modeling, can more accurately simulate the characteristics and patterns of human speech, generating more natural and fluent audio content. They can accurately synthesize or imitate the speaker's timbre, prosody, emotion, and pronunciation habits. With the powerful learning capabilities of large-scale audio models, the difference between audio generated by different methods and real audio is decreasing, making source tracing tasks between different generation methods more difficult. Therefore, improving the accuracy of audio source determination has become a significant technical challenge.

[0050] Based on this, this application provides a speech source tracing and evidence collection method. By adding a channel attention mechanism to the speech source tracing and evidence collection model, the model's ability to learn fine-grained audio features is improved. During the training process, a gradient inversion network layer is introduced to improve the robustness of the speech source tracing and evidence collection model, so that the speech source tracing and evidence collection result corresponding to the speech can be quickly and accurately determined through the speech source tracing and evidence collection model.

[0051] Please see Figure 1 , Figure 1 This is a flowchart illustrating a voice source tracing and forensics method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the voice source tracing and evidence collection method includes:

[0052] S101: Acquire the speech to be detected, process the speech to be detected based on an audio bandpass filter, and determine the valid audio of the speech to be detected.

[0053] In this step, the speech to be detected is acquired, and the speech is filtered using an audio bandpass filter to determine the valid audio of the speech to be detected.

[0054] Here, in addition to the speech to be detected, it can also be the audio to be detected; this part is not specifically limited.

[0055] Here, a fifth-order Butterworth bandpass filter is used, with a frequency range of [50Hz, 6000Hz]. This is because below 50Hz, the frequencies are generally low-frequency noise from the device itself, which does not contain the effective information needed for audio source tracing. If this frequency band information is retained, the source tracing task converges more slowly. In addition, with the development of large model speech generation technology, the differences between different generation algorithms at high frequencies are very small, and it is difficult for the source tracing engine to trace the audio source based on high-frequency features. Therefore, this invention proposes to filter the frequency domain features above 6000Hz, retaining only the effective audio information in the [50Hz, 6000Hz] range for audio source tracing.

[0056] S102: Input the valid audio into the pre-trained speech tracing and evidence collection model, extract the high-level audio features of the valid audio, perform convolution processing on the high-level audio features based on the convolution pooling network layer of the speech tracing and evidence collection model, and determine the high-level audio features after convolution processing.

[0057] In this step, valid audio is input into the speech tracing and evidence collection model, and the high-level audio features of the valid audio are extracted. The high-level audio features are then convolved by the convolutional pooling network layer of the speech tracing and evidence collection model to determine the high-level audio features after convolution.

[0058] The voice source tracing and evidence collection model was determined by adding a channel attention mechanism and a gradient inversion network layer to the neural network model to simulate adversarial training.

[0059] Here, audio source tracing algorithms typically use one or two audio features as input. However, these often have limitations in source tracing tasks involving large-scale audio generation. Furthermore, supervised training of neural network models heavily relies on large labeled datasets; generally, the larger the dataset, the better the model's generalization performance. However, collecting large labeled datasets for different tasks is costly, and low-quality labels can lead to non-convergence during model training. To alleviate the dependence on supervised training, this patent proposes a strategy of self-supervised pre-training and supervised fine-tuning to improve the performance of source tracing models. The self-supervised pre-trained model extracts representation vectors, compensating for insufficient feature extraction due to insufficient data volume in supervised training. High-level audio features can be extracted using models based on large-scale unsupervised training, such as Wav2vec, CPC, Wav2vec2.0, HuBERT, Whisper, and WavLM.

[0060] S103: Based on the channel attention mechanism of the speech tracing and evidence collection model and the fully connected network layer, the high-level audio features after convolution are processed to determine the target high-level features. Based on the activation function, the probability distribution of each tracing category corresponding to the target high-level features is calculated to predict the speech tracing and evidence collection result of the speech to be detected.

[0061] In this step, the channel attention mechanism of the speech source tracing and evidence collection model and the fully connected network layer process the high-level audio features after convolution to determine the target high-level features. Then, the probability distribution of each source tracing category corresponding to the target high-level features is calculated according to the activation function to predict the speech source tracing and evidence collection result of the speech to be detected.

[0062] The tracing categories can be tracing between different large audio models, such as Sora, Audiobox, VASA-1, Mega-TTS, and chatTTS; tracing between different commercial software, such as ACE studio, Suno, FreeTTS, TTSMaker, and Uberduck; tracing between audio forgery methods, such as speech synthesis, timbre conversion, speech splicing, speech playback, and playback after forgery; tracing between different vocoders, such as those based on autoregression, Flows, generative adversarial networks, VAEs, and diffusion; and tracing between different audio synthesis vendors, such as iFlytek, Baidu, Alibaba, Tencent, and ByteDance.

[0063] In one possible implementation, the channel attention mechanism and fully connected network layer of the speech source tracing and evidence collection model process the high-level audio features after convolution to determine the target high-level features. Based on the activation function, the probability distribution of each source tracing category corresponding to the target high-level features is calculated to predict the speech source tracing and evidence collection result of the speech to be detected, including:

[0064] A: Attention processing is performed on the high-level audio features after convolution based on the channel attention mechanism to determine the high-level audio features after attention processing.

[0065] Here, attention processing is performed on the high-level audio features after convolution based on the channel attention mechanism to determine the high-level audio features after attention processing.

[0066] B: Based on the fully connected network layer, the high-level audio features after attention processing are processed to determine the target high-level features.

[0067] Here, the high-level audio features after attention processing are processed by the fully connected network layer to determine the target high-level features.

[0068] C: Based on the activation function, calculate the probability distribution of each source tracing category corresponding to the target high-level feature, determine multiple probability values, and determine the source tracing category corresponding to the largest probability value as the speech source tracing and evidence collection result of the speech to be detected.

[0069] Here, the probability distribution of each source tracing category corresponding to the target high-level features is calculated based on the activation function to determine multiple probability values. The source tracing category corresponding to the maximum probability value is determined as the speech source tracing and evidence collection result of the speech to be detected.

[0070] In one possible implementation, the voice source tracing and evidence collection model is trained through the following steps:

[0071] (1): Select sample audio data from the training data after data augmentation, filter the sample audio data, and determine the effective audio samples of the sample audio data.

[0072] Here, sample audio data is selected from the augmented training data, filtered, and the valid audio samples are determined.

[0073] In one possible implementation, the augmented training data is determined in the following way:

[0074] Adding noise with different signal-to-noise ratios to the training data can help determine the augmented training data; alternatively, gradient-based adversarial example generation algorithms can generate different adversarial examples to help determine the augmented training data.

[0075] Here, different audio generation algorithms often produce audio with varying degrees of reverberation or metallic sounds. However, it is difficult to cover all generated audio in the training data for audio source tracing. Furthermore, some large-scale audio model generation algorithms can generate AI songs online, some even including synthesized accompaniments. These accompaniments can mask subtle differences in the synthesized songs, further increasing the difficulty of audio source tracing tasks. This invention proposes adding noise with different signal-to-noise ratios online to the training data to simulate various possible scenarios, thereby expanding the data. The noise dataset can select music and some accompaniment songs, with the signal-to-noise ratio set according to [5, 10, 15, 20, 25, 30] dB. The calculation of the signal-to-noise ratio is as follows:

[0076]

[0077] In noisy environments, the signal-to-noise ratio (SNR) is commonly used to measure the level of noise in speech. r The signal-to-noise ratio (SNR) represents the power of the clean speech signal (P). s ) and noise speech signal power (P) n The ratio of s(i) to n(i) is given by the formula, where i is the sampling point number, s(i) is the clean signal, n(i) is the noise signal, and N is the length of the speech sequence. Then the noisy speech signal y(i) = s(i) + n(i) is given by the formula, where i is the sampling point number, s(i) is the clean signal, n(i) is the noise signal, and N is the length of the speech sequence.

[0078] Here, this invention proposes using adversarial examples to robustly train the source tracing model in order to prevent attacks. Gradient-based adversarial example generation algorithms are very common, such as Fast Gradient Signed Method (FGSM), PGD, BIM, etc., as well as gradient optimization-based adversarial example generation methods like C&W, L-BFGS, and black-box attack methods like JSMA, One Pixel, etc.

[0079] (2): Input the valid audio samples into the neural network model, extract the high-level features of the valid audio samples based on self-supervised pre-training, and perform adversarial training on the high-level features of the audio samples until the neural network model converges and the training of the neural network model is stopped.

[0080] Here, the valid audio samples are input into the neural network model. The high-level features of the valid audio samples are extracted using a self-supervised pre-training method. Adversarial training is then performed on the high-level features of the audio samples until the neural network model converges, at which point the training of the neural network model is stopped.

[0081] In one possible implementation, the step of inputting the sample valid audio into a neural network model, extracting the sample audio high-level features of the sample valid audio based on self-supervised pre-training, and performing adversarial training on the sample audio high-level features until the neural network model converges and training of the neural network model is stopped includes:

[0082] I: Perform convolutional pooling and channel attention processing on the high-level features of the sample audio to determine the high-level features of the sample audio after channel attention processing.

[0083] Here, the high-level features of this audio are processed by convolutional pooling and channel attention mechanism to determine the high-level features of the sample audio after channel attention mechanism processing.

[0084] In the model training phase, the high-level features of the sample audio are reduced in dimensionality through convolutional pooling. Then, each channel of the feature map is treated as a feature detector through the channel attention mechanism, so that the channel features focus on the effective information in the feature map.

[0085] In one possible implementation, the step of performing convolutional pooling and channel attention processing on the high-level features of the sample audio to determine the high-level features of the sample audio after channel attention processing includes:

[0086] i: Perform global max pooling and global average pooling on the high-level features of the sample audio to determine the global average pooling features and max pooling features.

[0087] Here, global max pooling and global average pooling are performed on the high-level features of the sample audio to determine the global average pooling features and max pooling features.

[0088] Specifically, a high-level feature map F of sample audio with size H×W×C is subjected to global max pooling and global average pooling in the spatial dimension to obtain two 1×1×C feature maps, which are the feature maps corresponding to the global average pooling feature and the max pooling feature, respectively.

[0089] ii: Input the global average pooling feature and the max pooling feature into the multilayer perceptron network layer respectively for channel-dimensional feature learning, and output the global average pooling feature and the max pooling feature after processing by the multilayer perceptron network layer.

[0090] Here, the global average pooling features and the max pooling features are respectively input into the multilayer perceptron network layer for channel-dimensional feature learning, and the global average pooling features and the max pooling features processed by the multilayer perceptron network layer are output.

[0091] In this approach, the global max-pooling features and global average-pooling features are respectively fed into a shared multilayer perceptron (MLP) for learning. The first layer of the MLP has C / r neurons with ReLU activation function, and the second layer has C neurons.

[0092] iii: The global average pooling features processed by the multilayer perceptron network layer and the max pooling features processed by the multilayer perceptron network layer are concatenated to determine the concatenated features; the concatenated features are mapped based on the activation function to determine the channel attention weight matrix.

[0093] Here, the global average pooling features and the max pooling features processed by the multilayer perceptron network layers are concatenated to determine the concatenated features. The concatenated features are then mapped according to the activation function to determine the channel attention weight matrix.

[0094] The output of the MLP is added, then processed by the Sigmoid activation function to obtain the channel attention weight matrix M. c .

[0095] M c ∈R C×1×1

[0096] To reduce computational parameters, a dimensionality reduction coefficient r is used in MLP. c ∈R C / r×1×1 , where R is the dimension.

[0097] iv: Based on the channel attention weight matrix, the global average pooling feature, and the max pooling feature, determine the high-level features of the sample audio after the channel attention mechanism processing.

[0098] Here, the high-level features of the sample audio after processing by the channel attention mechanism are determined using the following formula:

[0099]

[0100] in, This is a global average pooling feature. For max pooling features, W1 and W2 are the weights of the MLP, σ(·) is the sigmoid function, and M c (F) represents the high-level features of the sample audio after processing by the channel attention mechanism.

[0101] II: Based on the first fully connected network layer, the high-level features of the sample audio after the channel attention mechanism are processed, and the first high-level features of the sample audio and the predicted source label corresponding to the first high-level features of the sample audio are output.

[0102] Here, the high-level features of the sample audio after the channel attention mechanism is processed by the first fully connected network layer, and the high-level features of the first sample audio and the predicted source label corresponding to the high-level features of the first sample audio are output.

[0103] Among them, the source label is the label corresponding to the audio source category corresponding to the predicted high-level features of the first sample audio.

[0104] III: Add a gradient inversion network layer after the first fully connected network layer. Input the first sample audio high-level features into the gradient inversion network layer for processing and output the processed first sample audio high-level features. Input the processed first sample audio high-level features into the second fully connected network layer after the gradient inversion network layer for processing and output the processed first sample audio high-level features of the second fully connected network layer. Input the processed first sample audio high-level features of the second fully connected network layer into the third fully connected network layer after the second fully connected network layer to output the second sample audio high-level features and the adversarial label of the second sample audio high-level features.

[0105] Here, a gradient inversion network layer is added after the first fully connected network layer. The high-level features of the first sample audio are input into the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are then input into the second fully connected network layer after the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are then input into the third fully connected network layer after the second fully connected network layer to output the high-level features of the second sample audio and the adversarial label of the high-level features of the second sample audio.

[0106] IV: Based on the high-level features of the first sample audio, the high-level features of the second sample audio, the predicted source label, the adversarial label, and the true source label of the sample audio data, determine the total loss value of the neural network model, and stop training the neural network model when the neural network model converges.

[0107] Here, the total loss value of the neural network model is determined based on the high-level features of the first sample audio, the high-level features of the second sample audio, the predicted source label, the adversarial label, and the real source label, and the training of the neural network model is stopped when the neural network model converges.

[0108] In one possible implementation, determining the total loss value of the neural network model based on the first sample audio high-level features, the second sample audio high-level features, the predicted source label, the adversarial label, and the true source label of the sample audio data, and stopping the training of the neural network model when the neural network model converges, includes:

[0109] The classification loss value is determined by calculating the high-level features of the first sample audio, the predicted source label, and the real source label based on the weighted cross-entropy loss function; the adversarial loss value is determined by calculating the high-level features of the second sample audio and the adversarial label based on the binary cross-entropy loss function; the total loss value of the neural network model is determined based on the classification loss value and the adversarial loss value, and the training of the neural network model is stopped when the neural network model converges.

[0110] The high-level features F of the first sample audio are obtained through the first fully connected network layer. s , will F s The classification loss L is calculated using the weighted cross-entropy (WCE) loss function, along with the source label y and the true source label. norm A gradient reversal layer (GRL) is added after the first fully connected network layer. The gradient fed into the GRL is multiplied by a negative number λ, so that the training objectives of the networks before and after the GRL are reversed. In the initial stage of the model, λ approaches 0, and as the number of training iterations increases, its value approaches -1. The calculation formula is shown below:

[0111]

[0112] In the formula C iLet K be the current iteration number, K be the total number of iterations set for model training, and T be an empirical coefficient. After incorporating GRL, the fully connected network layer has two objectives to satisfy: first, the fully connected network layer needs to generate features that can predict the correct labels; second, the features extracted by the fully connected layer should be as indistinguishable from their task domain as possible. These two objectives are adversarial. When calculating the adversarial loss, the features after passing through the GRL layer are passed through two fully connected network layers sequentially, and then the output is compared with the adversarial label. (If all labels are 0 or all labels are 1) Calculate the adversarial loss L using the Binary Cross Entropy (BCE) function. adv Ultimately, the classification loss L norm and combat losses L adv The weighted average yields the model's loss function as follows:

[0113] L = L norm +α×L adv

[0114] In the formula, α is an adjustment factor used to adjust the influence of adversarial loss on the model loss, and L is the total loss value.

[0115] Here, by adding a GRL layer to the model, the model can achieve an identity transformation during forward propagation and use negative coefficients for the gradient during backward propagation, thus adding constraints to the model training and enabling the trained speech tracing and evidence collection model to better identify unknown tracing algorithms.

[0116] c: The parameters of each layer of the neural network model with the best performance during the model training process are averaged, and the voice source tracing and evidence collection model is constructed based on the averaged parameters of each layer.

[0117] Here, during model training, the ten best-performing model parameters from the validation set are saved. Because the source tracing performance of different model parameters varies, to improve the stability of the source tracing model, the parameters of each layer of the Top 10 models are averaged to obtain the final speech source tracing and evidence collection model. In the usage phase, the test audio is input into the speech source tracing and evidence collection model, and the scores for each audio source tracing category are obtained. Then, the probability distribution of each category is obtained through the softmax function. The category corresponding to the column with the maximum value is the audio source tracing category, which is then compared with the training labels.

[0118] To address the issue that existing source tracing algorithms cannot meet the requirements of large-scale audio generation source tracing tasks, this invention proposes an end-to-end network structure to extract high-level audio representations from the original audio sequences. This simplifies the source tracing task while more effectively extracting feature parameters specific to the task. To address the limited data volume for some source tracing types, adversarial examples are used to augment the training data. Compared to previous methods of adding noise to augment data, this approach significantly improves the model's generalization and robustness. To address the poor generalization of existing source tracing models to the same generation method and its variants, this invention adjusts the traditional source tracing model structure, proposing a self-supervised model pre-trained on large-scale data and incorporating a channel attention mechanism to enhance the model's learning ability for fine-grained audio features. Furthermore, a gradient reversal layer is introduced during model training to further improve the robustness of the speech source tracing and evidence collection model.

[0119] In a specific implementation, step 1: The original audio signal is encoded and decoded using a unified frequency format; step 2: Various noises are randomly added to the audio, requiring control of different signal-to-noise ratios; step 3: Different adversarial example generation methods are used to construct an adversarial example dataset; methods include gradient-based FGSM, PGD, BIM, and optimization-based adversarial example generation methods C&W, L-BFGS, as well as black-box attack methods JSMA, One Pixel, etc.; step 4: All data are passed through a bandpass filter of [50Hz, 6000Hz] to retain the effective information of the audio; step 5: The filtered audio is input into an end-to-end model, high-level representations of the audio are extracted, and adversarial training is performed until the model converges; step 6: The top 10 model parameters are averaged to obtain a speech tracing and evidence collection model; step 7: The test audio is input into the speech tracing and evidence collection model to obtain the scores of each tracing category. The probability distribution of each category is obtained through the Softmax function, and the category corresponding to the column with the maximum value is the audio tracing category.

[0120] Here, during model training, a method combining adversarial examples and noise addition is proposed to augment the source tracing data. Then, an audio preprocessing module removes low-frequency and high-frequency interference (audio signals in these frequency bands often have a negative impact on audio source tracing tasks) and inputs it into an end-to-end source tracing network. The intermediate nodes of a self-supervised pre-trained large-scale speech model are used as high-level representations of audio features. During training, a gradient reversal layer (GRL) is used to simulate the generative adversarial mechanism, enhancing the model's generalization ability during training. Finally, the top 10 source tracing models are fused using parameter weighting. This invention addresses the problems of limited source tracing data samples, slow source tracing tasks, poor performance, and poor generalization capabilities in the context of large-scale model generation.

[0121] This application provides a speech source tracing and evidence collection method, which includes: acquiring a speech to be detected; processing the speech to be detected based on an audio bandpass filter to determine the valid audio of the speech to be detected; inputting the valid audio into a pre-trained speech source tracing and evidence collection model to extract the high-level audio features of the valid audio; performing convolution processing on the high-level audio features based on the convolution pooling network layer of the speech source tracing and evidence collection model to determine the high-level audio features after convolution processing; wherein, the speech source tracing and evidence collection model is determined by simulating generative adversarial training by adding a channel attention mechanism and a gradient inversion network layer to a neural network model; processing the high-level audio features after convolution processing based on the channel attention mechanism and fully connected network layer of the speech source tracing and evidence collection model to determine the target high-level features; calculating the probability distribution of each source tracing category corresponding to the target high-level features based on an activation function to predict the speech source tracing and evidence collection result of the speech to be detected; wherein, source tracing and evidence collection refers to verifying the source or method of generating the speech. By incorporating a channel attention mechanism into the speech tracing and evidence collection model, the model's ability to learn fine-grained audio features is improved. During training, a gradient inversion network layer is introduced to enhance the robustness of the speech tracing and evidence collection model, enabling the model to quickly and accurately determine the speech tracing and evidence collection results corresponding to the speech.

[0122] Please see Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of a voice source tracing and evidence collection device provided in the embodiments of this application; Figure 3 This is a second schematic diagram of a voice source tracing and evidence collection device provided in an embodiment of this application. Figure 2 As shown, the voice source tracing and evidence collection device 200 includes:

[0123] The filtering module 210 is used to acquire the speech to be detected, process the speech to be detected based on the audio bandpass filter, and determine the effective audio of the speech to be detected.

[0124] The first processing module 220 is used to input the valid audio into a pre-trained speech tracing and evidence collection model, extract the high-level audio features of the valid audio, and perform convolution processing on the high-level audio features based on the convolutional pooling network layer of the speech tracing and evidence collection model to determine the high-level audio features after convolution processing; wherein, the speech tracing and evidence collection model is determined by adding a channel attention mechanism and a gradient inversion network layer to the neural network model to simulate adversarial training;

[0125] The second processing module 230 is used to process the high-level audio features after convolution based on the channel attention mechanism and fully connected network layer of the speech tracing and evidence collection model, determine the target high-level features, calculate the probability distribution of each tracing category corresponding to the target high-level features based on the activation function, and predict the speech tracing and evidence collection result of the speech to be detected; wherein, tracing and evidence collection refers to verifying the source or method of generating the speech.

[0126] Furthermore, when the second processing module 230 processes the high-level audio features after convolution processing using the channel attention mechanism and fully connected network layer based on the speech source tracing and evidence collection model to determine the target high-level features, and calculates the probability distribution of each source tracing category corresponding to the target high-level features based on the activation function to predict the speech source tracing and evidence collection result of the speech to be detected, the second processing module 230 is specifically used for:

[0127] The high-level audio features after convolution are subjected to attention processing based on the channel attention mechanism to determine the high-level audio features after attention processing.

[0128] The high-level audio features after attention processing are processed based on the fully connected network layer to determine the target high-level features;

[0129] The probability distribution of each source tracing category corresponding to the target high-level feature is calculated based on the activation function, and multiple probability values ​​are determined. The source tracing category corresponding to the largest probability value is determined as the speech source tracing and evidence collection result of the speech to be detected.

[0130] Furthermore, such as Figure 3 As shown, the voice source tracing and evidence collection model 200 also includes a model training module 240, which trains the voice source tracing and evidence collection model through the following steps:

[0131] Sample audio data is selected from the augmented training data, and the sample audio data is filtered to determine the valid audio samples.

[0132] The valid audio samples are input into a neural network model. High-level features of the valid audio samples are extracted based on self-supervised pre-training. Adversarial training is then performed on the high-level features of the audio samples until the neural network model converges, at which point the training of the neural network model is stopped.

[0133] The parameters of each layer of the neural network model with the best performance during the model training process are averaged, and the averaged parameters of each layer are used to construct the speech source tracing and evidence collection model.

[0134] Furthermore, such as Figure 3 As shown, the voice source tracing and evidence collection model 200 also includes a data expansion module 250, which is used for:

[0135] Add noise with different signal-to-noise ratios to the training data to determine the training data after data augmentation;

[0136] Alternatively, gradient-based adversarial example generation algorithms can generate different adversarial examples to determine the augmented training data.

[0137] Furthermore, when the model training module 240 is used to input the sample valid audio into the neural network model, extract the sample audio high-level features of the sample valid audio based on self-supervised pre-training, and perform adversarial training on the sample audio high-level features until the neural network model converges and training of the neural network model is stopped, the model training module 240 is specifically used for:

[0138] The high-level features of the sample audio are subjected to convolutional pooling and channel attention mechanisms to determine the high-level features of the sample audio after channel attention mechanism processing.

[0139] The sample audio high-level features after the channel attention mechanism are processed based on the first fully connected network layer, and the first sample audio high-level features and the predicted source tag corresponding to the first sample audio high-level features are output.

[0140] A gradient inversion network layer is added after the first fully connected network layer. The high-level features of the first sample audio are input into the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are input into the second fully connected network layer after the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are input into the third fully connected network layer after the second fully connected network layer to output the high-level features of the second sample audio and the adversarial label of the high-level features of the second sample audio.

[0141] Based on the high-level features of the first sample audio, the high-level features of the second sample audio, the predicted source label, the adversarial label, and the true source label of the sample audio data, the total loss value of the neural network model is determined, and training of the neural network model is stopped when the neural network model converges.

[0142] Furthermore, when the model training module 240 performs convolutional pooling and channel attention processing on the high-level features of the sample audio to determine the high-level features of the sample audio after channel attention processing, the model training module 240 specifically performs the following:

[0143] The high-level features of the sample audio are subjected to global max pooling and global average pooling in the spatial dimension to determine the global average pooling features and max pooling features.

[0144] The global average pooling feature and the max pooling feature are respectively input into the multilayer perceptron network layer for channel dimension feature learning, and the global average pooling feature and the max pooling feature processed by the multilayer perceptron network layer are output.

[0145] The global average pooling feature processed by the multilayer perceptron network layer and the max pooling feature processed by the multilayer perceptron network layer are concatenated to determine the concatenated feature.

[0146] The concatenated features are mapped based on the activation function to determine the channel attention weight matrix;

[0147] Based on the channel attention weight matrix, the global average pooling feature, and the max pooling feature, the high-level features of the sample audio after processing by the channel attention mechanism are determined.

[0148] Furthermore, when the model training module 240 determines the total loss value of the neural network model based on the first sample audio high-level features, the second sample audio high-level features, the predicted source label, the adversarial label, and the true source label of the sample audio data, and stops training the neural network model when the neural network model converges, the model training module 240 is specifically used for:

[0149] The classification loss value is determined by calculating the high-level features of the first sample audio, the predicted source label, and the real source label based on the weighted cross-entropy loss function.

[0150] The adversarial loss value is determined by calculating the high-level features of the second sample audio and the adversarial label based on the binary cross-entropy loss function.

[0151] Based on the classification loss value and the adversarial loss value, the total loss value of the neural network model is determined, and training of the neural network model is stopped when the neural network model converges.

[0152] This application provides a speech source tracing and evidence collection device, comprising: a filtering module for acquiring speech to be detected, processing the speech to be detected based on an audio bandpass filter, and determining the valid audio of the speech to be detected; a first processing module for inputting the valid audio into a pre-trained speech source tracing and evidence collection model, extracting the high-level audio features of the valid audio, and performing convolution processing on the high-level audio features based on the convolution pooling network layer of the speech source tracing and evidence collection model, and determining the high-level audio features after convolution processing; wherein, the speech source tracing and evidence collection model is determined by simulating adversarial training by adding a channel attention mechanism and a gradient inversion network layer to a neural network model; a second processing module for processing the high-level audio features after convolution processing based on the channel attention mechanism and fully connected network layer of the speech source tracing and evidence collection model, determining the target high-level features, calculating the probability distribution of each source tracing category corresponding to the target high-level features based on an activation function, and predicting the speech source tracing and evidence collection result of the speech to be detected; wherein, source tracing and evidence collection refers to verifying the source or generation method of the generated speech. By incorporating a channel attention mechanism into the speech tracing and evidence collection model, the model's ability to learn fine-grained audio features is improved. During training, a gradient inversion network layer is introduced to enhance the robustness of the speech tracing and evidence collection model, enabling the model to quickly and accurately determine the speech tracing and evidence collection results corresponding to the speech.

[0153] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0154] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the voice source tracing and evidence collection method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0155] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the voice source tracing and evidence collection method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for voice source tracing and evidence collection, characterized in that, The voice source tracing and evidence collection method includes: The speech to be detected is acquired, and the speech is processed based on an audio bandpass filter to determine the effective audio of the speech to be detected. The valid audio is input into a pre-trained speech tracing and evidence collection model to extract the high-level audio features of the valid audio. The high-level audio features are then convolved based on the convolutional pooling network layer of the speech tracing and evidence collection model to determine the high-level audio features after convolution. The speech tracing and evidence collection model is determined by simulating adversarial training by adding a channel attention mechanism and a gradient inversion network layer to the neural network model. Based on the channel attention mechanism and the fully connected network layer of the speech tracing and evidence collection model, the high-level features of the audio after convolution are processed to determine the target high-level features. Based on the activation function, the probability distribution of each tracing category corresponding to the target high-level features is calculated to predict the speech tracing and evidence collection result of the speech to be detected. Here, tracing and evidence collection refers to verifying the source or method of generating the speech. The voice source tracing and evidence collection model is trained through the following steps: Sample audio data is selected from the augmented training data, and the sample audio data is filtered to determine the valid audio samples. The valid audio samples are input into a neural network model. High-level features of the valid audio samples are extracted based on self-supervised pre-training. Adversarial training is then performed on the high-level features of the audio samples until the neural network model converges, at which point the training of the neural network model is stopped. The parameters of each layer of the neural network model with the best performance during the model training process are averaged, and the voice source tracing and evidence collection model is constructed based on the averaged parameters of each layer. The process of performing convolutional pooling and channel attention processing on the high-level features of the sample audio to determine the high-level features of the sample audio after channel attention processing includes: The high-level features of the sample audio are subjected to global max pooling and global average pooling in the spatial dimension to determine the global average pooling features and max pooling features. The global average pooling feature and the max pooling feature are respectively input into the multilayer perceptron network layer for channel dimension feature learning, and the global average pooling feature and the max pooling feature processed by the multilayer perceptron network layer are output. The global average pooling feature processed by the multilayer perceptron network layer and the max pooling feature processed by the multilayer perceptron network layer are concatenated to determine the concatenated feature. The concatenated features are mapped based on the activation function to determine the channel attention weight matrix; Based on the channel attention weight matrix, the global average pooling feature, and the max pooling feature, the high-level features of the sample audio after processing by the channel attention mechanism are determined.

2. The voice source tracing and evidence collection method according to claim 1, characterized in that, The channel attention mechanism and fully connected network layer of the speech tracing and evidence collection model are used to process the high-level audio features after convolution to determine the target high-level features. Based on the activation function, the probability distribution of each tracing category corresponding to the target high-level features is calculated to predict the speech tracing and evidence collection result of the speech to be detected, including: The high-level audio features after convolution are subjected to attention processing based on the channel attention mechanism to determine the high-level audio features after attention processing. The high-level audio features after attention processing are processed based on the fully connected network layer to determine the target high-level features; The probability distribution of each source tracing category corresponding to the target high-level feature is calculated based on the activation function, and multiple probability values ​​are determined. The source tracing category corresponding to the largest probability value is determined as the speech source tracing and evidence collection result of the speech to be detected.

3. The voice source tracing and evidence collection method according to claim 2, characterized in that, The augmented training data was determined using the following method: Add noise with different signal-to-noise ratios to the training data to determine the training data after data augmentation; Alternatively, gradient-based adversarial example generation algorithms can generate different adversarial examples to determine the augmented training data.

4. The voice source tracing and evidence collection method according to claim 1, characterized in that, The step of inputting the valid audio samples into a neural network model, extracting high-level features of the valid audio samples based on self-supervised pre-training, and performing adversarial training on the high-level features until the neural network model converges and training of the neural network model is stopped includes: The high-level features of the sample audio are subjected to convolutional pooling and channel attention mechanisms to determine the high-level features of the sample audio after channel attention mechanism processing. The sample audio high-level features after the channel attention mechanism are processed based on the first fully connected network layer, and the first sample audio high-level features and the predicted source tag corresponding to the first sample audio high-level features are output. A gradient inversion network layer is added after the first fully connected network layer. The high-level features of the first sample audio are input into the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are input into the second fully connected network layer after the gradient inversion network layer for processing, and the processed high-level features of the first sample audio are output. The processed high-level features of the first sample audio are input into the third fully connected network layer after the second fully connected network layer to output the high-level features of the second sample audio and the adversarial label of the high-level features of the second sample audio. Based on the high-level features of the first sample audio, the high-level features of the second sample audio, the predicted source label, the adversarial label, and the true source label of the sample audio data, the total loss value of the neural network model is determined, and training of the neural network model is stopped when the neural network model converges.

5. The voice source tracing and evidence collection method according to claim 4, characterized in that, The step of determining the total loss value of the neural network model based on the high-level features of the first sample audio, the high-level features of the second sample audio, the predicted source label, the adversarial label, and the true source label of the sample audio data, and stopping the training of the neural network model when the neural network model converges, includes: The classification loss value is determined by calculating the high-level features of the first sample audio, the predicted source label, and the real source label based on the weighted cross-entropy loss function. The adversarial loss value is determined by calculating the high-level features of the second sample audio and the adversarial label based on the binary cross-entropy loss function. Based on the classification loss value and the adversarial loss value, the total loss value of the neural network model is determined, and training of the neural network model is stopped when the neural network model converges.

6. A voice-based evidence tracing device, characterized in that, The voice source tracing and evidence collection device includes: The filtering module is used to acquire the speech to be detected, process the speech to be detected based on the audio bandpass filter, and determine the effective audio of the speech to be detected. The first processing module is used to input the valid audio into a pre-trained speech tracing and evidence collection model, extract the high-level audio features of the valid audio, and perform convolution processing on the high-level audio features based on the convolutional pooling network layer of the speech tracing and evidence collection model to determine the high-level audio features after convolution processing; wherein, the speech tracing and evidence collection model is determined by adding a channel attention mechanism and a gradient inversion network layer to the neural network model to simulate adversarial training; The second processing module is used to process the high-level audio features after convolution based on the channel attention mechanism and fully connected network layer of the speech source tracing and evidence collection model, determine the target high-level features, calculate the probability distribution of each source tracing category corresponding to the target high-level features based on the activation function, and predict the speech source tracing and evidence collection result of the speech to be detected; wherein, source tracing and evidence collection refers to verifying the source or method of generating the speech. The voice source tracing and evidence collection device also includes a model training module, which trains the voice source tracing and evidence collection model through the following steps: Sample audio data is selected from the augmented training data, and the sample audio data is filtered to determine the valid audio samples. The valid audio samples are input into a neural network model. High-level features of the valid audio samples are extracted based on self-supervised pre-training. Adversarial training is then performed on the high-level features of the audio samples until the neural network model converges, at which point the training of the neural network model is stopped. The parameters of each layer of the neural network model with the best performance during the model training process are averaged, and the voice source tracing and evidence collection model is constructed based on the averaged parameters of each layer. The model training module is used to perform convolutional pooling and channel attention processing on the high-level features of the sample audio, and to determine the high-level features of the sample audio after channel attention processing. The high-level features of the sample audio are subjected to global max pooling and global average pooling in the spatial dimension to determine the global average pooling features and max pooling features. The global average pooling feature and the max pooling feature are respectively input into the multilayer perceptron network layer for channel dimension feature learning, and the global average pooling feature and the max pooling feature processed by the multilayer perceptron network layer are output. The global average pooling feature processed by the multilayer perceptron network layer and the max pooling feature processed by the multilayer perceptron network layer are concatenated to determine the concatenated feature. The concatenated features are mapped based on the activation function to determine the channel attention weight matrix; Based on the channel attention weight matrix, the global average pooling feature, and the max pooling feature, the high-level features of the sample audio after processing by the channel attention mechanism are determined.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the voice tracing and evidence collection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the voice tracing and evidence collection method as described in any one of claims 1 to 5.

Images