Audio authenticity identification method and device, and storage medium

Abstract

The application discloses an audio authenticity identification method and device and a storage medium. It relates to the field of artificial intelligence. The method comprises the following steps: obtaining input audio to be identified, and inputting the input audio into a target voiceprint identifier. The target voiceprint identifier comprises an audio processing network, a voiceprint extraction network and a true-false discrimination network connected in sequence. The target voiceprint identifier is obtained through an adversarial training strategy based on multiple initial speech generators. The input audio is subjected to feature extraction through the audio processing network to obtain an audio feature sequence. The audio feature sequence is subjected to multi-layer feature mapping and attention mechanism processing through the voiceprint extraction network to obtain target speech features. The true-false discrimination network outputs an identification result representing the authenticity of the input audio based on the target speech features. Through the application, the problem that it is difficult to effectively distinguish between real audio and fake audio in the related art, resulting in low audio identification accuracy, is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and storage medium for identifying the authenticity of audio files. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning-based speech synthesis and cloning technologies have matured, significantly reducing forgery costs and improving realism. Attackers can use illegally obtained customer voice samples to generate highly realistic forged voiceprints, thereby remotely bypassing the voice authentication systems of institutions such as banks (e.g., telephone banking, voice transfers), leading to a sharp increase in financial fraud risks. Existing voiceprint identification technologies mainly rely on traditional manual feature extraction combined with probabilistic modeling or early deep neural network models. However, these traditional technologies have significant limitations. On the one hand, the manual acoustic features they rely on are difficult to capture fine-grained physiological dynamics across channels and devices, resulting in insufficient model generalization ability and a lack of robustness against novel forged samples outside the training set. On the other hand, existing identification models are usually trained for a single type of generation method or a specific dataset, lacking adversarial learning mechanisms for various generator structures, making it difficult to effectively distinguish the complex physiological characteristics of human voices from the static acoustic artifacts of AI-synthesized speech. Therefore, when faced with diverse forged audio synthesized based on advanced generative AI technology, the existing technologies have low identification accuracy and cannot provide reliable security defenses.

[0003] There is currently no effective solution to the problem that related technologies have difficulty in effectively distinguishing between real and fake audio, resulting in low accuracy in audio identification. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, and storage medium for identifying the authenticity of audio, in order to solve the problem in related technologies that it is difficult to effectively distinguish between real and fake audio, resulting in low accuracy in audio identification.

[0005] To achieve the above objectives, according to one aspect of this application, an audio authenticity verification method is provided. The method includes: acquiring input audio to be verified and inputting the input audio to a target voiceprint discriminator, wherein the target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and an authenticity discrimination network connected in sequence, and the target voiceprint discriminator is trained using an adversarial training strategy based on multiple initial speech generators; extracting features from the input audio through the audio processing network to obtain an audio feature sequence; processing the audio feature sequence through multi-layer feature mapping and attention mechanisms by the voiceprint extraction network to obtain target speech features; and outputting an authentication result representing the authenticity of the input audio based on the target speech features through the authenticity discrimination network.

[0006] Optionally, the training steps of the target voiceprint discriminator include: acquiring real audio samples and corresponding text data; generating multiple first forged audio samples based on the text data using multiple initial speech generators; training the initial voiceprint discriminator based on the real audio samples and the multiple first forged audio samples to obtain the first voiceprint discriminator; jointly training the first voiceprint discriminator and the multiple initial speech generators using an alternating optimization strategy, alternately updating the parameters of the first voiceprint discriminator and the multiple initial speech generators according to a preset ratio until a stopping condition is reached to obtain a second voiceprint discriminator and multiple target speech generators; generating second forged audio samples based on the text data using multiple target speech generators; adding noise to the real audio samples to obtain noise-added real audio samples; adding noise to the second forged audio samples to obtain noise-added second forged audio samples; training the second voiceprint discriminator based on the noise-added real audio samples and the noise-added second forged audio samples to obtain the target voiceprint discriminator.

[0007] Optionally, generating multiple first forged audio samples from text data using multiple initial speech generators, and training an initial voiceprint discriminator based on real audio samples and multiple first forged audio samples to obtain the first voiceprint discriminator includes: generating multiple first forged audio samples from text data using multiple initial speech generators; inputting real audio samples and multiple first forged audio samples into the initial voiceprint discriminator; the initial voiceprint discriminator outputting a first prediction vector based on real audio samples and a second prediction vector corresponding to each first forged audio sample; calculating a first loss value based on the real sample label of the real audio sample and the first prediction vector; calculating a second loss value based on the fake sample label of each first forged audio sample and the second prediction vector corresponding to each first forged audio sample; adding the first loss value and the second loss value to obtain a first total loss value; and adjusting the parameters of the initial voiceprint discriminator based on the first total loss value to obtain the first voiceprint discriminator.

[0008] Optionally, with a preset ratio of N:1, an alternating optimization strategy is used to jointly train the first voiceprint discriminator and multiple initial speech generators. The parameters of the first voiceprint discriminator and the multiple initial speech generators are alternately updated according to the preset ratio until a stopping condition is reached, resulting in a second voiceprint discriminator and multiple target speech generators. This includes: alternately updating the parameters of the first voiceprint discriminator and the multiple initial speech generators according to the preset ratio; fixing the parameters of the multiple initial speech generators; updating the parameters of the first voiceprint discriminator N times to obtain an updated first voiceprint discriminator; fixing the updated parameters of the first voiceprint discriminator; updating the parameters of each initial speech generator once to obtain an updated speech generator corresponding to each initial speech generator, until a stopping condition is reached, resulting in a second voiceprint discriminator and multiple target speech generators.

[0009] Optionally, fixing the parameters of multiple initial speech generators and updating the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator includes: inputting real audio samples and multiple first fake audio samples into the first voiceprint discriminator; the first voiceprint discriminator outputs a third prediction vector based on the real audio samples and outputs a fourth prediction vector corresponding to each first fake audio sample; calculating a third loss value based on the real sample label of the real audio sample and the third prediction vector; calculating a fourth loss value based on the fake sample label of each first fake audio sample and the fourth prediction vector corresponding to each first fake audio sample; adding the third loss value and the fourth loss value to obtain a second total loss value; adjusting the parameters of the first voiceprint discriminator based on the second total loss value; repeating the step of adjusting the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator.

[0010] Optionally, fixing the parameters of the updated first voiceprint discriminator and updating the parameters of each initial speech generator once to obtain the updated speech generator corresponding to each initial speech generator includes: inputting real audio samples and multiple first fake audio samples into the updated first voiceprint discriminator; the updated first voiceprint discriminator outputs a fifth prediction vector based on the real audio samples and outputs a sixth prediction vector corresponding to each first fake audio sample; calculating a fifth loss value based on the real sample label of the real audio sample and the fifth prediction vector; calculating a sixth loss value based on the fake sample label of each first fake audio sample and the sixth prediction vector corresponding to each first fake audio sample; adding the fifth loss value and the sixth loss value to obtain a third total loss value; and adjusting the parameters of each initial speech generator based on the third total loss value to obtain the updated speech generator corresponding to each initial speech generator.

[0011] Optionally, the audio feature sequence is processed through a multi-layer feature mapping and attention mechanism via a speaker extraction network to obtain target speech features, including: mapping the audio feature sequence using the first linear layer of the speaker extraction network to obtain a query vector matrix; mapping the audio feature sequence using the second linear layer to obtain a key vector matrix; transposing the key vector matrix to obtain a transposed vector matrix; performing matrix multiplication between the transposed vector matrix and the query vector matrix to obtain an attention score matrix; compressing and normalizing the attention score matrix to obtain an attention weight matrix; and performing matrix multiplication between the attention weight matrix and the audio feature sequence. The attention aggregation feature is obtained through matrix multiplication. A third linear layer is used to linearly map the attention aggregation feature to obtain the first intermediate feature. The first intermediate feature is added to the audio feature sequence to obtain the second intermediate feature. A fourth linear layer is used to map the second intermediate feature to obtain the third intermediate feature. A non-linear activation function is applied to the third intermediate feature to obtain the activation feature. The activation feature is multiplied element-wise with the audio feature sequence to obtain the fourth intermediate feature. A fifth linear layer is used to map the fourth intermediate feature to obtain the fifth intermediate feature. The fifth intermediate feature is added to the audio feature sequence to obtain the target speech feature.

[0012] To achieve the above objectives, according to another aspect of this application, an audio authenticity verification device is provided. The device includes: a first acquisition unit, configured to acquire input audio to be verified and input the input audio to a target voiceprint discriminator, wherein the target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and an authenticity discrimination network connected in sequence, and the target voiceprint discriminator is trained using an adversarial training strategy based on multiple initial speech generators; an extraction unit, configured to extract features from the input audio through the audio processing network to obtain an audio feature sequence; a processing unit, configured to perform multi-layer feature mapping and attention mechanism processing on the audio feature sequence through the voiceprint extraction network to obtain target speech features; and an output unit, configured to output an authentication result representing the authenticity of the input audio based on the target speech features through the authenticity discrimination network.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, which includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any audio authenticity verification method.

[0014] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any audio authenticity verification method.

[0015] According to another aspect of this application, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the steps of the audio authenticity identification method described above.

[0016] In this embodiment, the input audio to be identified is acquired and fed into a target voiceprint discriminator. The target voiceprint discriminator comprises an audio processing network, a voiceprint extraction network, and a authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained using an adversarial training strategy based on multiple initial speech generators. The audio processing network extracts features from the input audio to obtain an audio feature sequence. The voiceprint extraction network processes the audio feature sequence using multi-layer feature mapping and attention mechanisms to obtain target speech features. The authenticity discrimination network outputs a discrimination result representing the authenticity of the input audio based on the target speech features. This solves the technical problem of difficulty in effectively distinguishing between real and fake audio, resulting in low accuracy in audio identification.

[0017] In this application, the input audio to be identified is processed by a target voiceprint discriminator. During the training phase, the target voiceprint discriminator uses multiple speech generators with different structures to generate diverse forgery samples, which improves the ability to identify forged audio in adversarial optimization. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and a authenticity discrimination network connected in sequence. First, the audio processing network converts the original input audio into an audio feature sequence. Then, the voiceprint extraction network performs multi-layer feature mapping and attention mechanism processing on the audio feature sequence to extract a deep voiceprint representation with high discriminativeness, thereby obtaining the target speech features. Finally, the authenticity discrimination network outputs the identification result based on the target speech features, thus achieving the technical effect of improving the accuracy of audio identification. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A hardware block diagram of a computer terminal for implementing an audio authenticity verification method is shown.

[0020] Figure 2 This is a flowchart of an audio authenticity verification method provided according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an audio authenticity verification method provided according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of an audio authenticity verification device provided according to an embodiment of this application;

[0023] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should be noted that all relevant information (including at least one of the following audio information, user personal information, etc.) and data (including at least one of the following data used for display, data used for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it needs to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0027] Example 1

[0028] According to an embodiment of this application, a method embodiment for identifying the authenticity of audio is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1A hardware block diagram of a computer terminal (or mobile device) for implementing an audio authenticity verification method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include at least one of the following: a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the audio authenticity identification method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned audio authenticity identification method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include at least one of the following: the Internet, corporate intranet, local area network, mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0033] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0034] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for identifying the authenticity of audio files is shown. Figure 2 This is a flowchart of the audio authenticity identification method according to Embodiment 1 of this application.

[0035] Step S201: Obtain the input audio to be identified and input the input audio to the target voiceprint discriminator. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and a authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained by an adversarial training strategy based on multiple initial speech generators.

[0036] Optionally, the input audio to be authenticated can originate from offline transaction scenarios, electronic communication scenarios, or audio / video recording scenarios. Before inputting the input audio to the target voiceprint discriminator, zero-padding can be performed on the input audio, and the duration of the input audio can be standardized to a preset duration (e.g., 30 seconds) to adapt to subsequent fixed-dimensional network input requirements.

[0037] Optionally, the training of the target voiceprint discriminator may include three stages. The first stage is the initial discrimination stage, in which the initial voiceprint discriminator is routinely optimized. The second stage is the adversarial joint training stage, in which the optimization suppression of the speech generator is removed, and an alternating training strategy is adopted to alternately update the parameters of the voiceprint discriminator and the speech generator according to a preset ratio (e.g., the ratio of training rounds of the discriminator to the generator is 3:1) until the accuracy of the voiceprint discriminator is stable. The third stage is the anti-interference training stage, in which noise interference is applied to real audio and fake audio, and only the voiceprint discriminator is optimized to improve the robustness of the voiceprint discriminator in complex channel environments.

[0038] Step S202: Extract features from the input audio using an audio processing network to obtain an audio feature sequence.

[0039] Optionally, a short-time Fourier transform can be performed on the input audio to obtain a spectrum graph with a linear frequency scale. Then, the frequency scale of the spectrum graph is converted to a Mel frequency scale through a Mel filter bank, and the energy value is logarithmically processed to obtain a Mel spectrogram to match the human ear's perception of audio. The Mel spectrogram is then input into a small convolutional neural network, and the frequency dimension is expanded and the time dimension is compressed through convolution operations. Finally, the processed features are mapped to an audio feature sequence with dimensions [1500, 512], where 1500 represents the time step and 512 represents the feature dimension.

[0040] Step S203: The audio feature sequence is processed by multi-layer feature mapping and attention mechanism through a voiceprint extraction network to obtain the target speech features.

[0041] Optionally, the voiceprint extraction network consists of multiple stacked feature extraction sub-networks (e.g., 8 layers). Two linear network layers can be used to map the audio feature sequence to obtain a query vector and a key vector, respectively. The key vector is transposed and then multiplied with the query vector. After scaling and softmax normalization, an attention weight matrix is ​​obtained. Then, the attention weight matrix is ​​multiplied with the input features to obtain the attention aggregation feature. After linear mapping, the attention aggregation feature is added to the input features to obtain the intermediate feature. Then, the intermediate feature is linearly mapped and processed by a non-linear activation function to obtain the activation feature. Then, the activation feature is multiplied element-wise with the input features to obtain the gated feature. Finally, the gated feature is linearly mapped and added to the input features again to obtain the target speech feature.

[0042] Step S204: The authenticity discrimination network outputs the identification result representing the authenticity of the input audio based on the target speech features.

[0043] Optionally, the first unit vector can be selected as the global feature representation from the target speech feature sequence output by the speaker extraction network. The first unit vector aggregates the global context information of the entire audio segment and the unique speaker features at the beginning stage. Then, a linear network layer can be used to compress the first unit vector into a classification vector of length 2. Softmax normalization is performed on the classification vector to obtain the true probability distribution. It is then determined whether the value of the corresponding true category in the normalized vector is greater than a preset threshold (e.g., 0.75). If it is greater, the input audio is determined to be real audio; otherwise, the input audio is determined to be fake audio.

[0044] The audio authenticity identification method provided in this application's embodiments processes the input audio to be identified through a target voiceprint discriminator. During the training phase, the target voiceprint discriminator utilizes multiple speech generators with different structures to generate diverse forged samples, improving the ability to identify forged audio through adversarial optimization. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and an authenticity discrimination network connected in sequence. First, the audio processing network converts the original input audio into an audio feature sequence. Then, the voiceprint extraction network performs multi-layer feature mapping and attention mechanism processing on the audio feature sequence to extract a deep voiceprint representation with high discriminative power, obtaining the target speech features. Finally, the authenticity discrimination network outputs the identification result based on the target speech features. This solves the technical problem of difficulty in effectively distinguishing between real and forged audio, resulting in low accuracy in audio identification, and achieves the technical effect of improving the accuracy of audio identification.

[0045] To improve the accuracy of target voiceprint identification in identifying input audio, optionally, in the audio authenticity identification method provided in this application embodiment, the training steps of the target voiceprint discriminator include:

[0046] The first step is to obtain real audio samples and the corresponding text data.

[0047] Optionally, the real audio samples can be collected from bank voice transaction scenarios or public voiceprint datasets, and the text data is a text sequence describing the content corresponding to the real audio samples.

[0048] The second step involves generating multiple first-forged audio samples from text data using multiple initial speech generators, and training an initial voiceprint discriminator based on the real audio samples and the multiple first-forged audio samples to obtain the first voiceprint discriminator.

[0049] Optionally, the multiple initial speech generators are generators with different network architectures, such as a text-to-speech model based on the Transformer architecture, a speech cloning model based on generative adversarial networks, or a speech generation model based on a diffusion model. By using multiple initial speech generators, multiple first forged audio samples can be generated based on text data, which can cover a variety of mainstream AI speech forgery technologies.

[0050] Optionally, the loss function used to train the initial voiceprint discriminator can be a binary classification cross-entropy loss, where the label of the real audio sample is 1 and the label of the first fake audio sample is 0. During the training of the initial voiceprint discriminator, by minimizing the loss function, the system learns to distinguish the differences between real audio and fake audio generated by the speech generator at the current stage, thus obtaining the first voiceprint discriminator.

[0051] The third step involves using an alternating optimization strategy to jointly train the first voiceprint discriminator and multiple initial speech generators. The parameters of the first voiceprint discriminator and multiple initial speech generators are updated alternately according to a preset ratio until the stopping condition is met, resulting in a second voiceprint discriminator and multiple target speech generators.

[0052] Optionally, the preset ratio is the ratio of training steps between the voiceprint discriminator and the speech generator, for example, 3:1, meaning that for every 3 batches trained on the first voiceprint discriminator, all initial speech generators are jointly trained in 1 batch. This alternating training strategy helps maintain a balance in capabilities between the voiceprint discriminator and the speech generator, preventing the voiceprint discriminator from becoming too powerful and causing the speech generator to fail, or the speech generator from becoming too powerful and causing the voiceprint discriminator to fail, thus promoting the co-evolution of both. Stopping conditions can include the discriminator's accuracy on the validation set no longer significantly improving, or reaching the preset maximum number of training rounds, etc.

[0053] The fourth step involves generating a second fake audio sample based on text data using multiple target speech generators. Noise is added to the real audio sample to obtain a real audio sample with added noise. Noise is added to the second fake audio sample to obtain a second fake audio sample with added noise. A second voiceprint discriminator is trained based on the real audio sample with added noise and the second fake audio sample with added noise to obtain the target voiceprint discriminator.

[0054] Optionally, the applied noise may include at least one of white noise, background noise, wind noise, and reverberation noise. The type of noise, signal-to-noise ratio, or reverberation time can be randomly generated to simulate complex communication channel environments in the real world. By updating only the parameters of the second voiceprint discriminator while fixing the parameters of multiple target speech generators, and training the second voiceprint discriminator on samples containing various noise interferences, the ability to combat channel noise and background interference in practical application scenarios can be improved.

[0055] In summary, through the above steps, multi-stage adversarial training by integrating various speech generators with different structures, and combined with anti-interference training, the target voiceprint discriminator's ability to identify forged audio has been improved.

[0056] To improve the ability of the initial voiceprint discriminator to identify forged audio, optionally, in the audio authenticity verification method provided in this application embodiment, multiple initial speech generators generate multiple first forged audio samples based on text data, and the initial voiceprint discriminator is trained based on real audio samples and multiple first forged audio samples to obtain the first voiceprint discriminator, including:

[0057] The first step involves generating multiple initial speech generators based on text data to produce multiple first-stage fake audio samples.

[0058] Optionally, multiple initial speech generators include generators with various network structures, which can generate multiple first fake audio samples with different acoustic artifact features based on text data in the early stages of training, thereby enriching the diversity of training data.

[0059] The second step involves inputting real audio samples and multiple first fake audio samples into the initial voiceprint discriminator. The initial voiceprint discriminator outputs a first prediction vector based on the real audio samples and a second prediction vector corresponding to each first fake audio sample.

[0060] Optionally, the initial voiceprint discriminator first converts the audio waveform into an audio feature sequence through an audio processing network, then extracts deep voiceprint features through a voiceprint extraction network, and finally outputs a classification vector of length 2 (i.e., the first prediction vector and the second prediction vector) through a authenticity discrimination network, representing the probability distribution of the input audio belonging to the real category and the fake category.

[0061] The third step is to calculate the first loss value based on the real sample label of the real audio sample and the first prediction vector, and to calculate the second loss value based on the fake sample label of each first fake audio sample and the second prediction vector corresponding to each first fake audio sample. The first loss value and the second loss value are added together to obtain the first total loss value.

[0062] Optionally, the first and second loss values ​​can be calculated using binary cross-entropy loss. The first loss value measures the error of the voiceprint discriminator in identifying genuine audio, while the second loss value measures the error of the voiceprint discriminator in identifying forged audio. When calculating the second loss value, a suppression coefficient (e.g., 0.1) can be introduced. By introducing a suppression coefficient less than 1, the influence of the speech generator on the gradient of the voiceprint discriminator in the early stages of training can be reduced, preventing the insufficiently trained voiceprint discriminator from generating incorrect gradient feedback to the speech generator.

[0063] The fourth step is to adjust the parameters of the initial voiceprint discriminator based on the first total loss value to obtain the first voiceprint discriminator.

[0064] Optionally, the parameters of multiple initial speech generators are kept fixed, and only the parameters of the initial voiceprint discriminator are updated. This can be done by using stochastic gradient descent or adaptive moment estimation optimizer. The parameters of all network layers (including audio processing network, voiceprint extraction network and authenticity discrimination network) of the initial voiceprint discriminator are updated by backpropagation based on the first total loss value to obtain the first voiceprint discriminator.

[0065] In summary, by fixing the parameters of multiple initial speech generators and training an initial voiceprint discriminator based on real audio samples and multiple first-forged audio samples, a first voiceprint discriminator is obtained. This avoids the negative interference of an immature initial speech generator on the initial voiceprint discriminator model, and the obtained first voiceprint discriminator has a higher discrimination accuracy than the initial voiceprint discriminator.

[0066] To improve the performance of the voiceprint discriminator and the speech generator through adversarial game during training, optionally, in the audio authenticity identification method provided in this application embodiment, the preset ratio is N:1, and an alternating optimization strategy is used to jointly train the first voiceprint discriminator and multiple initial speech generators. The parameters of the first voiceprint discriminator and the multiple initial speech generators are alternately updated according to the preset ratio until a stopping condition is reached, resulting in a second voiceprint discriminator and multiple target speech generators. This includes: alternately updating the parameters of the first voiceprint discriminator and the multiple initial speech generators according to the preset ratio; fixing the parameters of the multiple initial speech generators; updating the parameters of the first voiceprint discriminator N times to obtain an updated first voiceprint discriminator; fixing the updated parameters of the first voiceprint discriminator; updating the parameters of each initial speech generator once to obtain an updated speech generator corresponding to each initial speech generator, until a stopping condition is reached, resulting in a second voiceprint discriminator and multiple target speech generators.

[0067] Optionally, to maximize the ability of the obtained second voiceprint discriminator to distinguish between real and fake samples, the parameters of all initial speech synthesis generators are kept unchanged when updating the parameters of the first voiceprint discriminator N times. To optimize the realism of the fake audio generated by the speech generator, the parameters of the updated first voiceprint discriminator are kept unchanged when updating the parameters of the initial speech synthesis generator once. The preset ratio N:1 has N as an integer greater than 1, for example, N=3.

[0068] In summary, by using an alternating optimization strategy based on a preset ratio, a dynamic balance was achieved between the first voiceprint discriminator and the initial speech generator during the adversarial process. The first voiceprint discriminator learns to distinguish between genuine and fake audio through training, while the initial speech generator generates highly realistic fake audio through training. This results in a second voiceprint discriminator with higher accuracy and a target speech generator that generates more realistic fake audio.

[0069] To fully improve the discrimination capability of the first voiceprint discriminator in each alternating training cycle, optionally, in the audio authenticity identification method provided in this application embodiment, the parameters of multiple initial speech generators are fixed, and the parameters of the first voiceprint discriminator are updated N times to obtain the updated first voiceprint discriminator, including:

[0070] The first step involves inputting real audio samples and multiple first fake audio samples into a first voiceprint discriminator. The first voiceprint discriminator outputs a third prediction vector based on the real audio samples and a fourth prediction vector corresponding to each first fake audio sample.

[0071] Optionally, during the input process, real audio samples and multiple first-stage fake audio samples can be input simultaneously in a batch, or they can be input independently in sequence. The third and fourth prediction vectors output by the first voiceprint discriminator are probability distribution vectors of length 2, representing the confidence scores of the input sample belonging to the real and fake categories, respectively. The higher the value of the real category in the third prediction vector, the more confident the first voiceprint discriminator is that the audio is real; the higher the value of the fake category in the fourth prediction vector, the more confident the first voiceprint discriminator is that the audio is fake.

[0072] The second step is to calculate the third loss value based on the real sample label of the real audio sample and the third prediction vector, and to calculate the fourth loss value based on the fake sample label of each first fake audio sample and the fourth prediction vector corresponding to each first fake audio sample. The third loss value and the fourth loss value are added together to obtain the second total loss value. The parameters of the first voiceprint discriminator are adjusted based on the second total loss value.

[0073] Optionally, the third and fourth loss values ​​can be calculated using binary cross-entropy loss. The second total loss is the sum of all the third and fourth loss values. The gradient of the second total loss value relative to the weights of each layer of the first voiceprint discriminator can be calculated using the backpropagation algorithm, and the optimizer can be used to update the weights and biases of the first voiceprint discriminator according to the gradient to minimize the total loss value.

[0074] The third step is to repeat the process of adjusting the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator.

[0075] Optionally, during each repetition, a new batch of real audio samples and fake audio samples generated by the same set of speech generators can be resampled to prevent overfitting to a single batch of samples. The purpose of repeating the adjustment N times is to allow the first voiceprint discriminator to converge to a more robust local optimum while keeping the initial speech generator parameters unchanged, thus providing more accurate discrimination results for subsequent optimization of the initial speech generator.

[0076] In summary, by fixing the parameters of the initial speech generator and repeatedly updating the parameters of the first voiceprint discriminator N times through the above steps, the first voiceprint discriminator achieves rapid convergence under the current distribution of forged samples, and the updated first voiceprint discriminator has a stronger discrimination ability.

[0077] To improve the ability of the initial speech synthesis generator to generate highly realistic forged audio, optionally, in the audio authenticity verification method provided in this application embodiment, the parameters of the updated first voiceprint discriminator are fixed, and the parameters of each initial speech generator are updated once to obtain the updated speech generator corresponding to each initial speech generator, including:

[0078] The first step is to input the real audio sample and multiple first fake audio samples into the updated first voiceprint discriminator. The updated first voiceprint discriminator outputs a fifth prediction vector based on the real audio sample and a sixth prediction vector corresponding to each first fake audio sample.

[0079] Optionally, real audio samples and multiple first fake audio samples are input into the updated first voiceprint discriminator. In the output sixth prediction vector, the higher the probability value corresponding to the real category, the more realistic the fake sample is and the easier it is to deceive the voiceprint discriminator.

[0080] The second step is to calculate the fifth loss value based on the real sample label of the real audio sample and the fifth prediction vector, and to calculate the sixth loss value based on the fake sample label of each first fake audio sample and the sixth prediction vector corresponding to each first fake audio sample. The fifth loss value and the sixth loss value are added together to obtain the third total loss value. The parameters of each initial speech generator are adjusted based on the third total loss value to obtain the updated speech generator corresponding to each initial speech generator.

[0081] Optionally, the fifth and sixth loss values ​​can be calculated using binary cross-entropy loss. The third total loss is the sum of all the fifth and sixth loss values. Since each initial speech generator has a different structure, it can be optimized in parallel or serially according to the loss function of each initial speech generator to obtain the updated speech generator corresponding to each initial speech generator, thereby enhancing the diversity of fake audio from multiple dimensions.

[0082] In summary, by taking the above steps and fixing the parameters of the first voiceprint discriminator, each initial speech generator is updated to obtain the updated speech generator. This enhances the diversity of forged audio from multiple dimensions, thereby promoting the co-evolution of the voiceprint discriminator and the speech generator in adversarial processes and improving the discrimination accuracy of the target voiceprint discriminator.

[0083] To effectively distinguish between genuine and fake audio, optionally, in the audio authenticity identification method provided in this application embodiment, a multi-layer feature mapping and attention mechanism is used to process the audio feature sequence through a voiceprint extraction network to obtain target speech features including:

[0084] The first step involves mapping the audio feature sequence using the first linear layer of the voiceprint extraction network to obtain the query vector matrix. The second linear layer is then used to map the audio feature sequence again to obtain the key vector matrix. The key vector matrix is ​​transposed to obtain the transposed vector matrix. This transposed vector matrix is ​​then multiplied with the query vector matrix to obtain the attention score matrix. The attention score matrix is ​​then compressed and normalized to obtain the attention weight matrix. Finally, the attention weight matrix is ​​multiplied with the audio feature sequence to obtain the attention aggregation feature.

[0085] Alternatively, two linear network layers can be used. and For audio feature sequences Perform separate mappings to obtain the query vector matrix. and key vector matrix :

[0086] , .

[0087] in, and The dimension is represented as [seq, ]( (768).

[0088] Next, regarding Transpose the vector to obtain the transposed vector matrix. and Perform matrix multiplication to obtain the attention score matrix, and then compress and sum the attention score matrix. Function normalization is performed to obtain the attention weight matrix. :

[0089]

[0090] Next, the attention weight matrix With audio feature sequences Perform matrix multiplication to obtain attention aggregation features.

[0091] The second step involves using a third linear layer to linearly map the attention aggregation features to obtain the first intermediate feature. The first intermediate feature is then added to the audio feature sequence to obtain the second intermediate feature.

[0092] Optionally, the third linear network layer is Second intermediate feature With audio feature sequences The dimensions are the same, the second intermediate feature for:

[0093]

[0094] The third step involves mapping the second intermediate feature using a fourth linear layer to obtain the third intermediate feature; applying a nonlinear activation function to the third intermediate feature to obtain the activation feature; and performing element-wise multiplication of the activation feature with the audio feature sequence to obtain the fourth intermediate feature.

[0095] Alternatively, a linear network layer can be used. For the second intermediate feature A mapping is performed to transform the second dimension into a higher dimension, and after processing with the non-linear function Swish, it is compared with the audio feature sequence. Element-wise multiplication yields the fourth intermediate feature. The dimension is represented as [seq, The fourth intermediate feature is:

[0096]

[0097] The formula for calculating the nonlinear function Swish is as follows:

[0098]

[0099] in, This indicates element-wise multiplication.

[0100] The fourth step involves mapping the fourth intermediate feature using the fifth linear layer to obtain the fifth intermediate feature; then adding the fifth intermediate feature to the audio feature sequence to obtain the target speech feature.

[0101] Alternatively, a linear network layer can be used. For the fourth intermediate feature Perform a mapping, mapping the second dimension back to the dim dimension, and then comparing it with the audio feature sequence. Add them together to obtain the target speech features. Linear network layer The weight is The matrix of target speech features for:

[0102]

[0103] In summary, by combining the above steps with a multi-layer feature processing flow that integrates self-attention mechanism, residual connection, high-dimensional feature mapping, and nonlinear activation, the local details and global contextual information in the audio feature sequence are fused and enhanced to obtain the target speech features, which helps to improve the accuracy of the target voiceprint discriminator in identifying forged audio.

[0104] Figure 3The diagram illustrates an audio authenticity verification method. First, the system receives the client's voice input and enters the audio processing stage. The waveform processing step in this stage first pads the original audio waveform with zeros to standardize its length to a preset duration. Then, a short-time Fourier transform is performed on the standardized waveform to obtain a linear frequency-scale spectrogram. Next, a Mel filter bank is used to convert the linear frequency scale to a Mel frequency scale, and logarithmic energy compression is performed to generate a Mel spectrogram that closely matches the characteristics of human hearing. Finally, the Mel spectrogram is input into a sequential convolutional network for processing. The convolutional network expands the frequency dimension and compresses the time dimension through convolution operations, extracting lightweight and efficient sequence features to obtain the audio feature sequence.

[0105] Next, the audio feature sequence enters the voiceprint extraction stage, which includes multiple (e.g., 8 layers) identical feature extraction sub-networks. Each feature extraction sub-network performs the following operations in sequence: First, a linear layer is used to map the input features into query vectors and key vectors, and a self-attention weight matrix is ​​calculated; the input features are weighted and aggregated using the attention weight matrix to obtain attention-aggregated features; the aggregated features are nonlinearly transformed and feature-selected through linear mapping, the Swish nonlinear activation function, and element-wise multiplication gating mechanism; finally, the transformed features are added to the original input through residual connections to output the target speech features refined by deep feature extraction.

[0106] Next, the authenticity verification stage begins. The authenticity discrimination network extracts the unit vector from the beginning of the target speech feature sequence as a global speaker representation. This vector is compressed into a classification vector of length 2 through a linear network layer, and a Softmax normalization network layer is used to calculate the probability distribution of this vector belonging to the real and fake categories. If the probability value of the real category is greater than a preset threshold (e.g., 0.75), it is determined to be real audio; otherwise, it is determined to be fake audio.

[0107] In summary, through audio processing, voiceprint extraction, and authenticity verification stages, the global dependencies and subtle physiological dynamics of voiceprints are effectively captured. Furthermore, the adversarial training strategy significantly enhances the model's generalization and identification capabilities for various AI-generated speech (such as speech generated by GANs and diffusion models). Thus, even in complex channel environments and under noise interference, it can still maintain high-precision voiceprint authenticity verification, providing a reliable security barrier for bank voice authentication systems.

[0108] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0109] Example 2

[0110] This application also provides an audio authenticity verification device. It should be noted that the audio authenticity verification device of this application can be used to execute the audio authenticity verification method provided in this application. The audio authenticity verification device provided in this application is described below.

[0111] According to embodiments of this application, an apparatus for implementing the above-described audio authenticity verification method is also provided, such as... Figure 4 As shown, the device includes: a first acquisition unit 401, an extraction unit 402, a processing unit 403, and an output unit 404.

[0112] The first acquisition unit 401 is used to acquire the input audio to be identified and input the input audio to the target voiceprint discriminator. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network and a authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained by an adversarial training strategy based on multiple initial speech generators.

[0113] Extraction unit 402 is used to extract features from input audio through an audio processing network to obtain an audio feature sequence;

[0114] Processing unit 403 is used to perform multi-layer feature mapping and attention mechanism processing on audio feature sequences through a speaker extraction network to obtain target speech features;

[0115] The output unit 404 is used to output the identification result representing the authenticity of the input audio based on the target speech features through the authenticity discrimination network.

[0116] The audio authenticity identification device provided in this application embodiment acquires the input audio to be identified through a first acquisition unit 401 and inputs the input audio to a target voiceprint discriminator. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and an authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained by an adversarial training strategy based on multiple initial speech generators. The extraction unit 402 extracts features from the input audio through the audio processing network to obtain an audio feature sequence. The processing unit 403 performs multi-layer feature mapping and attention mechanism processing on the audio feature sequence through the voiceprint extraction network to obtain target speech features. The output unit 404 outputs the identification result representing the authenticity of the input audio based on the target speech features through the authenticity discrimination network. This solves the problem in related technologies that it is difficult to effectively distinguish between real audio and fake audio, resulting in low accuracy in audio identification, and achieves the technical effect of improving the accuracy of audio identification.

[0117] Optionally, in the audio authenticity identification device provided in this application embodiment, the device further includes: a second acquisition unit, used to acquire real audio samples and text data corresponding to the real audio samples; a first training unit, used to generate multiple first forged audio samples based on the text data using multiple initial speech generators, and to train an initial voiceprint discriminator based on the real audio samples and the multiple first forged audio samples to obtain a first voiceprint discriminator; a second training unit, used to jointly train the first voiceprint discriminator and the multiple initial speech generators using an alternating optimization strategy, and to alternately update the parameters of the first voiceprint discriminator and the parameters of the multiple initial speech generators according to a preset ratio until a stopping condition is reached to obtain a second voiceprint discriminator and multiple target speech generators; a third training unit, used to generate second forged audio samples based on the text data using multiple target speech generators, add noise to the real audio samples to obtain real audio samples with added noise, add noise to the second forged audio samples to obtain second forged audio samples with added noise, and train a second voiceprint discriminator based on the real audio samples with added noise and the second forged audio samples with added noise to obtain a target voiceprint discriminator.

[0118] Optionally, in the audio authenticity identification device provided in this application embodiment, the first training unit includes: a generation module, used to generate multiple first forged audio samples based on text data through multiple initial speech generators; an input module, used to input real audio samples and multiple first forged audio samples into an initial voiceprint discriminator, wherein the initial voiceprint discriminator outputs a first prediction vector based on the real audio samples and outputs a second prediction vector corresponding to each first forged audio sample; a calculation module, used to calculate a first loss value based on the real sample label of the real audio sample and the first prediction vector, calculate a second loss value based on the fake sample label of each first forged audio sample and the second prediction vector corresponding to each first forged audio sample, and add the first loss value and the second loss value to obtain a first total loss value; and an adjustment module, used to adjust the parameters of the initial voiceprint discriminator based on the first total loss value to obtain a first voiceprint discriminator.

[0119] Optionally, in the audio authenticity identification device provided in this application embodiment, the second training unit includes: an update module, used to alternately update the parameters of the first voiceprint discriminator and the parameters of multiple initial speech generators according to a preset ratio of N:1, fix the parameters of multiple initial speech generators, update the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator, fix the updated parameters of the first voiceprint discriminator, update the parameters of each initial speech generator once to obtain the updated speech generator corresponding to each initial speech generator, until a stopping condition is reached, to obtain the second voiceprint discriminator and multiple target speech generators.

[0120] Optionally, in the audio authenticity identification device provided in this application embodiment, the update module includes: a first input submodule, used to input real audio samples and multiple first fake audio samples into a first voiceprint discriminator, the first voiceprint discriminator outputs a third prediction vector based on the real audio samples, and outputs a fourth prediction vector corresponding to each first fake audio sample based on each first fake audio sample; a calculation submodule, used to calculate a third loss value based on the real sample label of the real audio sample and the third prediction vector, calculate a fourth loss value based on the fake sample label of each first fake audio sample and the fourth prediction vector corresponding to each first fake audio sample, add the third loss value and the fourth loss value to obtain a second total loss value, and adjust the parameters of the first voiceprint discriminator based on the second total loss value; and a first adjustment submodule, used to repeatedly execute the step of adjusting the parameters of the first voiceprint discriminator N times to obtain an updated first voiceprint discriminator.

[0121] Optionally, in the audio authenticity identification device provided in this application embodiment, the update module includes: a second input submodule, used to input real audio samples and multiple first forged audio samples into an updated first voiceprint discriminator, the updated first voiceprint discriminator outputs a fifth prediction vector based on the real audio samples, and outputs a sixth prediction vector corresponding to each first forged audio sample based on each first forged audio sample; a second adjustment submodule, used to calculate a fifth loss value based on the real sample label of the real audio sample and the fifth prediction vector, calculate a sixth loss value based on the fake sample label of each first forged audio sample and the sixth prediction vector corresponding to each first forged audio sample, add the fifth loss value and the sixth loss value to obtain a third total loss value, and adjust the parameters of each initial speech generator based on the third total loss value to obtain an updated speech generator corresponding to each initial speech generator.

[0122] Optionally, in the audio authenticity verification device provided in this application embodiment, the processing unit 404 includes: a first mapping module, configured to map the audio feature sequence using a first linear layer in the speaker extraction network to obtain a query vector matrix; map the audio feature sequence using a second linear layer to obtain a key vector matrix; transpose the key vector matrix to obtain a transposed vector matrix; perform matrix multiplication between the transposed vector matrix and the query vector matrix to obtain an attention score matrix; compress and normalize the attention score matrix to obtain an attention weight matrix; and perform matrix multiplication between the attention weight matrix and the audio feature sequence to obtain attention aggregation features. The second mapping module is used to linearly map the attention aggregation features using a third linear layer to obtain the first intermediate feature, and then add the first intermediate feature to the audio feature sequence to obtain the second intermediate feature. The third mapping module is used to map the second intermediate feature using a fourth linear layer to obtain the third intermediate feature. A non-linear activation function is applied to the third intermediate feature to obtain the activation feature. The activation feature is then multiplied element-wise with the audio feature sequence to obtain the fourth intermediate feature. The fourth mapping module is used to map the fourth intermediate feature using a fifth linear layer to obtain the fifth intermediate feature. The fifth intermediate feature is then added to the audio feature sequence to obtain the target speech feature.

[0123] It should be noted that the first acquisition unit 401, extraction unit 402, processing unit 403, and output unit 404 mentioned above correspond to steps S201 to S204 in Embodiment 1. The four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0124] Example 3

[0125] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced with a mobile terminal or an electronic device, etc.

[0126] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0127] In this embodiment, the computer terminal described above can execute the program code for the following steps in the audio authenticity identification method: acquiring the input audio to be identified and inputting the input audio to the target voiceprint discriminator, wherein the target voiceprint discriminator includes an audio processing network, a voiceprint extraction network, and an authenticity discrimination network connected in sequence, and the target voiceprint discriminator is trained by an adversarial training strategy based on multiple initial speech generators; extracting features from the input audio through the audio processing network to obtain an audio feature sequence; processing the audio feature sequence through multi-layer feature mapping and attention mechanism through the voiceprint extraction network to obtain target speech features; and outputting an identification result representing the authenticity of the input audio based on the target speech features through the authenticity discrimination network.

[0128] Optionally, the aforementioned computer terminal can execute program code for the following steps in the audio authenticity verification method: The training steps of the target voiceprint discriminator include: acquiring real audio samples and corresponding text data; generating multiple first forged audio samples based on the text data using multiple initial speech generators; training the initial voiceprint discriminator based on the real audio samples and the multiple first forged audio samples to obtain the first voiceprint discriminator; jointly training the first voiceprint discriminator and the multiple initial speech generators using an alternating optimization strategy; alternately updating the parameters of the first voiceprint discriminator and the multiple initial speech generators according to a preset ratio until a stopping condition is reached to obtain a second voiceprint discriminator and multiple target speech generators; generating second forged audio samples based on the text data using multiple target speech generators; adding noise to the real audio samples to obtain noise-added real audio samples; adding noise to the second forged audio samples to obtain noise-added second forged audio samples; training the second voiceprint discriminator based on the noise-added real audio samples and the noise-added second forged audio samples to obtain the target voiceprint discriminator.

[0129] Optionally, the aforementioned computer terminal can execute program code for the following steps in the audio authenticity verification method: generating multiple first forged audio samples based on text data using multiple initial speech generators, and training an initial voiceprint discriminator based on real audio samples and multiple first forged audio samples to obtain the first voiceprint discriminator includes: generating multiple first forged audio samples based on text data using multiple initial speech generators; inputting real audio samples and multiple first forged audio samples into the initial voiceprint discriminator; the initial voiceprint discriminator outputs a first prediction vector based on real audio samples and outputs a second prediction vector corresponding to each first forged audio sample based on each first forged audio sample; calculating a first loss value based on the real sample label of the real audio sample and the first prediction vector; calculating a second loss value based on the fake sample label of each first forged audio sample and the second prediction vector corresponding to each first forged audio sample; adding the first loss value and the second loss value to obtain a first total loss value; and adjusting the parameters of the initial voiceprint discriminator based on the first total loss value to obtain the first voiceprint discriminator.

[0130] Optionally, the aforementioned computer terminal can execute the following steps in the audio authenticity verification method: A preset ratio of N:1 is used to jointly train the first voiceprint discriminator and multiple initial speech generators using an alternating optimization strategy. The parameters of the first voiceprint discriminator and multiple initial speech generators are alternately updated according to a preset ratio until a stopping condition is met, resulting in a second voiceprint discriminator and multiple target speech generators. This includes: alternately updating the parameters of the first voiceprint discriminator and multiple initial speech generators according to a preset ratio; fixing the parameters of multiple initial speech generators; updating the parameters of the first voiceprint discriminator N times to obtain an updated first voiceprint discriminator; fixing the updated parameters of the first voiceprint discriminator; updating the parameters of each initial speech generator once to obtain an updated speech generator corresponding to each initial speech generator, until a stopping condition is met, resulting in a second voiceprint discriminator and multiple target speech generators.

[0131] Optionally, the aforementioned computer terminal can execute the following steps in the audio authenticity verification method: fixing the parameters of multiple initial speech generators, updating the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator includes: inputting real audio samples and multiple first fake audio samples into the first voiceprint discriminator; the first voiceprint discriminator outputs a third prediction vector based on the real audio samples, and outputs a fourth prediction vector corresponding to each first fake audio sample based on each first fake audio sample; calculating a third loss value based on the real sample label of the real audio sample and the third prediction vector; calculating a fourth loss value based on the fake sample label of each first fake audio sample and the fourth prediction vector corresponding to each first fake audio sample; adding the third loss value and the fourth loss value to obtain a second total loss value; adjusting the parameters of the first voiceprint discriminator based on the second total loss value; repeating the step of adjusting the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator.

[0132] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the audio authenticity verification method: fixing the parameters of the updated first voiceprint discriminator, updating the parameters of each initial speech generator once, and obtaining the updated speech generator corresponding to each initial speech generator includes: inputting real audio samples and multiple first forged audio samples into the updated first voiceprint discriminator; the updated first voiceprint discriminator outputs a fifth prediction vector based on the real audio samples, and outputs a sixth prediction vector corresponding to each first forged audio sample based on each first forged audio sample; calculating a fifth loss value based on the real sample label of the real audio sample and the fifth prediction vector; calculating a sixth loss value based on the fake sample label of each first forged audio sample and the sixth prediction vector corresponding to each first forged audio sample; adding the fifth loss value and the sixth loss value to obtain a third total loss value; adjusting the parameters of each initial speech generator based on the third total loss value to obtain the updated speech generator corresponding to each initial speech generator.

[0133] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the audio authenticity verification method: processing the audio feature sequence through multi-layer feature mapping and attention mechanisms using a speaker extraction network to obtain target speech features, including: mapping the audio feature sequence using the first linear layer of the speaker extraction network to obtain a query vector matrix; mapping the audio feature sequence using the second linear layer to obtain a key vector matrix; transposing the key vector matrix to obtain a transposed vector matrix; performing matrix multiplication between the transposed vector matrix and the query vector matrix to obtain an attention score matrix; compressing and normalizing the attention score matrix to obtain an attention weight matrix; and then applying the attention weights to the target speech signal. The force weight matrix is ​​multiplied with the audio feature sequence to obtain the attention aggregation feature. The attention aggregation feature is linearly mapped using a third linear layer to obtain the first intermediate feature. The first intermediate feature is added to the audio feature sequence to obtain the second intermediate feature. The second intermediate feature is mapped using a fourth linear layer to obtain the third intermediate feature. A nonlinear activation function is applied to the third intermediate feature to obtain the activation feature. The activation feature is multiplied element-wise with the audio feature sequence to obtain the fourth intermediate feature. The fourth intermediate feature is mapped using a fifth linear layer to obtain the fifth intermediate feature. The fifth intermediate feature is added to the audio feature sequence to obtain the target speech feature.

[0134] Optionally, Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (Only one is shown) processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0135] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio authenticity identification method and device in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned audio authenticity identification method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the aforementioned networks include at least one of the following: the Internet, corporate intranet, local area network, mobile communication network, and combinations thereof.

[0136] The processor can access the information and application programs stored in the memory via the transmission device to execute the steps described above in the audio authenticity verification method.

[0137] This application provides a scheme for audio authenticity verification. It involves acquiring the input audio to be verified and feeding it into a target voiceprint discriminator. The target voiceprint discriminator comprises an audio processing network, a voiceprint extraction network, and an authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained using an adversarial training strategy based on multiple initial speech generators. The audio processing network extracts features from the input audio to obtain an audio feature sequence. The voiceprint extraction network processes the audio feature sequence using multi-layer feature mapping and an attention mechanism to obtain target speech features. The authenticity discrimination network outputs a verification result representing the authenticity of the input audio based on the target speech features. This solves the technical problem in existing technologies where it is difficult to effectively distinguish between real and fake audio, resulting in low accuracy in audio verification, and achieves the technical effect of improving the accuracy of audio verification.

[0138] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0139] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0140] Example 4

[0141] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the audio authenticity verification method provided in Embodiment 1.

[0142] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0143] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing steps of an audio authenticity verification method.

[0144] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0145] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0150] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for identifying the authenticity of audio files, characterized in that, include: The input audio to be identified is acquired and input to the target voiceprint discriminator. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network and a authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained by an adversarial training strategy based on multiple initial speech generators. The audio processing network extracts features from the input audio to obtain an audio feature sequence. The target speech features are obtained by performing multi-layer feature mapping and attention mechanism processing on the audio feature sequence through the voiceprint extraction network. The authenticity discrimination network outputs a discrimination result representing the authenticity of the input audio based on the target speech features.

2. The method according to claim 1, characterized in that, The training steps for the target voiceprint discriminator include: Obtain real audio samples and the corresponding text data of the real audio samples; The multiple initial speech generators generate multiple first fake audio samples based on the text data. An initial voiceprint discriminator is trained based on the real audio samples and the multiple first fake audio samples to obtain the first voiceprint discriminator. An alternating optimization strategy is used to jointly train the first voiceprint discriminator and the multiple initial speech generators. The parameters of the first voiceprint discriminator and the multiple initial speech generators are alternately updated according to a preset ratio until a stopping condition is reached, thereby obtaining a second voiceprint discriminator and multiple target speech generators. The multiple target speech generators generate a second fake audio sample based on the text data. Noise is added to the real audio sample to obtain a real audio sample with added noise. Noise is added to the second fake audio sample to obtain a second fake audio sample with added noise. The second voiceprint discriminator is trained based on the real audio sample with added noise and the second fake audio sample with added noise to obtain the target voiceprint discriminator.

3. The method according to claim 2, characterized in that, Using the plurality of initial speech generators, a plurality of first forged audio samples are generated based on the text data. An initial voiceprint discriminator is trained based on the real audio samples and the plurality of first forged audio samples to obtain the first voiceprint discriminator, which includes: Multiple initial speech generators are used to generate multiple first forged audio samples based on the text data. The real audio sample and the plurality of first fake audio samples are input into the initial voiceprint discriminator. The initial voiceprint discriminator outputs a first prediction vector based on the real audio sample and outputs a second prediction vector corresponding to each first fake audio sample based on each first fake audio sample. A first loss value is calculated based on the real sample label of the real audio sample and the first prediction vector. A second loss value is calculated based on the fake sample label of each first fake audio sample and the second prediction vector corresponding to each first fake audio sample. The first loss value and the second loss value are added together to obtain the first total loss value. The parameters of the initial voiceprint discriminator are adjusted based on the first total loss value to obtain the first voiceprint discriminator.

4. The method according to claim 2, characterized in that, The preset ratio is N:

1. An alternating optimization strategy is used to jointly train the first voiceprint discriminator and the multiple initial speech generators. The parameters of the first voiceprint discriminator and the multiple initial speech generators are alternately updated according to the preset ratio until a stopping condition is met, resulting in a second voiceprint discriminator and multiple target speech generators, including: The parameters of the first voiceprint discriminator and the parameters of the plurality of initial speech generators are alternately updated according to a preset ratio. The parameters of the plurality of initial speech generators are fixed, and the parameters of the first voiceprint discriminator are updated N times to obtain the updated first voiceprint discriminator. The parameters of the updated first voiceprint discriminator are fixed, and the parameters of each initial speech generator are updated once to obtain the updated speech generator corresponding to each initial speech generator. This process continues until a stopping condition is met, resulting in the second voiceprint discriminator and the plurality of target speech generators.

5. The method according to claim 4, characterized in that, By fixing the parameters of the plurality of initial speech generators and updating the parameters of the first voiceprint discriminator N times, the updated first voiceprint discriminator is obtained, including: The real audio sample and the plurality of first fake audio samples are input into the first voiceprint discriminator. The first voiceprint discriminator outputs a third prediction vector based on the real audio sample and outputs a fourth prediction vector corresponding to each first fake audio sample based on each first fake audio sample. The third loss value is calculated based on the real sample label of the real audio sample and the third prediction vector. The fourth loss value is calculated based on the fake sample label of each first fake audio sample and the fourth prediction vector corresponding to each first fake audio sample. The third loss value and the fourth loss value are added together to obtain the second total loss value. The parameters of the first voiceprint discriminator are adjusted based on the second total loss value. Repeat the step of adjusting the parameters of the first voiceprint discriminator N times to obtain the updated first voiceprint discriminator.

6. The method according to claim 5, characterized in that, With the parameters of the updated first voiceprint discriminator fixed, the parameters of each initial speech generator are updated once to obtain the updated speech generator corresponding to each initial speech generator, including: The real audio sample and the plurality of first fake audio samples are input into the updated first voiceprint discriminator. The updated first voiceprint discriminator outputs a fifth prediction vector based on the real audio sample and outputs a sixth prediction vector corresponding to each first fake audio sample based on each first fake audio sample. The fifth loss value is calculated based on the real sample label of the real audio sample and the fifth prediction vector. The sixth loss value is calculated based on the fake sample label of each first fake audio sample and the sixth prediction vector corresponding to each first fake audio sample. The fifth loss value and the sixth loss value are added together to obtain the third total loss value. The parameters of each initial speech generator are adjusted based on the third total loss value to obtain the updated speech generator corresponding to each initial speech generator.

7. The method according to claim 1, characterized in that, The target speech features are obtained by performing multi-layer feature mapping and attention mechanism processing on the audio feature sequence through the speaker extraction network, including: The audio feature sequence is mapped using the first linear layer of the voiceprint extraction network to obtain a query vector matrix. A second linear layer is then used to map the audio feature sequence to obtain a key vector matrix. The key vector matrix is ​​transposed to obtain a transposed vector matrix. The transposed vector matrix is ​​then multiplied by the query vector matrix to obtain an attention score matrix. The attention score matrix is ​​compressed and normalized to obtain an attention weight matrix. Finally, the attention weight matrix is ​​multiplied by the audio feature sequence to obtain attention aggregation features. The attention aggregation features are linearly mapped using a third linear layer to obtain a first intermediate feature. The first intermediate feature is then added to the audio feature sequence to obtain a second intermediate feature. The second intermediate feature is mapped using a fourth linear layer to obtain a third intermediate feature; a nonlinear activation function is applied to the third intermediate feature to obtain an activation feature; the activation feature is then multiplied element-wise with the audio feature sequence to obtain a fourth intermediate feature. The fifth linear layer is used to map the fourth intermediate feature to obtain the fifth intermediate feature; the fifth intermediate feature is added to the audio feature sequence to obtain the target speech feature.

8. An audio authenticity verification device, characterized in that, include: The first acquisition unit is used to acquire the input audio to be identified and input the input audio to the target voiceprint discriminator. The target voiceprint discriminator includes an audio processing network, a voiceprint extraction network and a authenticity discrimination network connected in sequence. The target voiceprint discriminator is trained by an adversarial training strategy based on multiple initial speech generators. An extraction unit is used to extract features from the input audio through the audio processing network to obtain an audio feature sequence; The processing unit is used to perform multi-layer feature mapping and attention mechanism processing on the audio feature sequence through the speaker extraction network to obtain target speech features; The output unit is used to output a discrimination result representing the authenticity of the input audio based on the target speech features through the authenticity discrimination network.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the audio authenticity identification method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program executes the audio authenticity identification method according to any one of claims 1 to 7 when it runs.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the audio authenticity identification method according to any one of claims 1 to 7.

Images