Forgery audio detection method, device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and financial science and technology, and provides a forged audio detection method, device and equipment and a medium, and the method comprises the steps: carrying out the data preprocessing and feature extraction of to-be-detected audio data, so as to obtain an original feature embedding vector; performing feature compression on the original feature embedded vector to obtain a compressed embedded vector; key feature extraction is carried out on the original feature embedding vector through an attention statistics pool layer and a multi-layer perceptron to obtain a key feature embedding vector; fusing the compressed embedded vector and the key feature embedded vector through a multi-head attention mechanism to obtain a fused feature vector; and detecting the fused feature vector through the classification head to determine whether the to-be-detected audio data is a forged audio or not. Through the technical scheme of the invention, the recognition precision of the forged audio is improved, and meanwhile, better generalization ability is shown on data in different fields.

Description

[Technical field]

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a method, device, equipment and medium for detecting forged audio. [Background technology]

[0002] With the rapid development of artificial intelligence technology, voice deep fake technology has made significant progress. This technology uses deep learning algorithms to synthesize or generate voices, which not only reaches unprecedented levels of naturalness and realism, but also makes breakthroughs in voice diversity. However, it also brings unprecedented challenges to voice deep fake detection technology. On social media and online platforms, realistic fake voices are increasingly difficult to distinguish by hearing, which provides opportunities for criminals, making the abuse of fake voices increasingly serious, posing a serious threat to personal privacy, social trust and even national security.

[0003] Faced with this challenge, the academic community has begun to pay extensive attention to and conduct in-depth research on voice deep fake technology, especially speech synthesis (SS) and voice conversion (VC). The goal of speech synthesis technology is to generate speech that is highly similar to real speech, while voice conversion technology replaces the speaker of the original speech without changing the semantic content. In order to effectively prevent the potential security risks brought by forged voice, how to improve the accuracy and efficiency of forged audio detection and improve the generalization ability of the detection model is an urgent problem to be solved. [Summary of the invention]

[0004] The embodiments of the present application provide a forged audio detection method, apparatus, device and medium, which aim to solve technical problems existing in the related art, such as insufficient generalization ability of audio deep forgery detection models on in-domain and out-of-domain data, low accuracy and low efficiency of forged audio detection.

[0005] In a first aspect, an embodiment of the present application provides a forged audio detection method, comprising:

[0006] Perform data preprocessing and feature extraction on the audio data to be detected to obtain the original feature embedding vector;

[0007] Performing feature compression on the original feature embedding vector to obtain a compressed embedding vector;

[0008] Extracting key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector;

[0009] The compressed embedding vector and the key feature embedding vector are fused through a multi-head attention mechanism to obtain a fused feature vector;

[0010] The fused feature vector is detected by a classification head to determine whether the audio data to be detected is forged audio.

[0011] In a second aspect, an embodiment of the present application provides a forged audio detection device, comprising:

[0012] A processing module, used for performing data preprocessing and feature extraction on the audio data to be detected to obtain an original feature embedding vector;

[0013] A compression module, used for performing feature compression on the original feature embedding vector to obtain a compressed embedding vector;

[0014] An extraction module, configured to extract key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector;

[0015] A fusion module, used for fusing the compressed embedding vector and the key feature embedding vector through a multi-head attention mechanism to obtain a fused feature vector;

[0016] The detection module is used to detect the fused feature vector through a classification head to determine whether the audio data to be detected is forged audio.

[0017] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the forged audio detection method when executing the computer program.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the forged audio detection method are implemented.

[0019] In the above schemes implemented based on the forged audio detection method, device, equipment and medium, the audio data to be detected can be preprocessed and feature extracted to obtain the original feature embedding vector; the original feature embedding vector can be feature compressed to obtain the compressed embedding vector; the original feature embedding vector can be key feature extracted through the attention statistical pooling layer and the multi-layer perceptron to obtain the key feature embedding vector; the compressed embedding vector and the key feature embedding vector can be fused through the multi-head attention mechanism to obtain the fused feature vector; the fused feature vector can be detected through the classification head to determine whether the audio data to be detected is forged audio. Through the above technical scheme of the present invention, feature compression is performed on the original feature embedding vector after preprocessing and feature extraction, which helps to reduce the computational complexity and number of parameters of subsequent processing, and can also improve the generalization ability of the model. When extracting and fusing key features, the attention statistical pooling layer, multi-layer perceptron and multi-head attention mechanism are used, which not only maintains the sensitivity to the real speech features, but also improves the ability to detect forged audio. This enables the model to more accurately integrate different features, highlight the key attributes of real audio, and capture subtle differences, thereby improving the recognition accuracy of forged audio, while showing better generalization capabilities on data from different fields.

Brief Description of the Drawings

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic flow chart of a forged audio detection method according to an embodiment of the present application is shown.

[0022] Figure 2 A schematic flow chart of step S101 in a forged audio detection method according to an embodiment of the present application is shown.

[0023] Figure 3 A schematic flow chart of a model training process in a forged audio detection method according to an embodiment of the present application is shown.

[0024] Figure 4 A schematic flow chart of step S105 in a forged audio detection method according to an embodiment of the present application is shown.

[0025] Figure 5 A specific flow chart of a method for detecting forged audio according to an embodiment of the present application is shown.

[0026] Figure 6 A block diagram of a counterfeit audio detection device according to an embodiment of the present application is shown.

[0027] Figure 7 A block diagram of a computer device according to an embodiment of the present application is shown. [Specific implementation method]

[0028] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0030] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0031] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0032] It should be noted that the embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0034] See also Figure 1 , Figure 1 A schematic flow chart of a forged audio detection method according to an embodiment of the present application is shown. The forged audio detection method is used to solve technical problems existing in the related art, such as insufficient generalization ability of the audio deep forgery detection model on in-domain and out-of-domain data, low accuracy and low efficiency of forged audio detection, etc.

[0035] like Figure 1 As shown, in the audio detection stage, the process of the forged audio detection method according to an embodiment of the present application includes:

[0036] Step S101, performing data preprocessing and feature extraction on the audio data to be detected to obtain an original feature embedding vector;

[0037] like Figure 2 As shown, in one embodiment, optionally, step S101 includes:

[0038] Step S201, resampling the audio data to be detected to obtain target audio data with a preset sampling rate;

[0039] In this step, the preset sampling rate can be set according to actual needs. For example, if a sampling rate of 16 kHz is required, the preset sampling rate can be set to 16 kHz.

[0040] Step S202, normalizing the target audio data so that the values ​​of the processed target audio data are distributed within a preset range;

[0041] In this step, the resampled audio is normalized so that the values ​​of the audio data are distributed within a reasonable range, such as [-1, 1].

[0042] Step S203: input the target audio data of the preprocessed audio data input into a pre-trained cross-language representation model to output the original feature embedding vector.

[0043] In this step, a pre-trained cross-language representation model is used as an encoder, and the pre-processed audio data is input into the model. The model extracts the feature representation of the audio through self-supervised learning on a large amount of multilingual unlabeled speech data. A three-dimensional tensor with a shape of [batch size, sequence length, feature dimension] is obtained as the speech embedding.

[0044] Among them, the cross-language representation model is based on the wav2vec 2.0 framework and is designed for speech recognition tasks. The cross-language representation model wav2vec2-xlsr achieves cross-language speech feature representation through self-supervised learning on a large amount of multilingual unlabeled speech data, providing strong feature support for a variety of downstream tasks. The model has a large scale and is trained through self-supervised learning on a diverse dataset, which can capture rich speech features, including the speaker's timbre, intonation, emotion, etc. These features are crucial to improving the accuracy of audio forgery detection.

[0045] Step S102, performing feature compression on the original feature embedding vector to obtain a compressed embedding vector;

[0046] In one embodiment, optionally, step S102 includes:

[0047] The original feature embedding vector is feature compressed by a pre-trained compression model to obtain a compressed embedding vector, wherein the dimension of the compressed embedding vector is lower than the dimension of the original feature embedding vector.

[0048] The compression model works by averaging the Transformer layer outputs to compress the input high-dimensional feature representation into a lower-dimensional representation while retaining the most important information. This compression helps reduce the computational complexity and number of parameters for subsequent processing, while also improving the generalization ability of the model. Finally, the compressed feature representation of the speech is output to prepare for the classification task in the subsequent processing stage.

[0049] The above are the steps of the audio detection stage. Before audio detection, the model needs to be trained based on the training set. Model training can be divided into two stages. The main purpose of the first stage is to learn the dependent features of real audio. These features are expected to be highly correlated in real samples and least correlated in deep fake samples.

[0050] like Figure 3 As shown, the first stage of training process includes:

[0051] Step S301, acquiring a massive amount of real audio data, and preprocessing the real audio data to obtain preprocessed real audio data;

[0052] Step S302, using the preprocessed real audio data to train the cross-language representation model to obtain a pre-trained cross-language representation model;

[0053] Step S303: training the compression model using the real audio embedding vector output by the pre-trained cross-language representation model to obtain the pre-trained compression model.

[0054] In this embodiment, since only real samples are needed, other open source speech data sets can be used to diversify style changes. Given a speech sample, the original audio waveform data is first preprocessed, resampled to the sampling rate required by the model (16kHz), and then normalized. In the feature extraction stage, the pre-trained wav2vec2-xlsr network is used to extract the speech representation, that is, the preprocessed audio data is converted into a series of feature representations. The obtained speech embedding is a three-dimensional tensor with a shape of [batch size, sequence length, feature dimension], and then these subspace embeddings are sent to the compress module. The working principle of the Compress module is to average the Transformer layer output, compress the input high-dimensional feature representation into a lower-dimensional representation, while trying to retain the most important information. Finally, the compressed feature representation of the speech is output to prepare for the classification task in the subsequent processing stage.

[0055] After completing the first stage of training, the parameters of the pre-trained cross-lingual representation model and the compression model are fixed, i.e., frozen.

[0056] In the above embodiment, the pre-trained wav2vec2-xlsr model is used as the core encoder. The model is trained on massive data with the help of self-supervised learning technology, and can capture and extract highly discriminative feature representations. In the first stage of training, multiple open source speech datasets are further utilized, which not only enriches the real speech features learned by the model, but also significantly improves the adaptability and robustness of the model to diverse speech environments.

[0057] Step S103, extracting key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector;

[0058] In one embodiment, optionally, step S103 includes:

[0059] Performing weighted aggregation on the original feature embedding vectors through an attention statistical pooling layer to extract a key feature embedding vector, wherein the attention statistical pooling layer includes a plurality of attention heads, each of which learns different aspects of the input features;

[0060] The key feature embedding vector is subjected to feature extraction by a multi-layer perceptron to obtain the key feature embedding vector, wherein the multi-layer perceptron includes multiple layers of fully connected layers, and activation functions are used between each fully connected layer to introduce nonlinearity.

[0061] In this embodiment, the embedding of the original features, i.e., the output from the SSL encoder, and the embedding obtained by the compression model are fused to produce a new vector through an attention statistics pooling (ASP) layer and a multi-layer perceptron (MLP) network. During the fusion process, the attention statistics pooling (ASP) layer first performs weighted aggregation on the input features to highlight important features and suppress unimportant features, and usually contains multiple attention heads, each of which learns different aspects of the input features. The MLP network is a fully connected neural network that performs nonlinear transformations through a series of layers, including activation functions such as ReLU to introduce nonlinearity, enabling it to learn complex feature combinations. The MLP network receives the output of the ASP layer and further processes it to extract features that help distinguish between real and fake audio.

[0062] Step S104, fusing the compressed embedding vector and the key feature embedding vector through a multi-head attention mechanism to obtain a fused feature vector;

[0063] In one embodiment, optionally, step S104 includes:

[0064] The key feature embedding vector is used as a query vector, and the compressed embedding vector is used as a key vector and a value vector, which are fused through a multi-head attention mechanism to obtain a fused feature vector.

[0065] In this embodiment, the key feature embedding vector is used as the query vector Q, and the compressed embedding vector is used as the key vector K and the value vector V, which are targetedly fused through the Transformer's multi-head attention mechanism.

[0066] Step S105: Detect the fused feature vector through a classification head to determine whether the audio data to be detected is forged audio.

[0067] like Figure 4 As shown, in one embodiment, optionally, step S105 includes:

[0068] Step S401, detecting the fused feature vector through a classification head, and outputting the category probability value corresponding to the audio data to be detected through a softmax function;

[0069] Step S402: when the category probability value is greater than or equal to a preset probability value, determining that the audio data to be detected is real audio;

[0070] Step S403: When the category probability value is less than the preset probability value, it is determined that the audio data to be detected is forged audio.

[0071] In this embodiment, a classification head is used to distinguish between real audio and forged audio. The classification head consists of multiple fully connected layers, and the feature vector processed by the ASP and MLP networks is used to make the final classification decision. The output of the classification head is converted into a probability distribution through a softmax function, and a suitable threshold is selected according to the task requirements to determine whether the audio is real or forged. For example, a preset probability value can be set to 0.6. If the probability value is greater than or equal to 0.6, the audio is determined to be real audio; if it is less than 0.6, it is determined to be forged audio.

[0072] In one embodiment, optionally, the method further comprises:

[0073] Using the preprocessed real audio data and the compressed real audio data, jointly training the attention statistics pooling layer, the multi-layer perceptron and the multi-head attention mechanism by calculating the cross entropy loss function and the classification head;

[0074] The gradient of the loss function with respect to each parameter is calculated by back propagation, and in each iteration, the parameters corresponding to the multi-layer perceptron, the multi-head attention mechanism, and the classification head are updated respectively according to the gradient.

[0075] In this embodiment, a cross entropy loss function is used to jointly train ASP, MLP, and multi-head attention mechanisms with the classification head. During the training process, the output of the model obtained by the ASP, MLP modules, and multi-head attention mechanism is passed to the classification head, and the classification head generates the final category probability. After each forward propagation, the value of the cross entropy loss function is calculated. Then, the gradient of the loss function with respect to the model parameters is calculated by the back propagation process algorithm. In each iteration, the parameters of the MLP and multi-head attention mechanism and the parameters of the classification head are updated according to the gradient.

[0076] Through the above technical solution of the present invention, the original feature embedding vector after preprocessing and feature extraction is compressed, which helps to reduce the computational complexity and number of parameters of subsequent processing, and also improves the generalization ability of the model. When extracting and fusing key features, the attention statistical pooling layer, multi-layer perceptron and multi-head attention mechanism are used, which not only maintains the sensitivity to the real speech features, but also improves the detection ability of forged audio, so that the model can more accurately integrate different features, highlight the key attributes of real audio, and capture subtle differences, thereby improving the recognition accuracy of forged audio, while showing better generalization ability on data in different fields.

[0077] The above technical solution of the present invention can be applied to the financial field, and specifically can be applied to forgery detection of telephone conference audio in the financial field to ensure the authenticity and reliability of important business decisions.

[0078] The above technical solution of the present invention is described in detail below with a specific embodiment.

[0079] 1. Scene Setting

[0080] Suppose you want to perform deep fake detection on conference call audio in the financial field to ensure the authenticity and reliability of important business decisions.

[0081] 2. Data Preparation

[0082] We collect a large amount of real financial conference call audio data, and select different types of voice samples from other open source voice databases to increase the diversity and style of the data.

[0083] Phase 1: Learning real audio-dependent features

[0084] 1. Data preprocessing

[0085] 1) Randomly select an audio clip from the collected financial conference call audio dataset.

[0086] 2) Resample the audio waveform data to a sampling rate of 16 kHz to ensure that it meets the model requirements.

[0087] 3) Normalize the resampled audio so that the values ​​of the audio data are distributed within a reasonable range, such as [-1, 1].

[0088] 2. Feature extraction

[0089] 1) Use the pre-trained wav2vec2-xlsr model as the encoder.

[0090] 2) The preprocessed audio data is input into the model, and the model extracts the feature representation of the audio through self-supervised learning on a large amount of multilingual unlabeled speech data.

[0091] 3) Get a three-dimensional tensor of shape [batch size, sequence length, feature dimension] as speech embedding.

[0092] 3. Feature Compression

[0093] 1) Send the speech embedding to the compress module.

[0094] 2) The compression model averages the Transformer layer output to compress the high-dimensional speech embedding into a lower-dimensional representation. For example, from the original [1, 1000, 512] to [1, 1000, 128], while retaining the most important information about the real audio dependency features.

[0095] 3) Output the compressed feature representation to prepare for the subsequent classification task.

[0096] Phase 2: Distinguishing between real and fake audio

[0097] 1. Freeze parameters

[0098] After the first phase of training is completed, the parameters of the subspace SSL encoder and compression model are fixed and no longer updated.

[0099] 2. Feature fusion and classification

[0100] 1) Assume that there is an audio sample to be detected, which also undergoes the preprocessing and feature extraction steps in the first stage to obtain the embedding of the original features.

[0101] 2) Through the ASP layer, assuming that this layer has 4 attention heads. Each attention head focuses on different features in the audio, such as one attention head focuses on the speaker's speaking speed, another focuses on the change of intonation, another focuses on the intensity of emotion, and the last one focuses on the characteristics of timbre.

[0102] 3) The MLP network receives the output of the ASP layer. The MLP network consists of three fully connected layers, and the ReLU activation function is used between each layer to introduce nonlinearity. The MLP network further processes the output of the ASP layer to extract features that help distinguish real and fake audio.

[0103] 4) The embedding of the original feature is used as the query (Q), and the embedding obtained by the first stage compression model is used as the key (K) and value (V), which are fused through the Transformer's multi-head attention mechanism. For example, the fused features highlight the unique timbre and emotional expression of the speaker in the real audio, while suppressing possible fake features.

[0104] 5) The fused features are processed through a classification head consisting of three fully connected layers. The output of the classification head is converted into a probability distribution through a softmax function.

[0105] 6) Set a threshold value of 0.6. If the probability value is greater than 0.6, the audio is judged to be real audio; if it is less than 0.6, it is judged to be fake audio. The above process is as follows Figure 5 shown.

[0106] 3. Model training and updating

[0107] Use the cross entropy loss function to jointly train ASP, MLP, and multi-head attention mechanism with the classification head. Pass the output of the audio sample to be detected through the above modules to the classification head to generate the final category probability. Assume that after one forward propagation, the cross entropy loss function value is calculated to be 0.4. The gradient of the loss function with respect to the model parameters is calculated through the back propagation process algorithm. In each iteration, the parameters of the MLP and multi-head attention mechanism and the parameters of the classification head are updated according to the gradient. For example, a parameter is increased by 0.02 in this iteration according to the gradient descent algorithm.

[0108] Through this two-stage implementation process, deep fake detection can be effectively performed on conference call audio in the financial field, improving the authenticity and reliability of the audio and providing strong protection for business decision-making.

[0109] Of course, this case can also be applied to other application scenarios that require audio authenticity identification, which are not listed here one by one.

[0110] Figure 6 A block diagram of a counterfeit audio detection device according to an embodiment of the present application is shown.

[0111] like Figure 6 As shown, in a second aspect, an embodiment of the present application provides a forged audio detection device 60, comprising:

[0112] A processing module 61 is used to perform data preprocessing and feature extraction on the audio data to be detected to obtain an original feature embedding vector;

[0113] A compression module 62, configured to perform feature compression on the original feature embedding vector to obtain a compressed embedding vector;

[0114] An extraction module 63, configured to extract key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector;

[0115] A fusion module 64 is used to fuse the compressed embedding vector and the key feature embedding vector through a multi-head attention mechanism to obtain a fused feature vector;

[0116] The detection module 65 is used to detect the fused feature vector through a classification head to determine whether the audio data to be detected is forged audio.

[0117] In one embodiment, optionally, the processing module 61 includes:

[0118] A sampling unit, used for resampling the audio data to be detected to obtain target audio data with a preset sampling rate;

[0119] A preprocessing unit, configured to perform normalization processing on the target audio data so that the values ​​of the processed target audio data are distributed within a preset interval;

[0120] The first extraction unit is used to input the processed target audio data into a pre-trained cross-language representation model to output the original feature embedding vector.

[0121] In one embodiment, optionally, the compression module 62 includes:

[0122] The original feature embedding vector is feature compressed by a pre-trained compression model to obtain a compressed embedding vector, wherein the dimension of the compressed embedding vector is lower than the dimension of the original feature embedding vector.

[0123] In one embodiment, optionally, the device further comprises:

[0124] An acquisition module, used to acquire a large amount of real audio data, and preprocess the real audio data to obtain preprocessed real audio data;

[0125] A first training module is used to train the cross-language representation model using the pre-processed real audio data to obtain a pre-trained cross-language representation model;

[0126] The second training module is used to train the compression model using the real audio embedding vector output by the pre-trained cross-language representation model to obtain the pre-trained compression model.

[0127] In one embodiment, optionally, the extraction module 63 includes:

[0128] A second extraction unit is used to perform weighted aggregation on the original feature embedding vector through an attention statistical pooling layer to extract a key feature embedding vector, wherein the attention statistical pooling layer includes a plurality of attention heads, each attention head learns different aspects of the input feature;

[0129] The third extraction unit is used to perform feature extraction on the key feature embedding vector through a multi-layer perceptron to obtain the key feature embedding vector, wherein the multi-layer perceptron includes multiple layers of fully connected layers, and activation functions are used between each fully connected layer to introduce nonlinearity.

[0130] In one embodiment, optionally, the fusion module 64 is used to:

[0131] The key feature embedding vector is used as a query vector, and the compressed embedding vector is used as a key vector and a value vector, which are fused through a multi-head attention mechanism to obtain a fused feature vector.

[0132] In one embodiment, optionally, the detection module 65 includes:

[0133] A detection unit, used to detect the fused feature vector through a classification head, and output a category probability value corresponding to the audio data to be detected through a softmax function;

[0134] a first determining unit, configured to determine that the audio data to be detected is real audio when the category probability value is greater than or equal to a preset probability value;

[0135] The second determination unit is configured to determine that the audio data to be detected is forged audio when the category probability value is less than the preset probability value.

[0136] In one embodiment, optionally, the device further comprises:

[0137] A parameter training module, for using the preprocessed real audio data and the compressed real audio data to jointly train the attention statistics pool layer, the multi-layer perceptron and the multi-head attention mechanism by calculating the cross entropy loss function and the classification head;

[0138] The parameter updating module is used to calculate the gradient of the loss function with respect to each parameter by back propagation, and in each iteration, update the parameters corresponding to the multi-layer perceptron, the multi-head attention mechanism and the classification head respectively according to the gradient.

[0139] In the above schemes implemented based on the forged audio detection method, device, equipment and medium, the audio data to be detected can be preprocessed and feature extracted to obtain the original feature embedding vector; the original feature embedding vector can be feature compressed to obtain the compressed embedding vector; the original feature embedding vector can be key feature extracted through the attention statistical pooling layer and the multi-layer perceptron to obtain the key feature embedding vector; the compressed embedding vector and the key feature embedding vector can be fused through the multi-head attention mechanism to obtain the fused feature vector; the fused feature vector can be detected through the classification head to determine whether the audio data to be detected is forged audio. Through the above technical scheme of the present invention, feature compression is performed on the original feature embedding vector after preprocessing and feature extraction, which helps to reduce the computational complexity and number of parameters of subsequent processing, and can also improve the generalization ability of the model. When extracting and fusing key features, the attention statistical pooling layer, multi-layer perceptron and multi-head attention mechanism are used, which not only maintains the sensitivity to the real speech features, but also improves the ability to detect forged audio. This enables the model to more accurately integrate different features, highlight the key attributes of real audio, and capture subtle differences, thereby improving the recognition accuracy of forged audio, while showing better generalization capabilities on data from different fields.

[0140] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the forged audio detection method when executing the computer program.

[0141] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the forged audio detection method are implemented.

[0142] It should be noted that, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the counterfeit audio detection device and each module described above can refer to the corresponding process in the aforementioned counterfeit audio detection method embodiment, and will not be repeated here.

[0143] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the model training device and each module described above can refer to the corresponding process in the aforementioned forged audio detection method embodiment, and will not be repeated here.

[0144] The forged audio detection device can be implemented in the form of a computer program. Figure 7 Runs on the computer device shown.

[0145] Figure 7 A block diagram of a computer device according to an embodiment of the present application is shown.

[0146] See also Figure 7 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a storage medium and an internal memory.

[0147] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the forged audio detection methods for multi-source data provided in the embodiments of the present application.

[0148] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0149] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any infectious disease transmission path analysis method or prediction neural network training method. The storage medium can be non-volatile or volatile.

[0150] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0152] The computer device of the embodiment of the present application exists in various forms, including but not limited to:

[0153] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0154] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0155] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0156] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0157] (5) Other electronic devices with data interaction functions.

[0158] In addition, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to perform the following steps:

[0159] Perform data preprocessing and feature extraction on the audio data to be detected to obtain the original feature embedding vector;

[0160] Performing feature compression on the original feature embedding vector to obtain a compressed embedding vector;

[0161] Extracting key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector;

[0162] The compressed embedding vector and the key feature embedding vector are fused through a multi-head attention mechanism to obtain a fused feature vector;

[0163] The fused feature vector is detected by a classification head to determine whether the audio data to be detected is forged audio.

[0164] In one embodiment, optionally, data preprocessing and feature extraction are performed on the audio data to be detected to obtain an original feature embedding vector, including:

[0165] Resampling the audio data to be detected to obtain target audio data with a preset sampling rate;

[0166] Normalizing the target audio data so that the values ​​of the processed target audio data are distributed within a preset interval;

[0167] The processed target audio data is input into a pre-trained cross-language representation model to output the original feature embedding vector.

[0168] In one embodiment, optionally, feature compression is performed on the original feature embedding vector to obtain a compressed embedding vector, including:

[0169] The original feature embedding vector is feature compressed by a pre-trained compression model to obtain a compressed embedding vector, wherein the dimension of the compressed embedding vector is lower than the dimension of the original feature embedding vector.

[0170] In one embodiment, optionally, the method further comprises:

[0171] Acquire a massive amount of real audio data, and preprocess the real audio data to obtain preprocessed real audio data;

[0172] Using the preprocessed real audio data to train the cross-language representation model to obtain a pre-trained cross-language representation model;

[0173] The compression model is trained using the real audio embedding vector output by the pre-trained cross-language representation model to obtain the pre-trained compression model.

[0174] In one embodiment, optionally, extracting key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector includes:

[0175] Performing weighted aggregation on the original feature embedding vectors through an attention statistical pooling layer to extract a key feature embedding vector, wherein the attention statistical pooling layer includes a plurality of attention heads, each of which learns different aspects of the input features;

[0176] The key feature embedding vector is subjected to feature extraction by a multi-layer perceptron to obtain the key feature embedding vector, wherein the multi-layer perceptron includes multiple layers of fully connected layers, and activation functions are used between each fully connected layer to introduce nonlinearity.

[0177] In one embodiment, optionally, the compressed embedding vector and the key feature embedding vector are fused through a multi-head attention mechanism to obtain a fused feature vector, including:

[0178] The key feature embedding vector is used as a query vector, and the compressed embedding vector is used as a key vector and a value vector, which are fused through a multi-head attention mechanism to obtain a fused feature vector.

[0179] In one embodiment, optionally, the detecting the fused feature vector by a classification head to determine whether the audio data to be detected is forged audio includes:

[0180] Detecting the fused feature vector through a classification head, and outputting the category probability value corresponding to the audio data to be detected through a softmax function;

[0181] When the category probability value is greater than or equal to a preset probability value, determining that the audio data to be detected is real audio;

[0182] When the category probability value is less than the preset probability value, it is determined that the audio data to be detected is forged audio.

[0183] In one embodiment, optionally, the method further comprises:

[0184] Using the preprocessed real audio data and the compressed real audio data, jointly training the attention statistics pooling layer, the multi-layer perceptron and the multi-head attention mechanism by calculating the cross entropy loss function and the classification head;

[0185] The gradient of the loss function with respect to each parameter is calculated by back propagation, and in each iteration, the parameters corresponding to the multi-layer perceptron, the multi-head attention mechanism, and the classification head are updated respectively according to the gradient.

[0186] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or electronic device can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0187] The above, in conjunction with the accompanying drawings, describes in detail the technical solution of the present application. Through the technical solution of the present application, the relevant operations of grayscale release can be integrated into the release system. Developers only need to perform integrated settings in the release system to enable the release system to call the deployment system to perform corresponding deployment for the grayscale release, thereby reducing the complexity of the grayscale release deployment work and improving the efficiency and reliability of the grayscale release.

[0188] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0189] It should be understood that, although the terms first, second, etc. may be used to describe the setting unit in the embodiments of the present application, these setting units should not be limited to these terms. These terms are only used to distinguish the setting units from each other. For example, without departing from the scope of the embodiments of the present application, the first setting unit may also be referred to as the second setting unit, and similarly, the second setting unit may also be referred to as the first setting unit.

[0190] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0191] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0194] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for detecting forged audio, characterized in that: The method comprises: Perform data preprocessing and feature extraction on the audio data to be detected to obtain the original feature embedding vector; Performing feature compression on the original feature embedding vector to obtain a compressed embedding vector; Extracting key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector; The compressed embedding vector and the key feature embedding vector are fused through a multi-head attention mechanism to obtain a fused feature vector; The fused feature vector is detected by a classification head to determine whether the audio data to be detected is forged audio.

2. The forged audio detection method according to claim 1, characterized in that: Perform data preprocessing and feature extraction on the audio data to be detected to obtain the original feature embedding vector, including: Resampling the audio data to be detected to obtain target audio data with a preset sampling rate; Normalizing the target audio data so that the values ​​of the processed target audio data are distributed within a preset interval; The processed target audio data is input into a pre-trained cross-language representation model to output the original feature embedding vector.

3. The forged audio detection method according to claim 1, characterized in that: Performing feature compression on the original feature embedding vector to obtain a compressed embedding vector includes: The original feature embedding vector is feature compressed by a pre-trained compression model to obtain a compressed embedding vector, wherein the dimension of the compressed embedding vector is lower than the dimension of the original feature embedding vector.

4. The forged audio detection method according to claim 3, characterized in that: The method further comprises: Acquire a massive amount of real audio data, and preprocess the real audio data to obtain preprocessed real audio data; Using the preprocessed real audio data to train the cross-language representation model to obtain a pre-trained cross-language representation model; The compression model is trained using the real audio embedding vector output by the pre-trained cross-language representation model to obtain the pre-trained compression model.

5. The forged audio detection method according to claim 1, characterized in that: The extracting key features from the original feature embedding vector by using the attention statistical pooling layer and the multi-layer perceptron to obtain the key feature embedding vector includes: Performing weighted aggregation on the original feature embedding vectors through an attention statistical pooling layer to extract a key feature embedding vector, wherein the attention statistical pooling layer includes a plurality of attention heads, each of which learns different aspects of the input features; The key feature embedding vector is subjected to feature extraction by a multi-layer perceptron to obtain the key feature embedding vector, wherein the multi-layer perceptron includes multiple layers of fully connected layers, and activation functions are used between each fully connected layer to introduce nonlinearity.

6. The forged audio detection method according to claim 1, characterized in that: The compressed embedding vector and the key feature embedding vector are fused through a multi-head attention mechanism to obtain a fused feature vector, including: The key feature embedding vector is used as a query vector, and the compressed embedding vector is used as a key vector and a value vector, which are fused through a multi-head attention mechanism to obtain a fused feature vector.

7. The forged audio detection method according to claim 1, characterized in that: The detecting the fused feature vector by the classification head to determine whether the audio data to be detected is forged audio includes: Detecting the fused feature vector through a classification head, and outputting the category probability value corresponding to the audio data to be detected through a softmax function; When the category probability value is greater than or equal to a preset probability value, determining that the audio data to be detected is real audio; When the category probability value is less than the preset probability value, it is determined that the audio data to be detected is forged audio.

8. A forged audio detection device, characterized in that: include: A processing module, used for performing data preprocessing and feature extraction on the audio data to be detected to obtain an original feature embedding vector; A compression module, used for performing feature compression on the original feature embedding vector to obtain a compressed embedding vector; An extraction module, configured to extract key features from the original feature embedding vector through an attention statistical pooling layer and a multi-layer perceptron to obtain a key feature embedding vector; A fusion module, used for fusing the compressed embedding vector and the key feature embedding vector through a multi-head attention mechanism to obtain a fused feature vector; The detection module is used to detect the fused feature vector through a classification head to determine whether the audio data to be detected is forged audio.

9. A computer device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, wherein the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the method process according to any one of claims 1 to 7.

Images