Deep synthesis audio cross-domain detection method and device based on self-supervised auxiliary task

Abstract

The present application relates to a method and device for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks, wherein the method includes: obtaining the audio to be detected as a test data set; augmenting the test data set based on a preset self-supervised auxiliary task to obtain an augmented data set; based on the augmented data set, taking the self-supervised auxiliary task as the optimization target, fine-tuning the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and a self-supervised auxiliary task; inputting the test data set into the target fine-tuning model to obtain a detection result. Through this application, the problems of insufficient generalization and significantly reduced cross-domain detection effect in the related technology are solved, and while improving the generalization of the model, the overhead of model training can be greatly reduced; and it is able to adapt to the distribution differences of deep forgery attributes in the test data set, thereby improving cross-domain detection performance.

Description

Technical Field

[0001] The present application relates to the field of computer artificial intelligence security, and in particular to a method and device for cross-domain detection of deep synthesized audio based on self-supervised auxiliary tasks. Background Art

[0002] With the development of artificial intelligence technology, the quality of AI synthesized speech continues to improve and is difficult to distinguish by human ears. The speech audio generated by speech synthesis technology based on deep neural networks is called deepfake audio. The latest zero-shot TTS technology only needs a few sentences of the target speaker's speech to generate audio of any content with the same tone. While speech synthesis technology brings convenience and innovation, it also brings great security risks.

[0003] Existing cross-domain detection methods for synthetic audio are mainly based on the idea of ​​domain generalization, which improves the detection generalization of the model by adjusting the training process, thereby improving the performance in cross-domain testing. The specific method is: during the training phase, white noise, reverberation, masking and other operations are added to the training data. The main drawback of the domain generalization method is that this type of method is only optimized based on the information contained in the training data, and there is an upper limit on cross-domain performance. At the same time, the existing methods need to be deployed during training. When a new synthetic audio data set appears and the detection effect of the existing method is still poor after improvement, it is necessary to redesign the generalization improvement method and retrain, which will bring huge computational overhead under the current situation of the continuous development of audio synthesis technology.

[0004] There is currently no effective solution to the problems of insufficient generalization and significantly reduced cross-domain detection effects in related technologies. Summary of the invention

[0005] In this embodiment, a method and device for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks are provided to solve the problems of insufficient generalization and significantly reduced cross-domain detection effect in related technologies.

[0006] In the first aspect, in this embodiment, a method for cross-domain detection of deep synthesized audio based on a self-supervised auxiliary task is provided, including:

[0007] Get the audio to be detected as a test data set;

[0008] Based on a preset self-supervised auxiliary task, the test data set is augmented to obtain an augmented data set; based on the augmented data set, the self-supervised auxiliary task is used as an optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and the self-supervised auxiliary task;

[0009] The test data set is input into the target fine-tuning model to obtain a detection result.

[0010] In some of the embodiments, the self-supervisory auxiliary task includes a task of determining an audio speed regulation state and / or a task of reconstructing an audio decoupling feature encoding task.

[0011] In some of the embodiments, the architecture of the task of determining the audio speed adjustment state includes a common layer and a second classification linear layer;

[0012] The common layer includes a feature extractor and a Conformer module;

[0013] The feature extractor is connected to the Conformer module and is used to process the input audio samples and output a series of feature vectors;

[0014] The Conformer module is connected to the second classification linear layer, and is used to encode the feature vector and a preset classification indication vector, and output a sequence vector;

[0015] The second classification linear layer is used to process any one of the sequence vectors and output a speed adjustment state label.

[0016] In some of the embodiments, the architecture of the reconstructed audio decoupled feature encoding task includes a shared layer, a pre-trained decoupled encoder, and a reconstructed linear layer;

[0017] The common layer includes a feature extractor and a Conformer module;

[0018] The feature extractor is connected to the Conformer module and is used to process the input audio samples and output a series of feature vectors;

[0019] The Conformer module and the reconstruction linear layer are used to encode the feature vector and the preset classification indication vector and output a sequence vector;

[0020] The pre-trained decoupled encoder is connected to the reconstruction linear layer to process the input audio sample and output the acoustic detail code;

[0021] The reconstruction linear layer is used to process the sequence vector and the acoustic detail code, and output an acoustic reconstruction label.

[0022] In some embodiments, the method further comprises:

[0023] Constructing a target loss function according to a preset auxiliary loss function and a deep fake classification loss function; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function;

[0024] According to the acquired training data set and the target loss function, the initial detection model is pre-trained to obtain a target detection model.

[0025] In some embodiments, based on the augmented dataset, the self-supervised auxiliary task is used as an optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model, including:

[0026] Based on the augmented data set and the corresponding auxiliary loss function, the pre-trained target detection model is fine-tuned to obtain a target fine-tuning model; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function.

[0027] In some of the embodiments, the forgery detection task and the self-supervised auxiliary task are both constructed based on a baseline detection neural network architecture; the baseline detection neural network architecture is a Conformer architecture, a Transformer architecture, or a fully connected architecture.

[0028] In a second aspect, in this embodiment, a deep synthesis audio cross-domain detection device based on a self-supervised auxiliary task is provided, including: an acquisition module, a fine-tuning module, and a testing module;

[0029] The acquisition module is used to acquire the audio to be detected as a test data set;

[0030] The fine-tuning module is used to perform augmentation processing on the test data set based on a preset self-supervised auxiliary task to obtain an augmented data set; based on the augmented data set, the self-supervised auxiliary task is used as an optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and the self-supervised auxiliary task;

[0031] The testing module is used to input the testing data set into the target fine-tuning model to obtain a testing result.

[0032] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the method for cross-domain detection of deep synthesized audio based on self-supervised auxiliary tasks as described in the first aspect above is implemented.

[0033] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the method for deep synthesis audio cross-domain detection based on self-supervised auxiliary tasks described in the first aspect above is implemented.

[0034] Compared with the related art, the method and device for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks provided in this embodiment obtain the audio to be detected as a test data set; based on the preset self-supervised auxiliary tasks, the test data set is augmented to obtain an augmented data set matching the test data set, thereby improving the fine-tuning effect; based on the augmented data set, the self-supervised auxiliary tasks are used as the optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model; the training tasks of the target detection model include forgery detection tasks and self-supervised auxiliary tasks. In the test process, the target fine-tuning model is obtained by adding fine-tuning instead of retraining, which can greatly reduce the cost of model training while improving generalization. Then the test data set is input into the target fine-tuning model to obtain the detection result; wherein, the target fine-tuning model can adapt to the distribution differences of deep forgery attributes in the test data set, thereby improving the cross-domain detection performance; thereby solving the problems of insufficient generalization and significantly reduced cross-domain detection effect in the related art.

[0035] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 It is a hardware structure block diagram of a terminal device of a deep synthesis audio cross-domain detection method based on a self-supervised auxiliary task provided by an embodiment of the present application;

[0038] Figure 2 It is a flowchart of a deep synthesis audio cross-domain detection method based on a self-supervised auxiliary task provided by an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the architecture of a counterfeit detection task provided by an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the architecture of a task for determining an audio speed adjustment state provided by an embodiment of the present application;

[0041] Figure 5It is a schematic diagram of the architecture of the reconstructed audio decoupling feature encoding task provided by an embodiment of the present application;

[0042] Figure 6 is a schematic diagram of the architecture of a target detection model provided in one embodiment of the present application;

[0043] Figure 7 This is a structural block diagram of a deep synthesis audio cross-domain detection device based on self-supervised auxiliary tasks provided in one embodiment of the present application.

[0044] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, acquisition module; 220, fine-tuning module; 230, test module. DETAILED DESCRIPTION

[0045] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0046] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the method for detecting cross-domain deep synthesized audio based on self-supervised auxiliary tasks in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the deep synthetic audio cross-domain detection method based on self-supervised auxiliary tasks in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0049] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0050] In this embodiment, a method for cross-domain detection of deep synthesized audio based on self-supervised auxiliary tasks is provided. Figure 2 is a flowchart of the deep synthesis audio cross-domain detection method based on the self-supervised auxiliary task of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0051] Step S210, obtaining the audio to be detected as a test data set;

[0052] Step S220, augmenting the test data set based on the preset self-supervised auxiliary task to obtain an augmented data set; based on the augmented data set, taking the self-supervised auxiliary task as the optimization target, fine-tuning the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and a self-supervised auxiliary task;

[0053] Step S230, input the test data set into the target fine-tuning model to obtain the detection result.

[0054] Specifically, the target detection model is pre-trained by a relevant training method, and its training tasks include a forgery detection task and a self-supervised auxiliary task, that is, the target detection model can output a detection result (real or forged) corresponding to the forgery detection task; and can output a detection result corresponding to the self-supervised auxiliary task (self-supervised task label; for example: speed adjustment state label and acoustic reconstruction label, etc.). In this embodiment, there is no restriction on the training method and training data set of the target detection model.

[0055] In practical application, the method of obtaining the audio to be detected in the embodiment of the present application includes but is not limited to obtaining the audio to be detected that meets the above requirements from a database pre-stored therein; or downloading the audio to be detected that meets the requirements from a network platform; the embodiment of the present application does not limit the method of obtaining the audio to be detected. Among them, the audio to be detected is the test data set, and the audio to be detected can be a deep synthetic audio.

[0056] The above process of this embodiment can be considered as using the target detection model to perform cross-domain detection of deep synthetic audio. Before the actual test, the process needs to augment the test data set based on the preset self-supervised auxiliary task to obtain the augmented data set as a fine-tuned data set to match the newly appeared synthetic audio, without obtaining an additional data set; based on the augmented data set, the self-supervised auxiliary task is used as the optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model, instead of retraining. While improving generalization, it can greatly reduce the cost of model training. After fine-tuning, the test data set is re-input into the target fine-tuning model to obtain the detection result. Since the target fine-tuning model can adapt to the distribution differences of deep fake attributes in the test data set, the cross-domain detection performance is improved. Among them, the fine-tuning process of the target detection model can be considered as an improvement on the target detection model training process, which can be different data sets, different loss functions, etc., and there is no restriction on this.

[0057] In the related art, the existing cross-domain detection method of synthetic audio is mainly based on the idea of ​​domain generalization, which improves the detection generalization of the model by adjusting the training process, thereby improving the performance in cross-domain testing. The specific method is: in the training stage, white noise, reverberation, masking and other operations are added to the training data. The main defect of the domain generalization method is that this type of method is optimized only based on the information contained in the training data, and there is an upper limit on cross-domain performance. At the same time, the existing method needs to be deployed during training. When a new synthetic audio data set appears and the detection effect of the existing method after improvement is still poor, it is necessary to redesign the generalization improvement method and retrain, which will bring huge computational overhead under the current situation of the continuous development of audio synthesis technology. In this embodiment, by obtaining the audio to be detected as a test data set; based on the preset self-supervised auxiliary task, the test data set is augmented to obtain an augmented data set matching the test data set, and the fine-tuning effect is improved; based on the augmented data set, the self-supervised auxiliary task is used as the optimization target, and the pre-trained target detection model is fine-tuned to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and a self-supervised auxiliary task. During the test process, fine-tuning is added to obtain the target fine-tuning model instead of retraining. This can greatly reduce the cost of model training while improving generalization. The test data set is then input into the target fine-tuning model to obtain the detection results. The target fine-tuning model can adapt to the distribution differences of deep fake attributes in the test data set, thereby improving cross-domain detection performance. This solves the problem of insufficient generalization and significantly reduced cross-domain detection effects in related technologies.

[0058] In some of the embodiments, the self-supervised auxiliary task includes a task of determining an audio speed regulation state and / or a task of reconstructing an audio decoupling feature encoding task.

[0059] Specifically, the guiding principle for selecting self-supervised auxiliary tasks is that by optimizing these tasks, the target fine-tuning model can learn the distribution of deep fake-specific attributes in the test data set, thereby improving the detection performance of the target fine-tuning model on the test set. Therefore, the self-supervised tasks in this embodiment are related to or similar to the generation process of deep synthetic audio. Based on this principle, the self-supervised auxiliary tasks set include the task of judging the audio speed regulation state and / or the task of reconstructing the audio decoupling feature coding task; the task of judging the audio speed regulation state and / or the task of reconstructing the audio decoupling feature coding task are both related to deep synthetic audio. In other embodiments, self-supervised auxiliary tasks may also include fill-in-the-blank tasks, etc., which are not limited to this.

[0060] It should be noted that the model architecture of the target detection model is the same as the model architecture of the target fine-tuning model. For example: the self-supervised auxiliary task in the target detection model is the task of judging the audio speed regulation state; then the self-supervised auxiliary task in the target fine-tuning model is the task of judging the audio speed regulation state. For another example: the self-supervised auxiliary task in the target detection model is the task of judging the audio speed regulation state and the task of reconstructing the audio decoupling feature encoding; then the self-supervised auxiliary task in the target fine-tuning model is the task of judging the audio speed regulation state and the task of reconstructing the audio decoupling feature encoding. Among them, the forgery detection task and the self-supervised auxiliary task are both related to the model architecture. In other words, the model architectures of the target detection model and the target fine-tuning model will be associated with the forgery detection task and the self-supervised auxiliary task.

[0061] In some of the embodiments, both the forgery detection task and the self-supervised auxiliary task are constructed based on a baseline detection neural network architecture; the baseline detection neural network architecture is a Conformer architecture, a Transformer architecture, or a fully connected architecture.

[0062] Preferably, the Conformer architecture is used, which performs well in capturing audio features in both time and frequency domains. In addition, the Conformer architecture is highly compatible with self-supervised tasks, making it an ideal choice for detection frameworks. That is, the forgery detection task, the forgery detection task, and the self-supervised auxiliary task all include common layers and unique layers.

[0063] Among them, the architecture of the forgery detection task is as follows Figure 3 It includes a common layer and a first classification linear layer; the common layer includes a feature extractor and a Conformer module; the first classification linear layer is a deep fake classification head. The feature extractor is connected to the Conformer module; it uses the pre-trained wav2vec 2.0 XLS-R model to process the input audio sample α and output a series of feature vectors M is determined by the audio length and the CNN stride of XLS-R (e.g. 20 milliseconds). The Conformer module is connected to the first classification linear layer, which adds a learnable classification indicator vector in front of the feature vector O Encode and output the sequence in the last Conformer module The first sequence vector (can also be any sequence vector) is passed to the first classification linear layer for processing, and finally determines whether the input audio sample is classified as real or fake. A binary classification loss function for the fake detection task is constructed to control the learning of the model. The expression of the binary classification loss function is:

[0064] ;

[0065] Where N represents the number of input audio samples. represents the cross entropy loss function, represents the output of the deepfake classification head for the i-th input audio sample, Represents a deepfake label.

[0066] In this embodiment, the above architecture can improve the ability to detect audio deep forgeries and is compatible with self-supervised auxiliary tasks, thereby simplifying the training process of subsequent models and improving training efficiency.

[0067] In order to further improve the generalization ability of the model, the data enhancement technology from RawBoost can be introduced to retrain the target detection model. The data enhancement technology includes noise addition, time masking and frequency masking, which will not be explained in detail.

[0068] In some of the embodiments, the architecture of the task of determining the audio speed adjustment state includes a common layer and a second classification linear layer;

[0069] The shared layers include feature extractor and Conformer modules;

[0070] The feature extractor, connected to the Conformer module, is used to process the input audio samples and output a series of feature vectors;

[0071] A Conformer module, connected to the second classification linear layer, is used to encode the feature vector and the preset classification indicator vector and output a sequence vector;

[0072] The second classification linear layer is used to process any sequence vector and output the speed adjustment state label.

[0073] Specifically, adjusting the speed of a segment in an audio sample can change the fundamental frequency of the sound, and increasing the speed will increase the fundamental frequency of the human voice. Fundamental frequency is a key attribute of timbre, and in voice conversion (VC) methods used to generate deep synthetic audio, fundamental frequency usually changes with timbre. Therefore, audio speed adjustment is used as a self-supervised auxiliary task for cross-domain detection of deep synthetic audio to learn to identify audio speed adjustment states; this self-supervised auxiliary task captures deep fake-specific properties related to fundamental frequency changes, allowing the model to better adapt to the feature distribution of the test dataset.

[0074] Among them, the architecture of the task of judging the audio speed regulation status is as follows Figure 4As shown. The shared layer in the task of determining the audio speed regulation status is shared with the shared layer in the forgery detection task, and their structure and function are the same, so that the tasks in the model are compatible with each other and the overall training overhead can be simplified. The difference is that the second classification linear layer is connected to the Conformer module in the shared layer, which serves as the classification head of the auxiliary task. In other embodiments, the shared layer in the task of determining the audio speed regulation status can also be set separately, but it needs to be trained separately, and there is no restriction on this.

[0075] The input audio samples are processed at three different speeds: increasing the speed by 1.5 times, slowing down the speed to 0.7 times, and keeping the speed unchanged, which are marked as 1.5 times, 0.7 times, and 1.0 times respectively; the feature extractor uses the pre-trained wav2vec 2.0 XLS-R model to process the input audio samples and output a series of feature vectors. The Conformer module is connected to the second classification linear layer, which adds a learnable classification indicator vector in front of the feature vector for encoding. Any sequence vector of the output sequence of the last Conformer module (the first sequence vector is preferred) is passed to the second classification linear layer for processing, and finally the classification of the input audio sample is determined as a specific speed adjustment state label (for example: 1.5 times (acceleration), 0.7 times (deceleration) and 1.0 times (normal)). A three-classification loss function for the task of judging the audio speed adjustment state is constructed to control the learning of the model. The expression of the three-classification loss function is:

[0076] ;

[0077] Where N represents the number of input audio samples. represents the cross entropy loss function, Represents the audio sample of the i-th input The speed adjustment state of the classification head output, Represents an audio sample Speed ​​adjustment status label.

[0078] In some of the embodiments, the architecture for reconstructing the audio decoupled feature encoding task includes a shared layer, a pre-trained decoupled encoder, and a reconstructed linear layer;

[0079] The shared layers include feature extractor and Conformer modules;

[0080] The feature extractor, connected to the Conformer module, is used to process the input audio samples and output a series of feature vectors;

[0081] The Conformer module and the reconstruction linear layer are used to encode the feature vector and the preset classification indicator vector and output a sequence vector;

[0082] A pre-trained decoupled encoder, connected to a reconstruction linear layer, processes the input audio samples and outputs an acoustic detail code;

[0083] Reconstruction linear layer, which processes sequence vectors and acoustic detail encoding and outputs acoustic reconstruction labels.

[0084] Specifically, a self-supervised auxiliary task focusing on reconstructing the decoupled audio feature coding is proposed. Compared with reconstructing fragments of the entire audio sample, reconstructing the decoupled audio feature coding focuses on properties that are closely related to audio deepfakes, thereby reducing computational costs. Since the rhythm, semantic content, and timbre properties extracted from the real audio and the synthesized audio through the decoupled encoder should ideally be the same, it is believed that the acoustic detail coding contains the deepfake-specific properties inherent in the synthesized speech. Based on this, it is proposed to reconstruct the acoustic detail coding obtained by decoupling as a self-supervised auxiliary task. This self-supervised auxiliary task enables the model to learn the deepfake properties embedded in the decoupled acoustic detail coding, thereby helping it adapt to the feature distribution of the test set and ultimately improving the detection performance.

[0085] Among them, the architecture of the reconstructed audio decoupling feature encoding task is as follows Figure 5 As shown. The shared layers in the reconstructed audio decoupling feature encoding task are shared with the shared layers in the forgery detection task, and their structures and functions are the same, so that the tasks in the model are compatible with each other and the overall training overhead can be simplified. The difference is that a pre-trained decoupled encoder and a reconstructed linear layer are also included. The pre-trained decoupled encoder adopts the pre-trained decoupled encoder FACodec; the reconstructed linear layer serves as the reconstruction head of the auxiliary task. In other embodiments, the shared layers in the reconstructed audio decoupling feature encoding task can also be set separately, but they need to be trained separately, and there is no restriction on this.

[0086] Among them, the feature extractor uses the pre-trained wav2vec 2.0 XLS-R model to process the input audio samples and output a series of feature vectors. The Conformer module is connected to the reconstruction linear layer, which adds a learnable classification indicator vector in front of the feature vector for encoding. In the last Conformer module, all sequence vectors of the output sequence are passed to the reconstruction linear layer for processing; the decoupled encoder FACodec extracts the acoustic detail encoding of three attributes from the input audio sample, and the acoustic detail encoding is passed to the reconstruction linear layer for processing; the acoustic detail encoding is the encoding of rhythm, semantic content and acoustic detail. Finally, in the reconstruction linear layer, the sequence vector and acoustic detail encoding are processed, and the acoustic reconstruction label (rhythm, semantic content and acoustic detail) is output. And the reconstruction loss function of the reconstructed audio decoupled feature encoding task is constructed to control the learning of the model. The expression of the reconstruction loss function is:

[0087] ;

[0088] Where N represents the number of input audio samples. represents the hinge vector loss function, Represents the output of the reconstruction header for the i-th input audio sample.

[0089] In some of the embodiments, the method for cross-domain detection of deep synthesized audio based on self-supervised auxiliary tasks further includes the following steps:

[0090] Constructing a target loss function according to a preset auxiliary loss function and a deep fake classification loss function; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function;

[0091] According to the acquired training data set and target loss function, the initial detection model is pre-trained to obtain the target detection model.

[0092] Specifically, the target loss function can control the pre-training of the initial detection model to obtain the target detection model. It is constructed based on the preset auxiliary loss function and deep fake classification loss function. And the architecture of the initial detection model matches it.

[0093] For example: the auxiliary loss function is the reconstruction loss function; then the target loss function can be expressed as: ;

[0094] In the formula, β is an adjustable hyperparameter, which is empirically taken as 1.

[0095] The architecture of the initial detection model at this time is as follows Figure 6 shown.

[0096] After the target loss function is constructed, the initial detection model is trained in combination with the acquired training data set, and when the target loss function meets the cutoff condition, the target detection model is obtained. There is no restriction on the training method.

[0097] For example: the auxiliary loss function is a three-classification loss function; then the target loss function can be expressed as: ;

[0098] In the formula, γ is an adjustable hyperparameter, which is empirically set to 1. The model training process under this objective loss function will not be repeated.

[0099] Through this embodiment, the training of the target detection model can be completed quickly. The target detection model not only needs to classify the authenticity of the audio, but also needs to predict the labels of the auxiliary tasks. For example, in the self-supervised audio speed adjustment state task, the target detection model needs to output the authenticity score and the label corresponding to the speed adjustment at the same time; thus, the target detection model can better adapt to the feature distribution of the training data set and improve the detection performance.

[0100] In some embodiments, based on the augmented data set, the self-supervised auxiliary task is used as the optimization target, and the pre-trained target detection model is fine-tuned to obtain the target fine-tuning model, including the following steps:

[0101] Based on the augmented data set and the corresponding auxiliary loss function, the pre-trained target detection model is fine-tuned to obtain a target fine-tuning model; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function.

[0102] Specifically, the augmented dataset is obtained by augmenting the test dataset based on the characteristics of self-supervision. It can expand the amount of data without diluting the effective features in the test dataset, and does not introduce additional data, which would dilute the effective features in the test dataset.

[0103] Among them, fine-tuning the pre-trained object detection model is related to its self-supervised auxiliary task. For example, if the self-supervised auxiliary task of the object detection model is to reconstruct the audio decoupling feature encoding task, then other loss functions and other related parameters will be frozen during fine-tuning, and only the reconstruction loss function loss will be used. fa For another example: the self-supervised auxiliary tasks of the target detection model are to judge the audio speed adjustment state task and reconstruct the audio decoupling feature encoding task. In this case, other loss functions and other related parameters will be frozen during fine-tuning, and only the three-classification loss function loss sp And reconstruction loss function loss fa Train the subject.

[0104] Since the purpose of fine-tuning is to more effectively improve the performance of the model on various cross-domain data and adapt to various emerging speech synthesis methods, a preset number of training rounds will be used. The preferred number of training rounds for fine-tuning is 10, 15, 20, etc. This allows the fine-tuning process to greatly reduce the computational overhead compared to retraining.

[0105] Example:

[0106] The detection process with the task of judging the audio speed state as the self-supervised auxiliary task is as follows: 1. After the training data set is augmented by speed adjustment, audios with 1.5 times speed, 0.7 times speed and normal speed are obtained, and the corresponding speed adjustment state labels 1.5, 0.7, 1.0 are marked to obtain the augmented data set for training. 2. The forgery detection task and the task of judging the audio speed state are jointly trained to obtain the target detection model. 3. After the test data set of the audio to be detected is augmented by speed adjustment, audios with 1.5 times speed, 0.7 times speed and normal speed are obtained, and the corresponding speed adjustment state labels 1.5, 0.7, 1.0 are marked to obtain the augmented data set for detection. 4. The classification head of the first classification linear layer is frozen, and only the speed adjustment state label of the audio to be detected and the task of judging the audio speed state are used to fine-tune the rest of the target model to obtain the target fine-tuning model. 5. The original audio to be detected is input into the target fine-tuning model, and the score output by the above classification head is used to determine whether the audio to be detected is real audio or synthetic audio (forged audio).

[0107] Example:

[0108] The detection process with the reconstructed audio decoupled feature coding task as the self-supervised auxiliary task is as follows: 1. Use the pre-trained decoupled encoder to extract the acoustic detail coding in the training dataset, obtain the augmented dataset for training, and save it as the reconstruction target of the self-supervised auxiliary task. 2. Use the forgery detection task and the reconstructed audio decoupled feature coding task for training to obtain the target detection model. 3. Use the pre-trained decoupled encoder to extract the acoustic detail coding in the audio to be detected, obtain the augmented dataset for detection, and save it as the auxiliary task reconstruction target. 4. Freeze the classification head of the first classification linear layer, and only use the acoustic detail coding of the audio to be detected and the reconstructed audio decoupled feature coding task to fine-tune the rest of the target model to obtain the target fine-tuning model. 5. Input the original audio to be detected into the target fine-tuning model, and judge whether the audio to be detected is real audio or synthetic audio (forged audio) by the score output by the above classification head.

[0109] Through this embodiment, the target model is fine-tuned using the forged feature distribution of the audio to be detected during the test phase, which can more effectively improve the performance of the model on various cross-domain data and adapt to various emerging speech synthesis methods; and the computational overhead of fine-tuning the target model is much smaller than that of retraining.

[0110] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0111] In this embodiment, a deep synthesis audio cross-domain detection device based on self-supervised auxiliary tasks is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0112] Figure 7 is a structural block diagram of a deep synthesis audio cross-domain detection device based on a self-supervised auxiliary task in this embodiment, such as Figure 7 As shown, the device includes: an acquisition module 210, a fine-tuning module 220 and a testing module 230;

[0113] An acquisition module 210 is used to acquire the audio to be detected as a test data set;

[0114] The fine-tuning module 220 is used to perform augmentation processing on the test data set based on the preset self-supervised auxiliary task to obtain an augmented data set; based on the augmented data set, the self-supervised auxiliary task is used as the optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and a self-supervised auxiliary task;

[0115] The testing module 230 is used to input the test data set into the target fine-tuning model to obtain the detection result.

[0116] Through the above-mentioned device, the model generalization can be improved while the cost of model training can be greatly reduced; it can also adapt to the distribution differences of deep fake attributes in the test data set, thereby improving cross-domain detection performance; it solves the problem of insufficient generalization and significantly reduced cross-domain detection effect in related technologies.

[0117] In some of the embodiments, the self-supervised auxiliary task includes a task of determining an audio speed regulation state and / or a task of reconstructing an audio decoupling feature encoding task.

[0118] In some of the embodiments, the architecture of the task of determining the audio speed adjustment state includes a common layer and a second classification linear layer;

[0119] The shared layers include feature extractor and Conformer modules;

[0120] The feature extractor, connected to the Conformer module, is used to process the input audio samples and output a series of feature vectors;

[0121] A Conformer module, connected to the second classification linear layer, is used to encode the feature vector and the preset classification indicator vector and output a sequence vector;

[0122] The second classification linear layer is used to process any sequence vector and output the speed adjustment state label.

[0123] In some of the embodiments, the architecture for reconstructing the audio decoupled feature encoding task includes a shared layer, a pre-trained decoupled encoder, and a reconstructed linear layer;

[0124] The shared layers include feature extractor and Conformer modules;

[0125] The feature extractor, connected to the Conformer module, is used to process the input audio samples and output a series of feature vectors;

[0126] The Conformer module and the reconstruction linear layer are used to encode the feature vector and the preset classification indicator vector and output a sequence vector;

[0127] A pre-trained decoupled encoder, connected to a reconstruction linear layer, processes the input audio samples and outputs an acoustic detail code;

[0128] Reconstruction linear layer, which processes sequence vectors and acoustic detail encoding and outputs acoustic reconstruction labels.

[0129] In some of the embodiments, the deep synthetic audio cross-domain detection device based on self-supervised auxiliary tasks further includes a training module;

[0130] A training module, used to construct a target loss function based on a preset auxiliary loss function and a deep fake classification loss function; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function;

[0131] According to the acquired training data set and target loss function, the initial detection model is pre-trained to obtain the target detection model.

[0132] In some of the embodiments, the fine-tuning module 220 is further used to fine-tune the pre-trained target detection model based on the augmented data set and the corresponding auxiliary loss function to obtain a target fine-tuning model; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function.

[0133] In some of the embodiments, both the forgery detection task and the self-supervised auxiliary task are constructed based on a baseline detection neural network architecture; the baseline detection neural network architecture is a Conformer architecture, a Transformer architecture, or a fully connected architecture.

[0134] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0135] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0136] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0137] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0138] S1, obtain the audio to be detected as a test data set;

[0139] S2, based on the preset self-supervised auxiliary task, the test data set is augmented to obtain an augmented data set; based on the augmented data set, the self-supervised auxiliary task is used as the optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and a self-supervised auxiliary task;

[0140] S3, input the test dataset into the target fine-tuning model to obtain the detection results.

[0141] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0142] In addition, in combination with the self-supervised auxiliary task-based deep synthesized audio cross-domain detection method provided in the above embodiments, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the self-supervised auxiliary task-based deep synthesized audio cross-domain detection methods in the above embodiments is implemented.

[0143] It should be noted that the information and data involved in this application are all authorized by the user or fully authorized by all parties and will be used legally.

[0144] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0145] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0146] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0147] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks, characterized in that: include: Get the audio to be detected as a test data set; Based on a preset self-supervised auxiliary task, the test data set is augmented to obtain an augmented data set; Based on the augmented data set, the pre-trained target detection model is fine-tuned by taking the self-supervised auxiliary task as the optimization target to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and the self-supervised auxiliary task; the self-supervised auxiliary task includes a task of determining an audio speed regulation state and / or a task of reconstructing an audio decoupling feature coding; the task of determining an audio speed regulation state is used to learn to identify the audio speed regulation state to capture deep forgery properties related to fundamental frequency changes; the task of reconstructing an audio decoupling feature coding is used to reconstruct and decouple the acoustic detail coding, and learn the deep forgery properties embedded in the acoustic detail coding; The test data set is input into the target fine-tuning model to obtain a detection result.

2. According to claim 1, the method for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks is characterized in that: The architecture of the task of determining the audio speed regulation state includes a common layer and a second classification linear layer; The common layer includes a feature extractor and a Conformer module; The feature extractor is connected to the Conformer module and is used to process the input audio samples and output a series of feature vectors; The Conformer module is connected to the second classification linear layer, and is used to encode the feature vector and a preset classification indication vector, and output a sequence vector; The second classification linear layer is used to process any one of the sequence vectors and output a speed adjustment state label.

3. The method for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks according to claim 1 is characterized in that: The architecture of the reconstructed audio decoupled feature encoding task includes a shared layer, a pre-trained decoupled encoder, and a reconstructed linear layer; The common layer includes a feature extractor and a Conformer module; The feature extractor is connected to the Conformer module and is used to process the input audio samples and output a series of feature vectors; The Conformer module and the reconstruction linear layer are used to encode the feature vector and the preset classification indication vector and output a sequence vector; The pre-trained decoupled encoder is connected to the reconstruction linear layer to process the input audio sample and output the acoustic detail code; The reconstruction linear layer is used to process the sequence vector and the acoustic detail code, and output an acoustic reconstruction label.

4. The method for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks according to claim 1 is characterized in that: The method further comprises: Constructing a target loss function according to a preset auxiliary loss function and a deep fake classification loss function; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function; According to the acquired training data set and the target loss function, the initial detection model is pre-trained to obtain a target detection model.

5. The method for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks according to claim 1 is characterized in that: Based on the augmented dataset, the self-supervised auxiliary task is used as an optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model, including: Based on the augmented data set and the corresponding auxiliary loss function, the pre-trained target detection model is fine-tuned to obtain a target fine-tuning model; the auxiliary loss function includes a three-classification loss function and / or a reconstruction loss function.

6. The method for cross-domain detection of deep synthetic audio based on self-supervised auxiliary tasks according to claim 1, characterized in that: The forgery detection task and the self-supervised auxiliary task are both constructed based on a baseline detection neural network architecture; the baseline detection neural network architecture is a Conformer architecture, a Transformer architecture, or a fully connected architecture.

7. A deep synthesis audio cross-domain detection device based on self-supervised auxiliary tasks, characterized in that: include: Acquisition module, fine-tuning module and testing module; The acquisition module is used to acquire the audio to be detected as a test data set; The fine-tuning module is used to perform augmentation processing on the test data set based on a preset self-supervised auxiliary task to obtain an augmented data set; based on the augmented data set, the self-supervised auxiliary task is used as an optimization target to fine-tune the pre-trained target detection model to obtain a target fine-tuning model; the training task of the target detection model includes a forgery detection task and the self-supervised auxiliary task; the self-supervised auxiliary task includes a task of determining an audio speed regulation state and / or a task of reconstructing an audio decoupling feature coding task; the task of determining an audio speed regulation state is used to learn to identify the audio speed regulation state to capture deep forgery properties related to fundamental frequency changes; the task of reconstructing an audio decoupling feature coding task is used to reconstruct and decouple to obtain acoustic detail coding, and learn deep forgery properties embedded in the acoustic detail coding; The testing module is used to input the testing data set into the target fine-tuning model to obtain a testing result.

8. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the deep synthetic audio cross-domain detection method based on self-supervised auxiliary tasks described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deep synthetic audio cross-domain detection method based on self-supervised auxiliary tasks described in any one of claims 1 to 6 are implemented.

Images