Replay-free domain increment audio forgery detection method and device, equipment and medium

Abstract

The invention provides a domain incremental audio forgery detection method and device without playback, equipment and a medium, and relates to the field of audio detection, and the domain incremental audio forgery detection method comprises the steps: obtaining an original pre-training model backbone network; receiving the audio data field and preprocessing the audio data field to obtain at least one one-dimensional feature sequence; distributing a domain exclusive adapter and a domain exclusive asymmetric classifier for the one-dimensional feature sequence, performing training to obtain a target model, and learning prototype representation of the audio data domain at the current stage; and processing the target model through a soft routing technology and a dynamic merging technology to obtain an instance-level inference model. According to the method and the device, the domain exclusive parameters are subjected to weight recombination by using the probability obtained by soft routing prediction to obtain the instance-level exclusive parameters, so that the robustness of the model on error routing prediction is improved; and various factors such as data memory, storage load and detection precision existing in domain incremental audio forging detection can be effectively balanced.

Description

Technical Field

[0001] The present application relates to the field of audio detection technology, and in particular to a method, device, equipment and medium for detecting domain incremental audio forgery without replay. Background Art

[0002] With the rapid development of generation technology, the audio generated by speech synthesis and speech conversion technology is becoming more and more realistic. This technology has brought positive impacts but also brought security risks. Therefore, it is of great significance to carry out work on audio forgery detection. Voice forgery detection is usually based on a fixed data set, but in real scenarios, new forgery methods will continue to emerge with the advancement of technology. In addition, due to the diversity of real environments and collection equipment, there are certain deviations between real data. Therefore, when a detection model with high performance on a fixed data set faces these data from the new data distribution domain, it often encounters a more obvious performance degradation.

[0003] In the related art, in order to make the forgery detection model adapt to the continuously arriving data from the new data distribution domain; on the one hand, the new data can be used to fine-tune the old model, but the old model will forget the past knowledge after fine-tuning, resulting in a sharp decline in the performance of the model for the old data; the forgetting problem can be alleviated by utilizing the replay buffer to store the old data of the previous stage and replaying it in the subsequent incremental stage; on the other hand, in the field of speech forgery detection, the regularization-based continuous learning method adds an additional loss regularization term to the training loss, and utilizes the loss to constrain the update of the weights, thereby retaining the knowledge of the old task.

[0004] However, the replay-based incremental learning method requires the use of a replay buffer, which creates an additional storage burden and has significant limitations. The current regularization-based continuous learning method does not need to store old data, but it uses the same model parameters for different data domains that arrive sequentially. The model will produce poor results when faced with situations where the distribution of different data domains that arrive sequentially is significantly different. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method, device, equipment and medium for detecting domain incremental audio forgery without the need for replay, which solves the problem of difficulty in balancing various factors such as data memory, storage load and detection accuracy in domain incremental audio forgery detection when facing the continuous arrival of new data distribution domains.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a domain-incremental audio forgery detection method without replay, and the domain-incremental audio forgery detection method includes: obtaining an original pre-trained model backbone network, the pre-trained model backbone network is a convolution-free, attention-based audio classification network model that does not include any domain-specific parameters; receiving an audio data domain that arrives sequentially, and pre-processing the audio data domain to obtain at least one one-dimensional feature sequence; assigning a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model; based on the pre-trained model backbone network, learning a prototype representation of the audio data domain at the current stage from the one-dimensional feature sequence training to support instance-level soft routing selection during inference; processing the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio forgery identification.

[0008] According to the first aspect of the embodiment of the present application, the aforementioned receiving of the audio data domain that arrives sequentially and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence includes: based on a preset window length and frame shift, processing the audio data domain that arrives sequentially through a Hamming window to obtain a Mel-spectrogram; dividing the Mel-spectrogram into image blocks with a preset number and size, and overlapping them in the time dimension and the frequency dimension to obtain a first image sequence; expanding the first image sequence into a one-dimensional embedded feature through a linear mapping layer, and adding an output class feature at the starting position of the one-dimensional embedded feature to obtain a second image sequence; adding a trainable position encoding feature to each element of the second image sequence to obtain a one-dimensional feature sequence.

[0009] According to a first aspect of an embodiment of the present application, the domain-specific adapter is a convolutional adapter and includes multiple adapter layers, each adapter layer includes a linear downward projection layer, a two-dimensional convolution layer, a Gaussian error linear unit layer, and a linear upward projection layer; in a one-dimensional feature sequence, features other than output class features are reshaped into a two-dimensional structure before being input into the two-dimensional convolution layer.

[0010] According to the first aspect of the embodiment of the present application, the domain-specific asymmetric classifier has real audio class center parameters and fake audio class center parameters, the real audio class center parameters are shared parameters between different audio domains, and the fake audio class center parameters are exclusive parameters of each audio domain.

[0011] According to the first aspect of an embodiment of the present application, the intermediate model includes 12 transformer encoders, each transformer encoder includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, a multi-layer perceptron layer and an adapter layer; in each transformer encoder, the adapter layer runs in parallel with the multi-head self-attention layer and the multi-layer perceptron layer.

[0012] According to the first aspect of the embodiment of the present application, the aforementioned one-dimensional feature sequence is assigned a domain-specific adapter and a domain-specific asymmetric classifier to obtain an intermediate model, and the intermediate model is trained based on the one-dimensional feature sequence to obtain a target model, including: processing the one-dimensional feature sequence in sequence through 12 transformer encoders, and assigning a domain-specific adapter to obtain a target feature; assigning a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on a pre-trained model backbone network; optimizing the intermediate model through a cross entropy loss function to obtain a target model; wherein one transformer encoder corresponds to one processing stage.

[0013] According to the first aspect of the embodiment of the present application, the aforementioned target model includes forged audio class center parameters and domain-specific adapters corresponding to multiple audio data domains; the aforementioned target model is processed by soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication, including: using soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain the routing prediction probability; based on the forged audio class center parameters and the routing prediction probability, determining the forged audio class center weight by parameter merging; dynamically merging the domain-specific adapters and routing prediction probabilities corresponding to multiple audio data domains to determine the instance-level adapter; dynamically merging the forged audio class center weight and the real audio class center weight shared by different audio data domains to determine the instance-level asymmetric classifier; merging the original pre-trained model backbone network, the instance-level reorganization adapter and the instance-level asymmetric classifier to obtain the instance-level inference model for audio authentication.

[0014] In the second aspect, an embodiment of the present application provides a domain-incremental audio forgery detection device that does not require replay, and the domain-incremental audio forgery detection device includes an acquisition module, a preprocessing module, a training module, a learning module and a processing module; the acquisition module is used to acquire the original pre-trained model backbone network, and the pre-trained model backbone network is a convolution-free, attention-based audio classification network model that does not include any domain-specific parameters; the pre-processing module is used to receive audio data domains that arrive sequentially, and pre-process the audio data domains to obtain at least one one-dimensional feature sequence; the training module is used to assign domain-specific adapters and domain-specific asymmetric classifiers to the one-dimensional feature sequence to obtain an intermediate model, and train the intermediate model based on the one-dimensional feature sequence to obtain a target model; the learning module is used to train and learn a prototype representation of the audio data domain at the current stage from the one-dimensional feature sequence based on the pre-trained model backbone network to support instance-level soft routing selection during inference; the processing module is used to process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for detecting domain incremental audio forgery without replay in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for detecting domain incremental audio forgery without replay in the first aspect is implemented.

[0017] The present invention provides a domain incremental audio forgery detection method, device, equipment and medium without replay. Compared with the prior art, it has the following beneficial effects:

[0018] The present application is capable of receiving audio data domains that arrive sequentially and converting the audio data domains into one-dimensional feature sequences. In the process of training the pre-trained model backbone network, domain-specific adapters and domain-specific asymmetric classifiers are allocated to the one-dimensional feature sequences. Domain-specific knowledge is stored through domain-specific adapters and domain-specific asymmetric classifiers. While solving the problem of catastrophic forgetting, the need for old data is avoided. The asymmetric classifier avoids the problem of inconsistency in prediction probability distribution that may be caused by the use of multiple independent classifiers and the problem of catastrophic forgetting that may be caused by the use of a single shared classifier. At the same time, the present application processes the target model through soft routing technology and dynamic merging technology, thereby enhancing the model's error correction capability for erroneous routing selection and realizing knowledge sharing of different domain-specific parameters. No replay is required and the network architecture can be dynamically adjusted, which helps to solve the problem of identifying fake audio data generated by new forgery methods that continue to emerge in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a domain incremental audio forgery detection method without replay provided in an embodiment of the present application;

[0021] Figure 2 yes Figure 1 An exemplary flow chart of S120;

[0022] Figure 3 It is a flowchart of an audio data domain processing provided by an embodiment of the present application;

[0023] Figure 4 is a structural schematic diagram of a convolution adapter provided in an embodiment of the present application;

[0024] Figure 5 It is a structural schematic diagram of a domain incremental audio forgery detection device without replay provided in an embodiment of the present application;

[0025] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0028] The embodiments of the present application solve the problem of difficulty in balancing various factors such as data memory, storage load and detection accuracy in domain incremental audio forgery detection when facing the continuous arrival of new data distribution domains, by providing a domain incremental audio forgery detection method, device, equipment and medium that does not require replay.

[0029] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0030] With the rapid development of generation technology, the audio generated by speech synthesis and speech conversion technology is becoming more and more realistic. While this technology has brought positive impacts to human life, it has also brought security risks. Therefore, it is of great significance to carry out work on audio authentication.

[0031] Existing speech forgery detection methods are usually based on a fixed data set. However, in real scenarios, new forgery methods will continue to emerge with the advancement of technology. In addition, due to the diversity of real environments and collection equipment, there are certain deviations between real data. Therefore, when a detection model with high performance on a fixed data set faces data from new data distribution domains, it often encounters a more obvious performance degradation.

[0032] Therefore, an ideal forgery detection model needs to be able to adapt to the data that is constantly arriving from the new data distribution domain; there are two simple and intuitive ways to solve this problem. One solution is to retrain a model on the old and new data, but this method is time-consuming and takes up a lot of storage space, and in most cases, the old data may not be available due to data privacy issues. Another more intuitive solution is to fine-tune the old model only with the new data, but after fine-tuning, the old model will forget the past knowledge, which will cause the model's performance on the old data to drop sharply, that is, the catastrophic forgetting problem occurs, and this problem becomes more and more serious with the increase in the number of fine-tuning times.

[0033] In order to alleviate the problem of catastrophic forgetting, incremental learning methods are used by many researchers. Incremental learning methods enable the model to avoid forgetting old tasks while sequentially learning each new task. Replay-based incremental learning methods generally use a replay buffer to store old data from previous stages for replay in subsequent incremental stages to alleviate catastrophic forgetting. Since this method requires the use of a replay buffer, it will cause additional storage burden. In addition, for the task of voice forgery detection, its old data may not be available due to privacy issues. Therefore, for the task of voice forgery detection, the replay-based continuous learning method is not the best choice and has some limitations. Regularization-based continuous learning methods retain the knowledge of old tasks by adding an additional loss regularization term to the training loss and using the loss to constrain the update of weights. At present, there are some regularization-based continuous learning methods in the field of voice forgery detection. Although these methods do not need to store old data, they use the same model parameters for different data domains that arrive sequentially, which makes the model produce poor results when facing the situation where the distribution of different data domains that arrive sequentially is very different.

[0034] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] The following first introduces a domain incremental audio forgery detection method without replay provided in an embodiment of the present application.

[0036] A flowchart of a domain incremental audio forgery detection method without replay is provided in an embodiment of the present application, such as Figure 1 As shown, the domain incremental audio forgery detection method without replay may include the following steps S110-S150.

[0037] S110. Obtain the original pre-trained model backbone network, where the pre-trained model backbone network is a non-convolutional audio classification network model based on an attention mechanism and does not include any domain-specific parameters.

[0038] In the embodiments of the present application, it can be understood that before preprocessing the audio data domain to obtain a one-dimensional feature sequence, the present application obtains the original pre-trained model backbone network. The pre-trained model backbone network is usually trained on large-scale data. The pre-trained model backbone network has learned rich common semantic information and patterns and has good generalization ability. Good generalization ability can enhance the model's anti-forgetting ability to a certain extent, so the present application adopts a pre-trained model backbone network.

[0039] S120, receiving the audio data domain that arrives sequentially, and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence.

[0040] It should be noted that each audio data domain that arrives sequentially contains a certain number of audios, and each audio can be processed into a one-dimensional feature sequence. Therefore, one audio data domain can be processed to obtain one or more one-dimensional feature sequences.

[0041] S130, allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model.

[0042] In the embodiments of the present application, it can be understood that a simple and effective way to solve the problem of catastrophic forgetting is to assign a set of domain-specific parameters to the audio data domain of the current stage that arrives sequentially. These domain-specific parameters will be optimized on the audio data domain to model and learn the data distribution of the current stage.

[0043] At the same time, the other parameters of the pre-trained model backbone network remain fixed, achieving knowledge sharing across data domains. The convolution adapter performs comparable to the fully fine-tuned model with sufficient training data, and may even perform better when data is scarce. In addition, the method using domain-specific adapters requires fewer training parameters. This method based on domain-specific adapters can not only reduce parameters, but also effectively alleviate the dependence on the scale of training data.

[0044] S140. Based on the pre-trained model backbone network, a prototype representation of the audio data domain of the current stage is learned from the one-dimensional feature sequence training to support instance-level soft routing selection during inference.

[0045] S150, processing the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

[0046] In the embodiments of the present application, it can be understood that the present application stores domain-specific knowledge through domain-specific adapters and domain-specific asymmetric classifiers, which not only solves the problem of catastrophic forgetting, but also avoids the need for old data. The asymmetric classifier avoids the problem of inconsistency in prediction probability distribution that may be caused by using multiple independent classifiers and the problem of catastrophic forgetting that may be caused by using a single shared classifier.

[0047] It should be emphasized that the present application uses a soft routing technology based on learnable domain prototypes for route selection, and uses dynamic reconstruction technology to enhance the model's error correction capability for erroneous route selection and realize knowledge sharing of parameters specific to different domains; in addition, the present application uses an asymmetric classifier to avoid the inconsistency problem of predicted probability distribution that may be caused by the use of multiple independent classifiers and the catastrophic forgetting problem that may be caused by the use of a single shared classifier; it helps to solve the problem of identifying fake audio data generated by new forgery methods that continue to emerge in real scenarios.

[0048] In some embodiments, please refer to Figure 2 , the aforementioned receiving the audio data domain that arrives in sequence, and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence, that is, S120 may specifically include the following steps:

[0049] S210 , based on a preset window length and frame shift, the sequentially arriving audio data domain is processed through a Hamming window to obtain a Mel-spectrogram.

[0050] S220 , dividing the mel-spectrogram into image blocks with a preset number and size, and overlapping them in the time dimension and the frequency dimension to obtain a first image sequence.

[0051] S230, expanding the first image sequence into a one-dimensional embedding feature through a linear mapping layer, and adding an output class feature at the beginning position of the one-dimensional embedding feature to obtain a second image sequence.

[0052] S240. Add a trainable position encoding feature to each element of the second image sequence to obtain a one-dimensional feature sequence.

[0053] For details, please refer to Figure 3, for an audio data domain with a duration of t seconds, firstly, a Hamming window with a window length of 25ms is used and the frame shift is set to 10ms to obtain a Mel spectrogram with a size of 128×100t; then the obtained Mel spectrogram is divided into N image blocks with a size of 16×16, and overlapped by 6 in the time and frequency dimensions to obtain the first image sequence; wherein, Then, each 16X16 image block in the first image sequence is expanded into a one-dimensional embedding feature of size 768 through a linear mapping layer, and an output class feature cls is added at the beginning of the image block feature sequence. This feature can be regarded as an abstract representation of the entire image block feature sequence, which will be used as the input of the classifier; finally, a trainable position encoding feature is added to each one-dimensional embedding feature, so that a one-dimensional feature sequence Z input to the transformer encoder can be obtained. The one-dimensional feature sequence Z can be expressed as: Z = {z1, z2, ..., z N ,z cls},in, d is the embedding dimension, N is the total number of image patches, and cls represents the output class feature.

[0054] In some embodiments, please refer to Figure 3 , the intermediate model includes 12 transformer encoders, each transformer encoder includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, a multi-layer perceptron layer, and an adapter layer; in each transformer encoder, the adapter layer runs in parallel with the multi-head self-attention layer and the multi-layer perceptron layer. It can be understood that the first normalization layer and the second normalization layer are only used to represent different entities; specifically, the pre-trained model backbone network is a ViT / B-16 model with 12 Transformer encoders, and the model is initialized using pre-trained parameters on the ImageNet-21k dataset.

[0055] In one example, the one-dimensional feature sequence Z is processed by 12 Transformer encoders in sequence, where the processing of one Transformer encoder can be expressed as

[0056] Formula 1 is used to express:

[0057]

[0058] Among them, MHSA represents the multi-head self-attention layer, MLP represents the multi-layer perceptron layer, LN represents the layer normalization layer, which includes the first normalization layer and the second normalization layer mentioned above, l represents the number of layers, and Z l is the feature extracted from the lth layer.

[0059] In some embodiments, the aforementioned method allocates a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and trains the intermediate model based on the one-dimensional feature sequence to obtain a target model, that is, S130 may specifically include the following steps:

[0060] S310, processing the one-dimensional feature sequence in sequence through 12 transformer encoders, and assigning domain-specific adapters to obtain target features; wherein one transformer encoder corresponds to one processing stage.

[0061] In one example, assume that there are T audio data domains D = {(D1, ..., D T )} will arrive sequentially, the audio data domain of the current stage in, Indicates audio input, Indicates the label corresponding to the audio input, n t Indicates the current audio data domain D t The application uses a domain-specific adapter A consisting of L adapter layers. t For the current stage audio data domain D t For modeling, the parameters of the domain-specific adapter are The domain-specific adapter At is represented by Expression 2:

[0062]

[0063] In another example, see Figure 4 The domain-specific adapter is a convolutional adapter and includes multiple adapter layers, each of which includes a linear down-projection layer MLP down , a two-dimensional convolution layer Conv2D, a Gaussian error linear unit layer GELU, and a linear upward projection layer MLP up .

[0064] In the input one-dimensional feature sequence Z = {z1,z2,...,z N ,z cls}, except for the output class feature z cls In addition, the remaining input features {z1,z2,...,z N} will be reshaped into a two-dimensional structure before performing a two-dimensional convolution operation. Please refer to Figure 3 , the adapter layer will be inserted into the pre-trained model backbone network at the corresponding position of each multi-head self-attention layer MHSA and multi-layer perceptron layer MLP. The process of the adapter layer processing the one-dimensional feature sequence Z can be expressed by Expression 3:

[0065]

[0066] Domain-specific adapters in the lth adapter layer The multi-head self-attention layer MHSA layer and the multi-layer perceptron layer MLP layer of each transformer encoder run in parallel. The operation process can be expressed by Expression 4:

[0067]

[0068] S320, assigning a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on the pre-trained model backbone network.

[0069] In the embodiment of the present application, it can be understood that the feature distribution of real audio is compact, while the feature distribution of forged audio is scattered across different data domains. Therefore, the present invention uses an asymmetric classifier to treat real audio from different domains as the same category, while treating forged audio from different domains as independent categories. This avoids the problem of inconsistency in predicted probability distribution that may be caused by using multiple independent classifiers and the problem of catastrophic forgetting that may be caused by using a single shared classifier.

[0070] In one example, the domain-specific asymmetric classifier has real audio class center parameters and fake audio class center parameters, where the real audio class center parameters are shared parameters between different audio domains, and the fake audio class center parameters are exclusive parameters of each audio domain.

[0071] Specifically, the present application is to assign a domain-specific asymmetric classifier to the target feature obtained based on the one-dimensional feature sequence Its parameters include θ r and θ ft , where θ r represents the real audio class center parameter shared between different audio data domains, θ ft Indicates the audio data domain D of the current stage t Exclusive forged audio class center parameter; It can be understood that the asymmetric classifier up to the current stage t can be expressed as: f c ={θ r ,θ f1 ,...,θ ft}. During the training process, the asymmetric classifier at the current stage t The output probability It can be expressed by Expression 5:

[0072]

[0073] in, Yes Audio Input After 12 embedded domain-specific adapters A t The target features after the transformer encoder.

[0074] It should be emphasized that the aforementioned intermediate model corresponds to the model to be trained at the current stage f curr , f curr The parameters include the pre-trained model backbone network parameters ω, the domain-specific adapter parameters of the current stage Real audio class center parameter θ r and the forged audio class center parameter θ ft ; Among them, domain-specific adapter parameters Real audio class center parameter θ r and the forged audio class center parameter θ ft can be updated, and the pre-trained model backbone network parameters ω are in a frozen state.

[0075] It should also be emphasized that, in addition, the domain-specific adapter A at the current stage t The domain-specific adapter A from the previous stage will be used t-1 Initialize the parameters of the domain; the current stage domain-specific asymmetric classifier The real audio class center parameter θ in r The real audio class center parameter θ from the previous stage will be used r , and the forged audio class center parameter θ ft will be randomly initialized; this setting ensures that the domain-specific parameters can focus on the data domain at the current stage, which helps to alleviate catastrophic forgetting.

[0076] S330. Optimize the intermediate model through the cross entropy loss function to obtain the target model.

[0077] In one example, the cross entropy loss function can be expressed by Expression 6:

[0078]

[0079] in, Represents an asymmetric classifier The output probability of Indicates the label corresponding to the audio input, 1≤i≤n.

[0080] In some embodiments, regarding the aforementioned pre-trained model backbone network, a prototype representation of the audio data domain of the current stage is learned from the one-dimensional feature sequence training to support instance-level soft routing selection during inference, that is, S140. It can be understood that since the present application is for each audio data domain D that arrives sequentially t Domain-specific parameters are set, so this requires the model to know the probabilistic relationship between the test instance and different audio data domains during inference; this application learns the prototype representation C = {c1,...,c t} to solve this problem, that is, a randomly initialized domain prototype representation type c t The current audio data domain D t In the original pre-trained model backbone network f ast Align the features on the

[0081] In one example, assume that the audio in the current stage audio data domain that arrives sequentially is in the pre-trained model backbone network f ast The high-level feature space on has a similar distribution, so in the current stage, the audio data domain D t There must be a feature with the output class Aligned domain prototypes Specifically, this application minimizes the cross entropy loss L proto To make the domain prototype c t Close to the current audio data field D t The feature center of the minimization cross entropy loss L proto It can be expressed by Expression 7:

[0082]

[0083] Among them, f ast represents the pre-trained model backbone network, Indicates the current stage audio data domain D t The i-th audio input on n t Indicates the current audio data domain D t The total number of audio frequencies, τ is the temperature coefficient, is the cosine similarity.

[0084] During model inference, this application uses soft routing technology to predict the test instance and different audio domain prototypes C = {c1,...,c t Specifically, for a test sample x, the present application calculates the probability between it and T audio domain prototypes The calculation process is shown in Expression 8:

[0085]

[0086] In some embodiments, the target model includes a forged audio class central parameter and domain-specific adapters corresponding to multiple audio data domains; the target model is processed by the soft routing technology and the dynamic merging technology to obtain an instance-level inference model for audio authentication, that is, S150 may specifically include the following steps:

[0087] S410: Use soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain a routing prediction probability.

[0088] S420: Based on the forged audio class center parameter and the route prediction probability, determine the forged audio class center weight by parameter merging.

[0089] S430: Dynamically merge domain-specific adapters and routing prediction probabilities corresponding to multiple audio data domains to determine an instance-level adapter.

[0090] S440: Dynamically merge the forged audio class center weights and the real audio class center weights shared by different audio data domains to determine an instance-level asymmetric classifier.

[0091] S450, merging the original pre-trained model backbone network, the instance-level reorganization adapter, and the instance-level asymmetric classifier to obtain an instance-level inference model for audio authentication.

[0092] In the embodiments of the present application, it can be understood that relevant research shows that it is effective to merge multiple model parameters trained on different data domains into one model, and the merged model can be generalized to all data domains, thereby effectively avoiding the need for the domain to which the test instance belongs, and enabling knowledge sharing between different domains. Unlike the existing model merging technology that generates a single fixed model for all test samples, the present application tailors an instance-level model for each test instance, providing more powerful performance. In order to improve the robustness of the model to erroneous route predictions and to achieve knowledge sharing of domain-specific parameters of different domains, the present application designs a parameter dynamic merging technology, which obtains instance-level specific parameters by weighting domain-specific parameters using the probability obtained from soft route predictions.

[0093] Specifically, the adapter weight A obtained using the parameter merging technique m and the forged audio class center weight θ fm The process is shown in Expression 9:

[0094]

[0095] Where A={A1,...,A T}、θ f ={θ f1 ,...,θ ft}and They are the domain-specific adapters for the T audio data domains that arrive sequentially, the forged audio class center parameters, and the routing prediction probability of the test instance.

[0096] Use the merged fake audio class center weight θ fm The real audio class center weight θ shared with the domain t Get an instance-level asymmetric classifier f cm , and combine instance-level adapter A mWith the pre-trained model backbone network f ast Merge to obtain the instance-level inference model f instence To authenticate the audio.

[0097] Based on this, the dynamic merging technology used in this application enhances the robustness of the routing algorithm to erroneous domain predictions. Since the dynamic merging technology used will merge the exclusive parameters of multiple data domains, even if the routing algorithm produces some erroneous data domain predictions, the performance of the model will not be significantly reduced. In addition, the dynamic merging technology promotes the sharing of domain-specific knowledge through parameter space merging, especially when multiple data domains come from similar fields, the performance of this merging technology is more superior.

[0098] In some embodiments, please refer to Figure 5 The present application also provides a domain incremental audio forgery detection device 500 without replaying, and the domain incremental audio forgery detection device 500 may include the following modules:

[0099] The acquisition module 510 is used to acquire the original pre-trained model backbone network, which is a non-convolutional audio classification network model based on the attention mechanism and does not include any domain-specific parameters.

[0100] The preprocessing module 520 is used to receive the audio data domain that arrives sequentially, and preprocess the audio data domain to obtain at least one one-dimensional feature sequence.

[0101] The training module 530 is used to assign a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and train the intermediate model based on the one-dimensional feature sequence to obtain a target model.

[0102] The learning module 540 is used to learn a prototype representation of the audio data domain of the current stage from a one-dimensional feature sequence training based on a pre-trained model backbone network to support instance-level soft routing selection during inference.

[0103] The processing module 550 is used to process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

[0104] According to an embodiment of the present application, any multiple modules among the acquisition module 510, the preprocessing module 520, the training module 530, the learning module 540 and the processing module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0105] In some embodiments, the preprocessing module 520 can be specifically used to: based on a preset window length and frame shift, process the sequentially arriving audio data domain through a Hamming window to obtain a Mel-spectrogram; divide the Mel-spectrogram into image blocks with a preset number and size, and overlap them in the time dimension and frequency dimension to obtain a first image sequence; expand the first image sequence into a one-dimensional embedded feature through a linear mapping layer, and add an output class feature at the starting position of the one-dimensional embedded feature to obtain a second image sequence; add a trainable position encoding feature to each element of the second image sequence to obtain a one-dimensional feature sequence.

[0106] In some embodiments, the training module 530 can be specifically used to: process the one-dimensional feature sequence in sequence through 12 transformer encoders, and assign domain-specific adapters to obtain target features; wherein one transformer encoder corresponds to one processing stage; assign a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on a pre-trained model backbone network; optimize the intermediate model through a cross-entropy loss function to obtain a target model.

[0107] In some embodiments, the processing module 550 can be specifically used to: use soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain the routing prediction probability; determine the forged audio class center weight by parameter merging based on the forged audio class center parameters and the routing prediction probability; dynamically merge domain-specific adapters and routing prediction probabilities corresponding to multiple audio data domains to determine the instance-level adapter; dynamically merge the forged audio class center weight and the real audio class center weight shared by different audio data domains to determine the instance-level asymmetric classifier; merge the original pre-trained model backbone network, the instance-level reorganization adapter and the instance-level asymmetric classifier to obtain an instance-level inference model for audio authentication.

[0108] Figure 5 Each module in the device shown has the function of implementing each step in the aforementioned domain incremental audio forgery detection method without replay, and can achieve its corresponding technical effect. For the sake of concise description, it will not be repeated here.

[0109] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 6 shown.

[0110] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0111] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0112] The memory 620 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 620 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 620 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid-state memory.

[0113] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, typically, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the domain incremental audio forgery detection methods without replay in the above-mentioned embodiments.

[0114] The processor 610 implements any one of the domain incremental audio forgery detection methods without replay in the above embodiments by reading and executing computer program instructions stored in the memory 620 .

[0115] In one example, the electronic device may further include a communication interface 630 and a bus 600. Figure 6 As shown, the processor 610, the memory 620, and the communication interface 630 are connected via a bus 600 and communicate with each other.

[0116] The communication interface 630 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0117] Bus 600 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 600 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0118] In addition, in combination with the domain incremental audio forgery detection method without replay in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the domain incremental audio forgery detection methods without replay in the above embodiment is implemented.

[0119] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0120] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0121] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0122] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0123] In summary, compared with the prior art, this application has the following beneficial effects:

[0124] 1. This application sets up domain-specific adapters and domain-specific asymmetric classifiers in the pre-trained model backbone network to store domain-specific knowledge, which solves the catastrophic forgetting problem while also avoiding the need for old data; the adapter-based pre-trained model fine-tuning method greatly reduces the scale of parameter learning and reduces the model's requirements for the scale of training data, so that it can obtain better performance even when only a small amount of training data is provided.

[0125] 2. This application uses an asymmetric classifier, which regards real audio from different domains as the same category, and forged audio from different domains as independent categories. This design avoids the problem of inconsistency in predicted probability distribution that may be caused by using multiple independent classifiers and the problem of catastrophic forgetting that may be caused by using a single shared classification.

[0126] 3. This application adopts a soft routing technology based on learnable domain prototypes for route selection, and also adopts a parameter dynamic merging technology, that is, by using the probability obtained by soft routing prediction to reorganize the weights of domain-specific parameters to obtain instance-level specific parameters, thereby improving the model's robustness to erroneous route predictions and realizing knowledge sharing of different domain-specific parameters.

[0127] 4. The present application provides a domain incremental audio forgery detection method that does not require replay and can dynamically adjust the network architecture. When faced with the continuous arrival of new data distribution domains, it can effectively balance various factors such as data memory, storage load and detection accuracy in domain incremental audio forgery detection.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain incremental audio forgery detection method without replay, characterized in that: include: Obtain an original pre-trained model backbone network, where the pre-trained model backbone network is a non-convolutional, attention-based audio classification network model that does not include any domain-specific parameters; Receiving audio data domains that arrive sequentially, and preprocessing the audio data domains to obtain at least one one-dimensional feature sequence; Allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model; Based on the pre-trained model backbone network, a prototype representation of the audio data domain at the current stage is learned from the one-dimensional feature sequence training to support instance-level soft routing selection during inference; The target model is processed by soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

2. The domain incremental audio forgery detection method without replay as claimed in claim 1, characterized in that: The receiving of the audio data domain that arrives sequentially and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence includes: Based on the preset window length and frame shift, the sequentially arriving audio data domain is processed through the Hamming window to obtain a Mel spectrogram; The mel-spectrogram is divided into image blocks with a preset number and size, and overlapped in the time dimension and the frequency dimension to obtain a first image sequence; Expanding the first image sequence into a one-dimensional embedding feature through a linear mapping layer, and adding an output class feature at the beginning position of the one-dimensional embedding feature to obtain a second image sequence; A trainable position encoding feature is added to each element of the second image sequence to obtain a one-dimensional feature sequence.

3. The domain incremental audio forgery detection method without replay as claimed in claim 1 or 2, characterized in that: The domain-specific adapter is a convolution adapter and includes multiple adapter layers, each of which includes a linear down-projection layer, a two-dimensional convolution layer, a Gaussian error linear unit layer, and a linear up-projection layer; in the one-dimensional feature sequence, features other than the output class features are reshaped into a two-dimensional structure before being input into the two-dimensional convolution layer.

4. The domain incremental audio forgery detection method without replay as claimed in claim 1 or 2, characterized in that: The domain-specific asymmetric classifier has a real audio class center parameter and a fake audio class center parameter, wherein the real audio class center parameter is a shared parameter between different audio domains, and the fake audio class center parameter is a dedicated parameter of each audio domain.

5. The domain incremental audio forgery detection method without replay as claimed in claim 1 or 2, characterized in that: The intermediate model includes 12 transformer encoders, each of which includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, a multi-layer perceptron layer and an adapter layer; in each transformer encoder, the adapter layer runs in parallel with the multi-head self-attention layer and the multi-layer perceptron layer.

6. The domain incremental audio forgery detection method without replay as claimed in claim 1 or 2, characterized in that: The step of allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model includes: Processing the one-dimensional feature sequence in sequence through 12 transformer encoders and assigning domain-specific adapters to obtain target features; wherein one transformer encoder corresponds to one processing stage; Assigning a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on the pre-trained model backbone network; The intermediate model is optimized by a cross entropy loss function to obtain a target model.

7. The domain incremental audio forgery detection method without replay as claimed in claim 1 or 2, characterized in that: The target model includes forged audio class center parameters and domain-specific adapters corresponding to multiple audio data domains; The target model is processed by the soft routing technology and the dynamic merging technology to obtain an instance-level inference model for audio authentication, including: Use soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance, and obtain the routing prediction probability; Based on the forged audio class center parameter and the route prediction probability, determining the forged audio class center weight by parameter merging; Dynamically merge the domain-specific adapters and the routing prediction probabilities corresponding to multiple audio data domains to determine an instance-level adapter; Dynamically merging the forged audio class center weights and the real audio class center weights shared by different audio data domains to determine an instance-level asymmetric classifier; The original pre-trained model backbone network, instance-level reorganization adapter and instance-level asymmetric classifier are merged to obtain an instance-level inference model for audio authentication.

8. A domain incremental audio forgery detection device without replay, characterized in that: include: An acquisition module, used to acquire an original pre-trained model backbone network, where the pre-trained model backbone network is a non-convolutional audio classification network model based on an attention mechanism and does not include any domain-specific parameters; A preprocessing module, used for receiving the audio data domain that arrives sequentially, and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence; A training module, used for allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model; A learning module, configured to learn a prototype representation of the audio data domain at a current stage from the one-dimensional feature sequence training based on the pre-trained model backbone network, so as to support instance-level soft routing selection during inference; The processing module is used to process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for detecting domain incremental audio forgery without replay as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the domain incremental audio forgery detection method without replay as described in any one of claims 1 to 7 is implemented.

Images