Domain incremental audio forgery detection method, device, equipment and medium without replay

Abstract

The present application provides a domain-incremental audio forgery detection method, apparatus, device and medium that does not require replay, and relates to the field of audio detection. The domain-incremental audio forgery detection method includes obtaining an original pre-trained model backbone network; receiving an audio data domain and pre-processing it to obtain at least one one-dimensional feature sequence; allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence and training it to obtain a target model, learning the prototype representation of the audio data domain at the current stage; processing the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model. The present application improves the robustness of the model to erroneous routing predictions by reorganizing the weights of domain-specific parameters using the probability obtained from soft routing predictions, and can effectively balance various factors such as data memory, storage load and detection accuracy in domain-incremental audio forgery detection when facing continuously arriving new data distribution domains.

Description

Technical Field

[0001] The present application relates to the field of audio detection technology, and in particular to a method, apparatus, device and medium for detecting domain incremental audio forgery without the need for replay. Background Art

[0002] With the rapid development of generative technology, the audio generated by speech synthesis and voice conversion technologies is becoming increasingly realistic. While this technology has brought positive impacts, it also poses security risks. Therefore, the research on audio forgery detection is of great significance. Speech forgery detection is usually based on a fixed dataset. However, in real-world scenarios, new forgery methods continue to emerge with technological advancements. In addition, due to the diversity of real-world environments and collection equipment, there are certain deviations between real data. Therefore, when detection models that have high performance on fixed datasets are faced with data from new data distribution domains, they often experience significant performance degradation.

[0003] In related technologies, in order to enable the forgery detection model to adapt to the continuously arriving data from the new data distribution domain; on the one hand, the old model can be fine-tuned using new data, but the old model will forget the past knowledge after fine-tuning, resulting in a sharp decline in the model's performance on the old data; the forgetting problem is alleviated by utilizing a replay buffer to store old data from the previous stage and replaying it in subsequent incremental stages; on the other hand, in the field of speech forgery detection, the regularization-based continuous learning method adds an additional loss regularization term to the training loss, and uses this loss to constrain the update of the weights, thereby retaining the knowledge of the old task.

[0004] However, the replay-based incremental learning method requires the use of a replay buffer, which creates an additional storage burden and has significant limitations. The current regularization-based continuous learning method does not require the storage of old data, but uses the same model parameters for different data domains that arrive sequentially. This results in poor performance when faced with data domains that arrive sequentially and have large distribution differences. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a domain incremental audio forgery detection method, device, equipment and medium that does not require replay, which solves the problem of difficulty in balancing various factors such as data memory, storage load and detection accuracy in domain incremental audio forgery detection when facing the continuous arrival of new data distribution domains.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] In the first aspect, an embodiment of the present application provides a domain-incremental audio forgery detection method that does not require replay, and the domain-incremental audio forgery detection method includes: obtaining the original pre-trained model backbone network, which is a non-convolutional, attention-based audio classification network model that does not include any domain-specific parameters; receiving audio data domains that arrive sequentially, and preprocessing the audio data domains to obtain at least one one-dimensional feature sequence; assigning domain-specific adapters and domain-specific asymmetric classifiers to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model; based on the pre-trained model backbone network, learning a prototype representation of the audio data domain at the current stage from the one-dimensional feature sequence training to support instance-level soft routing selection during inference; processing the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio forgery authentication.

[0008] According to the first aspect of the embodiment of the present application, the aforementioned receiving audio data domain that arrives sequentially and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence includes: based on a preset window length and frame shift, processing the audio data domain that arrives sequentially through a Hamming window to obtain a Mel-spectrogram; dividing the Mel-spectrogram into image blocks with a preset number and size, and overlapping them in the time dimension and frequency dimension to obtain a first image sequence; expanding the first image sequence into a one-dimensional embedding feature through a linear mapping layer, and adding an output class feature at the starting position of the one-dimensional embedding feature to obtain a second image sequence; adding a trainable position encoding feature to each element of the second image sequence to obtain a one-dimensional feature sequence.

[0009] According to a first aspect of an embodiment of the present application, the domain-specific adapter is a convolutional adapter and includes multiple adapter layers, each adapter layer including a linear down-projection layer, a two-dimensional convolution layer, a Gaussian error linear unit layer, and a linear up-projection layer; in a one-dimensional feature sequence, features other than the output class features are reshaped into a two-dimensional structure before being input into the two-dimensional convolution layer.

[0010] According to the first aspect of the embodiment of the present application, the domain-specific asymmetric classifier has real audio class center parameters and fake audio class center parameters. The real audio class center parameters are shared parameters between different audio domains, and the fake audio class center parameters are exclusive parameters of each audio domain.

[0011] According to the first aspect of an embodiment of the present application, the intermediate model includes 12 transformer encoders, each transformer encoder includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, a multi-layer perceptron layer and an adapter layer; in each transformer encoder, the adapter layer runs in parallel with the multi-head self-attention layer and the multi-layer perceptron layer.

[0012] According to the first aspect of the embodiment of the present application, the aforementioned one-dimensional feature sequence is assigned a domain-specific adapter and a domain-specific asymmetric classifier to obtain an intermediate model, and the intermediate model is trained based on the one-dimensional feature sequence to obtain a target model, including: processing the one-dimensional feature sequence in sequence through 12 transformer encoders, and assigning a domain-specific adapter to obtain a target feature; assigning a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on a pre-trained model backbone network; optimizing the intermediate model through a cross-entropy loss function to obtain a target model; wherein, one transformer encoder corresponds to one processing stage.

[0013] According to the first aspect of the embodiment of the present application, the aforementioned target model includes forged audio class center parameters and domain-specific adapters corresponding to multiple audio data domains; the aforementioned target model is processed by soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication, including: using soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain the routing prediction probability; based on the forged audio class center parameters and the routing prediction probability, determining the forged audio class center weight by parameter merging; dynamically merging the domain-specific adapters and routing prediction probabilities corresponding to multiple audio data domains to determine the instance-level adapter; dynamically merging the forged audio class center weight and the real audio class center weight shared by different audio data domains to determine the instance-level asymmetric classifier; merging the original pre-trained model backbone network, the instance-level reorganization adapter and the instance-level asymmetric classifier to obtain an instance-level inference model for audio authentication.

[0014] In the second aspect, an embodiment of the present application provides a domain-incremental audio forgery detection device that does not require replay, and the domain-incremental audio forgery detection device includes an acquisition module, a preprocessing module, a training module, a learning module and a processing module; the acquisition module is used to obtain the original pre-trained model backbone network, which is a convolution-free, attention-based audio classification network model that does not include any domain-specific parameters; the pre-processing module is used to receive sequentially arriving audio data domains, and pre-process the audio data domains to obtain at least one one-dimensional feature sequence; the training module is used to assign domain-specific adapters and domain-specific asymmetric classifiers to the one-dimensional feature sequence to obtain an intermediate model, and train the intermediate model based on the one-dimensional feature sequence to obtain a target model; the learning module is used to learn a prototype representation of the audio data domain at the current stage from the one-dimensional feature sequence based on the pre-trained model backbone network to support instance-level soft routing selection during inference; the processing module is used to process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio forgery authentication.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a program stored in the memory and runnable on the processor, wherein when the program is executed by the processor, the method for detecting domain incremental audio forgery without replay in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for detecting domain incremental audio forgery without replay in the first aspect is implemented.

[0017] The present invention provides a method, apparatus, device, and medium for detecting domain-incremental audio forgery without replay. Compared with existing technologies, it has the following advantages:

[0018] The present application can receive audio data domains that arrive sequentially and convert the audio data domains into one-dimensional feature sequences. In the process of training the pre-trained model backbone network, domain-specific adapters and domain-specific asymmetric classifiers are assigned to the one-dimensional feature sequences. Domain-specific adapters and domain-specific asymmetric classifiers are used to store domain-specific knowledge. While solving the problem of catastrophic forgetting, the need for old data is also avoided. The asymmetric classifier avoids the problem of inconsistency in prediction probability distribution that may be caused by the use of multiple independent classifiers and the problem of catastrophic forgetting that may be caused by the use of a single shared classifier. At the same time, the present application processes the target model through soft routing technology and dynamic merging technology, thereby enhancing the model's error correction ability for erroneous routing selection and realizing knowledge sharing of different domain-specific parameters. No replay is required and the network architecture can be dynamically adjusted, which helps to solve the problem of identifying fake audio data generated by new forgery methods that continue to emerge in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of a method for detecting domain incremental audio forgery without replay provided by an embodiment of the present application;

[0021] Figure 2 yes Figure 1 An exemplary flow chart of S120;

[0022] Figure 3 This is a flowchart of an audio data domain processing method provided by an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of the structure of a convolution adapter provided in an embodiment of the present application;

[0024] Figure 5 This is a schematic structural diagram of a domain incremental audio forgery detection device without replay provided by an embodiment of the present application;

[0025] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0027] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0028] The embodiments of the present application solve the problem of difficulty in balancing various factors such as data memory, storage load and detection accuracy in domain incremental audio forgery detection when facing the continuous arrival of new data distribution domains by providing a domain incremental audio forgery detection method, device, equipment and medium that does not require replay.

[0029] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0030] With the rapid development of generation technology, the audio generated by speech synthesis and speech conversion technology is becoming more and more realistic. While this technology has brought positive impacts to human life, it also brings security risks. Therefore, it is of great significance to carry out audio authentication work.

[0031] Existing speech forgery detection methods are usually based on a fixed dataset. However, in real-world scenarios, new forgery methods will continue to emerge with technological advancements. In addition, due to the diversity of real-world environments and collection equipment, there are certain deviations between real data. Therefore, when detection models with high performance on fixed datasets are faced with data from these new data distribution domains, they often experience a significant performance degradation.

[0032] Therefore, an ideal forgery detection model needs to be able to adapt to the continuous arrival of data from new data distribution domains. There are two simple and intuitive approaches to this problem. One solution is to retrain the model on both the old and new data, but this approach is time-consuming and consumes a large amount of storage space. In addition, due to data privacy issues, the old data may not be accessible in most cases. Another more intuitive solution is to fine-tune the old model using only the new data. However, after fine-tuning, the old model will forget its past knowledge, resulting in a sharp decline in the model's performance on the old data. This is known as catastrophic forgetting, and this problem becomes increasingly severe with the number of fine-tuning cycles.

[0033] To mitigate the problem of catastrophic forgetting, incremental learning methods have been widely used by researchers. These methods enable models to sequentially learn each new task while avoiding forgetting previous tasks. Replay-based incremental learning methods typically utilize a replay buffer to store old data from previous stages for replay in subsequent incremental stages, thereby mitigating catastrophic forgetting. However, this requirement imposes an additional storage burden. Furthermore, for the task of speech forgery detection, old data may be inaccessible due to privacy concerns. Therefore, replay-based continuous learning methods are not optimal for this task and have several limitations. Regularization-based continuous learning methods add a loss regularization term to the training loss, leveraging this loss to constrain weight updates and thereby preserve knowledge from previous tasks. Currently, several regularization-based continuous learning methods have been developed in the field of speech forgery detection. While these methods do not require the storage of old data, they use the same model parameters for different sequentially arriving data domains. This results in poor performance when the distribution of data from different sequentially arriving domains differs significantly.

[0034] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] The following first introduces a domain incremental audio forgery detection method without replay provided by an embodiment of the present application.

[0036] The present invention provides a flowchart of a method for detecting domain incremental audio forgery without replaying, as shown in FIG. Figure 1 As shown, the domain incremental audio forgery detection method without replay may include the following steps S110-S150.

[0037] S110. Obtain the original pre-trained model backbone network, where the pre-trained model backbone network is a non-convolutional audio classification network model based on an attention mechanism and does not include any domain-specific parameters.

[0038] In the embodiments of the present application, it can be understood that before preprocessing the audio data domain to obtain a one-dimensional feature sequence, the present application obtains the original pre-trained model backbone network. The pre-trained model backbone network is usually trained on large-scale data. The pre-trained model backbone network has learned rich general semantic information and patterns and has good generalization ability; good generalization ability can enhance the model's anti-forgetting ability to a certain extent, so the present application adopts a pre-trained model backbone network.

[0039] S120: Receive audio data fields that arrive sequentially, and preprocess the audio data fields to obtain at least one one-dimensional feature sequence.

[0040] It should be noted that each audio data domain that arrives sequentially contains a certain number of audios, and each audio can be processed into a one-dimensional feature sequence. Therefore, one audio data domain can be processed to obtain one or more one-dimensional feature sequences.

[0041] S130 , allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model.

[0042] In the embodiments of the present application, it can be understood that a simple and effective way to solve the problem of catastrophic forgetting is to assign a set of domain-specific parameters to the audio data domain of the current stage that arrives sequentially. These domain-specific parameters will be optimized on the audio data domain to model and learn the data distribution of the current stage.

[0043] Meanwhile, the other parameters of the pre-trained model backbone network remain fixed, enabling knowledge sharing across data domains. Convolutional adapters perform comparable to fully fine-tuned models with sufficient training data, and may even outperform them when data is scarce. Furthermore, approaches using domain-specific adapters require fewer training parameters. This approach not only reduces parameters but also effectively alleviates the reliance on training data size.

[0044] S140. Based on the pre-trained model backbone network, a prototype representation of the audio data domain of the current stage is learned from the one-dimensional feature sequence training to support instance-level soft routing selection during inference.

[0045] S150: Process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

[0046] In the embodiments of the present application, it can be understood that the present application stores domain-specific knowledge through domain-specific adapters and domain-specific asymmetric classifiers, which not only solves the problem of catastrophic forgetting but also avoids the need for old data. The asymmetric classifier avoids the problem of inconsistency in prediction probability distribution that may be caused by using multiple independent classifiers and the problem of catastrophic forgetting that may be caused by using a single shared classifier.

[0047] It should be emphasized that this application uses a soft routing technology based on learnable domain prototypes for routing selection, and uses dynamic reconstruction technology to enhance the model's error correction ability for erroneous routing selection and realize knowledge sharing of parameters specific to different domains; in addition, this application uses an asymmetric classifier to avoid the problem of inconsistency in prediction probability distribution that may be caused by the use of multiple independent classifiers and the problem of catastrophic forgetting that may be caused by the use of a single shared classifier; it helps to solve the problem of identifying fake audio data generated by new forgery methods that continue to emerge in real scenarios.

[0048] In some embodiments, please refer to Figure 2 , the aforementioned receiving the audio data domain that arrives in sequence and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence, that is, S120 may specifically include the following steps:

[0049] S210 : Based on a preset window length and frame shift, the sequentially arriving audio data domain is processed through a Hamming window to obtain a mel-spectrogram.

[0050] S220 , dividing the mel-spectrogram into image blocks of a preset number and size, and overlapping them in the time dimension and the frequency dimension to obtain a first image sequence.

[0051] S230 , expanding the first image sequence into a one-dimensional embedding feature through a linear mapping layer, and adding an output class feature at the beginning position of the one-dimensional embedding feature to obtain a second image sequence.

[0052] S240: Add a trainable position encoding feature to each element of the second image sequence to obtain a one-dimensional feature sequence.

[0053] For details, please refer to Figure 3For an audio data domain with a duration of t seconds, a Hamming window with a window length of 25ms and a frame shift of 10ms is first used to obtain a Mel-spectrogram with a size of 128×100t; then the obtained Mel-spectrogram is divided into N image blocks with a size of 16×16, and overlapped by 6 in both the time and frequency dimensions to obtain the first image sequence; where, Then, each 16X16 image block in the first image sequence is expanded into a one-dimensional embedding feature of size 768 through a linear mapping layer, and an output class feature cls is added at the beginning of the image block feature sequence. This feature can be regarded as an abstract representation of the entire image block feature sequence, which will be used as the input of the classifier; finally, a trainable position encoding feature is added to each one-dimensional embedding feature, so that a one-dimensional feature sequence Z input to the transformer encoder can be obtained. The one-dimensional feature sequence Z can be expressed as: Z = {z1,z2,...,z N ,z cls},in, d is the embedding dimension, N is the total number of image patches, and cls represents the output class feature.

[0054] In some embodiments, please refer to Figure 3 The intermediate model includes 12 transformer encoders, each of which includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, a multi-layer perceptron layer, and an adapter layer. In each transformer encoder, the adapter layer runs in parallel with the multi-head self-attention layer and the multi-layer perceptron layer. It is understandable that the first and second normalization layers are only used to represent different entities. Specifically, the pre-trained model backbone network is a ViT / B-16 model with 12 transformer encoders, and the model is initialized using pre-trained parameters on the ImageNet-21k dataset.

[0055] In one example, the one-dimensional feature sequence Z is processed by 12 Transformer encoders in sequence, where the processing of one Transformer encoder can be expressed as

[0056] Formula 1 is used to express:

[0057]

[0058] Among them, MHSA represents the multi-head self-attention layer, MLP represents the multi-layer perceptron layer, LN represents the layer normalization layer, which includes the first normalization layer and the second normalization layer mentioned above, l represents the number of layers, and Z l is the feature extracted from the lth layer.

[0059] In some embodiments, the aforementioned process of allocating a domain-specific adapter and a domain-specific asymmetric classifier to a one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model, that is, S130 may specifically include the following steps:

[0060] S310. Process the one-dimensional feature sequence in sequence through 12 transformer encoders and assign domain-specific adapters to obtain target features; wherein one transformer encoder corresponds to one processing stage.

[0061] In one example, assume that there are T audio data domains D = {(D1, ..., D T )} will arrive in sequence, the audio data domain of the current stage in, Indicates audio input, Indicates the label corresponding to the audio input, n t Indicates the current audio data domain D t This application uses a domain-specific adapter A consisting of L adapter layers. t For the current stage audio data domain D t For modeling, the parameters of the domain-specific adapter are The domain-specific adapter At is represented by Expression 2:

[0062]

[0063] In another example, see Figure 4 The domain-specific adapter is a convolutional adapter and includes multiple adapter layers, each of which includes a linear down projection layer MLP down , two-dimensional convolution layer Conv2D, Gaussian error linear unit layer GELU and a linear upward projection layer MLP up .

[0064] In the input one-dimensional feature sequence Z={z1,z2,...,z N ,z cls}, in addition to the output class feature z cls In addition, the remaining input features {z1,z2,...,z N} will be reshaped into a two-dimensional structure before performing a two-dimensional convolution operation. Please refer to Figure 3 , the adapter layer will be inserted into the pre-trained model backbone network at the corresponding position of each multi-head self-attention layer MHSA and multi-layer perceptron layer MLP. The process of the adapter layer processing the one-dimensional feature sequence Z can be expressed by Expression 3:

[0065]

[0066] Domain-specific adapters in the lth adapter layer The multi-head self-attention layer MHSA layer and the multi-layer perceptron layer MLP layer of each transformer encoder run in parallel. The operation process can be expressed as Expression 4:

[0067]

[0068] S320. Assign a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on the pre-trained model backbone network.

[0069] In the embodiments of the present application, it is understood that the feature distribution of real audio is compact, while the feature distribution of forged audio is scattered across different data domains. Therefore, the present invention uses an asymmetric classifier, treating real audio from different domains as the same category, while treating forged audio from different domains as independent categories. This avoids the problem of inconsistent prediction probability distribution that may result from using multiple independent classifiers, as well as the problem of catastrophic forgetting that may result from using a single shared classifier.

[0070] In one example, the domain-specific asymmetric classifier has real audio class center parameters and fake audio class center parameters, where the real audio class center parameters are shared parameters between different audio domains, and the fake audio class center parameters are exclusive parameters of each audio domain.

[0071] Specifically, this application is to assign a domain-specific asymmetric classifier to the target feature obtained based on the one-dimensional feature sequence Its parameters include θ r and θ ft , where θ r represents the true audio class center parameter shared between different audio data domains, θ ft Indicates the audio data field D at the current stage t Exclusive forged audio class center parameters; It can be understood that the asymmetric classifier up to the current stage t can be expressed as: f c ={θ r ,θ f1 ,...,θ ft During the training process, the asymmetric classifier at the current stage t The output probability It can be expressed by Expression 5:

[0072]

[0073] in, Is audio input After 12 embedded domain-specific adapters A t The target features after the transformer encoder.

[0074] It should be emphasized that the aforementioned intermediate model corresponds to the model to be trained at the current stage f curr , f curr The parameters include the pre-trained model backbone network parameters ω, the domain-specific adapter parameters of the current stage Real audio class center parameter θ r and the forged audio class center parameter θ ft ; Among them, domain-specific adapter parameters Real audio class center parameter θ r and the forged audio class center parameter θ ft Can be updated, the pre-trained model backbone network parameters ω are in a frozen state.

[0075] It is also important to emphasize that, in addition, the domain-specific adapter A at the current stage t The domain-specific adapter A from the previous stage will be used t-1 Initialize the parameters of the domain-specific asymmetric classifier The true audio class center parameter θ in r The real audio class center parameter θ from the previous stage will be used r , and the forged audio class center parameter θ ft will be randomly initialized; this setting ensures that the domain-specific parameters can focus on the data domain of the current stage, which helps to alleviate catastrophic forgetting.

[0076] S330. Optimize the intermediate model through the cross entropy loss function to obtain the target model.

[0077] In one example, the cross entropy loss function can be expressed by Expression 6:

[0078]

[0079] in, Represents an asymmetric classifier The output probability of Indicates the label corresponding to the audio input, 1≤i≤n.

[0080] In some embodiments, regarding the aforementioned pre-trained model backbone network, a prototype representation of the audio data domain of the current stage is learned from the one-dimensional feature sequence training to support instance-level soft routing selection during inference, that is, S140. It can be understood that since the present application is for each audio data domain D that arrives sequentially, t Domain-specific parameters are set, so this requires the model to know the probabilistic relationship between the test instance and different audio data domains during inference; this application learns the prototype representation C={c1,...,c t} to solve this problem, a randomly initialized domain prototype representation type c t The current audio data domain D t In the original pre-trained model backbone network f ast Align the features on the

[0081] In one example, assuming that the audio in the current stage audio data domain arrives sequentially in the pre-trained model backbone network f ast The high-level feature space on the audio data domain D has a similar distribution, so in the current stage t There must be a feature that matches the output class Aligned domain prototypes Specifically, this application minimizes the cross entropy loss L proto To make the domain prototype c t Close to the current audio data field D t The feature center of the minimization cross entropy loss L proto It can be expressed by Expression 7:

[0082]

[0083] Among them, f ast represents the pre-trained model backbone network, Indicates the current stage audio data domain D t The i-th audio input on n t Indicates the current audio data domain D t The total number of audio frequencies, τ is the temperature coefficient, is the cosine similarity.

[0084] During model inference, this application uses soft routing technology to predict the relationship between the test instance and different audio domain prototypes C = {c1,...,c t Specifically, for a test sample x, this application calculates the probability between it and T audio domain prototypes The calculation process is shown in Expression 8:

[0085]

[0086] In some embodiments, the target model includes forged audio class center parameters and domain-specific adapters corresponding to multiple audio data domains; the target model is processed using soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication, that is, S150 may specifically include the following steps:

[0087] S410: Use soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain a routing prediction probability.

[0088] S420: Determine the weight of the forged audio class center by combining the parameters based on the forged audio class center parameter and the route prediction probability.

[0089] S430: Dynamically merge domain-specific adapters and routing prediction probabilities corresponding to multiple audio data domains to determine an instance-level adapter.

[0090] S440: Dynamically merge the forged audio class center weights and the real audio class center weights shared by different audio data domains to determine an instance-level asymmetric classifier.

[0091] S450: Merge the original pre-trained model backbone network, the instance-level reorganization adapter, and the instance-level asymmetric classifier to obtain an instance-level inference model for audio authentication.

[0092] In the embodiments of the present application, it can be understood that relevant research shows that it is effective to merge multiple model parameters obtained by training on different data domains into one model, and the merged model can be generalized to all data domains, thereby effectively avoiding the need for the domain to which the test instance belongs, and enabling knowledge sharing between different domains. Unlike the existing model merging technology that generates a single fixed model for all test samples, the present application tailors an instance-level model for each test instance, providing more powerful performance. In order to improve the robustness of the model to erroneous route predictions and to achieve knowledge sharing of parameters specific to different domains, the present application designs a parameter dynamic merging technology, which obtains instance-level specific parameters by weighting and merging domain-specific parameters using the probability obtained from soft route predictions.

[0093] Specifically, the adapter weight A obtained using the parameter merging technique m and the forged audio class center weight θ fm The process is shown in Expression 9:

[0094]

[0095] Where A={A1,...,A T}、θ f ={θ f1 ,...,θ ft}and are the domain-specific adapters for the T audio data domains that arrive sequentially, the forged audio class center parameters, and the routing prediction probabilities of the test instances.

[0096] Use the merged fake audio class center weight θ fm The real audio class center weight θ shared with the domain t Get an instance-level asymmetric classifier f cm , and combined with instance-level adapter A mWith the pre-trained model backbone network f ast Merge to obtain the instance-level inference model f instence For audio authentication.

[0097] Based on this, the dynamic merging technique used in this application enhances the routing algorithm's robustness against erroneous domain predictions. Because the dynamic merging technique combines the specific parameters of multiple data domains, even if the routing algorithm produces some erroneous data domain predictions, the model's performance will not be significantly reduced. Furthermore, dynamic merging facilitates the sharing of domain-specific knowledge by merging parameter spaces, particularly when multiple data domains come from similar domains, where this merging technique performs even better.

[0098] In some embodiments, please refer to Figure 5 The present application also provides a domain-incremental audio forgery detection device 500 without replaying. The domain-incremental audio forgery detection device 500 may include the following modules:

[0099] The acquisition module 510 is used to obtain the original pre-trained model backbone network, which is a non-convolutional audio classification network model based on the attention mechanism and does not include any domain-specific parameters.

[0100] The pre-processing module 520 is configured to receive the sequentially arriving audio data domain and pre-process the audio data domain to obtain at least one one-dimensional feature sequence.

[0101] The training module 530 is used to assign a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and train the intermediate model based on the one-dimensional feature sequence to obtain a target model.

[0102] The learning module 540 is used to learn a prototype representation of the audio data domain of the current stage from a one-dimensional feature sequence training based on a pre-trained model backbone network to support instance-level soft routing selection during inference.

[0103] The processing module 550 is used to process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

[0104] According to an embodiment of the present application, any multiple modules among the acquisition module 510, the preprocessing module 520, the training module 530, the learning module 540, and the processing module 550 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module.

[0105] In some embodiments, the preprocessing module 520 can be specifically used to: based on a preset window length and frame shift, process the sequentially arriving audio data domain through a Hamming window to obtain a mel-spectrogram; divide the mel-spectrogram into image blocks with a preset number and size, and overlap them in the time dimension and frequency dimension to obtain a first image sequence; expand the first image sequence into a one-dimensional embedding feature through a linear mapping layer, and add an output class feature at the starting position of the one-dimensional embedding feature to obtain a second image sequence; add a trainable position encoding feature to each element of the second image sequence to obtain a one-dimensional feature sequence.

[0106] In some embodiments, the training module 530 can be specifically used to: process the one-dimensional feature sequence in sequence through 12 transformer encoders, and assign domain-specific adapters to obtain target features; wherein, one transformer encoder corresponds to one processing stage; assign a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on the pre-trained model backbone network; optimize the intermediate model through the cross-entropy loss function to obtain the target model.

[0107] In some embodiments, the processing module 550 can be specifically used to: use soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain the routing prediction probability; determine the forged audio class center weight by parameter merging based on the forged audio class center parameters and the routing prediction probability; dynamically merge the domain-specific adapters and routing prediction probabilities corresponding to multiple audio data domains to determine the instance-level adapter; dynamically merge the forged audio class center weight and the real audio class center weight shared by different audio data domains to determine the instance-level asymmetric classifier; merge the original pre-trained model backbone network, the instance-level reorganization adapter and the instance-level asymmetric classifier to obtain an instance-level inference model for audio authentication.

[0108] Figure 5 Each module in the device shown has the function of implementing each step in the aforementioned domain incremental audio forgery detection method without replay, and can achieve its corresponding technical effects. For the sake of brevity, it will not be repeated here.

[0109] In some embodiments, the present application provides an electronic device, the structural diagram of the electronic device is as follows Figure 6 shown.

[0110] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0111] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0112] The memory 620 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 620 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 620 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid-state memory.

[0113] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Therefore, typically, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it may perform the operations described in any of the above-described embodiments of the domain-incremental audio forgery detection method without replay.

[0114] The processor 610 reads and executes computer program instructions stored in the memory 620 to implement any one of the domain incremental audio forgery detection methods without replay in the above embodiments.

[0115] In one example, the electronic device may further include a communication interface 630 and a bus 600. Figure 6 As shown, the processor 610 , the memory 620 , and the communication interface 630 are connected via a bus 600 and communicate with each other.

[0116] The communication interface 630 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0117] Bus 600 includes hardware, software or both, and the parts of online data flow metering equipment are coupled to each other. For example, but not limitation, bus can include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 600 can include one or more buses. Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0118] In addition, in conjunction with the domain-incremental audio forgery detection method without replay in the above embodiments, embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when executed by a processor, the computer program instructions implement any of the domain-incremental audio forgery detection methods without replay in the above embodiments.

[0119] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0120] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0121] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0122] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0123] In summary, compared with the prior art, this application has the following beneficial effects:

[0124] 1. This application sets up domain-specific adapters and domain-specific asymmetric classifiers in the pre-trained model backbone network to store domain-specific knowledge, which solves the problem of catastrophic forgetting while also avoiding the need for old data; the adapter-based pre-trained model fine-tuning method greatly reduces the scale of parameter learning and reduces the model's requirements for the scale of training data, so that it can obtain better performance even when only a small amount of training data is provided.

[0125] 2. This application uses an asymmetric classifier, which treats real audio from different domains as the same category, and forged audio from different domains as independent categories. This design avoids the problem of inconsistent prediction probability distribution that may be caused by using multiple independent classifiers and the problem of catastrophic forgetting that may be caused by using a single shared classification.

[0126] 3. This application adopts a soft routing technology based on learnable domain prototypes for route selection, and also adopts a parameter dynamic merging technology, that is, by using the probability obtained by soft routing prediction to restructure the domain-specific parameters to obtain instance-level specific parameters, it improves the model's robustness to erroneous route predictions and realizes the knowledge sharing of different domain-specific parameters.

[0127] 4. This application provides a domain incremental audio forgery detection method that does not require replay and can dynamically adjust the network architecture. When faced with continuously arriving data from new data distribution domains, it can effectively balance various factors such as data memory, storage load, and detection accuracy in domain incremental audio forgery detection.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A domain-incremental audio forgery detection method without replay, characterized in that: include: Obtaining an original pre-trained model backbone network, where the pre-trained model backbone network is a non-convolutional, attention-based audio classification network model that does not include any domain-specific parameters; receiving audio data domains arriving sequentially, and preprocessing the audio data domains to obtain at least one one-dimensional feature sequence; Allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model; Based on the pre-trained model backbone network, a prototype representation of the audio data domain at the current stage is learned from the one-dimensional feature sequence training to support instance-level soft routing selection during inference; The target model is processed by soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

2. The method for detecting domain-incremental audio forgery without replay as claimed in claim 1, wherein: The receiving of the audio data domain that arrives sequentially and preprocessing the audio data domain to obtain at least one one-dimensional feature sequence includes: Based on the preset window length and frame shift, the sequentially arriving audio data domain is processed through the Hamming window to obtain the Mel spectrogram; Dividing the mel-spectrogram into image blocks of a preset number and size, and overlapping them in the time dimension and the frequency dimension to obtain a first image sequence; Expanding the first image sequence into a one-dimensional embedding feature through a linear mapping layer, and adding an output class feature at the beginning position of the one-dimensional embedding feature to obtain a second image sequence; A trainable position encoding feature is added to each element of the second image sequence to obtain a one-dimensional feature sequence.

3. The method for detecting domain-incremental audio forgery without replay as claimed in claim 2, wherein: The domain-specific adapter is a convolutional adapter and includes multiple adapter layers, each of which includes a linear down-projection layer, a two-dimensional convolution layer, a Gaussian error linear unit layer, and a linear up-projection layer; in the one-dimensional feature sequence, features other than the output class features are reshaped into a two-dimensional structure before being input into the two-dimensional convolution layer.

4. The method for detecting domain-incremental audio forgery without replay according to claim 1 or 2, wherein: The domain-specific asymmetric classifier has real audio class center parameters and fake audio class center parameters, the real audio class center parameters are shared parameters between different audio domains, and the fake audio class center parameters are exclusive parameters of each audio domain.

5. The method for detecting domain-incremental audio forgery without replaying according to claim 1 or 2, wherein: The intermediate model includes 12 transformer encoders, each of which includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, a multi-layer perceptron layer and an adapter layer; in each transformer encoder, the adapter layer runs in parallel with the multi-head self-attention layer and the multi-layer perceptron layer.

6. The method for detecting domain-incremental audio forgery without replay according to claim 1 or 2, wherein: The allocating a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and training the intermediate model based on the one-dimensional feature sequence to obtain a target model includes: The one-dimensional feature sequence is processed sequentially by 12 transformer encoders and domain-specific adapters are assigned to obtain target features; wherein one transformer encoder corresponds to one processing stage; Assigning a domain-specific asymmetric classifier to the target feature to obtain an intermediate model based on the pre-trained model backbone network; The intermediate model is optimized by a cross entropy loss function to obtain a target model.

7. The method for detecting domain-incremental audio forgery without replay according to claim 1 or 2, wherein: The target model includes forged audio class center parameters and domain-specific adapters corresponding to multiple audio data domains; The target model is processed by the soft routing technology and the dynamic merging technology to obtain an instance-level inference model for audio authentication, including: Use soft routing technology to predict the similarity probability between the audio data domain prototype and the test instance to obtain the routing prediction probability; Determining a forged audio class center weight by parameter merging based on the forged audio class center parameter and the route prediction probability; Dynamically merging the domain-specific adapters and the routing prediction probabilities corresponding to multiple audio data domains to determine an instance-level adapter; Dynamically merging the forged audio class center weights and the real audio class center weights shared by different audio data domains to determine an instance-level asymmetric classifier; The original pre-trained model backbone network, instance-level reorganization adapter and instance-level asymmetric classifier are merged to obtain an instance-level inference model for audio authentication.

8. A domain incremental audio forgery detection device without replay, characterized in that: include: An acquisition module is used to obtain an original pre-trained model backbone network, where the pre-trained model backbone network is a non-convolutional audio classification network model based on an attention mechanism and does not include any domain-specific parameters; A preprocessing module, configured to receive sequentially arriving audio data domains and preprocess the audio data domains to obtain at least one one-dimensional feature sequence; a training module, configured to assign a domain-specific adapter and a domain-specific asymmetric classifier to the one-dimensional feature sequence to obtain an intermediate model, and train the intermediate model based on the one-dimensional feature sequence to obtain a target model; A learning module, configured to learn a prototype representation of the audio data domain at a current stage from the one-dimensional feature sequence training based on the pre-trained model backbone network, so as to support instance-level soft routing selection during inference; The processing module is used to process the target model through soft routing technology and dynamic merging technology to obtain an instance-level inference model for audio authentication.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for detecting domain incremental audio forgery without replay as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the domain incremental audio forgery detection method without replay as described in any one of claims 1 to 7 is implemented.

Images