A method and system for detecting continuous forged speech based on multi-expert confrontation against disturbance

Abstract

The application discloses a kind of based on the continuous counterfeit speech detection method and system of multi-expert counteracting disturbance, the initial stage of this method uses cross-entropy loss optimization detection model, in subsequent stage, single classification consistency loss is used to keep the stability of real class representation, combine multi-expert counteracting disturbance with single classification optimization goal, only maintain real speech representation center and disturbance vector, without storing historical data or model weight, to realize efficient and privacy-protected continuous counterfeit speech detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio technology, in particular to a method and system for continuous spoofed speech detection based on multi-expert confrontation disturbance. BACKGROUND

[0002] With the rapid development of deep learning based text-to-speech and speech conversion technology, high-quality spoofed speech emerges in an endless stream, bringing serious security risks to automatic speaker verification systems and other speech applications. Research communities such as the ASVspoof challenge have established standardized evaluation benchmarks and released publicly available datasets, driving the progress of spoofed speech detection technology. Existing research mostly focuses on optimizing model performance on existing public datasets. However, in actual deployment, if the model is trained only on target domain data, it often produces catastrophic forgetting phenomenon. This phenomenon refers to the model's rapid loss of ability to distinguish historical spoofing attack methods while learning to distinguish new spoofing attacks. The reason for this phenomenon is that the model usually adopts a data-driven training paradigm to adapt to the current data distribution when updating parameters, and lacks an explicit mechanism to remember historical knowledge. With iterative updates of model parameters, the current stage data dominates the optimization process, causing previously learned representation information to be gradually overwritten. In real-world scenarios, spoofing methods and acoustic conditions will change over time, and the model needs to continuously adapt to new spoofing attack methods while not forgetting historical knowledge, i.e., to achieve continuous learning.

[0003] Existing continuous learning methods can be divided into sample replay, structure expansion, and regularization constraints. Among them, the regularization constraint method is simple to operate and does not require explicit data storage, and has been widely applied. In the field of spoofed speech detection, existing methods mostly use regularization methods such as parameter constraints or gradient correction to reduce forgetting and maintain a balance between plasticity and stability. For example, RAWM adaptively corrects the update gradient of model weights according to the ratio of real class samples to spoofed class samples. Similarly, RWM designs a trainable gradient correction scheme that adjusts the model update direction according to the intra-class cosine distance. In addition, RegO further divides network neurons into four regions according to information importance, applies different gradient correction strategies to key regions, and allows non-important regions to update freely.

[0004] Existing spoofing speech detection techniques usually choose to directly combine historical data and new data for joint training to alleviate the forgetting phenomenon, but this training method requires a large amount of computing and storage resources, and is prone to privacy problems such as speech data leakage. Similarly, the replay-based continual learning method retains a subset of historical samples in the memory buffer and replays the samples during the training process of the new stage. The performance of this method is highly sensitive to the buffer size, and introduces additional storage overhead and privacy risks in speech-related applications. The method based on structural expansion assigns dedicated parameters to different tasks through network expansion or parameter isolation, greatly increasing the complexity of the model. The method based on regularization constraints avoids explicit data storage, mainly relying on parameter constraints or gradient correction strategies to update the model. However, such regularization constraints often compromise the plasticity of the model (the ability to adapt to new types of spoofing attacks), and as the data increases, the accumulated constraints can easily lead to model optimization difficulties and suboptimal balance between plasticity and stability.

[0005] In the previous work of the department, a knowledge retention mechanism is proposed, which uses adversarial perturbation as a lightweight knowledge carrier. This method only maintains a single perturbation vector to reconstruct the historical information of the spoofing discrimination boundary, without storing the original data. However, this method still requires additional storage of the weights of the historical model and applies knowledge distillation constraints in subsequent learning stages.

[0006] Based on the above analysis, the spoofing speech detection field needs a continual learning mechanism that efficiently remembers historical discrimination knowledge without saving the original speech data, while quickly adapting to new data domains. SUMMARY

[0007] To solve the problem of how to effectively maintain the model's discrimination ability for previous spoofing attacks without saving historical speech data, reduce catastrophic forgetting, and achieve continual learning of new attack methods during continuous spoofing speech detection, the present application proposes a multi-expert adversarial perturbation-based continuous spoofing speech detection method and system. This method has low storage overhead and is easily integrated into existing spoofing speech detection architectures.

[0008] According to one aspect of the present application, a multi-expert adversarial perturbation-based continuous spoofing speech detection method is proposed, comprising the following steps:

[0009] S1, constructing an audio classification network, the audio classification network comprising an encoder, a detection module comprising a plurality of expert models, and an information bottleneck layer;

[0010] S2, inputting historical audio signals into the audio classification network for initial stage training;

[0011] S3, after the initial stage training is completed, a plurality of rounds of adversarial perturbation training is performed, each expert model generates a corresponding adversarial perturbation vector, the encoder extracts features from real audio signals and fake audio signals in the historical audio signals, and outputs real class feature vectors and fake class feature vectors respectively, the adversarial perturbation vector is added to the real class feature vector to form a proxy fake feature vector, the expert model is caused to misclassify the real class feature vector as a fake class feature vector, the adversarial perturbation vector is continuously optimized until the plurality of rounds of adversarial perturbation training is completed, and the adversarial perturbation vector at the end of the plurality of rounds of adversarial perturbation training is saved to a historical memory bank;

[0012] S4, inputting a current audio signal into the audio classification network for new stage training, and randomly extracting an adversarial perturbation vector from the historical memory bank and adding it to a real class feature vector corresponding to the current audio signal to generate the proxy fake feature vector, assigning a fake class label to the proxy fake feature vector and inputting the same into the training of the audio classification network.

[0013] Specifically, the S2 inputs the historical audio signal into the audio classification network for initial stage training, which specifically includes: after the historical audio signal is input into the audio classification network, the encoder extracts feature vectors , wherein T represents the time domain dimension size, and S represents the frequency domain dimension size, the detection module includes expert models , the features are processed by each expert model in the detection module, each expert model outputs a feature vector correspondingly, the feature vectors output by all expert models are compressed by the information bottleneck layer to generate a final discriminant vector for audio classification.

[0014] Specifically, the S2 adopts a plurality of loss functions to optimize the audio classification network in the initial stage training, the plurality of loss functions adopts hyperparameters for controlling orthogonality and information compression strength to weight and combine a mutual information loss function based on prediction distribution difference of the expert model, an information bottleneck loss function based on task-related information retained in the compression process of the information bottleneck layer, and a cross-entropy loss function based on classification results of the audio classification network.

[0015] The embodiment provides a joint training mechanism of multi-expert mutual information orthogonalization and information bottleneck fusion: by designing a mutual information loss function and an information bottleneck loss function , the multi-expert features are caused to be complementary and the representation vector after fusion is caused to sufficiently retain discriminant information.

[0016] Specifically, the multi-loss function is defined as:

[0017] ,

[0018] wherein, respectively represent hyperparameters for controlling orthogonality and information compression strength, represents the mutual information loss function, which is defined as:

[0019] ,

[0020] wherein, represents a joint feature vector of all expert models, represents a joint feature vector excluding , respectively represent corresponding prediction distributions, represents calculating Kullback-Leibler divergence;

[0021] represents the information bottleneck loss function, which is defined as: , represents parameters of the information bottleneck layer;

[0022] represents the cross-entropy loss function, which is defined as:

[0023] ,

[0024] wherein, represents the audio classification network a set of historical audio signals input in the initial stage training, represents a data volume of the corresponding set of historical audio signals, represents a feature vector extracted by the encoder, represents a corresponding category label.

[0025] Specifically, the S4 adopts an updated multi-loss function to optimize the audio classification network in the new stage training, wherein the updated multi-loss function adopts a single-class consistency loss function based on ensuring consistency of the real class feature between different training stages replaces one of the multi-loss functions, and the specific definition of the single-class consistency loss function is as follows:

[0026] ;

[0027] wherein represents a discrimination vector corresponding to the fake class feature vector and the proxy fake feature vector, denotes a discriminative vector corresponding to the real class feature vector, and denote the number of corresponding discriminative vectors in the training round, respectively, denotes a real class feature center discriminative vector, which is updated by calculating the average of the discriminative vectors corresponding to the real class feature vectors.

[0028] The embodiment provides a scheme for stabilizing real speech representation based on single-class consistency constraint: by maintaining a real class feature center discriminative vector C, and by Loss pulls real class samples close to the center and pushes away fake class samples far from the center, thereby ensuring the consistency and effectiveness of the disturbance generation and proxy feature reconstruction.

[0029] Specifically, the updated multi-loss function replaces the cross-entropy loss with a single-class consistency loss function based on ensuring the consistency of the real class features between different training stages, and the updated multi-loss function is defined as: .

[0030] Specifically, the S3 continuously optimizes the adversarial disturbance vector until the multi-round adversarial disturbance training ends, specifically including: updating the adversarial disturbance vector corresponding to each expert model by calculating the mean square loss of the proxy fake feature vector processed by each expert model and the fake class feature vector, and the amplitude of the disturbance of the adversarial disturbance vector is constrained to be less than or equal to an amplitude threshold, and when the misclassification rate of the expert model exceeds a predetermined misclassification rate threshold, the multi-round adversarial disturbance training is defined as ending.

[0031] The embodiment provides a memory scheme for generating and storing adversarial disturbances based on multiple experts: in each training stage, an exclusive adversarial disturbance vector is learned and stored for each expert model, which is used to disturb the real class feature to misjudge it as a fake class feature to remember the discriminative boundary information, and a specific optimization strategy for adversarial disturbance generation is provided: using the mean square error to minimize the difference between the fake representation and the proxy representation, and using the stochastic gradient descent to update the disturbance until the misjudgment threshold is reached, while the disturbance amplitude is truncated. In the subsequent stage, the disturbance vector library is randomly extracted and added to the current real class feature as a fake class sample and incorporated into the training batch to realize the history knowledge retention.

[0032] Specifically, the proxy fake feature vector is defined as an element-level additive formula as follows:

[0033] ,

[0034] denotes the i-th The encoder described in the adversarial perturbation training uses a set of real audio signals. The true category features extracted from them, Indicates the first The proxy-forged feature vector generated during adversarial perturbation training. This indicates that the nth expert model is at the th . The adversarial perturbation vector obtained from the round of adversarial perturbation training, the adversarial perturbation vector The optimization objective is:

[0035] ,

[0036] s.t. ,

[0037] The anti-disturbance vector The mean squared error is updated using the following stochastic gradient descent method after optimization:

[0038] ,

[0039] This means transforming the joint feature vectors of all expert models into predicted values. Represents the norm, This indicates the threshold value for the disturbance amplitude. This indicates that the category is classified as forgery. The corresponding predicted value, This refers to the sign function. This represents the perturbation learning rate. This indicates the calculation of the mean square error. This indicates that the calculation of the nth expert model at the 1st... Adversarial perturbation vectors obtained from wheel adversarial perturbation training The gradient.

[0040] Specifically, it also includes: S5, repeating the steps of S3 after the new stage training is completed to perform multi-round adversarial perturbation training for the new stage, and saving the adversarial perturbation vector at the end of the multi-round adversarial perturbation training for the new stage to the history memory bank.

[0041] In this embodiment, at the end of each stage, the corresponding adversarial perturbation vector is learned from the experts of that stage and saved for use as boundary information for historical forgery classes in subsequent stages.

[0042] According to one aspect of the present invention, a persistent forged speech detection system based on multi-expert adversarial perturbation is proposed, comprising the following modules according to any one of the first aspects:

[0043] The audio classification network construction module is configured to construct an audio classification network, the audio classification network comprising an encoder, a detection module comprising a plurality of expert models, and an information bottleneck layer;

[0044] The initial stage training module is configured to input historical audio signals into the audio classification network for initial stage training.

[0045] The adversarial perturbation training module is configured to perform multiple rounds of adversarial perturbation training after the initial stage training ends, each expert model generating a corresponding adversarial perturbation vector, the encoder extracting features from real audio signals and fake audio signals in the historical audio signals and outputting real class feature vectors and fake class feature vectors respectively, adding the adversarial perturbation vector to the real class feature vector to form a proxy fake feature vector, causing the expert model to misclassify the real class feature vector as a fake class feature vector, and constantly optimizing the adversarial perturbation vector until the multiple rounds of adversarial perturbation training end, and saving the adversarial perturbation vector at the end of the multiple rounds of adversarial perturbation training to a historical memory bank.

[0046] The new stage training module is configured to input current audio signals into the audio classification network for new stage training, randomly extract an adversarial perturbation vector from the historical memory bank and add it to a real class feature vector corresponding to the current audio signal to generate the proxy fake feature vector, assign a fake class label to the proxy fake feature vector, and input the proxy fake feature vector into the training of the audio classification network.

[0047] The present application has the following advantages:

[0048] No need to save historical voice samples: only a compact set of perturbation vectors is stored, which fundamentally reduces storage and privacy pressure;

[0049] Improved stability-plasticity balance: based on the reconstruction of fake class samples in the feature space by the adversarial perturbation rather than directly imposing constraints on the model parameters, the system can not only retain the historical discrimination boundary but also efficiently adapt to new types of fake attack means;

[0050] Multiple expert design improves representation diversity: mutual information optimization between multiple experts and information bottleneck fusion can obtain complementary and orthogonal feature subspaces, thereby obtaining stronger discrimination ability with less parameter expansion;

[0051] Universal front-end and back-end structure: the present application has no strong dependence on the architecture of the encoder and the expert model, and is easy to integrate into existing fake voice detection systems. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain principles of the application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0053] Figure 1 Fig. 1 shows a flow chart of a method for detecting persistent fake speech based on multi-expert adversarial perturbation according to the present application;

[0054] Fig. 2(a) shows an architecture and principle diagram of an audio classification network in an initial stage of training according to the present application;

[0055] Fig. 2(b) shows a principle diagram of an audio classification network in an adversarial perturbation training according to the present application;

[0056] Fig. 2(c) shows a principle diagram of an audio classification network in a new stage of training according to the present application;

[0057] Figure 3 Fig. 3 shows a flow chart of an adversarial perturbation vector optimization of an audio classification network in training according to the present application;

[0058] Figure 4 Fig. 4 shows a structure diagram of a system for detecting persistent fake speech based on multi-expert adversarial perturbation according to the present application;

[0059] Figure 5 Fig. 5 shows a computer system structure diagram of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0060] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for ease of description, only the parts related to the application are shown in the drawings.

[0061] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0062] Figure 1 A method for detecting persistent fake speech based on multi-expert adversarial perturbation is shown, comprising the following steps:

[0063] S1, construct an audio classification network, the audio classification network comprising an encoder, a detection module comprising a plurality of expert models, and an information bottleneck layer;

[0064] S2, input historical audio signals into the audio classification network for initial stage training;

[0065] S3, after the initial stage training is completed, a plurality of rounds of adversarial perturbation training are performed, each expert model generates a corresponding adversarial perturbation vector, the encoder extracts features of real audio signals and fake audio signals in the historical audio signals, and outputs real class feature vectors and fake class feature vectors respectively, the adversarial perturbation vector is added to the real class feature vector to form a proxy fake feature vector, the expert model is caused to misclassify the real class feature vector as a fake class feature vector, the adversarial perturbation vector is continuously optimized until the plurality of rounds of adversarial perturbation training are completed, and the adversarial perturbation vector at the end of the plurality of rounds of adversarial perturbation training is saved to a historical memory bank;

[0066] S4, input current audio signals into the audio classification network for new stage training, randomly extract an adversarial perturbation vector from the historical memory bank and add it to a real class feature vector corresponding to the current audio signal to generate the proxy fake feature vector, assign a fake class label to the proxy fake feature vector, and input the proxy fake feature vector into training of the audio classification network.

[0067] The multi-expert adversarial perturbation framework proposed in the application combines multi-expert adversarial perturbation and single classification optimization target, only maintains a real speech representation center and a perturbation vector, and does not need to store historical data or model weights, so that efficient and privacy-protected continuous fake speech detection is realized. As shown in FIG. 2, the application proposes three core components: (1) a multi-expert detection module that captures different fake attack modes; (2) an adversarial perturbation memory that retains historical decision boundaries; and (3) a single classification consistency target that stabilizes the real representation distribution at different stages. The overall goal of the application is to encode previously learned fake discrimination knowledge into a compact perturbation vector, so that the model can continuously adapt to new fake attack means while retaining the ability to discriminate against historical attacks.

[0068] As shown in FIG. 2 (a), the S2 inputting the historical audio signals into the audio classification network for initial stage training specifically comprises: after the historical audio signals are input into the audio classification network, the encoder extracts feature vectors wherein T represents a time domain dimension size, and S represents a frequency domain dimension size, the detection module comprises expert models , the feature vectors extracted by the encoder pass through the detection module comprising expert models The detection module processes each expert output feature vector To ensure that the joint expert features retain comprehensive and task-related information, an information bottleneck layer is used for compressed fusion to generate the final discriminative vector for classification.

[0069] In one embodiment, the encoder is a pre-trained self-supervised speech large model (e.g., Wav2Vec2.0-XLSR-53), and the input audio signal is first extracted by the encoder to obtain general frame-level embedding feature vectors The encoder is parameter frozen in the continuous learning stage and only serves as a feature extraction front end.

[0070] The multi-loss function is used to optimize the audio classification network trained in the initial stage, and the multi-loss function is used to weight and combine the mutual information loss function based on the prediction distribution difference of the expert model, the information bottleneck loss function based on the task-related information retained in the information bottleneck layer compression process, and the cross-entropy loss function based on the classification result of the audio classification network.

[0071] In one embodiment, the multi-loss function is defined as:

[0072]

[0073] respectively represent the hyperparameters that control orthogonality and information compression strength.

[0074] To encourage the complementarity and orthogonality of the representations extracted by experts, a mutual information loss function based on prediction distribution difference is proposed, denotes the mutual information loss function, which is defined as:

[0075]

[0076] denotes the joint feature vector of all expert models, denotes the joint feature vector excluding denotes the corresponding prediction distribution, denotes the corresponding prediction distribution, denotes the calculation of Kullback-Leibler divergence. The mutual information loss function quantifies the difference between the distributions through Kullback-Leibler divergence. Optimizing this objective can encourage each expert model to capture unique and complementary counterfeit clues, thereby enriching the overall representation.

[0077] To minimize the loss of task-related information in the compression process, the following information bottleneck loss function is proposed​​ , defined as: , denotes the parameter of the information bottleneck layer. This objective effectively preserves discriminative information while filtering out redundant information.

[0078] denotes the cross-entropy loss function, defined as:

[0079] ,

[0080] wherein, denotes the audio classification network the set of historical audio signals input in the initial phase of training, denotes the data volume of the corresponding set of historical audio signals, denotes the feature vector extracted by the encoder, denotes the corresponding class label.

[0081] As shown in FIG. 2(b), after completing the training of each round, a dedicated adversarial perturbation vector is generated for each expert model to preserve its learned historical knowledge. For the nthexpert model in the mthround of adversarial perturbation training, the objective is to learn the adversarial perturbation vector extracted from the real class features in the set of real audio signals and the fake class features in the set of fake audio signals , so that the expert model will misclassify the real class features added with the adversarial perturbation vector as fake class features, i.e., to construct a proxy fake feature vector:

[0082] ,

[0083] As shown in FIG. 2(b), after completing the training of each round, a dedicated adversarial perturbation vector is generated for each expert model to preserve its learned historical knowledge. For the nthexpert model in the mthround of adversarial perturbation training, the objective is to learn the adversarial perturbation vector Figure 3 , so that the expert model will misclassify the real class features added with the adversarial perturbation vector as fake class features, i.e., to construct a proxy fake feature vector:

[0084] The optimization objective of the adversarial perturbation vector

[0085] ,

[0086] s.t. ​​,

[0087] The truncation is performed when the magnitude of the adversarial perturbation vector is greater than a magnitude threshold, which is set to 0.15 of the feature magnitude of the input audio signal in one embodiment. When the magnitude of the adversarial perturbation vector is greater than the magnitude threshold after the iterative training, the magnitude of the adversarial perturbation vector is reset to the magnitude threshold and the iteration continues.

[0088] The adversarial perturbation vector is updated by the stochastic gradient descent method as follows:

[0089] ,

[0090] represents the joint feature vector of all expert models being converted into prediction values, represents the norm, represents the magnitude threshold of the perturbation, represents the feature classified as the fake class corresponding prediction value, represents the sign function, represents the perturbation learning rate, represents the calculation of the mean square error, represents the calculation of the gradient of the adversarial perturbation vector obtained by the adversarial perturbation training of the nth expert model in the mth iteration.

[0091] The adversarial perturbation vector of each expert model is iteratively trained until the misclassification rate exceeds a predefined threshold. The obtained perturbation vector is stored in the memory to assist the subsequent training phase. These compact adversarial perturbation vectors are encoded as dimension tensors, which are aligned with the output dimension of the pre-trained encoder. This design ensures a small storage overhead while effectively preserving the historical decision boundary.

[0092] As shown in FIG. 2(c), the S4 specifically includes: in the training of the subsequent new phase, randomly extracting a perturbation vector from the memory , adding it to the real class features of the current batch to generate proxy fake features , and assigning fake class labels to them and entering the model training. The purpose is to reconstruct the historical decision boundary without saving the original historical speech data to optimize the parameter update direction of the model. This mechanism does not directly impose constraints on the model parameters, thereby maintaining good plasticity.

[0093] ​In step S4, an updated multiple loss function is used to optimize the audio classification network trained in the new stage. The updated multiple loss function replaces one of the multiple loss functions with a single-class consistency loss function that ensures the consistency of the true class features across different training stages.

[0094] In a specific implementation, to ensure the consistency of true class features across different stages, a single-classification optimization strategy is proposed during training in a new stage to maintain a true class feature center discriminant vector. The center vector is updated incrementally using only real audio signals. In training epochs containing real audio signals, the real class feature center discriminant vector is updated by calculating the average of the discriminant vectors corresponding to the real class feature vectors. The optimization objective is to attract real class samples closer to the center and push fake class samples away from the center.

[0095] in, The single-class consistency loss is specifically defined as follows:

[0096] ;

[0097] in This represents the discriminant vector corresponding to the forgery category feature vector and the proxy forgery feature vector. This represents the discriminant vector corresponding to the true category feature vector. and These represent the number of corresponding discriminant vectors in each training round.

[0098] In one embodiment, after initial training, when the model encounters a new type of forgery attack during a new training phase, the cross-entropy loss in the overall training loss is replaced with a single-class consistency loss, and the updated multi-loss function... Defined as: .

[0099] In one embodiment, the method further includes: S5, repeating the steps of S3 after the new stage training is completed to perform multi-round adversarial perturbation training for the new stage, and saving the adversarial perturbation vector at the end of the multi-round adversarial perturbation training for the new stage to the history memory bank.

[0100] In the training process, the model is divided into stages. Sequential learning, at each stage Only the corresponding dataset It can be used for training, where These represent the real audio signal set and the fake audio signal set for this stage, respectively. As shown in Figure 2, the model is trained in the order of Figure 2(a)-(b)-(c)-(b)-(c)-(b)-(c)-…, in the initial training phase. Corresponding to the training process in Figure 2(a), this stage uses cross-entropy loss to optimize the detection model. Afterward, the detection model parameters are frozen, completing the adversarial perturbation training shown in Figure 2(b). The corresponding adversarial perturbation vectors are learned and saved from the expert model in this stage, serving as boundary information for historical forgery classes in subsequent stages. In the new stage ( 1) During the training of Figure 2(c), a single-class consistency loss is used to train the model and maintain the stability of the true class representation. After the expert model optimization is completed in this stage, the adversarial perturbation training of Figure 2(b) is completed again, and the new adversarial perturbation vector is saved. This process continues until a new audio signal is input, in the next new stage... Continue training as shown in Figure 2(c), repeating the previous training cycle.

[0101] like Figure 4 As shown, according to one aspect of the present invention, a persistent forged speech detection system based on multi-expert adversarial perturbation is proposed, comprising the following modules according to any one of the first aspects:

[0102] The audio classification network construction module 401 is configured to construct an audio classification network, which includes an encoder, a detection module containing multiple expert models, and an information bottleneck layer.

[0103] The initial training module 402 is configured to input historical audio signals into the audio classification network for initial training.

[0104] The adversarial perturbation training module 403 is configured to perform multiple rounds of adversarial perturbation training after the initial training phase. Each expert model generates a corresponding adversarial perturbation vector. The encoder extracts features from the real audio signal and the fake audio signal in the historical audio signal, and outputs real category feature vector and fake category feature vector respectively. The adversarial perturbation vector is added to the real category feature vector to form a proxy fake feature vector, so that the expert model misclassifies the real category feature vector as the fake category feature vector. The adversarial perturbation vector is continuously optimized until the multiple rounds of adversarial perturbation training ends. The adversarial perturbation vector at the end of the multiple rounds of adversarial perturbation training is saved to the history memory bank.

[0105] The new stage training module 404 is configured to input the current audio signal into the audio classification network for new stage training, randomly extract adversarial perturbation vectors from the historical memory bank and add them to the real category feature vector corresponding to the current audio signal, generate the proxy forgery feature vector, assign a forgery category label to the proxy forgery feature vector and incorporate it into the training of the audio classification network.

[0106] The following is for reference. Figure 5It shows a schematic diagram of the structure of a computer device 500 suitable for implementing an electronic device according to embodiments of the present invention. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0107] like Figure 5 As shown, the computer device 500 includes a central processing unit (CPU) 501 and a graphics processing unit (GPU) 502, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 503 or programs loaded from storage section 509 into random access memory (RAM) 504. The RAM 504 also stores various programs and data required for the operation of the device 500. The CPU 501, GPU 502, ROM 503, and RAM 504 are interconnected via a bus 505. An input / output (I / O) interface 506 is also connected to the bus 505.

[0108] The following components are connected to I / O interface 506: an input section 507 including a keyboard, mouse, etc.; an output section 508 including an LCD, speakers, etc.; a storage section 509 including a hard disk, etc.; and a communication section 510 including a network interface card, such as a LAN card or modem. The communication section 510 performs communication processing via a network such as the Internet. A drive 511 may also be connected to I / O interface 506 as needed. A removable medium 512, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 511 as needed so that computer programs read from it can be installed into storage section 509 as needed.

[0109] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 510, and / or installed from removable medium 512. When the computer program is executed by central processing unit (CPU) 501 and graphics processing unit (GPU) 502, the functions defined in the methods of this invention are performed.

[0110] It should be noted that the computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable medium may, for example, be— but is not limited to— an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the computer-readable medium can include— but are not limited to— an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium can include a data signal traveling in a baseband or an analog or digital carrier wave or other transport medium, in which the computer-readable program code is contained or stored. Such a program code traveling in a carrier wave can take any of a variety of forms, including but not limited to— electromagnetic, optical, or other types of signals. The computer-readable medium can also be any computer-readable medium other than the computer-readable medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The program code contained or stored by the computer-readable medium can be transmitted or propagated using any suitable medium, including but not limited to— wireless, wire line, optical fiber, RF, etc., or any suitable combination of the above.

[0111] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0112] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of the present application can be implemented by software or by hardware. The modules described can be arranged in a processor.

[0114] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following steps: S1, constructing an audio classification network, the audio classification network comprising an encoder, a detection module comprising a plurality of expert models, and an information bottleneck layer; S2, inputting historical audio signals into the audio classification network for initial stage training; S3, after the initial stage training is completed, performing a plurality of rounds of adversarial perturbation training, each expert model generating a corresponding adversarial perturbation vector, the encoder performing feature extraction on real audio signals and fake audio signals in the historical audio signals, and outputting real class feature vectors and fake class feature vectors respectively, adding the adversarial perturbation vector to the real class feature vector to form a proxy fake feature vector, causing the expert model to misclassify the real class feature vector as a fake class feature vector, and constantly optimizing the adversarial perturbation vector until the plurality of rounds of adversarial perturbation training are completed, and saving the adversarial perturbation vector at the end of the plurality of rounds of adversarial perturbation training to a historical memory bank; S4, inputting a current audio signal into the audio classification network for new stage training, and randomly extracting an adversarial perturbation vector from the historical memory bank and adding it to a real class feature vector corresponding to the current audio signal to generate the proxy fake feature vector, assigning a fake class label to the proxy fake feature vector and incorporating it into the training of the audio classification network.

[0115] The above description is only the preferred embodiment of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the inventive scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present application (but not limited to) with similar functions.

Claims

1. A method for detecting persistent forged speech based on multi-expert adversarial perturbation, characterized in that, Includes the following steps: S1. Construct an audio classification network, which includes an encoder, a detection module containing multiple expert models, and an information bottleneck layer; S2. Input the historical audio signal into the audio classification network for initial training. S3. After the initial training phase, multiple rounds of adversarial perturbation training are performed. Each expert model generates a corresponding adversarial perturbation vector. The encoder extracts features from the real audio signal and the fake audio signal in the historical audio signal, and outputs real category feature vector and fake category feature vector respectively. The adversarial perturbation vector is added to the real category feature vector to form a proxy fake feature vector, so that the expert model misclassifies the real category feature vector as the fake category feature vector. The adversarial perturbation vector is continuously optimized until the multiple rounds of adversarial perturbation training ends. The adversarial perturbation vector at the end of the multiple rounds of adversarial perturbation training is saved to the history memory bank. S4. Input the current audio signal into the audio classification network for a new stage of training, and randomly extract adversarial perturbation vectors from the historical memory bank and add them to the real category feature vector corresponding to the current audio signal to generate the proxy forgery feature vector. Assign a forgery category label to the proxy forgery feature vector and incorporate it into the training of the audio classification network.

2. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 1, characterized in that, The step S2, which involves inputting historical audio signals into the audio classification network for initial training, specifically includes: after the historical audio signals are input into the audio classification network, feature vectors are extracted by the encoder. Where T represents the size of the time domain dimension and S represents the size of the frequency domain dimension, the detection module includes An expert model The feature vector After processing by each expert model in the detection module, each expert model outputs a corresponding feature vector. The feature vectors output by all expert models are then compressed through the information bottleneck layer to generate the final discriminant vector. Used for audio classification.

3. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 2, characterized in that, In step S2, multiple loss functions are used to optimize the audio classification network trained in the initial stage. The multiple loss functions use hyperparameters that control orthogonality and information compression intensity to weight and combine the mutual information loss function based on the difference in the predicted distribution of the expert model, the information bottleneck loss function based on the information bottleneck layer compression process that preserves task-related information, and the cross-entropy loss function based on the classification results of the audio classification network.

4. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 3, characterized in that, The multiple loss function is defined as follows: ， in, represent the hyperparameters controlling orthogonality and information compression intensity, respectively. The mutual information loss function is defined as follows: ， in, This represents the joint feature vector of all expert models. Indicates except External joint feature vectors, These represent the corresponding predicted distributions. This indicates the calculation of the Kullback-Leibler divergence; The information bottleneck loss function is defined as follows: , The parameters representing the information bottleneck layer; The cross-entropy loss function is defined as follows: ， in, The audio classification network represents The set of historical audio signals input during the initial training phase. This indicates the data size of the corresponding historical audio signal set. This represents the feature vector extracted by the encoder. This indicates the corresponding category label.

5. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 3, characterized in that, In step S4, an updated multiple loss function is used to optimize the audio classification network trained in the new stage. The updated multiple loss function employs a single-class consistency loss function based on ensuring the consistency of the true class features across different training stages. Replacing one of the multiple loss functions, the specific definition of the single-class consistency loss function is as follows: in This represents the discriminant vector corresponding to the forgery category feature vector and the proxy forgery feature vector. This represents the discriminant vector corresponding to the true category feature vector. and These represent the number of corresponding discriminant vectors in each training round. The true category feature center discriminant vector is represented by the true category feature center discriminant vector, which is updated by calculating the average value of the discriminant vectors corresponding to the true category feature vector.

6. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 5, characterized in that, The updated multi-loss function replaces the cross-entropy loss with a single-class consistency loss function that guarantees the consistency of the true class features across different training stages. The updated multi-loss function is defined as follows: .

7. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 1, characterized in that, The S3 step of continuously optimizing the adversarial perturbation vector until the multi-round adversarial perturbation training ends includes: updating the adversarial perturbation vector corresponding to each expert model by calculating the mean square loss between the proxy forgery feature vector processed by each expert model and the forgery category feature vector; constraining the magnitude of the perturbation of the adversarial perturbation vector to be less than or equal to the magnitude threshold; and defining the multi-round adversarial perturbation training as ending when the misclassification rate of the expert model exceeds a predetermined misclassification rate threshold.

8. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 7, characterized in that, The proxy forgery feature vector is defined as the following element-level summation: ， Indicates the first The encoder described in the adversarial perturbation training uses a set of real audio signals. The true category features extracted from them Indicates the first The proxy-forged feature vector generated during adversarial perturbation training. This indicates that the nth expert model is at the th . The adversarial perturbation vector obtained from the round of adversarial perturbation training, the adversarial perturbation vector The optimization objective is: ， st ， The anti-disturbance vector The mean squared error is updated using the following stochastic gradient descent method after optimization: ， This means transforming the joint feature vectors of all expert models into predicted values. Represents the norm, This indicates the threshold value for the disturbance amplitude. This indicates that the category is classified as forgery. The corresponding predicted value, This refers to the sign function. This represents the perturbation learning rate. This indicates the calculation of the mean square error. This indicates that the calculation of the nth expert model at the 1st... Adversarial perturbation vectors obtained from wheel adversarial perturbation training The gradient.

9. The method for detecting persistent forged speech based on multi-expert adversarial perturbation according to claim 1, characterized in that, Also includes: S5. After the training of the new stage ends, repeat the steps of S3 to carry out multi-round adversarial perturbation training for the new stage, and save the adversarial perturbation vector at the end of the multi-round adversarial perturbation training of the new stage to the history memory bank.

10. A continuous forged speech detection system based on multi-expert adversarial perturbation, characterized in that, The method according to any one of claims 1 to 9 comprises the following modules: An audio classification network construction module is configured to construct an audio classification network, which includes an encoder, a detection module containing multiple expert models, and an information bottleneck layer. The initial training module is configured to input historical audio signals into the audio classification network for initial training. The adversarial perturbation training module is configured to perform multiple rounds of adversarial perturbation training after the initial training phase. Each expert model generates a corresponding adversarial perturbation vector. The encoder extracts features from the real audio signal and the fake audio signal in the historical audio signal, and outputs real category feature vector and fake category feature vector respectively. The adversarial perturbation vector is added to the real category feature vector to form a proxy fake feature vector, so that the expert model misclassifies the real category feature vector as the fake category feature vector. The adversarial perturbation vector is continuously optimized until the multiple rounds of adversarial perturbation training are completed. The adversarial perturbation vector at the end of the multiple rounds of adversarial perturbation training is saved to the history memory bank. The new stage training module is configured to input the current audio signal into the audio classification network for new stage training, randomly extract adversarial perturbation vectors from the historical memory bank and add them to the real category feature vector corresponding to the current audio signal, generate the proxy forgery feature vector, assign a forgery category label to the proxy forgery feature vector and incorporate it into the training of the audio classification network.

Images