A method, device, electronic device and storage medium for generating speech adversarial samples
Through the multi-dimensional weighted fusion and optimization technology of voiceprint, false identification and sound quality models, voice adversarial samples are generated, which solves the problems of low generation efficiency and great influence of sound quality in the existing technology, and achieves efficient and low-impact adversarial samples generation.
Patent Information
- Application Number
- CN202410970241.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-19
AI Technical Summary
The prior art is inefficient in generating speech adversarial samples, making it difficult to quickly generate adversarial samples that can effectively attack neural network models, and at the same time has a great impact on the sound quality of the original speech samples.
Through the weighted fusion of the voiceprint dimension of the voiceprint classification model, the false-analysis dimension of the false-analysis classification model, and the quality dimension of the sound quality evaluation model, the anti-analysis signal is generated, and the disturbance signal is optimized using gradient backpropagation and zero-order optimization methods, combined with editing Distance processing method and partial sampling perturbation technology improve the generation efficiency and sound quality fidelity of countermeasure samples.
While ensuring high audio similarity and sound quality, the generation time of the adversarial sample is significantly reduced, and the attack efficiency against perturbation is improved, and the effect on the sound quality of the original speech sample is reduced.
Smart Images

Figure CN118522290B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and in particular to a method, device, electronic device and storage medium for generating speech adversarial samples. Background Art
[0002] Since deep neural networks are susceptible to slight disturbances, adversarial sample attacks and defenses against voice intelligence technology systems have become a hot topic of research in recent years. Adversarial samples refer to input samples generated by deliberately adding slight disturbances to the data, which can cause the neural network model to give an incorrect prediction result. There are two basic goals: one is to change the prediction result of the model; the other is that the disturbance added to the input is not enough to cause the change of the model prediction result in the eyes of humans, and is superficially harmless. In the process of generating adversarial samples, the recognition probability of the target model instructions is maximized by optimizing the characteristics of the audio signal without affecting the audibility of the original audio signal. It is necessary to collect the voice samples of the target model instructions first, and then construct an optimization target based on the characteristics of the target model instructions and the voice classification model, and finally generate an audio signal similar to the original voice. However, this method will lead to low efficiency in generating samples, so how to quickly generate voice adversarial samples has become a technical problem that cannot be underestimated. Summary of the invention
[0003] In view of this, the purpose of the present application is to provide a method, device, electronic device and storage medium for generating speech adversarial samples, which improves the attack efficiency of adversarial disturbances by weighted fusion of the first disturbance signal on the voiceprint dimension of the voiceprint classification model, the first disturbance signal on the false identification dimension of the false identification classification model and the first disturbance signal on the quality dimension, and reduces the influence of the disturbance on the sound quality of the original speech sample, and greatly reduces the time for generating adversarial samples while ensuring high audio similarity and sound quality.
[0004] The present invention provides a method for generating a speech adversarial sample, and the method for generating a speech adversarial sample includes:
[0005] Inputting the target speech signal into the voiceprint classification model and the forgery detection classification model respectively, processing the target speech signal, and determining the voiceprint loss value and the forgery detection loss value;
[0006] Based on the voiceprint classification model, a gradient back propagation calculation is performed on the voiceprint loss value to determine a first disturbance signal in the voiceprint dimension; based on the false identification classification model, a gradient back propagation calculation is performed on the false identification loss value to determine a first disturbance signal in the false identification dimension;
[0007] Based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the authentication dimension, anti-disturbance addition is performed on the target voice signal to determine an initial adversarial sample, and the initial adversarial sample and the target voice signal are input into a sound quality evaluation model to determine an audio difference score between the initial adversarial sample and the target voice signal;
[0008] Determine a first disturbance signal in the quality dimension based on the audio difference score value, process the initial adversarial sample based on the voiceprint classification model and the counterfeit detection classification model, and determine a second disturbance signal in the voiceprint dimension and a second disturbance signal in the counterfeit detection dimension;
[0009] Based on the second disturbance signal on the voiceprint dimension, the second disturbance signal on the authentication dimension and the first disturbance signal on the quality dimension, weighted fusion is performed, and the target speech signal is adversarially disturbed and added using the disturbance signal after weighted fusion. The disturbance signals on different dimensions are continuously generated until the number of iterations is met, and the target adversarial sample of the target speech signal is determined.
[0010] In a possible implementation manner, the target speech signal is input into a voiceprint classification model and a false identification classification model respectively, the target speech signal is processed, and a voiceprint loss value and a false identification loss value are determined, including:
[0011] Inputting the target voice signal into the voiceprint classification model, performing voiceprint classification prediction processing on the target voice signal, and outputting a first voiceprint classification result of the target voice signal;
[0012] Obtaining a real voiceprint classification result of the target voice signal, and determining the voiceprint loss value based on the real voiceprint classification result and the first voiceprint classification result; wherein the real voiceprint classification result is obtained by performing voiceprint classification prediction processing on the target voice signal through a target adversarial model;
[0013] Inputting the target speech signal into the false identification classification model, performing false identification classification prediction processing on the target speech signal, and outputting a first false identification classification result of the target speech signal;
[0014] Obtain a true false identification classification result of the target speech signal, and determine the false identification loss value based on the true false identification classification result and the first false identification classification result; wherein the true false identification classification result is obtained by performing false identification classification prediction processing on the target speech signal through a target adversarial model.
[0015] In a possible implementation manner, inputting the target voice signal into the voiceprint classification model, performing voiceprint classification prediction processing on the target voice signal, and outputting a first voiceprint classification result of the target voice signal includes:
[0016] Inputting the target voice signal into the voiceprint classification model, extracting a feature representation vector of the target voice signal, and determining a feature representation vector of the target voice signal;
[0017] Inputting the feature representation vector into the first stacked network layer of the voiceprint classification model, processing the feature representation vector, and determining the processed feature representation vector; wherein the stacked network layer includes a residual connection layer, a graph convolution attention layer, and a pooling layer;
[0018] The next stacked network layer continues to process the processed feature representation vector, determines the target feature representation vector, inputs the target feature representation vector into the long short-term memory network layer of the voiceprint classification model, performs voiceprint classification prediction processing on the target feature representation vector, and outputs the first voiceprint classification result of the target speech signal.
[0019] In a possible implementation manner, performing gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension includes:
[0020] Solving the gradient of the voiceprint loss value based on a zero-order optimization solution method to determine gradient information;
[0021] The gradient information is subjected to gradient back propagation calculation along the long short-term memory network layer and multiple stacked network layers in the voiceprint classification model, and a first disturbance signal in the voiceprint dimension is generated at the input end of the voiceprint classification model.
[0022] In a possible implementation manner, inputting the initial adversarial sample and the target speech signal into a sound quality evaluation model to determine an audio difference score between the initial adversarial sample and the target speech signal includes:
[0023] Inputting the initial adversarial sample and the target speech signal into a sound quality evaluation model, evaluating and scoring the audio quality of the initial adversarial sample and the target speech signal, and determining the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal;
[0024] The absolute value of the difference between the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal is used as the audio difference score.
[0025] In a possible implementation manner, determining the first disturbance signal in the quality dimension based on the audio difference score value includes:
[0026] Determining a disturbance signal difference between the target speech signal and the initial adversarial sample;
[0027] A first disturbance signal in the quality dimension is determined based on a negative product of the audio difference score value and the disturbance signal difference value.
[0028] In a possible implementation manner, before weighted fusion is performed on the second disturbance signal based on the voiceprint dimension, the second disturbance signal based on the authentication dimension, and the first disturbance signal based on the quality dimension, and adversarial disturbance is added to the target speech signal using the weighted fused disturbance signal, the speech adversarial sample generation method further includes:
[0029] Based on the edit distance processing method, multiple key waveform segments of the weighted fused disturbance signal are determined, the disturbance signal is retained for the waveform signal corresponding to the key waveform segments, and the disturbance signal is not retained for the waveform signal corresponding to the non-key waveform segments.
[0030] The embodiment of the present application also provides a speech adversarial sample generation device, the speech adversarial sample generation device comprising:
[0031] A loss value determination module, used to input the target speech signal into the voiceprint classification model and the forgery detection classification model respectively, process the target speech signal, and determine the voiceprint loss value and the forgery detection loss value;
[0032] a first disturbance signal generating module, configured to perform a gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension, and perform a gradient back propagation calculation on the authentication loss value based on the authentication classification model to determine a first disturbance signal in the authentication dimension;
[0033] A sound quality scoring module, configured to perform anti-disturbance addition on the target voice signal based on the first disturbance signal on the voiceprint dimension and the first disturbance signal on the authentication dimension, determine an initial adversarial sample, input the initial adversarial sample and the target voice signal into a sound quality evaluation model, and determine an audio difference score between the initial adversarial sample and the target voice signal;
[0034] A second disturbance signal generating module, configured to determine a first disturbance signal in a quality dimension based on the audio difference score value, and process the initial adversarial sample based on the voiceprint classification model and the counterfeit detection classification model to determine a second disturbance signal in a voiceprint dimension and a second disturbance signal in a counterfeit detection dimension;
[0035] A determination module is used to perform weighted fusion based on the second disturbance signal in the voiceprint dimension, the second disturbance signal in the authentication dimension, and the first disturbance signal in the quality dimension, and to add adversarial disturbance to the target speech signal using the disturbance signal after weighted fusion, and to continue to generate disturbance signals in different dimensions until the number of iterations is met, thereby determining a target adversarial sample of the target speech signal.
[0036] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of generating speech adversarial samples as described above are performed.
[0037] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of generating speech adversarial samples as described above are performed.
[0038] The embodiments of the present application provide a method, device, electronic device and storage medium for generating speech adversarial samples. The method for generating speech adversarial samples comprises: inputting a target speech signal into a voiceprint classification model and a false identification classification model respectively, processing the target speech signal, and determining a voiceprint loss value and a false identification loss value; performing gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension, and performing gradient back propagation calculation on the false identification loss value based on the false identification classification model to determine a first disturbance signal in the false identification dimension; performing anti-disturbance addition on the target speech signal based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the false identification dimension to determine an initial adversarial sample, and adding the initial adversarial sample to the target speech signal. The target speech signal is input into the sound quality evaluation model to determine the audio difference score between the initial adversarial sample and the target speech signal; the first disturbance signal on the quality dimension is determined based on the audio difference score, the initial adversarial sample is processed based on the voiceprint classification model and the false identification classification model to determine the second disturbance signal on the voiceprint dimension and the second disturbance signal on the false identification dimension; the second disturbance signal on the voiceprint dimension, the second disturbance signal on the false identification dimension and the first disturbance signal on the quality dimension are weighted and fused, the target speech signal is adversarially disturbed and added by using the disturbance signal after weighted fusion, and disturbance signals on different dimensions are continuously generated until the number of iterations is met, and the target adversarial sample of the target speech signal is determined. The attack efficiency of the adversarial disturbance is improved by weighted fusion of the first disturbance signal on the voiceprint dimension of the voiceprint classification model, the first disturbance signal on the false identification dimension of the false identification classification model and the first disturbance signal on the quality dimension, and the influence of the disturbance on the sound quality of the original speech sample is reduced, and the time for generating adversarial samples is greatly reduced under the premise of ensuring high audio similarity and sound quality.
[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A flowchart of a method for generating speech adversarial samples provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram of a method for generating speech adversarial samples provided in an embodiment of the present application;
[0043] Figure 3 One of the structural schematic diagrams of a speech adversarial sample generation device provided in an embodiment of the present application;
[0044] Figure 4 The second structural diagram of a speech adversarial sample generation device provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.
[0047] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of speech processing technology.
[0048] Research has found that because deep neural networks are easily affected by slight disturbances, adversarial sample attacks and defenses against voice intelligence technology systems have become a hot topic in recent years. Adversarial samples refer to input samples generated by deliberately adding slight disturbances to the data, which can cause the neural network model to give an incorrect prediction result. There are two basic goals: one is to change the prediction result of the model; the other is that the disturbance added to the input is not enough to cause the change of the model prediction result in the eyes of humans, and is harmless on the surface. In the process of generating adversarial samples, the recognition probability of the target model instructions is maximized by optimizing the characteristics of the audio signal without affecting the audibility of the original audio signal. It is necessary to collect the voice samples of the target model instructions first, and then construct an optimization target based on the characteristics of the target model instructions and the voice classification model, and finally generate an audio signal similar to the original voice. However, this method will lead to low efficiency in generating samples, so how to quickly generate voice adversarial samples has become a technical problem that cannot be underestimated.
[0049] Based on this, an embodiment of the present application provides a method for generating speech adversarial samples, which improves the attack efficiency of adversarial disturbances through weighted fusion of the first disturbance signal on the voiceprint dimension of the voiceprint classification model, the first disturbance signal on the false identification dimension of the false identification classification model, and the first disturbance signal on the quality dimension, and reduces the impact of the disturbance on the sound quality of the original speech sample. Under the premise of ensuring high audio similarity and sound quality, the time for generating adversarial samples is greatly reduced.
[0050] See also Figure 1 , Figure 1 This is a flow chart of a method for generating speech adversarial samples provided in an embodiment of the present application. Figure 1 As shown in , the speech adversarial sample generation method provided in the embodiment of the present application includes:
[0051] S101: Input the target speech signal into the voiceprint classification model and the authentication classification model respectively, process the target speech signal, and determine the voiceprint loss value and the authentication loss value.
[0052] In this step, the target speech signal is input into the voiceprint classification model and the forgery detection classification model, the target speech signal is processed, and the voiceprint loss value and the forgery detection loss value are determined.
[0053] Among them, the voiceprint classification model and the counterfeit detection classification model are obtained by pre-training the neural network learning model through self-supervision. The training process of the voiceprint classification model and the counterfeit detection classification model is the same, but their training labels and training tasks are different.
[0054] In a possible implementation manner, the target speech signal is input into a voiceprint classification model and a false identification classification model respectively, the target speech signal is processed, and a voiceprint loss value and a false identification loss value are determined, including:
[0055] A: Input the target voice signal into the voiceprint classification model, perform voiceprint classification prediction processing on the target voice signal, and output a first voiceprint classification result of the target voice signal.
[0056] Here, the target voice signal is input into the voiceprint classification model, the target voice signal is subjected to voiceprint classification prediction processing, and the first voiceprint classification result of the target voice signal is output.
[0057] The voiceprint classification result includes the voiceprint category of the target speech signal.
[0058] In a possible implementation manner, inputting the target voice signal into the voiceprint classification model, performing voiceprint classification prediction processing on the target voice signal, and outputting a first voiceprint classification result of the target voice signal includes:
[0059] (1): Input the target voice signal into the voiceprint classification model, extract the feature representation vector of the target voice signal, and determine the feature representation vector of the target voice signal.
[0060] Here, the target speech signal is input into the voiceprint classification model, the feature representation vector of the target speech signal is extracted, and the feature representation vector of the target speech signal is determined.
[0061] (2): Input the feature representation vector into the first stacked network layer of the voiceprint classification model, process the feature representation vector, and determine the processed feature representation vector; wherein the stacked network layer includes a residual connection layer, a graph convolution attention layer, and a pooling layer.
[0062] Here, the feature representation vector is input into the first stacked network layer of the voiceprint classification model, the feature representation vector is processed, and the processed feature representation vector is determined; wherein the stacked network layer includes a residual connection layer, a graph convolution attention layer and a pooling layer.
[0063] Among them, end-to-end feature extraction is used to quickly extract audio information, the residual connection layer can effectively connect the convolution context information, the graph convolution attention layer is used to learn the relationship between clues across different sub-bands and time intervals, and feature aggregation is achieved through the pooling layer. Here, the stacked network layer is composed of residual connections, graph convolution attention layers, and pooling layers. In the classification network, N such network layers can be stacked as needed. Long short-term memory networks can better capture long-term dependencies in sequence data, which effectively improves the classification model's ability to identify sound signals.
[0064] (3): The next stacked network layer continues to process the processed feature representation vector, determines the target feature representation vector, inputs the target feature representation vector into the long short-term memory network layer of the voiceprint classification model, performs voiceprint classification prediction processing on the target feature representation vector, and outputs the first voiceprint classification result of the target speech signal.
[0065] Here, the next stacked network layer continues to process the processed feature representation vector, determines the target feature representation vector, inputs the target feature representation vector into the long short-term memory network layer of the voiceprint classification model, performs voiceprint classification prediction processing on the target feature representation vector, and outputs the first voiceprint classification result of the target speech signal.
[0066] Among them, because the training of the voiceprint classification model uses the gradient descent method to minimize the target loss function, in order to attack the voiceprint classification model, it is only necessary to maximize the loss function in the direction of gradient ascent. By iterating the gradient of the current input data, the loss function is maximized to finally obtain the adversarial disturbance.
[0067] B: Obtain the real voiceprint classification result of the target voice signal, and determine the voiceprint loss value based on the real voiceprint classification result and the first voiceprint classification result; wherein the real voiceprint classification result is obtained by performing voiceprint classification prediction processing on the target voice signal through a target adversarial model.
[0068] Here, the loss function calculation formula is used to calculate the real voiceprint classification result and the first voiceprint classification result to determine the voiceprint loss value of the voiceprint classification model.
[0069] Here, the target adversarial model is the model that the speech adversarial sample is going to attack.
[0070] C: Inputting the target speech signal into the false identification and classification model, performing false identification and classification prediction processing on the target speech signal, and outputting a first false identification and classification result of the target speech signal.
[0071] Here, the target speech signal is input into the false identification and classification model, the false identification and classification prediction processing is performed on the target speech signal, and the first false identification and classification result of the target speech signal is output.
[0072] D: Obtain a true false identification classification result of the target speech signal, and determine the false identification loss value based on the true false identification classification result and the first false identification classification result; wherein the true false identification classification result is obtained by performing false identification classification prediction processing on the target speech signal through a target adversarial model.
[0073] Here, the real counterfeit detection and classification result and the first counterfeit detection and classification result are calculated according to the loss function calculation formula to determine the counterfeit detection and classification model's counterfeit detection loss value.
[0074] S102: performing gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension; performing gradient back propagation calculation on the authentication loss value based on the authentication classification model to determine a first disturbance signal in the authentication dimension.
[0075] In this step, a gradient back propagation calculation is performed on the voiceprint loss value according to the voiceprint classification model to determine the first disturbance signal in the voiceprint dimension, and a gradient back propagation calculation is performed on the false identification loss value according to the false identification classification model to determine the first disturbance signal in the false identification dimension.
[0076] Here, the voiceprint classification model inputs adversarial speech samples, and the voiceprint loss value of the voiceprint classification model is determined by using the output voiceprint authentication results and the query results for the black box target model. The larger the loss function value, the better the adversarial attack effect from the perspective of voiceprint classification. The authentication classification model inputs adversarial speech samples, and the authentication loss value of the authentication classification model is determined by using the output authentication results and the query results for the black box target model. The larger the loss function value, the better the adversarial attack effect from the perspective of audio authentication classification.
[0077] In a possible implementation manner, performing gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension includes:
[0078] The gradient of the voiceprint loss value is solved based on a zero-order optimization solution method to determine the gradient information; the gradient information is back-propagated along the long short-term memory network layer and multiple stacked network layers in the voiceprint classification model to generate a first disturbance signal in the voiceprint dimension at the input end of the voiceprint classification model.
[0079] Here, the gradient of the voiceprint loss value is solved according to the zero-order optimization solution method to determine the gradient information so that the gradient information can be back-propagated along the long short-term memory network layer and multiple stacked network layers in the voiceprint classification model to generate the first disturbance signal in the voiceprint dimension at the input end of the voiceprint classification model.
[0080] Here, the zero-order optimization method is used to solve the problem that black box attacks cannot obtain model gradient information. The gradient back propagation calculation formula is:
[0081]
[0082] In the formula, Indicates that the voiceprint classification model converts the sample Classification The probability of (the classification label in the present invention can be a voiceprint category or a true or false category), represents the adjustment parameters, L ( x,t ) is the loss function calculation result, using The advantage of the function is that it is a monotonic function, so when the voiceprint classification model converts the sample Classification The probability formula is:
[0083]
[0084] Since the black box attack cannot obtain the gradient information of the model, the symmetric difference quotient and Hessian matrix are used to estimate the gradient of the black box model, and then the zero-order optimization method is used to solve the gradient. After obtaining the gradient, the perturbation generation signal can be obtained by back propagating along the gradient.
[0085] Here, the process of generating the first disturbance signal in the authentication dimension is substantially consistent with the process of determining the first disturbance signal in the voiceprint dimension. The difference is that the authentication classification result is calculated in the gradient back-propagation calculation formula.
[0086] S103: Based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the authentication dimension, anti-disturbance addition is performed on the target speech signal to determine an initial adversarial sample, and the initial adversarial sample and the target speech signal are input into a sound quality evaluation model to determine an audio difference score between the initial adversarial sample and the target speech signal.
[0087] In this step, anti-disturbance is added to the target speech signal according to the first disturbance signal in the voiceprint dimension and the first disturbance signal in the authentication dimension to determine the initial adversarial sample. The initial adversarial sample and the target speech signal are input into the sound quality evaluation model to determine the audio difference score between the initial adversarial sample and the target speech signal.
[0088] Here, the first disturbance signal in the voiceprint dimension is weighted and fused with the first disturbance signal in the authentication dimension, and the disturbance signal after the weighted fusion is added to the target speech signal for anti-disturbance.
[0089] In order to minimize disturbance and achieve the effect of "imperceptible to human ears", the present invention does not simply calculate the signal-to-noise ratio, but adopts an objective evaluation method to evaluate the quality of speech samples before and after the ongoing adversarial disturbance, so as to better measure the sound quality changes of adversarial samples in terms of naturalness and fluency. The sound quality evaluation model is trained by using speech clips and their corresponding subjective quality scores for factors such as background noise, reverberation, packet loss and network jitter.
[0090] In a possible implementation manner, inputting the initial adversarial sample and the target speech signal into a sound quality evaluation model to determine an audio difference score between the initial adversarial sample and the target speech signal includes:
[0091] I: Input the initial adversarial sample and the target speech signal into the sound quality evaluation model, evaluate and score the audio quality of the initial adversarial sample and the target speech signal, and determine the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal.
[0092] Here, the initial adversarial sample and the target speech signal are input into the sound quality evaluation model, the audio quality of the initial adversarial sample and the target speech signal are evaluated and scored, and the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal are determined.
[0093] II: Taking the absolute value of the difference between the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal as the audio difference score.
[0094] Here, the absolute value of the difference between the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal is taken as the audio difference score.
[0095] The audio difference score is determined by the following formula:
[0096]
[0097] in, L is the audio difference score, Indicates taking the absolute value, is the audio quality score of the adversarial sample, is the audio quality score of the target speech signal.
[0098] S104: Determine a first disturbance signal in the quality dimension based on the audio difference score value, process the initial adversarial sample based on the voiceprint classification model and the false identification classification model, and determine a second disturbance signal in the voiceprint dimension and a second disturbance signal in the false identification dimension.
[0099] In this step, a first disturbance signal in the quality dimension is determined according to the audio difference score value, and the initial adversarial sample is processed according to the voiceprint classification model and the counterfeit detection classification model to determine a second disturbance signal in the voiceprint dimension and a second disturbance signal in the counterfeit detection dimension.
[0100] Here, the processing process of the second disturbance signal in the voiceprint dimension and the second disturbance signal in the authentication dimension is consistent with the processing process of the first disturbance signal in the voiceprint dimension and the first disturbance signal in the authentication dimension mentioned above, and will not be repeated in this part.
[0101] In a possible implementation manner, determining the first disturbance signal in the quality dimension based on the audio difference score value includes:
[0102] Determine a disturbance signal difference between the target speech signal and the initial adversarial sample; and determine a first disturbance signal in a quality dimension based on a product of a negative audio difference score value and the disturbance signal difference.
[0103] Here, the first disturbance signal in the quality dimension is determined by the following formula:
[0104]
[0105] in, is the first disturbance signal in the mass dimension, is the target speech signal, For adversarial samples.
[0106] S105: Perform weighted fusion based on the second disturbance signal in the voiceprint dimension, the second disturbance signal in the authentication dimension, and the first disturbance signal in the quality dimension, use the disturbance signal after weighted fusion to add adversarial disturbance to the target speech signal, continue to generate disturbance signals in different dimensions until the number of iterations is met, and determine the target adversarial sample of the target speech signal.
[0107] In this step, weighted fusion is performed according to the second disturbance signal on the voiceprint dimension, the second disturbance signal on the authentication dimension, and the first disturbance signal on the quality dimension. The disturbance signal after weighted fusion is used to add adversarial disturbance to the target speech signal, and disturbance signals on different dimensions are continuously generated until the number of iterations is met, thereby determining the target adversarial sample of the target speech signal.
[0108] Here, in order to better generate adversarial disturbances, a three-branch multi-task fusion method is adopted, namely, the effect of adversarial samples of the voiceprint classification model, the effect of adversarial samples of the false identification classification model, and the change in the sound quality of the speech adversarial samples, which constitutes a weighted fusion of the disturbance signals of the three branches. The formula is as follows:
[0109] + +
[0110] in, represents the joint countermeasure disturbance signal, represents the adversarial disturbance signal output by the voiceprint classification model, represents the adversarial perturbation signal output by the false identification classification model, Represents the adversarial perturbation signal output by the sound quality evaluation model. , and Represents the hyperparameters to be optimized and the initial values are set to equal weights. .
[0111] In a possible implementation manner, before weighted fusion is performed on the second disturbance signal based on the voiceprint dimension, the second disturbance signal based on the authentication dimension, and the first disturbance signal based on the quality dimension, and adversarial disturbance is added to the target speech signal using the weighted fused disturbance signal, the speech adversarial sample generation method further includes:
[0112] Based on the edit distance processing method, multiple key waveform segments of the weighted fused disturbance signal are determined, the disturbance signal is retained for the waveform signal corresponding to the key waveform segments, and the disturbance signal is not retained for the waveform signal corresponding to the non-key waveform segments.
[0113] Here, multiple key waveform segments of the weighted fused disturbance signal are determined according to the edit distance processing method, the disturbance signal is retained for the waveform signal corresponding to the key waveform segment, and the disturbance signal is not retained for the waveform signal corresponding to the non-key waveform segment.
[0114] The present invention accelerates the generation time of adversarial samples and improves the generation efficiency of adversarial samples through partial sampling perturbation technology. Usually, it is sufficient to add slight perturbations to the entire waveform signal, but only a small amount of perturbations can be added to make the attack successful. The sampling perturbation technology is to add perturbations by fixing a part of the original waveform signal and only changing another part of the points, which can reduce the number of perturbations of the audio signal, so that the adversarial samples generated are closer to the original audio and have better adversarial effects. In the process of generating adversarial samples, when the adversarial samples in the intermediate process gradually approach the target, most of the sampling points no longer need to be perturbed. The time cost is reduced by adjusting the distortion weights at different positions, and the edit distance (Levenshtein distance) is used to solve this problem. The edit distance is a quantitative measurement of the degree of difference between two audio waveform signals. The measurement method is to see how many times of processing is required to transform a segment of audio waveform signal into another segment of audio waveform signal. The partial sampling perturbation technology focuses on points with relatively small edit distances but not 0, also called key points, and at the same time, these points are given greater weights to reduce the attack time. In the present invention, in order to avoid the hard decision threshold setting of key points, 15% of the waveform segments with non-zero edit distance and small distance are selected as key points as the output of the partial sampling perturbation strategy module. The sampling perturbation technology can reduce the attack generation time on the one hand, and avoid excessive perturbation and miss the optimal value of the algorithm on the other hand.
[0115] In the present invention, the adversarial disturbance obtained through voiceprint classification model learning is perturbed from the dimension of identity information contained in the speech. The adversarial disturbance obtained through false identification classification model learning is perturbed from the dimension of true and false information contained in the speech. By reducing the voice quality change value of the original sample and the adversarial sample before and after the disturbance (that is, the impact of the fluency and naturalness of the speech before and after the disturbance on the human ear), the learned adversarial disturbance is adjusted to perturb from the dimension of the sound quality change information of the speech. The partial sampling perturbation technology is used to speed up the generation time of adversarial samples and improve the generation efficiency of adversarial samples. And the optimization problem of adversarial samples is converted into maximizing the loss function of the voiceprint classification model and the loss function of the false identification classification model (that is, the attack success rate is high), while minimizing the audio difference score value (that is, the adversarial disturbance is small), and setting hyperparameters to balance the maximization of the loss function and the minimization of the adversarial disturbance.
[0116] Here, in the present invention, a data enhancement attack strategy can also be used to improve the attack success rate of adversarial samples. The data enhancement attack strategy is to enhance the original sample before inputting it into the model to improve the transferability. The attack strategy based on data enhancement can be used in conjunction with other adversarial sample generation methods while maintaining excellent transferability of its own attack method, so as to achieve a higher attack success rate. Specifically, before each round of selection, the original speech waveform and the adversarial speech waveform are randomly resized and filled with a fixed probability p.
[0117] For further information, please refer to Figure 2 , Figure 2 A schematic diagram of a method for generating speech adversarial samples provided in an embodiment of the present application. Figure 2As shown, the target speech signal is added to the target disturbance signal to obtain an adversarial sample. The initial target disturbance signal is 0. The adversarial sample is input into the voiceprint classification model to extract the feature representation vector and determine the feature representation vector of the adversarial sample. The feature representation vector is input into the residual connection layer, graph convolution attention layer and pooling layer of the multiple stacked network layers of the voiceprint classification model for processing in sequence to obtain the target feature representation vector. The target feature representation vector is input into the long short-term memory network layer to obtain the first voiceprint classification result. The adversarial sample is input into the target adversarial model to obtain the real voiceprint classification result. The loss function is calculated based on the real voiceprint classification result and the first voiceprint classification result to obtain the voiceprint loss value. The voiceprint loss value is back-propagated along the direction of the dotted arrow to determine the first disturbance signal in the voiceprint dimension. The adversarial sample is input into the false identification classification model to obtain the first false identification classification result. The false identification loss value is determined based on the first false identification classification result and the real false identification classification result. The false identification loss value is back-propagated to determine the first disturbance signal in the false identification dimension. The first disturbance signal on the voiceprint dimension and the first disturbance signal on the authentication dimension are anti-disturbance added to the target speech signal to continue to determine the adversarial sample, and the adversarial sample is input into the sound quality evaluation model to obtain the first disturbance signal on the quality dimension. The above steps are repeated to weightedly fuse the second disturbance signal on the voiceprint dimension, the second disturbance signal on the authentication dimension, and the first disturbance signal on the quality dimension, and the weighted fused disturbance signal is sampled and disturbed to generate the target disturbance signal. The above steps are repeated until the number of iterations is met to determine the target adversarial sample.
[0118] The present invention proposes to adopt a multi-task learning method to obtain the adversarial generation effect of voiceprint interference and counterfeit interference, and the controllable evaluation of the sound quality change of sound disturbance. Compared with other single-task methods, the adversarial disturbance generation effect is better, the controllability of the change of the auditory perception of sound disturbance, and the loss function of speech quality evaluation describes the quality evaluability before and after speech disturbance. For the weighted summation of the three adversarial disturbance signals defined from different dimensions, the joint optimization of the overall system is realized, which has high practical application value. Construct a voiceprint classification model and a counterfeit identification classification model based on self-supervised pre-training front-end learning, extract the representation vector through the self-supervised pre-training model, and make up for the problem of insufficient feature extraction and low representation degree caused by insufficient data in supervised training. Accelerate the generation time of adversarial samples through partial sampling perturbation technology, and improve the generation efficiency of adversarial samples. Sampling perturbation technology can reduce the number of perturbations of audio vectors, so that the adversarial samples generated are closer to the original audio and have better robustness. The information of key positions can be obtained in any iteration process, and the weights can be changed at any time to accelerate convergence, thereby improving the generation rate of adversarial audio.
[0119] The embodiment of the present application provides a method for generating a speech adversarial sample, the method comprising: inputting a target speech signal into a voiceprint classification model and a false identification classification model respectively, processing the target speech signal, and determining a voiceprint loss value and a false identification loss value; performing gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension, performing gradient back propagation calculation on the false identification loss value based on the false identification classification model to determine a first disturbance signal in the false identification dimension; performing anti-disturbance addition on the target speech signal based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the false identification dimension to determine an initial adversarial sample, and adding the initial adversarial sample and the target speech The signal is input into the sound quality evaluation model to determine the audio difference score between the initial adversarial sample and the target speech signal; the first disturbance signal in the quality dimension is determined based on the audio difference score, and the initial adversarial sample is processed based on the voiceprint classification model and the false identification classification model to determine the second disturbance signal in the voiceprint dimension and the second disturbance signal in the false identification dimension; the second disturbance signal in the voiceprint dimension, the second disturbance signal in the false identification dimension, and the first disturbance signal in the quality dimension are weighted and fused, and the target speech signal is adversarially disturbed and added using the disturbance signal after weighted fusion, and disturbance signals in different dimensions are continuously generated until the number of iterations is met, and the target adversarial sample of the target speech signal is determined. The attack efficiency of the adversarial disturbance is improved by weighted fusion of the first disturbance signal in the voiceprint dimension of the voiceprint classification model, the first disturbance signal in the false identification dimension of the false identification classification model, and the first disturbance signal in the quality dimension, and the influence of the disturbance on the sound quality of the original speech sample is reduced, and the time for generating adversarial samples is greatly reduced under the premise of ensuring high audio similarity and sound quality.
[0120] See also Figure 3 , Figure 4 , Figure 3 One of the structural schematic diagrams of a speech adversarial sample generation device provided in an embodiment of the present application; Figure 4 This is a second structural diagram of a speech adversarial sample generation device provided in an embodiment of the present application. Figure 3 As shown in , the speech adversarial sample generating device 300 includes:
[0121] The loss value determination module 310 is used to input the target speech signal into the voiceprint classification model and the forgery detection classification model respectively, process the target speech signal, and determine the voiceprint loss value and the forgery detection loss value;
[0122] A first disturbance signal generating module 320 is used to perform a gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension, and perform a gradient back propagation calculation on the authentication loss value based on the authentication classification model to determine a first disturbance signal in the authentication dimension;
[0123] A sound quality scoring module 330 is used to perform anti-disturbance addition on the target voice signal based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the authentication dimension to determine an initial adversarial sample, input the initial adversarial sample and the target voice signal into a sound quality evaluation model, and determine an audio difference score between the initial adversarial sample and the target voice signal;
[0124] A second disturbance signal generating module 340 is used to determine a first disturbance signal in a quality dimension based on the audio difference score value, and to process the initial adversarial sample based on the voiceprint classification model and the counterfeit detection classification model to determine a second disturbance signal in a voiceprint dimension and a second disturbance signal in a counterfeit detection dimension;
[0125] The determination module 350 is used to perform weighted fusion based on the second disturbance signal in the voiceprint dimension, the second disturbance signal in the authentication dimension and the first disturbance signal in the quality dimension, and add adversarial disturbance to the target speech signal using the disturbance signal after weighted fusion, and continue to generate disturbance signals in different dimensions until the number of iterations is met, thereby determining the target adversarial sample of the target speech signal.
[0126] Furthermore, when the loss value determination module 310 is used to input the target speech signal into the voiceprint classification model and the authentication classification model respectively, process the target speech signal, and determine the voiceprint loss value and the authentication loss value, the loss value determination module 310 is specifically used to:
[0127] Inputting the target voice signal into the voiceprint classification model, performing voiceprint classification prediction processing on the target voice signal, and outputting a first voiceprint classification result of the target voice signal;
[0128] Obtaining a real voiceprint classification result of the target voice signal, and determining the voiceprint loss value based on the real voiceprint classification result and the first voiceprint classification result; wherein the real voiceprint classification result is obtained by performing voiceprint classification prediction processing on the target voice signal through a target adversarial model;
[0129] Inputting the target speech signal into the false identification classification model, performing false identification classification prediction processing on the target speech signal, and outputting a first false identification classification result of the target speech signal;
[0130] Obtain a true false identification classification result of the target speech signal, and determine the false identification loss value based on the true false identification classification result and the first false identification classification result; wherein the true false identification classification result is obtained by performing false identification classification prediction processing on the target speech signal through a target adversarial model.
[0131] Furthermore, when the loss value determination module 310 is used to input the target voice signal into the voiceprint classification model, perform voiceprint classification prediction processing on the target voice signal, and output the first voiceprint classification result of the target voice signal, the loss value determination module 310 is specifically used to:
[0132] Inputting the target voice signal into the voiceprint classification model, extracting a feature representation vector of the target voice signal, and determining a feature representation vector of the target voice signal;
[0133] Inputting the feature representation vector into the first stacked network layer of the voiceprint classification model, processing the feature representation vector, and determining the processed feature representation vector; wherein the stacked network layer includes a residual connection layer, a graph convolution attention layer, and a pooling layer;
[0134] The next stacked network layer continues to process the processed feature representation vector, determines the target feature representation vector, inputs the target feature representation vector into the long short-term memory network layer of the voiceprint classification model, performs voiceprint classification prediction processing on the target feature representation vector, and outputs the first voiceprint classification result of the target speech signal.
[0135] Further, when the first disturbance signal generating module 320 is used to perform gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine the first disturbance signal in the voiceprint dimension, the first disturbance signal generating module 320 is specifically used to:
[0136] Solving the gradient of the voiceprint loss value based on a zero-order optimization solution method to determine gradient information;
[0137] The gradient information is subjected to gradient back propagation calculation along the long short-term memory network layer and multiple stacked network layers in the voiceprint classification model, and a first disturbance signal in the voiceprint dimension is generated at the input end of the voiceprint classification model.
[0138] Further, when the sound quality scoring module 330 is used to input the initial adversarial sample and the target speech signal into the sound quality evaluation model to determine the audio difference score value between the initial adversarial sample and the target speech signal, the sound quality scoring module 330 is specifically used to:
[0139] Inputting the initial adversarial sample and the target speech signal into a sound quality evaluation model, evaluating and scoring the audio quality of the initial adversarial sample and the target speech signal, and determining the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal;
[0140] The absolute value of the difference between the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal is used as the audio difference score.
[0141] Further, when the second disturbance signal generating module 340 is used to determine the first disturbance signal on the quality dimension based on the audio difference score value, the second disturbance signal generating module 340 is specifically used to:
[0142] Determining a disturbance signal difference between the target speech signal and the initial adversarial sample;
[0143] A first disturbance signal in the quality dimension is determined based on a negative product of the audio difference score value and the disturbance signal difference value.
[0144] Further, such as Figure 4 As shown, the speech adversarial sample generation device 300 further includes a sampling disturbance optimization module 360, and the sampling disturbance optimization module 360 is used to:
[0145] Based on the edit distance processing method, multiple key waveform segments of the weighted fused disturbance signal are determined, the disturbance signal is retained for the waveform signal corresponding to the key waveform segments, and the disturbance signal is not retained for the waveform signal corresponding to the non-key waveform segments.
[0146] The embodiment of the present application provides a speech adversarial sample generation device, which includes: a loss value determination module, which is used to input the target speech signal into a voiceprint classification model and a false identification classification model respectively, process the target speech signal, and determine the voiceprint loss value and the false identification loss value; a first disturbance signal generation module, which is used to perform gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine the first disturbance signal in the voiceprint dimension, and perform gradient back propagation calculation on the false identification loss value based on the false identification classification model to determine the first disturbance signal in the false identification dimension; a sound quality scoring module, which is used to perform anti-disturbance addition on the target speech signal based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the false identification dimension to determine the initial adversarial sample, and add the initial adversarial sample and The target speech signal is input into the sound quality evaluation model to determine the audio difference score between the initial adversarial sample and the target speech signal; a second disturbance signal generating module is used to determine the first disturbance signal in the quality dimension based on the audio difference score, and process the initial adversarial sample based on the voiceprint classification model and the false identification classification model to determine the second disturbance signal in the voiceprint dimension and the second disturbance signal in the false identification dimension; a determination module is used to perform weighted fusion based on the second disturbance signal in the voiceprint dimension, the second disturbance signal in the false identification dimension and the first disturbance signal in the quality dimension, and perform adversarial disturbance addition on the target speech signal using the weighted fused disturbance signal, and continue to generate disturbance signals in different dimensions until the number of iterations is met, and then determine the target adversarial sample of the target speech signal. The attack efficiency of adversarial disturbance is improved by weighted fusion of the first disturbance signal on the voiceprint dimension of the voiceprint classification model, the first disturbance signal on the authentication dimension of the authentication classification model, and the first disturbance signal on the quality dimension, and the influence of the disturbance on the sound quality of the original speech sample is reduced. Moreover, the time for generating adversarial samples is greatly reduced while ensuring high audio similarity and sound quality.
[0147] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .
[0148] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 as well as Figure 2The steps of generating speech adversarial samples in the method embodiment shown are specifically implemented in the method embodiment and will not be described in detail here.
[0149] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The steps of the method for generating speech adversarial samples in the method embodiment shown in the figure, and the specific implementation method thereof can be found in the method embodiment, which will not be described in detail here.
[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0151] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0154] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0155] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for generating speech adversarial samples, characterized in that: The speech adversarial sample generation method comprises: Inputting the target speech signal into the voiceprint classification model and the forgery detection classification model respectively, processing the target speech signal, and determining the voiceprint loss value and the forgery detection loss value; Based on the voiceprint classification model, a gradient back propagation calculation is performed on the voiceprint loss value to determine a first disturbance signal in the voiceprint dimension; based on the false identification classification model, a gradient back propagation calculation is performed on the false identification loss value to determine a first disturbance signal in the false identification dimension; Based on the first disturbance signal in the voiceprint dimension and the first disturbance signal in the authentication dimension, anti-disturbance addition is performed on the target voice signal to determine an initial adversarial sample, and the initial adversarial sample and the target voice signal are input into a sound quality evaluation model to determine an audio difference score between the initial adversarial sample and the target voice signal; Determine a first disturbance signal in the quality dimension based on the audio difference score value, process the initial adversarial sample based on the voiceprint classification model and the counterfeit detection classification model, and determine a second disturbance signal in the voiceprint dimension and a second disturbance signal in the counterfeit detection dimension; Based on the second disturbance signal on the voiceprint dimension, the second disturbance signal on the authentication dimension and the first disturbance signal on the quality dimension, weighted fusion is performed, and the target speech signal is adversarially disturbed and added using the disturbance signal after weighted fusion. The disturbance signals on different dimensions are continuously generated until the number of iterations is met, and the target adversarial sample of the target speech signal is determined.
2. The method for generating speech adversarial samples according to claim 1, characterized in that: The target speech signal is input into the voiceprint classification model and the forgery detection classification model respectively, the target speech signal is processed, and the voiceprint loss value and the forgery detection loss value are determined, including: Inputting the target voice signal into the voiceprint classification model, performing voiceprint classification prediction processing on the target voice signal, and outputting a first voiceprint classification result of the target voice signal; Obtaining a real voiceprint classification result of the target voice signal, and determining the voiceprint loss value based on the real voiceprint classification result and the first voiceprint classification result; wherein the real voiceprint classification result is obtained by performing voiceprint classification prediction processing on the target voice signal through a target adversarial model; Inputting the target speech signal into the false identification classification model, performing false identification classification prediction processing on the target speech signal, and outputting a first false identification classification result of the target speech signal; Obtain a true false identification classification result of the target speech signal, and determine the false identification loss value based on the true false identification classification result and the first false identification classification result; wherein the true false identification classification result is obtained by performing false identification classification prediction processing on the target speech signal through a target adversarial model.
3. The method for generating speech adversarial samples according to claim 2, characterized in that: The step of inputting the target voice signal into the voiceprint classification model, performing voiceprint classification prediction processing on the target voice signal, and outputting a first voiceprint classification result of the target voice signal includes: Inputting the target voice signal into the voiceprint classification model, extracting a feature representation vector of the target voice signal, and determining a feature representation vector of the target voice signal; Inputting the feature representation vector into the first stacked network layer of the voiceprint classification model, processing the feature representation vector, and determining the processed feature representation vector; wherein the stacked network layer includes a residual connection layer, a graph convolution attention layer, and a pooling layer; The next stacked network layer continues to process the processed feature representation vector, determines the target feature representation vector, inputs the target feature representation vector into the long short-term memory network layer of the voiceprint classification model, performs voiceprint classification prediction processing on the target feature representation vector, and outputs the first voiceprint classification result of the target speech signal.
4. The method for generating speech adversarial samples according to claim 1, characterized in that: The performing gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension includes: Solving the gradient of the voiceprint loss value based on a zero-order optimization solution method to determine gradient information; The gradient information is subjected to gradient back propagation calculation along the long short-term memory network layer and multiple stacked network layers in the voiceprint classification model, and a first disturbance signal in the voiceprint dimension is generated at the input end of the voiceprint classification model.
5. The method for generating speech adversarial samples according to claim 1, characterized in that: The step of inputting the initial adversarial sample and the target speech signal into a sound quality evaluation model to determine an audio difference score between the initial adversarial sample and the target speech signal includes: Inputting the initial adversarial sample and the target speech signal into a sound quality evaluation model, evaluating and scoring the audio quality of the initial adversarial sample and the target speech signal, and determining the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal; The absolute value of the difference between the audio quality score of the initial adversarial sample and the audio quality score of the target speech signal is used as the audio difference score.
6. The method for generating speech adversarial samples according to claim 1, characterized in that: The determining of a first disturbance signal on a quality dimension based on the audio difference score value comprises: Determining a disturbance signal difference between the target speech signal and the initial adversarial sample; A first disturbance signal in the quality dimension is determined based on a negative product of the audio difference score value and the disturbance signal difference value.
7. The method for generating speech adversarial samples according to claim 1, characterized in that: Before weighted fusion of the second disturbance signal based on the voiceprint dimension, the second disturbance signal based on the authentication dimension, and the first disturbance signal based on the quality dimension, and adding adversarial disturbance to the target speech signal using the weighted fused disturbance signal, the speech adversarial sample generation method further includes: Based on the edit distance processing method, multiple key waveform segments of the weighted fused disturbance signal are determined, the disturbance signal is retained for the waveform signal corresponding to the key waveform segments, and the disturbance signal is not retained for the waveform signal corresponding to the non-key waveform segments.
8. A speech adversarial sample generation device, characterized in that: The speech adversarial sample generating device comprises: A loss value determination module, used to input the target speech signal into the voiceprint classification model and the forgery detection classification model respectively, process the target speech signal, and determine the voiceprint loss value and the forgery detection loss value; a first disturbance signal generating module, configured to perform a gradient back propagation calculation on the voiceprint loss value based on the voiceprint classification model to determine a first disturbance signal in the voiceprint dimension, and perform a gradient back propagation calculation on the authentication loss value based on the authentication classification model to determine a first disturbance signal in the authentication dimension; A sound quality scoring module, configured to perform anti-disturbance addition on the target voice signal based on the first disturbance signal on the voiceprint dimension and the first disturbance signal on the authentication dimension, determine an initial adversarial sample, input the initial adversarial sample and the target voice signal into a sound quality evaluation model, and determine an audio difference score between the initial adversarial sample and the target voice signal; A second disturbance signal generating module, configured to determine a first disturbance signal in a quality dimension based on the audio difference score value, and process the initial adversarial sample based on the voiceprint classification model and the counterfeit detection classification model to determine a second disturbance signal in a voiceprint dimension and a second disturbance signal in a counterfeit detection dimension; A determination module is used to perform weighted fusion based on the second disturbance signal in the voiceprint dimension, the second disturbance signal in the authentication dimension, and the first disturbance signal in the quality dimension, and to add adversarial disturbance to the target speech signal using the disturbance signal after weighted fusion, and to continue to generate disturbance signals in different dimensions until the number of iterations is met, thereby determining a target adversarial sample of the target speech signal.
9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for generating speech adversarial samples as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating speech adversarial samples according to any one of claims 1 to 7 are executed.
Citation Information
Patent Citations
False voice recognition method and device, electronic equipment and storage medium
CN115497481A
Data enhancement method and device for ultrasonic voiceprint anti-counterfeiting
CN116110417A
Voiceprint recognition method and device, voiceprint model training method and device, equipment and medium
CN117423341A
Target speaker separation system, device and storage medium
US20240005941A1
Speech recognition front-end processing method and apparatus, and terminal device
WO2021179717A1
