A method, device, terminal device and medium for generating speech adversarial samples

Abstract

The present application is applicable to the field of voice security technology, and provides a method, device, terminal equipment and medium for generating voice adversarial samples. The method obtains a set of voice samples to be disturbed; calculates the voice similarity between other speakers and the current speaker to determine the target speaker; performs voice endpoint detection on the set of voice samples to be disturbed to obtain candidate disturbed voice frames; calculates the priority and priority frequency to determine the priority overlap frame; according to the priority overlap frame, determines the disturbance point and disturbance value in the set of voice samples to be disturbed, disturbs the set of voice samples to be disturbed, and obtains the disturbance result; calculates the loss value between the disturbance result and the voice sample set corresponding to the target speaker; uses the projected gradient descent algorithm to update the disturbance point and disturbance value, and uses the disturbance result with a loss value less than a preset loss threshold as the voice adversarial sample corresponding to the current speaker. The present application can improve the de-identification effect and the quality of speech synthesis.

Description

Technical Field

[0001] The present application belongs to the field of voice security technology, and in particular, relates to a method, apparatus, terminal device and medium for generating voice adversarial samples. Background Art

[0002] In the information age, voice information plays a huge role in all aspects of people's daily lives. Usually, users' voice information is stored in the cloud by voice service providers. These massive voice information can be used as training samples for models such as speech recognition, speaker recognition, and emotion recognition, and can also be used as research materials for user behavior pattern surveys and medical analysis.

[0003] However, since the voice information itself contains the speaker's unique identity information - voiceprint, once the user's voice information is obtained by criminals, even if it is only a small amount of voice information, criminals can use the voiceprint to launch a link attack, thereby stealing the voice user's private information, such as bank card passwords, daily routines, etc., which will pose a huge threat to the user's property and personal safety.

[0004] In order to solve this problem, a common method is to perform speaker de-identification on voice information and anonymize the speaker's voiceprint so that attackers cannot obtain the target victim's private information through link attacks. Currently, there are three main solutions for speaker de-identification:

[0005] Solution 1: Voice conversion

[0006] Voice conversion refers to converting the source speaker's voiceprint into the target speaker's voiceprint using a conversion function while retaining the original content. However, this solution has obvious disadvantages. It requires a large number of parallel corpora as support and is easily vulnerable to reverse conversion attacks.

[0007] Solution 2: Speech Synthesis

[0008] Speech synthesis refers to extracting the speaker's identity features, semantic content, fundamental frequency F0 and other necessary information from the speech, and then re-synthesizing the new speech with the semantic content and F0 after modifying the identity features. Although this solution has a good identity recognition effect, it is limited by the current technical tool chain. The synthesized new speech is quite different from the real speech in terms of speech speed and emotional changes, and the semantic content of the synthesized speech also has a high error rate.

[0009] Solution 3: Reduce voice quality

[0010] Reducing speech quality is to reduce the speech signal-to-noise ratio by reducing the sampling rate, adding white noise, etc., making it difficult for the speaker recognition system to obtain valid identity information. Although this solution can reduce the speaker recognition rate to a certain extent, it also greatly reduces the speech quality.

[0011] It can be seen that the current speaker de-identification method has the problems of poor identity de-identification effect and low speech synthesis quality, and it is difficult to be effective in protecting the security of voice information. Summary of the invention

[0012] The present application provides a method, apparatus, terminal device and medium for generating speech adversarial samples, which can solve the problems of poor identity de-identification effect and low speech synthesis quality in traditional speaker de-identification methods.

[0013] In a first aspect, the present application provides a method for generating speech adversarial samples, comprising:

[0014] Step 1, obtaining a speech sample set to be disturbed corresponding to the current speaker and speech sample sets corresponding to multiple other speakers;

[0015] Step 2, respectively calculating the speech similarity between each other speaker and the current speaker, and determining the target speaker from other speakers whose speech similarity is less than a preset similarity threshold according to the speech similarity;

[0016] Step 3, performing speech endpoint detection on the speech sample set to be disturbed to obtain a plurality of candidate disturbed speech frames in the speech sample set to be disturbed; the candidate disturbed speech frames are used to determine the disturbed speech frames;

[0017] Step 4, using multiple different speaker recognition models to respectively calculate the priority and priority frequency corresponding to each candidate disturbed speech frame, and determine the priority overlap frame in the to-be-disturbed speech sample set according to the priority frequency and priority; the priority frequency is used to determine the disturbed speech frame from the multiple candidate disturbed speech frames, and the priority overlap frame represents the speech frame with the highest priority to be disturbed;

[0018] Step 5, according to the priority overlapped frames, determine the disturbance point in the to-be-disturbed speech sample set, initialize the disturbance value corresponding to the disturbance point, and according to the disturbance point and the disturbance value, disturb the to-be-disturbed speech sample set to obtain the disturbance result;

[0019] Step 6, calculating the loss value between the perturbation result and the speech sample set corresponding to the target speaker;

[0020] Step 7: If the loss value is greater than the preset loss threshold, the projected gradient descent algorithm is used to update the disturbance point and disturbance value, and the disturbance result is disturbed using the updated disturbance point and the updated disturbance value to obtain a new disturbance result. The new disturbance result is used as the disturbance result in step 6, and the process returns to step 6; otherwise, the disturbance result obtained in step 5 is used as the speech adversarial sample corresponding to the current speaker.

[0021] Optionally, the expression of voice similarity is:

[0022]

[0023] Among them, Sim(u orig ,u i ) indicates the current speaker u orig The speech similarity between the current speaker and the i-th speaker, i = 1, 2, ..., m, m represents the total number of other speakers, n represents the total number of speech samples corresponding to the current speaker, X k Indicates the current speaker u orig The corresponding speech sample set X to be disturbed orig The kth disturbed speech sample in X orig ={X1,X2,...,X k ,...,X d}, k = 1, 2, ..., d, d represents the speech sample set X to be disturbed orig The total number of speech samples to be disturbed, f0(X k ,u i ) indicates that the speaker recognition model f0(·) considers the perturbed speech sample X k The confidence level belonging to the i-th other speaker, The speaker recognition model f0(·) is any speaker recognition model, Indicates that the speaker recognition model f0 believes that the speech sample X k ∈u i confidence level.

[0024] Optionally, step 2 includes:

[0025] Other speakers whose speech similarity is less than a preset similarity threshold are arranged in order of speech similarity from small to large, and one other speaker is randomly selected from the top 20% of other speakers as the target speaker.

[0026] Optionally, step 4 includes:

[0027] Copy the disturbed speech sample set to obtain a new disturbed speech sample set in, {F1,F2,...,F H} represents the speech frame in the speech sample set to be disturbed, H represents the total number of speech frames, each speech frame includes C sampling points, X HC represents the Cth sampling point in the Hth speech frame;

[0028] For each candidate disturbed speech frame, the new to-be-perturbed speech sample set The sampling points corresponding to the candidate disturbed speech frames are set to zero;

[0029] For each speaker recognition model, the formula is calculated Get priority P sh ; Among them, P sh represents the priority of the hth candidate perturbed speech frame calculated by the sth speaker recognition model, s = 1, 2, ..., Q, Q represents the total number of preset speaker recognition models, Q speaker recognition models are different from each other, h = 1, 2, ..., L, L represents the total number of candidate perturbed speech frames obtained after processing in step 3, Indicates that the current speaker recognition model believes that the speech sample belongs to u orig confidence level;

[0030] By calculating the formula

[0031]

[0032] Get the priority frequency C h Among them, C h represents the priority frequency of the hth candidate disturbed speech frame, C sh represents the priority frequency of the speaker recognition model s for the hth candidate perturbed speech frame, represents a set of multiple candidate perturbed speech frames sorted from large to small according to the priority calculated by the s-th speaker recognition model. F t express The candidate perturbed speech frame with index t in Ω h represents a set of multiple candidate perturbed speech frames, Ω h ={F st ,F st+1 ,..,F ed}, F st ,F st+1 ,..,F ed All represent candidate perturbed speech frames. sort1(parameter 1, parameter 2, parameter 3) represents the sorting function. Parameter 1 represents the set to be sorted. Parameter 2 represents the comparison parameter. When parameter 3 is 1, it means that the elements in the set represented by parameter 1 are sorted from large to small according to the comparison parameter. top k Indicates the number of pre-set disturbance points;

[0033] By calculating the formula Ω order = sort(Ω h ,C h ,P h ,1), get the candidate priority overlap frame set Ω order; Among them, sort2(parameter 1, parameter 2, parameter 3, parameter 4) represents the sorting function, parameter 1 represents the set to be sorted, parameter 2 represents the first sorting condition, parameter 3 represents the second sorting condition, and when parameter 4 is 1, it means that the elements in the set represented by parameter 1 are sorted according to the first sorting condition or the second sorting condition.

[0034] The candidate priority overlap frame set Ω order Middle, front top k The candidate priority coincidence frames are used as the priority coincidence frames.

[0035] Optionally, step 5 includes:

[0036] The first k sampling points in the priority overlap frame are taken as the disturbance point G, and the disturbance value δ of the disturbance point is randomly initialized G ; Where G = {G1, G2, ..., G k},G1,G2,...,G k All represent disturbance points;

[0037] By calculating the formula

[0038] X result =X orig +G⊙δ G , and get the perturbation result X result .

[0039] Optionally, step 6 includes:

[0040] By calculating the formula

[0041]

[0042] The loss value L(.) is obtained; where γ represents the balance parameter, STFT(.) represents the short-time Fourier transform, It means solving for the norm of 2.

[0043] Optionally, in step 7, the disturbance point and the disturbance value are updated using a projected gradient descent algorithm, including:

[0044] By calculating the formula

[0045]

[0046] Get the updated disturbance point G' and the updated disturbance value δ' G ;in, η represents a hyperparameter, ||·|| ∞ represents the infinite norm, ∈ represents the hyperparameter, which is used to specify the upper limit of the perturbation that can be added to each sampling point, Represents dimension f s*The real number space of t.

[0047] In a second aspect, the present application provides a speech adversarial sample generation device, comprising:

[0048] A speech sample collection module is used to obtain a speech sample set to be disturbed corresponding to the current speaker, and speech sample sets corresponding to multiple other speakers;

[0049] A target speaker determination module is used to calculate the speech similarity between each other speaker and the current speaker, and determine the target speaker from other speakers whose speech similarity is less than a preset similarity threshold according to the speech similarity;

[0050] A speech endpoint detection module is used to perform speech endpoint detection on the speech sample set to be disturbed, and obtain a plurality of candidate disturbed speech frames in the speech sample set to be disturbed; the candidate disturbed speech frames are used to determine the disturbed speech frames;

[0051] The priority overlap frame determination module is used to calculate the priority and priority frequency corresponding to each candidate disturbed speech frame respectively by using multiple different speaker recognition models, and determine the priority overlap frame in the to-be-disturbed speech sample set according to the priority frequency and priority; the priority frequency is used to determine the disturbed speech frame from multiple candidate disturbed speech frames, and the priority overlap frame represents the speech frame with the highest priority to be disturbed;

[0052] The first disturbance module is used to determine the disturbance point in the speech sample set to be disturbed according to the priority overlap frame, initialize the disturbance value corresponding to the disturbance point, and disturb the speech sample set to be disturbed according to the disturbance point and the disturbance value to obtain the disturbance result;

[0053] A loss calculation module, used to calculate the loss value between the perturbation result and the speech sample set corresponding to the target speaker;

[0054] The speech adversarial sample generation module is used to determine whether the loss value is greater than the preset loss threshold. If the loss value is greater than the preset loss threshold, the perturbation point and the perturbation value are updated using the projected gradient descent algorithm, and the perturbation result is perturbed using the updated perturbation point and the updated perturbation value to obtain a new perturbation result. The new perturbation result is used as the perturbation result in the loss calculation module and returned to the execution loss calculation module; otherwise, the perturbation result obtained by the first perturbation module is used as the speech adversarial sample corresponding to the current speaker.

[0055] In a third aspect, the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating speech adversarial samples when executing the computer program.

[0056] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for generating speech adversarial samples is implemented.

[0057] The above solution of the present application has the following beneficial effects:

[0058] The speech adversarial sample generation method provided by the present application calculates the speech similarity between other speakers and the current speaker, and selects other speakers with lower speech similarity to the current speaker as target speakers based on the speech similarity, thereby preventing the disturbed speech sample from still being recognized as the current speaker due to excessive speech similarity, thereby improving the speech synthesis quality; determining the priority overlapping frames in the speech sample set to be disturbed based on the priority frequency and priority, thereby helping to screen out disturbance points with higher disturbance value, reducing the number of disturbance points, avoiding the problem of excessive difference between the disturbed speech sample and the current speaker's speech sample due to too many disturbance points, thereby improving the speech synthesis quality; using the projected gradient descent algorithm to update the disturbance points and disturbance values, thereby further improving the excellence of the disturbance points and disturbance values, thereby achieving the best identity de-identification effect.

[0059] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 A flowchart of a method for generating speech adversarial samples provided in an embodiment of the present application;

[0062] Figure 2 Schematic diagram of a process of setting the sampling points corresponding to the candidate disturbed speech frame to zero in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of the structure of a speech adversarial sample generation device provided in one embodiment of the present application;

[0064] Figure 4 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0065] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0066] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0067] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0068] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0069] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0071] In view of the problems of poor identity de-identification effect and low speech synthesis quality in traditional speaker de-identification methods, the present application provides a speech adversarial sample generation method, apparatus, terminal device and medium. The speech adversarial sample generation method calculates the speech similarity between other speakers and the current speaker, and selects other speakers with lower speech similarity to the current speaker as target speakers based on the speech similarity. This can avoid the situation where the disturbed speech sample can still be identified as the current speaker due to excessive speech similarity, thereby improving the speech synthesis quality. According to the priority frequency and priority, the priority overlapping frames in the speech sample set to be disturbed are determined, which helps to screen out disturbance points with higher disturbance value, reduce the number of disturbance points, avoid the problem of excessive difference between the disturbed speech sample and the current speaker's speech sample due to too many disturbance points, and improve the speech synthesis quality. The disturbance points and disturbance values ​​are updated using the projected gradient descent algorithm, which can further improve the excellence of the disturbance points and disturbance values, thereby achieving the best identity de-identification effect.

[0072] like Figure 1 As shown, the speech adversarial sample generation method provided in this application includes the following steps:

[0073] Step 1: Obtain a speech sample set to be disturbed corresponding to the current speaker and speech sample sets corresponding to multiple other speakers.

[0074] It should be understood that the above-mentioned speech sample set to be disturbed and speech sample set both represent a set consisting of multiple speech samples. In one embodiment of the present application, the speech sample set to be disturbed can be represented as X orig ={X1,X2,...,X k ,...,X d},X1,X2,...,X k ,...,X d In the embodiment of the present application, the voice sample is the voice data of the speaker recorded by a device (computer, mobile phone, etc.), and the voice sample usually includes multiple voice segments, which can be specifically classified into silent segments, noise segments and active segments.

[0075] Among them, a silent segment means that there is no sound in the voice segment. When the device recording fails, the device cannot convert the received sound wave signal into an electrical signal, which may generate a silent segment. A noisy segment means that there is only background noise in the voice segment. Usually, it is caused by the speaker not speaking after the recording device starts recording, or the background noise is too loud. An active segment means that there is voice data of the speaker in the voice segment.

[0076] The speech sample sets corresponding to the above-mentioned other speakers can be obtained from a public speech database or collected through a speech recording device, and the acquisition method is not limited here.

[0077] Step 2, respectively calculating the speech similarity between each other speaker and the current speaker, and determining the target speaker from other speakers whose speech similarity is less than a preset similarity threshold according to the speech similarity.

[0078] Among them, the expression of speech similarity is:

[0079]

[0080] Among them, Sim(u orig ,u i ) indicates the current speaker u orig The speech similarity between the i-th speaker and the current speaker, i = 1, 2, ..., m, m represents the total number of other speakers and the total number of speech samples corresponding to the current speaker, n represents, X k Indicates the current speaker u orig The corresponding speech sample set X to be disturbed orig The kth disturbed speech sample in X orig ={X1,X2,...,X k ,...,X d}, k = 1, 2, ..., d, d represents the speech sample set X to be disturbed orig The total number of speech samples to be disturbed, f0(X k ,u i ) indicates that the speaker recognition model f0(·) considers the perturbed speech sample X k The confidence level belonging to the i-th other speaker, The speaker recognition model f0(·) is any speaker recognition model, Indicates that the speaker recognition model f0 believes that the speech sample X k ∈u i confidence level.

[0081] It should be noted that in the embodiments of the present application, the above-mentioned speaker recognition model may adopt a Gaussian mixture model (GMM), a hidden Markov model (HMM), etc. The present application does not limit its specific form. As long as the model has the speaker recognition function, it can be used. This also reflects the simplicity of the sound adversarial sample generation method provided in the present application.

[0082] After calculating the speech similarity between the current speaker and other speakers, the target speaker is selected from other speakers based on the speech similarity. This is to facilitate the subsequent setting of the perturbation target, that is, by adding perturbations, the speech recognition model recognizes the current speaker's speech sample as the target speaker. In this way, even if the criminals obtain the speaker's speech data, they cannot use link attacks and other means to accurately identify the speaker, and cannot obtain the speaker's identity privacy information, thereby achieving identity de-identification and protecting the speaker's security and privacy. The specific process is:

[0083] Other speakers whose speech similarity is less than a preset similarity threshold are arranged in order of speech similarity from small to large, and one other speaker is randomly selected from the top 20% of other speakers as the target speaker.

[0084] In an embodiment of the present application, the above-mentioned preset similarity threshold can be set to 0.45.

[0085] Randomly selecting another speaker as the target speaker can prevent criminals from obtaining the specific pattern of the target speaker, thereby improving security.

[0086] It is worth mentioning that here other speakers with lower speech similarity to the current speaker are used as target speakers, which can avoid the problem that the speech similarity between the target speaker and the current speaker is too high, resulting in poor identity recognition of the disturbed speech sample (it can still be recognized as the current speaker).

[0087] It should be noted that, in another embodiment of the present application, when the number of other speakers whose voice similarity is less than a preset similarity threshold is less than ten, one speaker is directly randomly selected from them as the target speaker.

[0088] Step 3: perform speech endpoint detection on the speech sample set to be disturbed to obtain a plurality of candidate disturbed speech frames in the speech sample set to be disturbed.

[0089] The candidate disturbed speech frame represents the speech frame used for disturbance. According to the introduction of the speech sample in step 1 (including silent segments, noise segments and active segments), it is not difficult to deduce that the speech segment that can be used to identify the speaker is the active segment, and the silent segment and the noise segment are both redundant speech segments, which will interfere with the result. Therefore, this application uses speech endpoint detection to obtain the active segment (candidate disturbed speech frame) in the speech sample.

[0090] Common voice endpoint detection includes the following steps:

[0091] Step 31, frame division.

[0092] For example, firstly, the total duration t of the speech sample set to be disturbed (the sum of the durations of all speech samples to be disturbed) is counted, and then the speech sample set X to be disturbed is set according to the actual situation. orig The sampling rate f s (used to calculate the number of sampling points), frame length t f (e.g. 25ms), frame shift t m (e.g. 10ms), the speech sample set to be disturbed can be divided into speech frames, each of which includes C = f s *t f Sampling points, for example, the first 25ms of the speech sample set to be disturbed is taken as the first speech frame F1, after F1 is shifted to the right by 10ms, the next 25ms is taken as the second speech frame F2, and so on, and finally all the speech frames of the speech sample set to be disturbed can be expressed as X orig ={F1,F2,...,F H}.

[0093] Step 32, adding windows.

[0094] For each speech frame, the formula is calculated

[0095]

[0096] Get the windowed speech frame in, Indicates the bth speech frame F b The result after windowing, b = 1, 2, ..., H, n represents the sampling point of the speech sample.

[0097] Step 33, perform the following operations on each speech frame:

[0098] By calculating the formula Get the short-time energy e1 of each speech frame b , e1 b Represents the short-time energy of the bth speech frame. Short-time energy is used to determine the boundary between silent and voiced speech.

[0099] By calculating the formula

[0100]

[0101] Get the short-time zero-crossing rate e2 of each speech frame b , where e2 b represents the short-time zero-crossing rate of the b-th speech frame, sgn(·) represents the number of times the speech signal changes its sign within the frame, The short-term zero-crossing rate is used to find out whether there is speech and the starting point of the speech from the background noise.

[0102] Step 34: confirm the candidate disturbance speech frame.

[0103] First, start searching from the first speech frame F1 and find the first frame that is not lower than the high volume threshold TH. upper The speech frame is denoted as F st After traversing all the speech frames, the last one that is not lower than the high volume threshold TH upper The speech frame is denoted as F ed Then, from F st Search forward and find the first lower The speech frame is used as the new F st , from F ed Search backwards and find the first lower The speech frame is used as the new F ed Then, from the new F st Search forward and find the first zcr The speech frame of the active segment is taken as the starting frame of the new F ed Search backward and find the first zcr The speech frame of the active segment is taken as the end frame of the active segment. Finally, all the speech frames in the active segment are determined as candidate disturbance speech frames. Specifically, the candidate disturbance speech frame set Ω0 = {F st ,F st+1 ,…,F ed}={x st1 ,…,x stC ,…,x ed1 ,…,x edC}.

[0104] Step 4: Calculate the priority and priority frequency corresponding to each candidate disturbed speech frame using multiple different speaker recognition models, and determine the priority coincidence frame in the to-be-disturbed speech sample set according to the priority frequency and priority.

[0105] The priority frequency is used to determine the disturbed speech frame from a plurality of candidate disturbed speech frames, and the priority coincidence frame represents the disturbed speech frame with the highest priority.

[0106] It is worth mentioning that determining the priority overlapping frames in the speech sample set to be disturbed based on the priority frequency and priority helps to screen out disturbance points with higher disturbance value, reduce the number of disturbance points, avoid the problem of too many disturbance points leading to excessive differences between the disturbed speech sample and the current speaker's speech sample, and improve the quality of speech synthesis.

[0107] Step 5: determine the disturbance point in the to-be-disturbed speech sample set according to the priority overlapped frames, initialize the disturbance value corresponding to the disturbance point, and disturb the to-be-disturbed speech sample set according to the disturbance point and the disturbance value to obtain the disturbance result.

[0108] Specifically, step 5 includes the following steps:

[0109] Step 51: Take the first k sampling points in the priority overlap frame as the disturbance point G, and randomly initialize the disturbance value δ of the disturbance point G .

[0110] Where G = {G1, G2, ..., G k},G1,G2,...,G k Both represent disturbance points.

[0111] Step 52, by calculating formula

[0112] X result =X orig +G⊙δ G , and get the perturbation result X result .

[0113] Step 6: Calculate the loss value between the perturbation result and the speech sample set corresponding to the target speaker.

[0114] In the embodiment of the present application, the formula

[0115]

[0116] Get the loss value L(.).

[0117] Among them, γ represents the balance parameter, STFT(.) represents the short-time Fourier transform, It means solving for the norm of 2.

[0118] Step 7: If the loss value is greater than the preset loss threshold, the projected gradient descent algorithm is used to update the disturbance point and disturbance value, and the disturbance result is disturbed using the updated disturbance point and the updated disturbance value to obtain a new disturbance result. The new disturbance result is used as the disturbance result in step 6, and the process returns to step 6; otherwise, the disturbance result obtained in step 5 is used as the speech adversarial sample corresponding to the current speaker.

[0119] Taking voice assistant software as an example, after the voice assistant software obtains the voice sample of the current speaker, it perturbs the voice sample of the current speaker by implementing the voice adversarial sample generation method provided in this application to generate a voice adversarial sample, and then transmits the voice adversarial sample to the voice assistant developer through the network, and the voice assistant developer stores it in the cloud database. In this way, even if criminals obtain the voice adversarial sample in the cloud database by attacking the cloud database, they will not be able to obtain the source voice sample (the voice sample of the current speaker) through link attacks and other means (usually using various speaker recognition models), and will not be able to identify the identity privacy information of the current speaker, which greatly protects the privacy security of the speaker.

[0120] It should be noted that the speech adversarial sample generation method provided in this application regards finding the disturbance point and the disturbance value as an optimization problem. For each speaker recognition model, the expression of the optimization problem is as follows:

[0121]

[0122] Where G represents the set of perturbation points, When the value of the disturbance point is 1, it means that disturbance is added to the point, and when the value of the disturbance point is 0, it means that no disturbance is added to the point. G represents a set of disturbance values, including the disturbance value corresponding to each disturbance point, ⊙ represents the dot product operator, f s (X orig +G⊙δ G ,u tar ) indicates that the s-th speaker recognition model considers the speech adversarial sample X att =X orig +G⊙δ G ,u tar Belongs to the target speaker u tar , STFT(·) represents short-time Fourier transform, which reflects the frequency and phase of the signal in the time domain, γ represents the balance parameter, ∈ represents the hyperparameter, which is used to specify the upper limit of the disturbance that can be added to each sampling point, ∥δ∥ ∞ is the infinite norm, indicating that δ G The number of elements in cannot exceed ∈.

[0123] The process of updating the disturbance points and disturbance values ​​using the projected gradient descent algorithm in step 7 is as follows:

[0124] By calculating the formula

[0125]

[0126] Get the updated disturbance point G' and the updated disturbance value δ' G;in, η represents a hyperparameter, ||·|| ∞ represents the infinite norm, ∈ represents the hyperparameter, which is used to specify the upper limit of the perturbation that can be added to each sampling point, Represents dimension f s *The real number space of t.

[0127] It is worth mentioning that using the projected gradient descent algorithm to update the disturbance points and disturbance values ​​can further improve the excellence of the disturbance points and disturbance values, thereby achieving the best identity de-identification effect.

[0128] The specific process of step 4 (using multiple different speaker recognition models to respectively calculate the priority and priority frequency corresponding to each candidate disturbed speech frame, and determining the priority overlapping frames in the speech sample set to be disturbed according to the priority frequency and priority) is exemplified below.

[0129] Step 41: copy the speech sample set to be disturbed to obtain a new speech sample set to be disturbed. in, {F1,F2,...,F H} represents the speech frame in the speech sample set to be disturbed, H represents the total number of speech frames, each speech frame includes C sampling points, X HC represents the Cth sampling point in the Hth speech frame;

[0130] Step 42: for each candidate disturbed speech frame, the new to-be-disturbed speech sample set The sampling points corresponding to the candidate disturbed speech frames in are set to zero.

[0131] It should be noted that the sampling point is set to 0 in order to distinguish the two (disturbance point and non-disturbance point) when the disturbance point is set to 1 later.

[0132] The specific process of step 42 is as follows Figure 2 As shown, Figure 2 The slashed boxes in the figure represent candidate disturbed speech frames in the to-be-perturbed speech sample set, and the blank boxes represent speech frames that do not need to be disturbed.

[0133] Step 43, for each speaker recognition model, calculate the formula Get priority P sh .

[0134] Among them, P shrepresents the priority of the hth candidate perturbed speech frame calculated by the sth speaker recognition model, s = 1, 2, ..., Q, Q represents the total number of preset speaker recognition models, Q speaker recognition models are different from each other, h = 1, 2, ..., L, L represents the total number of candidate perturbed speech frames obtained after processing in step 3, Indicates that the current speaker recognition model believes that the speech sample belongs to u orig confidence level.

[0135] Step 44, by calculating the formula

[0136]

[0137] Get the priority frequency C h .

[0138] Among them, C h represents the priority frequency of the hth candidate disturbed speech frame, C sh represents the priority frequency of the speaker recognition model s for the hth candidate perturbed speech frame, represents a set of multiple candidate perturbed speech frames sorted from large to small according to the priority calculated by the s-th speaker recognition model. F t express The candidate perturbed speech frame with index t in Ω h represents a set of multiple candidate perturbed speech frames, Ω h ={F st ,F st+1 ,..,F ed}, F st ,F st+1 ,..,F ed All represent candidate perturbed speech frames. sort1(parameter 1, parameter 2, parameter 3) represents the sorting function. Parameter 1 represents the set to be sorted. Parameter 2 represents the comparison parameter. When parameter 3 is 1, it means that the elements in the set represented by parameter 1 are sorted from large to small according to the comparison parameter. top k Indicates the number of preset disturbance points.

[0139] Step 45, by calculating the formula Ω order = sort(Ω h ,C h ,P h ,1), get the candidate priority overlap frame set Ω order .

[0140] Among them, sort2(parameter 1, parameter 2, parameter 3, parameter 4) represents the sorting function, parameter 1 represents the set to be sorted, parameter 2 represents the first sorting condition, parameter 3 represents the second sorting condition, and when parameter 4 is 1, it means that the elements in the set represented by parameter 1 are sorted according to the first sorting condition or the second sorting condition.

[0141] Step 46: set the candidate priority overlap frame set Ω order Middle, front top k The candidate priority coincidence frames are used as the priority coincidence frames.

[0142] It can be seen from the above steps that the speech adversarial sample generation method provided by the present application calculates the speech similarity between other speakers and the current speaker, and selects other speakers with lower speech similarity to the current speaker as target speakers based on the speech similarity, so as to avoid the situation where the disturbed speech sample can still be identified as the current speaker due to excessive speech similarity, thereby improving the speech synthesis quality; according to the priority frequency and priority, the priority overlapping frames in the speech sample set to be disturbed are determined, which helps to screen out disturbance points with higher disturbance value, reduces the number of disturbance points, avoids the problem of excessive difference between the disturbed speech sample and the current speaker's speech sample due to too many disturbance points, and improves the speech synthesis quality; using the projected gradient descent algorithm to update the disturbance points and disturbance values, the excellence of the disturbance points and disturbance values ​​can be further improved, thereby achieving the best identity de-identification effect.

[0143] The following is an exemplary description of the speech adversarial sample generation device provided in this application.

[0144] like Figure 3 As shown, the speech adversarial sample generating device 300 includes:

[0145] The speech sample collection module 301 is used to obtain a speech sample set to be disturbed corresponding to the current speaker and speech sample sets corresponding to multiple other speakers;

[0146] The target speaker determination module 302 calculates the speech similarity between each other speaker and the current speaker, and determines the target speaker from other speakers whose speech similarity is less than a preset similarity threshold according to the speech similarity;

[0147] The speech endpoint detection module 303 is used to perform speech endpoint detection on the speech sample set to be disturbed, and obtain a plurality of candidate disturbed speech frames in the speech sample set to be disturbed; the candidate disturbed speech frames are used to determine the disturbed speech frames;

[0148] The priority overlap frame determination module 304 is used to calculate the priority and priority frequency corresponding to each candidate disturbed speech frame respectively by using multiple different speaker recognition models, and determine the priority overlap frame in the to-be-disturbed speech sample set according to the priority frequency and priority; the priority frequency is used to determine the disturbed speech frame from the multiple candidate disturbed speech frames, and the priority overlap frame represents the speech frame with the highest priority to be disturbed;

[0149] The first disturbance module 305 is used to determine the disturbance point in the speech sample set to be disturbed according to the priority overlap frame, initialize the disturbance value corresponding to the disturbance point, and disturb the speech sample set to be disturbed according to the disturbance point and the disturbance value to obtain the disturbance result;

[0150] A loss calculation module 306, used to calculate a loss value between the disturbance result and the speech sample set corresponding to the target speaker;

[0151] The speech adversarial sample generation module 307 is used to determine whether the loss value is greater than a preset loss threshold. If the loss value is greater than the preset loss threshold, the perturbation point and the perturbation value are updated using the projected gradient descent algorithm, and the perturbation result is perturbed using the updated perturbation point and the updated perturbation value to obtain a new perturbation result. The new perturbation result is used as the perturbation result in the loss calculation module and returned to the execution loss calculation module; otherwise, the perturbation result obtained by the first perturbation module is used as the speech adversarial sample corresponding to the current speaker.

[0152] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0153] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0154] like Figure 4As shown, an embodiment of the present application provides a terminal device, such as Figure 4 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0155] Specifically, when the processor D100 executes the computer program D102, first, a speech sample set to be disturbed corresponding to the current speaker and speech sample sets corresponding to multiple other speakers are obtained, and then the speech similarity between each other speaker and the current speaker is calculated respectively, and based on the speech similarity, the target speaker is determined from other speakers whose speech similarity is less than a preset similarity threshold, and then the speech endpoint detection is performed on the speech sample set to be disturbed to obtain multiple candidate disturbed speech frames in the speech sample set to be disturbed, and then the priority and priority frequency corresponding to each candidate disturbed speech frame are calculated respectively using multiple different speaker recognition models, and the priority coincidence frame in the speech sample set to be disturbed is determined according to the priority frequency and priority, and then, based on the priority frequency and priority, the target speaker is determined. According to the priority overlapping frames, the disturbance points in the speech sample set to be disturbed are determined, the disturbance values ​​corresponding to the disturbance points are initialized, and the speech sample set to be disturbed is disturbed according to the disturbance points and the disturbance values ​​to obtain the disturbance results, and then the loss value between the disturbance results and the speech sample set corresponding to the target speaker is calculated, and then the size between the loss value and the preset loss threshold is judged. If the loss value is greater than the preset loss threshold, the disturbance points and the disturbance values ​​are updated using the projected gradient descent algorithm, and the disturbance results are disturbed using the updated disturbance points and the updated disturbance values ​​to obtain new disturbance results. The steps of updating and calculating the loss value are repeatedly performed until the loss value of the new disturbance result is less than the preset loss threshold, and the disturbance result with a loss value less than the preset loss threshold is used as the speech adversarial sample corresponding to the current speaker. Among them, by calculating the speech similarity between other speakers and the current speaker, and selecting other speakers with lower speech similarity to the current speaker as target speakers based on the speech similarity, it is possible to avoid the disturbed speech sample still being able to be identified as the current speaker due to excessive speech similarity, thereby improving the quality of speech synthesis; according to the priority frequency and priority, the priority overlapping frames in the speech sample set to be disturbed are determined, which helps to screen out disturbance points with higher disturbance value, reduce the number of disturbance points, avoid the problem of too large difference between the disturbed speech sample and the current speaker's speech sample due to too many disturbance points, and improve the quality of speech synthesis; using the projected gradient descent algorithm to update the disturbance points and disturbance values, it is possible to further improve the excellence of the disturbance points and disturbance values, thereby achieving the best identity de-identification effect.

[0156] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0157] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0158] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0159] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the speech adversarial sample generation device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0161] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0162] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0163] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] The speech adversarial sample generation method provided in this application has the following advantages:

[0166] 1. The idea of ​​introducing adversarial samples is to reduce the size of the perturbation. Adversarial samples add almost imperceptible perturbations to the original samples, causing the target model to misclassify the adversarial samples.

[0167] 2. Sparse perturbations are introduced to reduce the number of perturbed sample points. A small number of suitable sample points are found through a random search algorithm, which not only avoids excessive deviation of the adversarial sample direction, but also further reduces the amount of changes to the original sample.

[0168] 3. The present invention proposes a priority sampling point calculation algorithm to narrow the range of disturbance candidates and reduce time overhead.

[0169] 4. Taking into account the transferability of disturbances, even if the attacker uses multiple speaker recognition models to perform link attacks, the present invention can still ensure the effectiveness of identity de-identification.

[0170] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating speech adversarial samples, characterized in that: include: Step 1, obtaining a speech sample set to be disturbed corresponding to the current speaker and speech sample sets corresponding to multiple other speakers; Step 2, respectively calculating the speech similarity between each of the other speakers and the current speaker, and determining the target speaker from the other speakers whose speech similarity is less than a preset similarity threshold; Step 3, performing speech endpoint detection on the speech sample set to be disturbed to obtain a plurality of candidate disturbed speech frames in the speech sample set to be disturbed; The candidate disturbed speech frame is used to determine the disturbed speech frame; Step 4, using a plurality of different speaker recognition models to respectively calculate the priority and priority frequency corresponding to each candidate disturbed speech frame, and determining the priority coincidence frame in the to-be-disturbed speech sample set according to the priority frequency and the priority; The priority frequency is used to determine the disturbed speech frame from the multiple candidate disturbed speech frames, and the priority coincidence frame represents the disturbed speech frame with the highest priority; Step 5, determining the disturbance point in the speech sample set to be disturbed according to the priority overlap frame, initializing the disturbance value corresponding to the disturbance point, and disturbing the speech sample set to be disturbed according to the disturbance point and the disturbance value to obtain a disturbance result; Step 6, calculating the loss value between the disturbance result and the speech sample set corresponding to the target speaker; Step 7, if the loss value is greater than the preset loss threshold, the disturbance point and the disturbance value are updated using the projected gradient descent algorithm, and the disturbance result is disturbed using the updated disturbance point and the updated disturbance value to obtain a new disturbance result, and the new disturbance result is used as the disturbance result in step 6, and the process returns to step 6; Otherwise, the disturbance result obtained in step 5 is used as the speech adversarial sample corresponding to the current speaker; Wherein, the step 4 comprises: The speech sample set to be disturbed is copied to obtain a new speech sample set to be disturbed in, {F1,F2,...,F H } represents the speech frame in the speech sample set to be disturbed, H represents the total number of the speech frames, each of the speech frames includes C sampling points, X HC represents the Cth sampling point in the Hth speech frame, X orig Indicates the current speaker u orig The corresponding speech sample set to be disturbed; For each candidate disturbed speech frame, the new to-be-perturbed speech sample set The sampling point corresponding to the candidate disturbed speech frame is set to zero; For each speaker recognition model, the formula Get the priority P sh ; Among them, P sh represents the priority of the hth candidate perturbed speech frame calculated by the sth speaker recognition model, s=1,2,...,Q, Q represents the total number of preset speaker recognition models, and the Q speaker recognition models are different from each other, h=1,2,...,L, L represents the total number of candidate perturbed speech frames obtained after the processing in step 3, Indicates that the current speaker recognition model believes that the speech sample belongs to u orig Confidence level; By calculating the formula Get the priority frequency C h Among them, C h represents the priority frequency of the hth candidate disturbed speech frame, C sh represents the priority frequency of the speaker recognition model s for the hth candidate perturbed speech frame, represents a set of multiple candidate perturbed speech frames sorted from large to small according to the priority calculated by the s-th speaker recognition model. F t express The candidate perturbed speech frame with index t in Ω h represents a set consisting of the plurality of candidate disturbed speech frames, Ω h ={F st ,F st+1 ,..,F ed }, F st ,F st+1 ,..,F ed All represent the candidate disturbed speech frames, sort1(parameter 1, parameter 2, parameter 3) represents the sorting function, parameter 1 represents the set to be sorted, parameter 2 represents the comparison parameter, and when parameter 3 is 1, it means that the elements in the set represented by parameter 1 are sorted from large to small according to the comparison parameter, top k Indicates the number of pre-set disturbance points; By calculating the formula Ω order = sort(Ω h ,C h ,P h ,1), get the candidate priority overlap frame set Ω order ; Among them, sort2(parameter 1, parameter 2, parameter 3, parameter 4) represents the sorting function, parameter 1 represents the set to be sorted, parameter 2 represents the first sorting condition, parameter 3 represents the second sorting condition, and when parameter 4 is 1, it means that the elements in the set represented by parameter 1 are sorted according to the first sorting condition or the second sorting condition. The candidate priority overlap frame set Ω order Middle, front top k A candidate priority coincidence frame is used as the priority coincidence frame.

2. The method for generating speech adversarial samples according to claim 1, characterized in that: The expression of the speech similarity is: Among them, Sim(u orig ,u i ) indicates the current speaker u orig The speech similarity between the current speaker and the i-th speaker, i = 1, 2, ..., m, m represents the total number of other speakers, n represents the total number of speech samples corresponding to the current speaker, X k Indicates the current speaker u orig The corresponding speech sample set X to be disturbed orig The kth disturbed speech sample in X orig ={X1,X2,...,X k ,...,X d }, k = 1, 2, ..., d, d represents the speech sample set X to be disturbed orig The total number of speech samples to be disturbed, f0(X k ,u i ) indicates that the speaker recognition model f0(·) considers the perturbed speech sample X k The confidence level belonging to the i-th other speaker, The speaker recognition model f0(·) is any speaker recognition model, Indicates that the speaker recognition model f0 believes that the speech sample X k ∈u i confidence level.

3. The method for generating speech adversarial samples according to claim 2, characterized in that: The step 2 comprises: Other speakers whose speech similarity is less than a preset similarity threshold are arranged in order of the speech similarity from small to large, and one other speaker is randomly selected from the top 20% of other speakers as the target speaker.

4. The method for generating speech adversarial samples according to claim 3, characterized in that: The step 5 comprises: The first k sampling points in the priority overlap frame are used as the disturbance point G, and the disturbance value δ of the disturbance point is randomly initialized. G ; Where G = {G1, G2, ..., G k },G1,G2,...,G k All represent the disturbance points; By calculating the formula X result =X orig +G⊙δ G , get the disturbance result X result , ⊙ represents the dot product operator.

5. The method for generating speech adversarial samples according to claim 4, characterized in that: The step 6 comprises: By calculating the formula The loss value L(.) is obtained; wherein γ represents a balance parameter, STFT(.) represents a short-time Fourier transform, It means solving for the norm of 2.

6. The method for generating speech adversarial samples according to claim 5, characterized in that: In step 7, the projected gradient descent algorithm is used to update the disturbance point and the disturbance value, including: By calculating the formula Get the updated disturbance point G' and the updated disturbance value δ' G ;in, η represents a hyperparameter, ∥·∥ ∞ represents the infinite norm, ∈ represents the hyperparameter, which is used to specify the upper limit of the perturbation that can be added to each sampling point, Represents dimension f s *The real number space of t.

7. A speech adversarial sample generation device, characterized in that: include: A speech sample collection module is used to obtain a speech sample set to be disturbed corresponding to the current speaker, and speech sample sets corresponding to multiple other speakers; A target speaker determination module, used to respectively calculate the speech similarity between each of the other speakers and the current speaker, and determine the target speaker from the other speakers whose speech similarity is less than a preset similarity threshold; A speech endpoint detection module, used for performing speech endpoint detection on the speech sample set to be disturbed to obtain a plurality of candidate disturbed speech frames in the speech sample set to be disturbed; The candidate disturbed speech frame is used to determine the disturbed speech frame; A priority overlap frame determination module is used to calculate the priority and priority frequency corresponding to each candidate disturbed speech frame by using a plurality of different speaker recognition models, and determine the priority overlap frame in the to-be-disturbed speech sample set according to the priority frequency and the priority; The priority frequency is used to determine the disturbed speech frame from the multiple candidate disturbed speech frames, and the priority coincidence frame represents the disturbed speech frame with the highest priority; A first disturbance module, used for determining a disturbance point in the to-be-disturbed speech sample set according to the priority overlapped frame, initializing a disturbance value corresponding to the disturbance point, and disturbing the to-be-disturbed speech sample set according to the disturbance point and the disturbance value to obtain a disturbance result; A loss calculation module, used to calculate the loss value between the disturbance result and the speech sample set corresponding to the target speaker; A speech adversarial sample generation module is used to determine whether the loss value is greater than a preset loss threshold. If the loss value is greater than the preset loss threshold, the disturbance point and the disturbance value are updated using a projected gradient descent algorithm, and the disturbance result is disturbed using the updated disturbance point and the updated disturbance value to obtain a new disturbance result, and the new disturbance result is used as the disturbance result in the loss calculation module, and the loss calculation module is returned for execution; otherwise, the disturbance result obtained by the first disturbance module is used as the speech adversarial sample corresponding to the current speaker; Wherein, the priority overlap frame determination module is specifically used to implement: The speech sample set to be disturbed is copied to obtain a new speech sample set to be disturbed in, {F1,F2,...,F H } represents the speech frame in the speech sample set to be disturbed, H represents the total number of the speech frames, each of the speech frames includes C sampling points, X HC represents the Cth sampling point in the Hth speech frame, X orig Indicates the current speaker u orig The corresponding speech sample set to be disturbed; For each candidate disturbed speech frame, the new to-be-perturbed speech sample set The sampling point corresponding to the candidate disturbed speech frame is set to zero; For each speaker recognition model, the formula Get the priority P sh ; Among them, P sh represents the priority of the hth candidate perturbed speech frame calculated by the sth speaker recognition model, s = 1, 2, ..., Q, Q represents the total number of preset speaker recognition models, Q speaker recognition models are different from each other, h = 1, 2, ..., L, L represents the total number of candidate perturbed speech frames obtained after processing, Indicates that the current speaker recognition model believes that the speech sample belongs to u orig Confidence level; By calculating the formula Get the priority frequency C h Among them, C h represents the priority frequency of the hth candidate disturbed speech frame, C sh represents the priority frequency of the speaker recognition model s for the hth candidate perturbed speech frame, represents a set of multiple candidate perturbed speech frames sorted from large to small according to the priority calculated by the s-th speaker recognition model. F t express The candidate perturbed speech frame with index t in Ω h represents a set consisting of the plurality of candidate disturbed speech frames, Ω h ={F st ,F st+1 ,..,F ed }, F st ,F st+1 ,..,F ed All represent the candidate disturbed speech frames, sort1(parameter 1, parameter 2, parameter 3) represents the sorting function, parameter 1 represents the set to be sorted, parameter 2 represents the comparison parameter, and when parameter 3 is 1, it means that the elements in the set represented by parameter 1 are sorted from large to small according to the comparison parameter, top k Indicates the number of pre-set disturbance points; By calculating the formula Ω order = sort(Ω h ,C h ,P h ,1), get the candidate priority overlap frame set Ω order ; Among them, sort2(parameter 1, parameter 2, parameter 3, parameter 4) represents the sorting function, parameter 1 represents the set to be sorted, parameter 2 represents the first sorting condition, parameter 3 represents the second sorting condition, and when parameter 4 is 1, it means that the elements in the set represented by parameter 1 are sorted according to the first sorting condition or the second sorting condition. The candidate priority overlap frame set Ω order Middle, front top k A candidate priority coincidence frame is used as the priority coincidence frame.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating speech adversarial samples according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating speech adversarial samples according to any one of claims 1 to 6 is implemented.

Images