High-fidelity speech desensitization method and device based on adversarial sample generation

Abstract

The present invention discloses a high-fidelity speech desensitization method and device based on adversarial sample generation technology, which belongs to the field of data privacy protection. By inputting the protected person's audio into the adversarial generative network desensitization model, the output audio can change the voiceprint features in the audio while maintaining the original semantic content, so that the artificial intelligence speaker recognition model cannot judge the speaker's identity through the input audio, thereby achieving the effect of removing the original speaker's identity information. The training of the desensitization model mainly achieves the training purpose by minimizing the generator and discriminator losses in the adversarial generative network. In a specific application scenario, the speaker can first process his or her audio through a trained desensitization model, and then provide the generated audio to the outside world without revealing his or her identity, thereby achieving the purpose of protecting personal privacy.

Description

Technical Field

[0001] The present invention relates to the fields of speech recognition and voiceprint feature modeling, and in particular to a high-fidelity speech desensitization method based on adversarial sample generation. Background Art

[0002] Deep learning technology has been applied to all aspects of daily life, especially in speech recognition, image recognition, target detection, etc. However, with the development of deep learning and IoT technology, the privacy issues it brings have also received constant attention from users. When users need to control IoT devices at home or convey text messages by voice, they often need to use the voice recognition or speaker recognition model of smart terminals to achieve this. This will expose the user's pure audio to the servers of some application systems that do not need to provide voiceprint authentication. Deep learning technology can also extract the user's voiceprint features from the leaked voice data and sell them illegally. Illegal personnel will use the victim's voiceprint to synthesize audio, disguise themselves as victims, and then obtain illegal authorization or achieve other illegal purposes.

[0003] In the era of big data, more and more voice data is being released to improve voice-based services or academic research. However, there is a risk of privacy leakage in the process of voice data release. For example, in the release of voice data, if an attacker obtains the voice data of a specific user through voiceprint recognition, he can understand the user's sensitive information by analyzing the voice data.

[0004] Although deep learning technology has been widely used, more and more studies have shown that this technology itself has many security risks. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention proposes a high-fidelity speech desensitization method and device based on adversarial sample generation. By constructing an adversarial sample generation model, the generated adversarial samples can change the voiceprint information in the audio while retaining the semantic information in the original audio, thereby preventing the leakage of the speaker's identity privacy.

[0006] The present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention proposes a high-fidelity speech desensitization method based on adversarial sample generation technology, comprising the following steps:

[0008] Construct a generative adversarial network, including a training adversarial sample generation model, a speech discrimination model, and a pre-trained speaker recognition model and speech recognition model;

[0009] The generative adversarial network is trained, and the adversarial sample generation model in the trained generative adversarial network is used as a speech desensitization model. The speech desensitization model is used to superimpose perturbations on the input clean audio to generate adversarial audio that retains the original semantic information of the clean audio and desensitizes the identity information. The adversarial audio is used to interact with the outside world.

[0010] Furthermore, the high-fidelity speech desensitization method comprises the following steps:

[0011] S01, build a generative adversarial network and randomly initialize the parameters of the speech identification model and adversarial sample generation model in the network;

[0012] S02, collect a clean audio data set and divide the data set into a training set and a test set;

[0013] S03, using the training set to train the generative adversarial network, is divided into two stages:

[0014] In the first stage, the parameters of the speech discrimination model, the pre-trained speaker recognition model, and the speech recognition model are fixed; the parameters of the adversarial sample generation model are iteratively updated through the back-propagation algorithm, and the parameters of the adversarial sample generation model that minimizes the loss function of the test set are selected as the optimal parameters.

[0015] In the second stage, the adversarial sample generation model is fixed according to the optimal parameters obtained in the first stage, and the parameters of the speech identification model are iteratively updated through the back propagation algorithm. The parameters of the speech identification model that make the loss function of the test set the lowest are selected as the optimal parameters and saved;

[0016] In the second stage, the adversarial sample generation model is fixed according to the optimal parameters obtained in the first stage, the parameters of the speech identification model are iteratively updated through the back propagation algorithm, and the optimal parameters of the speech identification model are selected using the test set;

[0017] The training process of the first and second stages is executed repeatedly until the parameters of the adversarial sample generation model and the parameters of the speech identification model meet the established optimization goals and the training is completed;

[0018] S04, using the adversarial sample generation model obtained after training as a speech desensitization model, using the speaker audio as the input of the speech desensitization model, using the speech desensitization model to superimpose disturbances on the input speaker audio, generating adversarial audio that retains the original semantic information of the speaker audio and desensitizes the speaker identity information, using the adversarial audio to interact with the outside world to achieve the purpose of hiding the speaker's identity.

[0019] Furthermore, the model parameters obtained in each training cycle are saved in a breakpoint file. After the current training cycle is completed, the test set is used to test whether the training loss function value of the model is reduced. If so, the previous breakpoint file is deleted and the current breakpoint file is retained; if not, the current breakpoint file is deleted and the previous breakpoint file is retained.

[0020] Furthermore, in the process of training the generative adversarial network, the minimax game algorithm is used to make the adversarial sample generation model and the speech identification model reach a dynamic balance after alternating training.

[0021] Furthermore, in the first stage of training the generative adversarial network using the training set, the loss of the back propagation algorithm includes three parts: voiceprint loss, semantic loss and similarity loss;

[0022] The calculation of the voiceprint loss is specifically as follows: the clean audio and the adversarial audio are encoded by a speaker recognition model to obtain a clean audio voiceprint matrix and an adversarial audio voiceprint matrix, and the cosine distance between the two is calculated as the voiceprint loss.

[0023] The calculation of the semantic loss is specifically as follows: performing content recognition on the clean audio and the adversarial audio through the speech recognition model, obtaining the word sequence X of the clean audio and the word sequence of the adversarial audio, and calculating the connection temporal classification loss of the two as the semantic loss;

[0024] The similarity loss is calculated specifically as follows: similarity features of the clean audio and the adversarial audio are extracted through a speech identification model, similarity confidence between the adversarial audio and the clean audio is predicted, and cross entropy loss is calculated as the similarity loss based on the similarity confidence and the true label.

[0025] Furthermore, in the second stage of training the generative adversarial network using the training set, the loss of the back propagation algorithm is the similarity loss; the calculation of the similarity loss is specifically as follows: extracting the similarity features of the clean audio and the adversarial audio through the speech identification model, predicting the similarity confidence between the adversarial audio and the clean audio, and calculating the cross entropy loss as the similarity loss based on the similarity confidence and the true label.

[0026] In a second aspect, the present invention proposes a high-fidelity speech desensitization device based on adversarial sample generation technology, which is used to implement the above-mentioned high-fidelity speech desensitization method.

[0027] The beneficial effects of the present invention are:

[0028] (1) The present invention uses adversarial sample generation technology to change the voiceprint features in the audio, which causes very little disturbance to the original audio, and the semantic content can be easily captured by the corresponding entity in both the physical domain and the digital domain.

[0029] (2) The speech desensitization model trained by the present invention does not rely on special hardware, has the characteristics of high portability, small storage resource occupation, timely response and rapid processing.

[0030] (3) When the speaker recognition model and speech recognition model are iteratively developed, it is only necessary to replace the corresponding model during network training to achieve the desensitization effect that adapts to the iterative development of the model, and to achieve more accurate and effective speech desensitization under the action of the iteratively developed model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the workflow of a high-fidelity speech desensitization method based on adversarial sample generation according to an embodiment of the present invention.

[0032] Figure 2 The figure is a schematic diagram of the speech desensitization model training process shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the invention is described below in conjunction with the accompanying drawings. The accompanying drawings are only schematic diagrams of the present invention. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0035] The present invention proposes a high-fidelity speech desensitization technology for generating adversarial samples, constructs a loss function for generating adversarial samples and performs iterative optimization to generate disturbances that are imperceptible to the human ear, so that after adding disturbances to the private speech data, the user's audio content can be heard clearly by people, and the speech recognition model can correctly parse the semantic messages therein, but the smart device cannot extract the user's real voiceprint, and the audio data cannot be illegally used after being uploaded, thereby protecting the user's privacy.

[0036] Figure 1 The figure is a schematic diagram of the workflow of the method of the present invention. The present invention uses a generative adversarial network to generate adversarial samples for speech. The network will be used as a desensitization model to blur the speaker information contained in the speech, protect the original semantic information and desensitize the identity information. The whole workflow can be divided into two stages:

[0037] The first stage is to train the speech desensitization model. First, build a generative adversarial network and initialize the training hyperparameters, such as training cycle, learning rate, etc. The generative adversarial network includes an adversarial sample generation model and a speech identification model; collect public or existing speech resources on the Internet as samples, and divide the samples into test sets and training sets. Then, train the adversarial sample generation model and speech identification model in turn, and save the model parameters with the smallest loss value of the training set until the training cycle reaches the set threshold.

[0038] The second stage is the application stage. The trained adversarial sample generation model is used as a speech desensitization model to process the input pure audio and generate desensitized speech for providing to the outside world, so as to protect the identity of the speaker.

[0039] In this embodiment, the adversarial audio is generated by applying slight perturbations to the input pure audio through the speech desensitization model, thereby achieving the effect of protecting the original semantic information and desensitizing the identity information.

[0040] The speech desensitization model training process diagram of the present invention is divided into two parts:

[0041] like Figure 2 As shown in (a) in the figure, the first part is to train the adversarial sample generation model. First, the parameters of the speech discrimination model, speaker recognition model, and speech recognition model are fixed; if there are previously saved adversarial sample generation model parameters, they are directly loaded, otherwise the model parameters are randomly initialized, and the adversarial sample generation model parameters are continuously updated and learned through the back propagation algorithm, and the parameters that make the model generalize better are selected and saved using the test set.

[0042] In this embodiment, the speaker recognition model and speech recognition model are pre-trained and do not need to update parameters. The adversarial audio output by the adversarial sample generation model is used as the input of the speech identification model, the speaker recognition model, and the speech recognition model respectively. The loss function of the back propagation algorithm is divided into three parts: voiceprint loss, semantic loss, and similarity loss.

[0043] The calculation of the voiceprint loss is as follows: the clean audio and the adversarial audio are encoded by the speaker recognition model to obtain the clean audio voiceprint matrix and the adversarial audio voiceprint matrix, and the cosine distance between the two is calculated. The formula is:

[0044] Loss 声纹 =cosine(A,A')

[0045] Among them, A represents the pure audio voiceprint matrix, A' represents the adversarial audio voiceprint matrix, and Loss 声纹 Indicates voiceprint loss.

[0046] The semantic loss is calculated as follows: using a speech recognition model to perform content recognition on the clean audio and the adversarial audio, the speech recognition model can decode the complete sentence or word information contained in the audio, and obtain a word sequence X of the clean audio and a word sequence Y of the adversarial audio, wherein the word sequence X is the true value of the semantic content of the adversarial audio, and the connection temporal classification loss of the word sequence X and the word sequence Y is calculated, and the formula is:

[0047] Loss 语义 =CTCLoss(X,Y)

[0048] Among them, Loss 语义 represents the connection temporal classification loss.

[0049] The similarity loss is calculated as follows: using the speech identification model to extract the similarity features of the pure audio and the adversarial audio, predicting the probability that the adversarial audio sample is a positive sample, that is, the degree of similarity between the adversarial audio and the pure audio (similarity confidence), and calculating the cross entropy loss based on the similarity confidence output by the speech identification model and the true label. The formula is:

[0050]

[0051] L i =-(y n *log(z n )+(1-y n )*log(1-z n ))

[0052] in, N is the sample size, z n Represents the probability of predicting that the nth adversarial audio sample is a positive sample, y n represents the label of the nth adversarial audio sample, L i represents the cross entropy loss of the i-th adversarial audio sample, Loss 相似 represents the average cross entropy loss over all samples.

[0053] The total loss function of the training adversarial sample generation model is a linear combination of the three. Technical personnel in this field can assign different weights to the three according to actual conditions to find the desensitization model that best meets the expectations.

[0054] like Figure 2As shown in (b) in the figure, the second part is to train the speech identification model. The adversarial sample generation model parameters are fixed. If there are previously saved speech identification model parameters, they are directly loaded. Otherwise, the speech identification model parameters are randomly initialized. During the training process, the speech identification model parameters are continuously updated and learned through the back propagation algorithm, and the parameters that enable the model to generalize better are selected and saved using the test set. During the training process of the second part, the loss function is the cross entropy loss between the similarity confidence output by the speech identification model and the true label.

[0055] The training process of the first part and the training process of the second part are performed alternately in a cycle until the parameters of the adversarial sample generation model and the parameters of the speech identification model meet the established optimization goals and the training is completed. In this embodiment, a threshold of a training cycle can be set, and a minimax game algorithm is used to make the adversarial sample generation model and the speech identification model reach a dynamic balance after alternating training.

[0056] Corresponding to the aforementioned embodiment of a high-fidelity speech desensitization method based on adversarial sample generation, the present application also provides an embodiment of a high-fidelity speech desensitization device based on adversarial sample generation, the device comprising:

[0057] A generative adversarial network module, which includes an adversarial sample generation model, a speech discrimination model, a speaker recognition model, and a speech recognition model;

[0058] A generative adversarial network training module, which is used to train the adversarial sample generation model and speech identification model in the generative adversarial network to obtain the optimal parameters;

[0059] A data acquisition module is used to obtain clean audio data sets to build training sets and test sets, and to obtain speaker audio that needs to be desensitized;

[0060] The speech desensitization module is used to load the parameters of the trained adversarial sample generation model and use the loaded adversarial sample generation model as the speech desensitization model; and to obtain the speaker audio that needs to be desensitized and use it as the input of the speech desensitization model, and output the desensitized adversarial audio for interaction with the outside world, so as to achieve the purpose of hiding the speaker's identity.

[0061] Regarding the device in the above embodiment, the specific manner in which each unit or module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0062] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein each module may or may not be physically separated. In addition, each functional module in the present invention may be integrated into a processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated modules or units may be implemented in the form of hardware or in the form of software functional units, so as to select some or all of the modules according to actual needs to achieve the purpose of the present application scheme.

[0063] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

Claims

1. A high-fidelity speech desensitization method based on adversarial sample generation technology, characterized in that: include: Construct a generative adversarial network, including a training adversarial sample generation model, a speech discrimination model, and a pre-trained speaker recognition model and speech recognition model; Train the generative adversarial network, use the adversarial sample generation model in the trained generative adversarial network as a speech desensitization model, use the speech desensitization model to superimpose disturbances on the input clean audio, generate adversarial audio that retains the original semantic information of the clean audio and desensitizes the identity information, and use the adversarial audio to interact with the outside world; The specific steps include: S01, build a generative adversarial network and randomly initialize the parameters of the speech identification model and adversarial sample generation model in the network; S02, collect a clean audio data set and divide the data set into a training set and a test set; S03, using the training set to train the generative adversarial network, is divided into two stages: In the first stage, the parameters of the speech discrimination model, the pre-trained speaker recognition model and the speech recognition model are fixed; the parameters of the adversarial sample generation model are iteratively updated through the back propagation algorithm, and the parameters of the adversarial sample generation model that makes the loss function of the test set the lowest are selected as the optimal parameters for storage; in the first stage, the loss of the back propagation algorithm includes three parts: voiceprint loss, semantic loss and similarity loss; In the second stage, the adversarial sample generation model is fixed according to the optimal parameters obtained in the first stage, and the parameters of the speech identification model are iteratively updated through the back propagation algorithm, and the parameters of the speech identification model that make the loss function of the test set the lowest are selected as the optimal parameters for storage; in the second stage, the loss of the back propagation algorithm is the similarity loss; The training process of the first and second stages is executed repeatedly until the parameters of the adversarial sample generation model and the parameters of the speech identification model meet the established optimization goals and the training is completed; S04, using the adversarial sample generation model obtained after training as a speech desensitization model, using the speaker audio as the input of the speech desensitization model, using the speech desensitization model to superimpose disturbances on the input speaker audio, generating adversarial audio that retains the original semantic information of the speaker audio and desensitizes the speaker identity information, using the adversarial audio to interact with the outside world to achieve the purpose of hiding the speaker's identity.

2. According to claim 1, a high-fidelity speech desensitization method based on adversarial sample generation technology is characterized in that: The model parameters obtained in each training cycle are saved in a breakpoint file. After the current training cycle is completed, the test set is used to test whether the training loss function value of the model is reduced. If so, the previous breakpoint file is deleted and the current breakpoint file is retained; if not, the current breakpoint file is deleted and the previous breakpoint file is retained.

3. According to claim 1, a high-fidelity speech desensitization method based on adversarial sample generation technology is characterized in that: In the process of training the generative adversarial network, the minimax game algorithm is used to make the adversarial sample generation model and the speech identification model reach a dynamic balance after alternating training.

4. According to claim 1, a high-fidelity speech desensitization method based on adversarial sample generation technology is characterized in that: The calculation of the voiceprint loss is specifically as follows: encoding the clean audio and the adversarial audio through a speaker recognition model, obtaining a clean audio voiceprint matrix and an adversarial audio voiceprint matrix, and calculating the cosine distance between the two as the voiceprint loss; The calculation of the semantic loss is specifically as follows: performing content recognition on the clean audio and the adversarial audio through the speech recognition model, obtaining the word sequence X of the clean audio and the word sequence of the adversarial audio, and calculating the connection temporal classification loss of the two as the semantic loss; The similarity loss is calculated specifically as follows: similarity features of the clean audio and the adversarial audio are extracted through a speech identification model, similarity confidence between the adversarial audio and the clean audio is predicted, and cross entropy loss is calculated as the similarity loss based on the similarity confidence and the true label.

5. According to claim 1, a high-fidelity speech desensitization method based on adversarial sample generation technology is characterized in that: The similarity loss is calculated specifically as follows: similarity features of the clean audio and the adversarial audio are extracted through a speech identification model, similarity confidence between the adversarial audio and the clean audio is predicted, and cross entropy loss is calculated as the similarity loss based on the similarity confidence and the true label.

6. A high-fidelity speech desensitization device based on adversarial sample generation technology, used to implement the high-fidelity speech desensitization method according to claim 1, the speech desensitization device comprising: A generative adversarial network module, which includes an adversarial sample generation model, a speech discrimination model, a speaker recognition model, and a speech recognition model; A generative adversarial network training module, which is used to train the adversarial sample generation model and speech identification model in the generative adversarial network to obtain the optimal parameters; A data acquisition module is used to obtain clean audio data sets to build training sets and test sets, and to obtain speaker audio that needs to be desensitized; A speech desensitization module, which is used to load the parameters of the trained adversarial sample generation model and use the loaded adversarial sample generation model as the speech desensitization model; Also, it is used to obtain the audio of the speaker that needs to be desensitized and use it as the input of the speech desensitization model, and output the desensitized adversarial audio for interaction with the outside world to achieve the purpose of hiding the identity of the speaker.

Images