Voice privacy protection method and storage medium
By using a machine learning-based feature separator, leveraging RSFE convolutional neural networks and multi-head attention mechanisms, combined with mutual information loss functions and customized protection strategies, this approach addresses the issue that voice privacy protection methods cannot balance privacy and data utility, thereby enabling secure transmission of voice data and personalized services.
Patent Information
- Application Number
- CN202310292210.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-23
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2043-03-23
AI Technical Summary
Existing voice privacy protection methods cannot guarantee the effective use of data while protecting user privacy, especially due to high computational costs and limited protection scope.
A machine learning-based feature separator is employed to extract speech features through RSFE convolutional neural networks and multi-head attention mechanisms. The feature extractor is optimized using mutual information loss functions to separate voiceprint, emotion, and health features from speech data. Custom protection strategies are implemented to ensure privacy protection during data transmission.
It enables customized privacy protection based on user needs while ensuring data utility, effectively separating sensitive information in voice data, and improving data transmission security and service personalization.
Smart Images
Figure CN116631440B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data protection technology, specifically to a method and storage medium for protecting voice privacy data based on machine learning anonymization. Background Technology
[0002] With the development of interactive devices such as voice assistants and smart speakers, smart voice devices need to collect users' voice data for analysis and prediction. Voice privacy protection technology must protect the privacy of users' voice data while ensuring high data availability. That is, it needs to protect the secondary information in users' voice data from being leaked, while also ensuring that users can enjoy the convenience brought by voice servers.
[0003] Common methods for protecting voice privacy include deletion, privacy computation, and anonymization. Deletion methods use acoustic sensors to interfere with ambient sound; however, this has a limited protection range—if the user goes beyond the sensor's effective range, protection is lost. Privacy computation methods apply traditional techniques like secure multi-party computation and differential privacy to speaker speech processing to address privacy leaks; however, their high computational cost makes them unsuitable for voice data transmission scenarios. Anonymization methods use machine learning models to generate adversarial noise to protect user privacy. Most of these solutions fail to balance data utility with privacy protection. Users need to fully enjoy the service while also having their privacy effectively protected—a critical issue that needs to be addressed in AI-powered voice data privacy protection. Summary of the Invention
[0004] This invention provides a voice privacy data protection method based on machine learning anonymization. This method separates the voiceprint, emotion and health features contained in the voice, which can fully protect the privacy information that users want to protect without affecting the effective use of the data.
[0005] The main steps of the technical solution of the present invention are as follows: the trained feature separator is deployed and distributed to the user terminal through an offline third-party model training module. The user customizes the feature information that needs to be protected. The feature separator selectively separates the user's real voice data according to the user's settings. The separated feature information that does not contain privacy information is transmitted to the voice server through symmetric encryption. Then, the voice server receives the encrypted feature value, decrypts and analyzes it, and pushes the corresponding personalized service.
[0006] The construction of the feature separator includes the following steps:
[0007] (1) Output the corresponding speech feature vector by passing the acoustic feature vector through the RSFE convolutional neural network model.
[0008] (2) Input the speech feature vector into the multi-head attention mechanism.
[0009] (3) Users select the elements to be separated from voiceprint, emotion, and health. The feature separator controls the objects of speech separation by setting intermediate parameters. By adding a mutual information loss function between the three feature extractors to optimize the feature extractors, the correlation and mutuality between the extracted features are reduced, thereby achieving feature separation.
[0010] The RSFE convolutional neural network model in step (1) can be a general model or an optimized model provided by this invention. The optimized RSFE convolutional neural network model construction method includes:
[0011] (a) The RSFE convolutional neural network model has 13 convolutional layers, a long short-term memory network (LSTM), an attention layer, and a fully connected layer.
[0012] (b) The activation function of the convolutional layer is the ReLU function, and the pooling layer is the max pooling method. The input features are connected to the results of the next 12 convolutional layers after passing through the first convolutional layer.
[0013] (c) The final fully connected layer uses the softmax activation function to regress the outputs of all dropout layers to obtain the corresponding feature output probabilities.
[0014] (d) Train the RSFE convolutional neural network model for a classification task and use the cross-entropy loss function to train the corresponding feature extractor.
[0015] The specific process of the multi-head attention mechanism described in step (2) is as follows:
[0016] (a) Input the output feature vectors of each speech feature extraction model into the input layer of the multi-head attention mechanism.
[0017] (b) Multi-head attention layer: The input feature vector is extracted through a multi-head attention mechanism. In the multi-head attention mechanism, the input feature vector is divided into multiple heads. Take head=10, calculate the attention score for each head, and then concatenate the outputs of all heads to obtain the output feature vector of the multi-head attention mechanism.
[0018] (c) Point linear transformation layer: The output feature vector of the multi-head attention layer is passed through a fully connected neural network layer to perform nonlinear transformation and extraction of features, resulting in an output feature vector with richer feature expression.
[0019] (d) Normalization layer: The output feature vector of the point linear transformation layer is normalized to eliminate the instability that may occur during the training of the neural network and the differences between features of different dimensions.
[0020] (e) Residual Connection: The output feature vectors of the input layer and the normalization layer are added together to obtain the final output feature vector of the multi-head attention mechanism. This residual connection design can improve the training effect of the model and avoid the gradient vanishing and gradient exploding problems in deep networks.
[0021] In step (3), the user can set the protection level for three elements (emotion, voiceprint, and health). The feature separation module then incorporates a mutual information loss function based on the user's settings to optimize the feature extractors, reducing the correlation between the features extracted by the three extractors and separating the information the user wants protected from the speech data. In this step, the speech feature vectors extracted by each speech extractor are subjected to feature separation. The specific implementation method is as follows:
[0022] (a) Users can select the elements to be separated from voiceprint, emotion, and health. The feature separator will control the objects of voice separation by setting intermediate parameters to protect user privacy without affecting the classification performance of the server.
[0023] (b) Feature separation is mainly achieved by optimizing the feature extractors by incorporating a mutual information loss function among the three feature extractors, thereby reducing the correlation between the features extracted by the three feature extractors. The mutual information expression is as follows:
[0024] I(u;v;w)=H(u)+H(v)+H(w)-H(u,v)-H(u,w)-H(v,w)+H(u,v,w);
[0025] Where u, v, and w represent three discrete random variables, H(u,w) represents the joint entropy of u and w, and H(·) represents the entropy.
[0026] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described voice privacy data protection method.
[0027] This invention also provides a speech feature separation system based on mutual information estimation, comprising the following modules:
[0028] Feature extraction module: This module trains corresponding feature extractors using voiceprint, emotion, and health datasets, respectively. The functions of this module are as follows:
[0029] The feature extractor was trained on a classification task using voiceprint, emotion, and health datasets respectively. When saving the model, the last classification layer was removed, resulting in three feature extractors that can extract the corresponding features.
[0030] Multi-head attention mechanism is incorporated into three speech feature extraction models to improve the accuracy of speech feature extraction.
[0031] User settings module: Allows users to set the level of protection for different elements.
[0032] Feature Separation Module: Based on user settings, this module incorporates a mutual information loss function to optimize the feature extractor in the feature extraction module, thereby separating the various element features contained in the speech. The functions of this module are as follows:
[0033] The degree of feature separation and the object can be controlled according to different user settings.
[0034] By incorporating a mutual information loss function, the correlation between features is reduced, thereby ensuring that the extracted features only contain the information that the user wants to share.
[0035] Technical features and effects of the present invention:
[0036] Existing technologies typically protect user privacy by adding noise, which can negatively impact data utility. This invention offers users multiple options, allowing them to customize privacy protection strategies according to their needs. The designed feature separator separates voice data on the user's end based on user settings, thus enabling data sharing while ensuring both data utility and privacy. Attached Figure Description
[0037] Figure 1 This is an overall framework diagram of the speech feature separation method in this invention;
[0038] Figure 2 This is a diagram of the RSFE convolutional neural network model designed in this invention. Detailed Implementation
[0039] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0040] like Figure 1 As shown, speech information obtained from speech recognition software and smart voice devices is processed by a speech feature extractor to extract voiceprint feature vectors, emotion feature vectors, and health feature vectors, respectively. A multi-head attention mechanism is incorporated into the speech feature extractor to help the neural network model better adapt to the data features, thereby improving model performance. Based on user settings, a mutual information loss function is added to selectively separate voiceprint, emotion, and health feature vectors from the speech data. The steps for establishing the above scheme are as follows:
[0041] (1) Collection of voice emotion, voiceprint and voice health datasets used for model training.
[0042] (1.1) In this case study, six categories of speech emotion data were extracted from the Chinese Emotion Dataset (CASIA) recorded by the Institute of Automation, Chinese Academy of Sciences, to train a speech emotion feature extractor. The CASIA dataset contains 9,600 different speech samples, recorded by four professional speakers (two men and two women). Each speaker recited 300 identical sentences and 100 different sentences, with each sentence having a version with six emotions (anger, happiness, fear, sadness, surprise, and neutral).
[0043] (1.2) The speaker feature extractor was trained using the VCTK dataset built by the Centre for Speech Technology at the University of Edinburgh (CSTR). The VCTK dataset contains speech data from 110 English speakers with different accents, each speaker reading about 400 sentences.
[0044] (1.3) The DementiaBank dataset, a collaboration between the University of Pittsburgh and Carnegie Mellon University, was used. This dataset contains speech samples from 117 Alzheimer's patients and 93 healthy individuals, all describing a picture. The Parkinson's Voice Initiative dataset, a collaboration between the University of London and the University of Cambridge, was also used. It contains 100,000 speech samples, of which 5,875 are from Parkinson's patients and 4,125 are from healthy individuals.
[0045] (2) Establishment of the feature extraction module.
[0046] (2.1) The input data consists of 13-dimensional MFCC feature values. The convolutional neural network model is trained on the dataset for classification tasks, resulting in a voiceprint feature extractor, a speech emotion feature extractor, and a speech health feature extractor. The specific model structure is as follows:
[0047] a. The feature extractor structure consists of 13 convolutional layers, a long short-term memory network (LSTM), an attention layer, and a fully connected layer.
[0048] b. The activation function of the convolutional layer is the ReLU function, and the pooling layer uses max pooling. After the input features pass through the first convolutional layer, they will be connected with the results of the subsequent 12 convolutional layers.
[0049] c. The final fully connected layer uses the softmax activation function to regress the outputs of all dropout layers to obtain the corresponding feature output probabilities.
[0050] d. Train the model for a classification task and use the cross-entropy loss function to train the corresponding feature extractor.
[0051] (2.2) Input the feature vector output by the feature extractor in step (2.1) into the multi-head attention mechanism to help the neural network model better adapt to the data features, thereby improving the model's performance, as follows:
[0052] a. Multi-head attention layer: First, the input feature vector is processed by a multi-head attention mechanism for feature extraction. In the multi-head attention mechanism, the input feature vector is divided into multiple heads, each of which calculates an attention score. Then, the outputs of all heads are concatenated to obtain the output feature vector of the multi-head attention mechanism. Here, head = 10 is selected.
[0053] b. Point linear transformation layer: The output feature vector of the multi-head attention layer is passed through a fully connected neural network layer to perform nonlinear transformation and extraction of features, resulting in a richer feature representation.
[0054] c. Normalization layer: Normalizes the output feature vector of the point linear transformation layer to eliminate instability that may occur during neural network training and the differences between features of different dimensions.
[0055] d. Residual connection: The output feature vectors of the input layer and the normalization layer are added together to obtain the final output feature vector of the multi-head attention mechanism.
[0056] (3) User settings module: Users can select elements they do not want to be shared according to their own situation, so that the feature separator can perform feature separation on the user's voice data in a targeted manner later.
[0057] (4) Establishment of the feature separation module.
[0058] (4.1) Based on the user settings module's custom settings in step (4), add a mutual information loss function between the three feature extractors described in step (3), as shown in the appendix. Figure 1 As shown. The mutual information between the three random variables is calculated using the following expression:
[0059] I(u;v;w)=H(u)+H(v)+H(w)-H(u,v)-H(u,w)-H(v,w)+H(u,v,w);
[0060] Where u, v, and w represent three discrete random variables, H(u,w) represents the joint entropy of u and w, and H(·) represents the entropy.
[0061] The expression for the mutual information loss function is as follows:
[0062]
[0063] Where f(x) i ,y i ) represents the mutual information between input data x and y, ω is a constant, and N is the number of input data.
[0064] (4.2) The mutual information loss function is added to the overall loss function. Specifically, the overall loss function can be calculated using the following formula: L loss =L classification +λ*L MI Among them, L classification This is the cross-entropy loss function of the classification model, where λ is a hyperparameter used to balance the weights of classification loss and mutual information loss. MI It is the mutual information loss function.
Claims
1. A voice privacy data protection method characterized by, The method comprises: training RSFE convolutional neural network using voiceprint, emotion and health data sets to obtain voiceprint, emotion and health feature extractors respectively; adding multi-head attention mechanism in the feature extractor and training in the manner of adding mutual information loss function between the voiceprint, emotion and health feature extractors to obtain the corresponding feature separator; deploying the trained feature separator, distributing it to the user end through the offline third-party model training module, and the user customizes the voiceprint, emotion and health and other three label information to be protected, the feature separator selectively separates the real voice data of the user according to the user's setting, transmits the separated feature information not containing any private information to the voice server through the symmetric encryption mode, then the voice server decrypts and analyzes the encrypted feature value after receiving it, and pushes the corresponding personalized service; the construction of the feature separator comprises the following steps: (1) output corresponding voiceprint, emotion and health feature vectors through the RSFE convolutional neural network model of acoustic feature vectors; (2) input the voiceprint, emotion and health feature vectors into the multi-head attention mechanism; (3) the user selects the elements to be separated in the voiceprint, emotion and health, and the feature separator controls the object of voice separation by setting the intermediate parameters, adds the mutual information loss function between the voiceprint, emotion and health feature extractors to optimize the feature extractor, and realizes feature separation.
2. The method of claim 1, wherein, The construction of the RSFE convolutional neural network model comprises the following steps: (1) the RSFE convolutional neural network model structure is 13 convolutional layers, one long short-term memory network LSTM, one attention layer and one fully connected layer; (2) the activation function of the convolutional layer adopts the relu function, the pooling layer adopts the maximum value pooling mode, and the input feature is connected after the first layer convolution with the results of the last 12 layers of convolution; (3) the last fully connected layer selects the softmax activation function to regress the output of all dropout layers to obtain the corresponding feature output probability; (4) the RSFE convolutional neural network model is trained for classification task, and the cross-entropy loss function is used to train the corresponding feature extractor.
3. A non-transitory computer-readable storage medium, comprising, A computer program is stored thereon, which, when executed by a processor, implements the steps of the method of claim 1 or 2.