Privacy information desensitization method and system for voice generation type large model

Abstract

The invention discloses a privacy information desensitization method and system for a voice generation type large model, and relates to the technical field of artificial intelligence. Input voice data is discretized, Gaussian noise disturbance sensitive features are injected, and a cross-modal voice generation model is constructed in combination with three-stage training; and meanwhile, a cross-modal privacy enhancement mechanism is applied to detect and fuzzify sensitive information in real time in an output stage. According to the invention, the adaptability of a large-scale voice generation model in a privacy protection scene is improved, and comprehensive protection of user privacy is realized. And moreover, the capability of extracting user privacy information by an adversarial attacker is effectively limited, and the risk of sensitive information leakage in the data transmission, storage and generation process of the voice generation model is reduced. On the premise that privacy is ensured, the voice generation model can still keep high-quality generation performance, generated voice output has high naturalness and accuracy, and actual application requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for desensitizing privacy information for speech generation large models. Background Art

[0002] Currently, large-scale speech models such as SpeechGPT have cross-modal dialogue capabilities, allowing the model to seamlessly generate and interpret multi-modal content (SpeechGPT Empowering La…). However, in a cloud-based environment, these models face significant privacy challenges. Privacy threats in a Machine Learning as a Service (MLaaS) environment can lead to the accidental leakage of sensitive information due to adversarial attacks on model outputs or parameters (Privacy-preserving Mach…).

[0003] Existing methods, such as differential privacy and homomorphic encryption, are effective in protecting certain types of privacy but are limited in covering all threat vectors.

[0004] Therefore, how to ensure comprehensive privacy protection for sensitive speech data is an urgent problem for those skilled in the art to solve. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for desensitizing privacy information for speech generation large models to ensure comprehensive privacy protection for sensitive speech data, especially during data transmission and model training.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for desensitizing privacy information for speech generation large models, comprising:

[0008] Performing discretization processing on input speech data to generate a discrete unit sequence;

[0009] Desensitizing the discrete unit sequence and perturbing sensitive features;

[0010] Determining the distribution and intensity of noise injection by analyzing the density of privacy information in the discrete unit sequence;

[0011] Using the desensitized speech data to perform three-stage training on a cross-modal speech generation model to obtain a trained cross-modal speech generation model;

[0012] By performing semantic analysis on the speech output of the cross-modal speech generation model, applying a cross-modal privacy enhancement mechanism to detect potential sensitive information and performing masking or obfuscation processing on it.

[0013] Optionally, the discretization process specifically includes: using a self-supervised learning-based speech unit extractor to convert the input speech data into a sequence of discrete units.

[0014] Optionally, the desensitization process specifically includes: performing selective noise injection on the privacy features in the discrete unit sequence, and the noise injection method is Gaussian noise.

[0015] Optionally, the three-stage training of the cross-modal speech generation model specifically includes:

[0016] A) Modal adaptation pre-training: Perform the next prediction task on the unlabeled speech data, introduce the discrete unit sequence into the model vocabulary, expand the speech processing ability of the language model, and avoid exposing sensitive data;

[0017] B) Cross-modal instruction fine-tuning: Use the speech-text paired data with privacy obfuscation for training to ensure that the model can process both speech and text inputs and generate corresponding outputs;

[0018] C) Multi-modal instruction fine-tuning: Further optimize the performance of the model in multi-modal tasks through chained modal instructions, while restricting the possibility of the model leaking privacy when generating speech outputs.

[0019] Optionally, the specific cross-modal privacy enhancement mechanism is:

[0020] When the model generates speech, detect the privacy features in the model output and weaken them to prevent the model from exposing user privacy information when providing services externally; this detection process uses a feature extraction algorithm to identify potential privacy information in the speech, and then desensitizes or weakens the relevant features through a specific algorithm to ensure that the output data of the model meets privacy requirements in cross-modal generation tasks.

[0021] Optionally, it further includes privacy evaluation and optimization: Design privacy risk assessment indicators to quantify the privacy leakage risk of the cross-modal speech generation model in different tasks, and optimize the obfuscation algorithm and training process based on experimental results to balance privacy protection and model performance.

[0022] A privacy information desensitization system for speech generation large models includes:

[0023] A data discretization module that performs discretization processing on the input speech data to generate a sequence of discrete units;

[0024] A desensitization module that desensitizes the discrete unit sequence and perturbs sensitive features;

[0025] A noise injection module that determines the distribution and intensity of noise injection by analyzing the density of privacy information in the discrete unit sequence;

[0026] The model establishment and training module uses the desensitized speech data to perform three-stage training on the cross-modal speech generation model to obtain a trained cross-modal speech generation model;

[0027] The privacy detection module detects potential sensitive information through semantic analysis of the speech output of the cross-modal speech generation model, applies the cross-modal privacy enhancement mechanism to mask or blur it.

[0028] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method and system for desensitizing privacy information of a speech generation large model. By discretizing the input speech data and injecting Gaussian noise to perturb sensitive features, a cross-modal speech generation model is constructed through three-stage training. At the same time, the cross-modal privacy enhancement mechanism is applied in the output stage to detect and blur sensitive information in real time. The present invention not only improves the adaptability of the large-scale speech generation model in privacy protection scenarios and realizes comprehensive protection of user privacy, but also effectively restricts the ability of adversarial attackers to extract user privacy information through the discretization of speech data, adaptive noise injection and privacy confusion technology, reducing the risk of sensitive information leakage in the process of data transmission, storage and generation of the speech generation model. On the premise of ensuring privacy, the speech generation model can still maintain high-quality generation performance, and the generated speech output has high naturalness and accuracy, meeting the actual application requirements. Description of the Drawings

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0030] Figure 1 It is a schematic flowchart of the method provided by the present invention;

[0031] Figure 2 It is a schematic flowchart of the privacy desensitization method provided by the present invention. Detailed Embodiments

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0033] An embodiment of the present invention discloses a privacy information desensitization method for speech generative large models, as follows Figure 1 shown, including:

[0034] 1. Discretize the input speech data to generate a discrete unit sequence;

[0035] 2. Desensitize the discrete unit sequence and perturb sensitive features;

[0036] 3. Determine the distribution and intensity of noise injection by analyzing the density of privacy information in the discrete unit sequence;

[0037] 4. Use the desensitized speech data to perform three-stage training on the cross-modal speech generation model to obtain a trained cross-modal speech generation model;

[0038] 5. Through semantic analysis of the speech output of the cross-modal speech generation model, apply a cross-modal privacy enhancement mechanism to detect potential sensitive information and mask or blur it.

[0039] In a specific embodiment, the discretization process specifically includes: using a speech unit extractor based on self-supervised learning to convert the input speech data into a discrete unit sequence. K-means clustering can also be applied to discretize the intermediate representation.

[0040] In a specific embodiment, the desensitization process specifically includes: selectively injecting noise into the privacy features in the discrete unit sequence, and the noise injection method is Gaussian noise.

[0041] The desensitization process specifically includes: selectively injecting noise into important privacy features (such as identity features and dialect features) in the discrete unit sequence, and the noise injection method is Gaussian noise. The purpose is to effectively reduce the inference accuracy of these features without significantly affecting the model generation effect; to ensure privacy protection, use a random noise mechanism to confuse the discrete unit sequence and interfere with the extraction of individual privacy attributes by adversarial attackers. Perform adaptive noise injection on the discrete unit sequence to achieve de-identification of the data by adding Gaussian noise.

[0042] Adaptive noise optimization, where the degree of noise injection is dynamically adjusted according to the importance and sensitivity of the data. To this end, by analyzing the density of privacy information in the speech content, the distribution and intensity of noise injection are determined. A privacy risk analysis module is used to mark high-sensitivity areas in the speech unit sequence. The privacy risk analysis module first collects data-related information, including type, storage location, and access method. Then it identifies possible risk scenarios and evaluates the likelihood and impact of the risks. When marking, different colors (such as red, yellow, green) are used to mark relevant data processing activities, etc. according to the risk level (high, medium, low), and it also includes a description of the risk type. High-intensity noise is added to the marked areas, while low-intensity noise is added to ordinary data to balance privacy protection and model performance.

[0043] By adding random noise or generating synthetic samples on specific sensitive features, the privacy information in the speech data is masked. In this embodiment, Gaussian noise is used to perturb the input speech features to reduce the identifiability of sensitive information. For example, for features related to user identity (such as timbre, accent), without significantly affecting the model generation ability, by adding different intensities of noise to weaken such features and ensure data security.

[0044] In a specific embodiment, as Figure 2 shown, the privacy desensitization method includes the following steps:

[0045] (a): Add Gaussian noise to the discrete unit sequence, and calculate the perturbed eigenvalue according to formulas (1) and (2) to achieve the purpose of privacy desensitization.

[0046] Formula (1) Dynamic calculation of noise intensity

[0047] (1)

[0048] d i : Privacy density value corresponding to the i-th discrete unit (calculated through a classifier or rule base, such as sensitive word frequency, voiceprint feature significance)

[0049] λ: Upper limit of noise intensity (controlling the maximum perturbation amplitude)

[0050] r: Density threshold (distinguishing high / low sensitivity areas)

[0051] γ: Steepness coefficient of the S-shaped function (controlling the transition interval)

[0052] Formula (2) Privacy perturbation operation

[0053] (2)

[0054] x i : Feature representation of the original discrete unit (such as the continuous vector after HuBERT encoding)

[0055] ε i : Random noise that follows a Gaussian distribution

[0056] The variance dynamically calculated by formula (1)

[0057] (b): Fine-tune the model with the cross-modal instruction dataset SpeechInstruct to train the model to process desensitized data with noise. In this embodiment, different noise levels are set to balance data privacy protection and generation quality. This process also includes chained-modal training to ensure that the model has strong instruction-following ability when generating speech responses by constructing chained instructions.

[0058] (c): In the inference stage, weaken the privacy features through feature detection and adaptive processing to ensure that the model output does not expose the sensitive information of the user. The privacy enhancement mechanism specifically designs two desensitization methods: speed weakening based on features and timbre adjustment. In practical applications, the privacy enhancement parameters can be adjusted according to user needs to achieve the best protection effect.

[0059] In a specific embodiment, the three-stage training of the cross-modal speech generation model specifically includes:

[0060] A) Modal adaptation pre-training: Perform the next prediction task on unlabeled speech data, introduce the discrete unit sequence into the model vocabulary, expand the speech processing ability of the language model, and avoid exposing sensitive data;

[0061] B) Cross-modal instruction fine-tuning: Train with speech-text paired data with privacy obfuscation to ensure that the model can process both speech and text inputs and generate corresponding outputs;

[0062] C) Multi-modal instruction fine-tuning: Further optimize the performance of the model in multi-modal tasks through chained-modal instructions, while restricting the possibility of the model leaking privacy when generating speech outputs.

[0063] The cross-modal instruction fine-tuning process includes: training the model with the cross-modal instruction dataset (such as SpeechInstruct), and improving the generalization ability of the model on multi-modal data through instruction fine-tuning and chained instruction fine-tuning. This process adopts a three-stage structure: modal adaptation pre-training, cross-modal instruction fine-tuning, and chained-modal instruction fine-tuning to ensure that the model can effectively process desensitized speech data (SpeechGPT Empowering La…), and achieve efficient processing of cross-modal tasks.

[0064] In a specific embodiment, the specific content of the cross-modal privacy enhancement mechanism is:

[0065] When the model generates speech, an embedded privacy filtering module is used to detect privacy features in the model output and weaken them to prevent the model from exposing users' privacy information when providing services externally; during this detection process, HuBERT is first used for iterative clustering of speech units, or Wav2Vec2.0 is adopted to discretize speech features through vector quantization (VQ) technology, so as to extract key acoustic information. In addition, MFCC is combined for voiceprint recognition and emotion analysis, and DTW (Dynamic Time Warping) is used to accurately locate sensitive audio segments. In terms of cross-modal feature fusion, Transformer is used to process speech-text paired data, and the semantic and acoustic features are aligned through the attention mechanism to identify possible privacy leakage risks. For privacy protection, Gaussian noise injection technology is adopted for adaptive noise adjustment and residual perturbation to prevent privacy information from being reversely recovered. At the same time, through feature replacement and masking technology, sensitive segments are replaced or voiceprint confusion is performed. In addition, BERT is combined to analyze text semantics, detect sensitive words related to privacy, and adjust the corresponding acoustic features (such as fundamental frequency and speech rate) through speech synthesis technology. Finally, HiFi-GAN is used to enhance the speech quality, and the residual privacy features are further weakened through the privacy filtering module to ensure that the model output meets the privacy protection requirements.

[0066] This mechanism ensures that the output speech data does not disclose users' sensitive information by detecting potential privacy information (such as pitch, speech rate, etc.) in the generated speech and adaptively adjusting these features.

[0067] In a specific embodiment, it also includes privacy assessment and optimization: First, the discriminability of voiceprint features is evaluated through the equal error rate (EER) of voiceprint recognition. The ECAPA-TDNN voiceprint recognition model is trained using VoxCeleb. If the EER before desensitization is 0.5% and it increases to 15% after desensitization, it indicates that voiceprint privacy is effectively protected. Second, the sensitive word detection rate measures the retention of privacy information in the text. The omission rate of sensitive words is statistically analyzed using SnipsNLU. For example, if the detection rate drops from 95% to 10%, it shows that the desensitization mechanism effectively reduces privacy leakage. Cross-modal consistency is measured by the BLEU score to evaluate the similarity between the generated speech and the original text. If the BLEU drops by no more than 5% after desensitization, it indicates that privacy protection does not significantly affect semantic information. In addition, the success rate of membership inference attack is used to evaluate whether the model leaks member information in the training data. If the success rate of the attack using the projected gradient descent method drops from 80% to 20%, it indicates that the model enhances the protection ability of member privacy. The accuracy of attribute inference attack measures the inference ability of user attributes such as age and gender. For example, if the gender recognition accuracy drops from 90% to 45%, it indicates that the privacy protection strategy is effective. The effectiveness of cross-modal forgetting is evaluated by MLLMU-Bench to determine whether the model forgets specific privacy information. For example, the memory rate of sensitive images under multi-modal input drops from 70% to 10%. Finally, the voiceprint-semantic decoupling degree quantifies the correlation between voiceprint features and semantic content through mutual information. The decoupling degree drops from 0.8 to 0.2, indicating that the separation of voiceprint privacy and semantic content is more thorough. The comprehensive evaluation of these indicators helps to optimize the privacy protection strategy of the model while maintaining a high cross-modal generation quality.

[0068] A privacy information desensitization system for a speech generation large model, comprising:

[0069] A data discretization module that discretizes the input speech data to generate a sequence of discrete units;

[0070] A desensitization module that desensitizes the sequence of discrete units and perturbs sensitive features;

[0071] A noise injection module that determines the distribution and intensity of noise injection by analyzing the density of privacy information in the sequence of discrete units;

[0072] A model establishment and training module that uses the desensitized speech data to perform three-stage training on a cross-modal speech generation model to obtain a trained cross-modal speech generation model;

[0073] A privacy detection module that performs semantic analysis on the speech output of the cross-modal speech generation model, applies a cross-modal privacy enhancement mechanism to detect potential sensitive information, and masks or blurs it.

[0074] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0075] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A privacy information desensitization method for speech generation large models, characterized in that, Including: Performing discretization processing on the input speech data to generate a discrete unit sequence; Desensitizing the discrete unit sequence and perturbing sensitive features; Determining the distribution and intensity of noise injection by analyzing the density of privacy information in the discrete unit sequence; Using the desensitized speech data to perform three-stage training on the cross-modal speech generation model to obtain a trained cross-modal speech generation model; By performing semantic analysis on the speech output of the cross-modal speech generation model, applying a cross-modal privacy enhancement mechanism to detect potential sensitive information and masking or obfuscating it.

2. The privacy information desensitization method for a speech generation large model according to claim 1, wherein, The discretization processing specifically includes: using a speech unit extractor based on self-supervised learning to convert the input speech data into a discrete unit sequence.

3. A privacy information desensitization method for speech generation large models according to claim 1, characterized in that The desensitization processing specifically includes: performing selective noise injection on the privacy features in the discrete unit sequence, and the noise injection method is Gaussian noise.

4. A privacy information desensitization method for a speech generation large model according to claim 1, characterized in that, The three-stage training of the cross-modal speech generation model specifically includes: A) Modal adaptation pre-training: Performing the next prediction task on unlabeled speech data, introducing the discrete unit sequence into the model vocabulary to expand the speech processing ability of the language model while avoiding exposing sensitive data; B) Cross-modal instruction fine-tuning: Training using speech-text paired data with privacy confusion to ensure that the model can process both speech and text inputs and generate corresponding outputs; C) Multi-modal instruction fine-tuning: Further optimizing the model's performance in multi-modal tasks through chained modal instructions while restricting the possibility of the model leaking privacy when generating speech outputs.

5. A privacy information desensitization method for a speech generation large model according to claim 1, characterized in that, The specific cross-modal privacy enhancement mechanism is: When the model generates speech, by detecting and weakening the privacy features in the model output, preventing the model from exposing user privacy information when providing services externally; this detection process uses a feature extraction algorithm to identify potential privacy information in the speech, and then desensitizes or weakens the relevant features through a specific algorithm to ensure that the output data of the model meets privacy requirements in cross-modal generation tasks.

6. A privacy information desensitization method for a speech generation large model according to claim 1, characterized in that, It also includes privacy evaluation and optimization: Designing privacy risk assessment metrics to quantify the privacy leakage risk of the cross-modal speech generation model in different tasks, and optimizing the confusion algorithm and training process based on experimental results to balance privacy protection and model performance.

7. A privacy information desensitization system for speech generation large models, characterized in that, Applying the privacy information desensitization method for a speech generation large model according to any one of claims 1-6, including: A data discretization module that performs discretization processing on the input speech data to generate a discrete unit sequence; A desensitization module that desensitizes the discrete unit sequence and perturbs sensitive features; A noise injection module that determines the distribution and intensity of noise injection by analyzing the density of privacy information in the discrete unit sequence; A model establishment and training module that uses the desensitized speech data to perform three-stage training on the cross-modal speech generation model to obtain a trained cross-modal speech generation model; A privacy detection module that performs semantic analysis on the speech output of the cross-modal speech generation model, applies a cross-modal privacy enhancement mechanism to detect potential sensitive information, and masks or obfuscates it.

Images