Data desensitization method and device based on TTS speech synthesis, equipment and medium

Abstract

The present application relates to the technical field of finance, medical health and artificial intelligence, and provides a data desensitization method and device based on TTS speech synthesis, equipment and medium, which can perform desensitization processing on initial text and user initial speech based on a multi-modal voiceprint confusion desensitization architecture, realize desensitization processing of various modes, embed watermarks based on a dynamic quantization watermark embedding mechanism, facilitate subsequent tracing and privacy verification, train a sub-TTS model at each node based on a federated learning driven differential privacy protection mechanism, and aggregate the sub-TTS models into a target TTS model, which can break data silos while preventing model inversion attacks, process real-time speech data streams output by the target TTS model based on a real-time streaming desensitization acceleration engine, which can realize real-time data desensitization and improve processing efficiency, and verify the synthesized speech based on a cross-modal privacy consistency verification mechanism, which can further ensure the speech synthesis effect.

Description

Technical Field

[0001] This invention relates to the fields of finance, healthcare and artificial intelligence, and in particular to a data desensitization method, apparatus, device and medium based on TTS speech synthesis. Background Technology

[0002] Currently, to protect data privacy, various fields involve scenarios where data based on TTS (Text-to-Speech) speech synthesis needs to be anonymized. For example, in the financial sector, when the TTS-synthesized speech returned by intelligent customer service includes sensitive words such as ID card numbers, the ID card numbers need to be anonymized to avoid privacy leaks; in the healthcare sector, sensitive information such as patient names and contact information also needs to be hidden in voice medical record retrieval scenarios.

[0003] However, the privacy protection technologies of existing TTS speech synthesis systems still have the following main problems:

[0004] (1) The contradiction between desensitization effect and speech quality.

[0005] Specifically, while traditional desensitization methods (such as static noise injection and speech feature masking) can hide speaker information, they significantly reduce the naturalness of the speech. Noise addition based on spectral subtraction can lower the MOS (Mean Opinion Score) to below 3.2, and result in a loss of up to 42% of high-frequency details (such as sibilants and plosives). Furthermore, the Texas College guidelines state that dynamic desensitization using row or column constraints can disrupt the coherence of speech data, leading to prosodic breaks in the synthesized speech.

[0006] (2) The watermark embedding lacks concealment and robustness.

[0007] Specifically, existing watermarking schemes (such as LSB (Least Significant Bit) pixel replacement) are easily compromised by frequency domain filtering attacks and are vulnerable to resampling and compression operations. JavaScript dynamic watermarking experiments show that frequency domain phase modulation-based schemes have a failure rate as high as 65% after MP3 (Moving Picture Experts Group Audio Layer III) compression (128kbps). While Android audio and video dynamic watermarking supports timestamp overlay, it does not achieve covert embedding at the acoustic feature level, making it easy for malicious users to locate the watermark position through waveform analysis.

[0008] (3) Risks of voiceprint cloning and privacy leakage.

[0009] Specifically, while mainstream TTS systems (such as MiniMax Speech-02-HD) support high-fidelity voice cloning, they lack a voiceprint obfuscation mechanism. Attackers can reconstruct the voiceprint characteristics of the target speaker through a 5-second reference voice, which poses a risk of identity forgery. Summary of the Invention

[0010] In view of the above, it is necessary to provide a data desensitization method, apparatus, device and medium based on TTS speech synthesis, in order to solve the problem of poor data desensitization effect of TTS speech synthesis.

[0011] A data desensitization method based on TTS speech synthesis, the data desensitization method based on TTS speech synthesis includes:

[0012] For each node, obtain the initial text and the user's initial voice corresponding to each initial text;

[0013] The initial text and the user's initial speech are desensitized based on a multimodal voiceprint obfuscation and desensitization architecture to obtain the first data;

[0014] The first data is watermarked using a dynamic quantum watermarking embedding mechanism to obtain the second data.

[0015] Based on a federated learning-driven differential privacy protection mechanism, a sub-TTS model is trained at each node using the second data of each node, and the sub-TTS models of each node are aggregated into the target TTS model.

[0016] In response to a speech synthesis instruction triggered by a target text and a target user's speech corresponding to the target text, the target text and the target user's speech are input into the target TTS model for speech synthesis, and the real-time speech data stream output by the target TTS model is processed based on a real-time streaming desensitization acceleration engine to obtain synthesized speech.

[0017] The synthesized speech is verified based on a cross-modal privacy consistency verification mechanism;

[0018] When the synthesized speech passes the verification, the synthesized speech is sent to the triggerer of the speech synthesis instruction.

[0019] A data desensitization device based on TTS speech synthesis, the data desensitization device based on TTS speech synthesis includes:

[0020] The acquisition unit is used to acquire the initial text and the user's initial speech corresponding to each initial text for each node;

[0021] The desensitization unit is used to desensitize the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain the first data;

[0022] An embedding unit is used to embed a watermark into the first data based on a dynamic quantum watermark embedding mechanism to obtain the second data.

[0023] The training unit is used for a differential privacy protection mechanism driven by federated learning. It uses the second data of each node to train a sub-TTS model at each node, and aggregates the sub-TTS models of each node into the target TTS model.

[0024] The synthesis unit is configured to respond to a speech synthesis instruction triggered by a target text and a target user's speech corresponding to the target text, input the target text and the target user's speech into the target TTS model for speech synthesis, and process the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine to obtain synthesized speech.

[0025] The verification unit is used to verify the synthesized speech based on a cross-modal privacy consistency verification mechanism;

[0026] The sending unit is used to send the synthesized speech to the triggerer of the speech synthesis instruction when the synthesized speech passes the verification.

[0027] A computer device, the computer device comprising:

[0028] Memory, storing at least one instruction; and

[0029] The processor executes the instructions stored in the memory to implement the data desensitization method based on TTS speech synthesis.

[0030] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the data desensitization method based on TTS speech synthesis.

[0031] As can be seen from the above technical solutions, this invention can desensitize the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain the first data, achieving desensitization processing for various modalities; it can also embed the first data with a watermark based on a dynamic quantum watermark embedding mechanism to obtain the second data, facilitating subsequent traceability and privacy verification; based on a federated learning-driven differential privacy protection mechanism, it can train a sub-TTS model at each node using the second data of each node, and aggregate the sub-TTS models of each node into a target TTS model, breaking down data silos and preventing model inversion attacks; it can process the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine, achieving real-time data desensitization and improving processing efficiency; and it can verify the synthesized speech based on a cross-modal privacy consistency verification mechanism, further ensuring the speech synthesis effect. Attached Figure Description

[0032] Figure 1 This is a flowchart of a preferred embodiment of the data desensitization method based on TTS speech synthesis of the present invention.

[0033] Figure 2 This is a functional block diagram of a preferred embodiment of the data desensitization device based on TTS speech synthesis of the present invention.

[0034] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the data desensitization method based on TTS speech synthesis according to the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the data desensitization method based on TTS speech synthesis according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0037] The data desensitization method based on TTS speech synthesis is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0038] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0039] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0040] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0041] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0042] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0043] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0044] S10: For each node, obtain the initial text and the user's initial voice corresponding to each initial text.

[0045] In this embodiment, the node can be a local node of each company or entity.

[0046] In this embodiment, the initial text and the user's initial voice corresponding to each initial text can be text and user voice under different tasks and scenarios.

[0047] For example, in the context of intelligent customer service in the financial sector, the initial text can be the text of the intelligent customer service (to be converted into speech), and the user's initial speech can be the input speech of the user in the dialogue; in the context of intelligent consultation in the healthcare sector, the initial text can be the output text of the medical consultation robot (to be converted into speech), and the user's initial speech can be the input speech of the patient seeking consultation.

[0048] S11, the initial text and the user's initial speech are desensitized based on a multimodal voiceprint obfuscation and desensitization architecture to obtain the first data.

[0049] In this embodiment, the desensitization processing of the initial text and the user's initial speech based on the multimodal voiceprint obfuscation desensitization architecture to obtain the first data includes:

[0050] The privacy information in the initial text is identified by a bidirectional long short-term memory sensitive word detection network, and the privacy information is replaced based on a local rewriting strategy to obtain the replaced text;

[0051] Generative adversarial networks are used to generate noise orthogonal to the corresponding user's fundamental frequency envelope in the user's initial speech, resulting in noisy speech;

[0052] The first data is obtained by combining the replaced text and the noisy speech.

[0053] The multimodal voiceprint obfuscation desensitization architecture is an acoustic-semantic joint desensitization framework.

[0054] For example, on the semantic side, when the bidirectional long short-term memory sensitive word detection network identifies the private information "Beijing" in the initial text, it replaces "Beijing" with "North China," accurately rewriting text privacy (such as phone numbers, addresses, etc.) and reducing the risk of leakage at the semantic level with an accuracy exceeding 98%. On the acoustic side, a generative adversarial network is used to generate noise (frequency band offset ±200Hz) orthogonal to the corresponding user's fundamental frequency envelope in the user's initial speech. After adversarial frequency domain perturbation, the voiceprint similarity can be reduced from 4.3 to 2.8. By changing the voiceprint features, the direct association between voiceprint and identity is severed, avoiding the leakage of user privacy caused by "identifying people by their voice."

[0055] S12, the first data is watermarked based on the dynamic quantum watermark embedding mechanism to obtain the second data.

[0056] In this embodiment, the dynamic quantized watermark embedding mechanism is equivalent to a dynamic watermarking system based on a quantum neural network.

[0057] Specifically, the dynamic quantized watermark embedding mechanism includes a generator; the process of embedding the watermark into the first data based on the dynamic quantized watermark embedding mechanism to obtain the second data includes:

[0058] In the generator, the user ID (Identity) of the user corresponding to the first data is obtained;

[0059] Perform a hash operation on the user ID to obtain the user ID hash value;

[0060] A quantum state watermark is generated based on the user ID hash value;

[0061] The quantum state watermark is encoded using Qubit entanglement coding to obtain the watermark code;

[0062] The watermark code is embedded into the zero-crossing rate feature of the speech Mel spectrogram of the first data to obtain the second data.

[0063] For example, quantum neural networks can be used to convert user ID hashes into 128-bit quantum watermarks, thus adding a "digital ID" to voice messages for subsequent tasks such as copyright tracking and privacy verification. At the same time, robustness against compression attacks can be improved by 3 times.

[0064] Specifically, the dynamic quantum watermark embedding mechanism further includes an extractor; after obtaining the second data, the method further includes:

[0065] In the extractor, the quantum annealing algorithm is used to decode the second data to obtain the watermark to be verified;

[0066] When the watermark to be verified is the same as the quantum state watermark, the second data is determined to pass verification; or

[0067] When the watermark to be verified is different from the quantum state watermark, it is determined that the second data has failed verification.

[0068] Experiments show that the quantum annealing algorithm can achieve fast watermark decoding, thereby verifying the integrity of the embedding, with a latency of ≤20ms and a bit error rate of less than 0.1%.

[0069] In the above embodiments, if the extractor can still decode the watermark after the speech is compressed and edited, it proves that the second data is data obtained after processing the initial text and the corresponding user's initial speech.

[0070] S13, based on a federated learning-driven differential privacy protection mechanism, uses the second data of each node to train a sub-TTS (Text-to-Speech) model at each node, and aggregates the sub-TTS models of each node into the target TTS model.

[0071] In this embodiment, the differential privacy protection mechanism driven by federated learning is equivalent to a hierarchical differential privacy federated training framework.

[0072] In this embodiment, the differential privacy protection mechanism based on federated learning, which uses the second data of each node to train a sub-TTS model at each node and aggregates the sub-TTS models of each node into a target TTS model, includes:

[0073] Using the second data from each node, train a sub-TTS model at each node and obtain the model parameters for each sub-model;

[0074] Laplacian noise is added to the model parameters of each sub-model using gradient perturbation;

[0075] The model parameters of each sub-model after adding noise are aggregated based on the homomorphic encryption algorithm to obtain the global model parameters;

[0076] The target TTS model is generated based on the global model parameters.

[0077] Federated learning can break down data silos and enable collaborative training using multi-source data.

[0078] Specifically, by adding Laplacian noise to the model parameters of each sub-model using gradient perturbation, and then aggregating the model parameters of each sub-model after adding noise using a homomorphic encryption algorithm, model inversion attacks can be prevented, making it impossible to infer user privacy from model parameters, thus reducing the cumulative error of privacy budget by 62%.

[0079] For example, companies A, B, and C each have local data. Through federated learning, they can jointly train a better TTS model without sharing the original data, and their data privacy is not compromised.

[0080] In the above embodiments, each node trains an acoustic model with desensitized data and adds Laplacian noise (ε = 0.5) to the gradient to hide individual data features; at the same time, the gradients of each node are aggregated through homomorphic encryption to eliminate the cumulative error of privacy budget.

[0081] S14, in response to a speech synthesis instruction triggered by the target text and the target user's speech corresponding to the target text, the target text and the target user's speech are input into the target TTS model for speech synthesis, and the real-time speech data stream output by the target TTS model is processed based on a real-time streaming desensitization acceleration engine to obtain synthesized speech.

[0082] In this embodiment, the voice synthesis command can be triggered by relevant staff according to actual needs.

[0083] In this embodiment, the real-time streaming desensitization acceleration engine is a GPU (Graphics Processing Unit)-CPU (Central Processing Unit) heterogeneous computing architecture.

[0084] In this embodiment, the process of processing the real-time speech data stream output by the target TTS model based on the real-time streaming desensitization acceleration engine to obtain synthesized speech includes:

[0085] The synthesized speech is obtained by using a WebGL (Web Graphics Library) parallel computing mechanism and a dynamic desensitization rule engine to perform real-time regular expression matching on sensitive fields in the real-time speech data stream, and then desensitizing the matched fields.

[0086] The dynamic desensitization rule engine can match sensitive fields in real time (throughput of 100,000 records / second) and then trigger desensitization to meet the needs of low-latency scenarios (such as real-time call scenarios, voice navigation scenarios, etc.).

[0087] Among them, by using WebGL parallel computing, the frequency domain watermark embedding speed can be increased to 50ms / sentence, which can meet the real-time synthesis requirements of speech data.

[0088] S15, the synthesized speech is verified based on the cross-modal privacy consistency verification mechanism.

[0089] In this embodiment, the cross-modal privacy consistency verification mechanism is equivalent to a voice-text-lip movement trimodal verification system.

[0090] Specifically, the verification of the synthesized speech based on the cross-modal privacy consistency verification mechanism includes:

[0091] The semantic consistency between the synthesized speech and the target text is compared based on the Transformer multi-task network.

[0092] The system detects whether there are sensitive word change events in the synthesized speech. When a sensitive word change event exists, the system adjusts the opening degree parameter of the 3D lip shape model to synchronize the 3D lip shape with the changed speech content.

[0093] Among them, semantic consistency detection can intercept abnormal privacy leakage requests, such as voiceprints that do not match ID numbers.

[0094] Among them, 3D lip shape adjustment can ensure that the synchronization error between visual and auditory information is ≤5ms, thereby ensuring the naturalness of multimodal scenes. For example, when a virtual human speaks, lip movements and speech can be perfectly synchronized to improve realism.

[0095] For example, when a virtual anchor synthesizes the voice "I am in North China", if the text is mistakenly written as "I am in Beijing", the system detects the semantic-speech inconsistency, intercepts the output, corrects the text, and adjusts the lip movement animation.

[0096] S16, when the synthesized speech passes the verification, the synthesized speech is sent to the triggerer of the speech synthesis instruction.

[0097] In the above embodiments, after the speech is quickly generated, the verified speech is sent to the triggerer of the speech synthesis instruction, thereby ensuring that the synthesized speech is natural and compliant.

[0098] This embodiment achieves a dual improvement in privacy protection and voice quality. Specifically, by using voiceprint obfuscation technology, the similarity of synthesized speech can be ≤2.8, while maintaining a MOS score ≥4.1 (an improvement of 0.9 points compared to traditional desensitization schemes).

[0099] This embodiment also provides resistance to attacks on the watermarking system. Specifically, the quantum watermark achieves an extraction success rate of ≥99.5% and a source tracing accuracy of 100% under attacks such as MP3 compression (64kbps) and resampling (8kHz→16kHz).

[0100] This embodiment also achieves real-time performance and resource optimization. Specifically, the GPU acceleration engine reduces the de-identification latency to 50ms, with a memory footprint of only 38MB, supporting embedded device deployment (CPU utilization ≤15%).

[0101] This embodiment also achieves a breakthrough in cross-scenario applicability. Specifically, the federated learning framework supports simultaneous training in 32 languages, and the adaptation cycle for privacy models in new languages ​​is shortened from 3 months to 2 weeks.

[0102] This embodiment also enables compliance and auditing capabilities. Specifically, the dynamic watermark log can be traced back to a single synthesis request, meeting the auditing requirements of regulations such as GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act), improving the efficiency of data breach liability determination by 90%.

[0103] This embodiment can be applied to data anonymization in TTS speech synthesis tasks in various scenarios such as finance and healthcare.

[0104] For example, in financial scenarios, intelligent customer service systems in banks and voice assistants in securities trading often need to read out information such as account balance and transaction password when responding to users. If the robot reads out this text directly, it will lead to privacy leaks. Therefore, it is necessary to complete the desensitization in the TTS synthesis stage to avoid the broadcast of sensitive information.

[0105] For example, in a healthcare setting, a hospital system's voice medical record query function needs to read out patient diagnosis records to doctors, but it needs to hide sensitive information such as names, ID numbers, and contact information, while ensuring accurate reading of medical terminology. This embodiment allows for data anonymization when the voice medical record query robot reads out patient diagnosis records, thereby avoiding the leakage of patient privacy.

[0106] As can be seen from the above technical solutions, this invention can desensitize the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain the first data, achieving desensitization processing for various modalities; it can also embed the first data with a watermark based on a dynamic quantum watermark embedding mechanism to obtain the second data, facilitating subsequent traceability and privacy verification; based on a federated learning-driven differential privacy protection mechanism, it can train a sub-TTS model at each node using the second data of each node, and aggregate the sub-TTS models of each node into a target TTS model, breaking down data silos and preventing model inversion attacks; it can process the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine, achieving real-time data desensitization and improving processing efficiency; and it can verify the synthesized speech based on a cross-modal privacy consistency verification mechanism, further ensuring the speech synthesis effect.

[0107] like Figure 2The diagram shown is a functional block diagram of a preferred embodiment of the data desensitization device based on TTS speech synthesis of the present invention. The data desensitization device 11 based on TTS speech synthesis includes an acquisition unit 110, a desensitization unit 111, an embedding unit 112, a training unit 113, a synthesis unit 114, a verification unit 115, and a sending unit 116. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0108] The acquisition unit 110 is used to acquire the initial text and the user's initial voice corresponding to each initial text for each node.

[0109] In this embodiment, the node can be a local node of each company or entity.

[0110] In this embodiment, the initial text and the user's initial voice corresponding to each initial text can be text and user voice under different tasks and scenarios.

[0111] For example, in the context of intelligent customer service in the financial sector, the initial text can be the text of the intelligent customer service (to be converted into speech), and the user's initial speech can be the input speech of the user in the dialogue; in the context of intelligent consultation in the healthcare sector, the initial text can be the output text of the medical consultation robot (to be converted into speech), and the user's initial speech can be the input speech of the patient seeking consultation.

[0112] The desensitization unit 111 is used to desensitize the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain first data.

[0113] In this embodiment, the desensitization unit 111 performs desensitization processing on the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain the first data, including:

[0114] The privacy information in the initial text is identified by a bidirectional long short-term memory sensitive word detection network, and the privacy information is replaced based on a local rewriting strategy to obtain the replaced text;

[0115] Generative adversarial networks are used to generate noise orthogonal to the corresponding user's fundamental frequency envelope in the user's initial speech, resulting in noisy speech;

[0116] The first data is obtained by combining the replaced text and the noisy speech.

[0117] The multimodal voiceprint obfuscation desensitization architecture is an acoustic-semantic joint desensitization framework.

[0118] For example, on the semantic side, when the bidirectional long short-term memory sensitive word detection network identifies the private information "Beijing" in the initial text, it replaces "Beijing" with "North China," accurately rewriting text privacy (such as phone numbers, addresses, etc.) and reducing the risk of leakage at the semantic level with an accuracy exceeding 98%. On the acoustic side, a generative adversarial network is used to generate noise (frequency band offset ±200Hz) orthogonal to the corresponding user's fundamental frequency envelope in the user's initial speech. After adversarial frequency domain perturbation, the voiceprint similarity can be reduced from 4.3 to 2.8. By changing the voiceprint features, the direct association between voiceprint and identity is severed, avoiding the leakage of user privacy caused by "identifying people by their voice."

[0119] The embedding unit 112 is used to embed a watermark into the first data based on a dynamic quantum watermark embedding mechanism to obtain the second data.

[0120] In this embodiment, the dynamic quantized watermark embedding mechanism is equivalent to a dynamic watermarking system based on a quantum neural network.

[0121] Specifically, the dynamic quantized watermark embedding mechanism includes a generator; the embedding unit 112 performs watermark embedding on the first data based on the dynamic quantized watermark embedding mechanism to obtain the second data, including:

[0122] In the generator, the user ID (Identity) of the user corresponding to the first data is obtained;

[0123] Perform a hash operation on the user ID to obtain the user ID hash value;

[0124] A quantum state watermark is generated based on the user ID hash value;

[0125] The quantum state watermark is encoded using Qubit entanglement coding to obtain the watermark code;

[0126] The watermark code is embedded into the zero-crossing rate feature of the speech Mel spectrogram of the first data to obtain the second data.

[0127] For example, quantum neural networks can be used to convert user ID hashes into 128-bit quantum watermarks, thus adding a "digital ID" to voice messages for subsequent tasks such as copyright tracking and privacy verification. At the same time, robustness against compression attacks can be improved by 3 times.

[0128] Specifically, the dynamic quantum watermark embedding mechanism further includes an extractor; after obtaining the second data, the extractor uses a quantum annealing algorithm to decode the second data to obtain the watermark to be verified.

[0129] When the watermark to be verified is the same as the quantum state watermark, the second data is determined to pass verification; or

[0130] When the watermark to be verified is different from the quantum state watermark, it is determined that the second data has failed verification.

[0131] Experiments show that the quantum annealing algorithm can achieve fast watermark decoding, thereby verifying the integrity of the embedding, with a latency of ≤20ms and a bit error rate of less than 0.1%.

[0132] In the above embodiments, if the extractor can still decode the watermark after the speech is compressed and edited, it proves that the second data is data obtained after processing the initial text and the corresponding user's initial speech.

[0133] The training unit 113 is used to train a sub-TTS (Text-to-Speech) model at each node using the second data of each node based on a federated learning-driven differential privacy protection mechanism, and to aggregate the sub-TTS models of each node into a target TTS model.

[0134] In this embodiment, the differential privacy protection mechanism driven by federated learning is equivalent to a hierarchical differential privacy federated training framework.

[0135] In this embodiment, the training unit 113, based on a federated learning-driven differential privacy protection mechanism, trains a sub-TTS model at each node using the second data from each node, and aggregates the sub-TTS models of each node into a target TTS model, including:

[0136] Using the second data from each node, train a sub-TTS model at each node and obtain the model parameters for each sub-model;

[0137] Laplacian noise is added to the model parameters of each sub-model using gradient perturbation;

[0138] The model parameters of each sub-model after adding noise are aggregated based on the homomorphic encryption algorithm to obtain the global model parameters;

[0139] The target TTS model is generated based on the global model parameters.

[0140] Federated learning can break down data silos and enable collaborative training using multi-source data.

[0141] Specifically, by adding Laplacian noise to the model parameters of each sub-model using gradient perturbation, and then aggregating the model parameters of each sub-model after adding noise using a homomorphic encryption algorithm, model inversion attacks can be prevented, making it impossible to infer user privacy from model parameters, thus reducing the cumulative error of privacy budget by 62%.

[0142] For example, companies A, B, and C each have local data. Through federated learning, they can jointly train a better TTS model without sharing the original data, and their data privacy is not compromised.

[0143] In the above embodiments, each node trains an acoustic model with desensitized data and adds Laplacian noise (ε = 0.5) to the gradient to hide individual data features; at the same time, the gradients of each node are aggregated through homomorphic encryption to eliminate the cumulative error of privacy budget.

[0144] The synthesis unit 114 is configured to respond to a speech synthesis instruction triggered by the target text and the target user's speech corresponding to the target text, input the target text and the target user's speech into the target TTS model for speech synthesis, and process the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine to obtain synthesized speech.

[0145] In this embodiment, the voice synthesis command can be triggered by relevant staff according to actual needs.

[0146] In this embodiment, the real-time streaming desensitization acceleration engine is a GPU (Graphics Processing Unit)-CPU (Central Processing Unit) heterogeneous computing architecture.

[0147] In this embodiment, the synthesis unit 114 processes the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine to obtain synthesized speech, including:

[0148] The synthesized speech is obtained by using a WebGL (Web Graphics Library) parallel computing mechanism and a dynamic desensitization rule engine to perform real-time regular expression matching on sensitive fields in the real-time speech data stream, and then desensitizing the matched fields.

[0149] The dynamic desensitization rule engine can match sensitive fields in real time (throughput of 100,000 records / second) and then trigger desensitization to meet the needs of low-latency scenarios (such as real-time call scenarios, voice navigation scenarios, etc.).

[0150] Among them, by using WebGL parallel computing, the frequency domain watermark embedding speed can be increased to 50ms / sentence, which can meet the real-time synthesis requirements of speech data.

[0151] The verification unit 115 is used to verify the synthesized speech based on a cross-modal privacy consistency verification mechanism.

[0152] In this embodiment, the cross-modal privacy consistency verification mechanism is equivalent to a voice-text-lip movement trimodal verification system.

[0153] Specifically, the verification unit 115 verifies the synthesized speech based on a cross-modal privacy consistency verification mechanism, including:

[0154] The semantic consistency between the synthesized speech and the target text is compared based on the Transformer multi-task network.

[0155] The system detects whether there are sensitive word change events in the synthesized speech. When a sensitive word change event exists, the system adjusts the opening degree parameter of the 3D lip shape model to synchronize the 3D lip shape with the changed speech content.

[0156] Among them, semantic consistency detection can intercept abnormal privacy leakage requests, such as voiceprints that do not match ID numbers.

[0157] Among them, 3D lip shape adjustment can ensure that the synchronization error between visual and auditory information is ≤5ms, thereby ensuring the naturalness of multimodal scenes. For example, when a virtual human speaks, lip movements and speech can be perfectly synchronized to improve realism.

[0158] For example, when a virtual anchor synthesizes the voice "I am in North China", if the text is mistakenly written as "I am in Beijing", the system detects the semantic-speech inconsistency, intercepts the output, corrects the text, and adjusts the lip movement animation.

[0159] The sending unit 116 is used to send the synthesized speech to the triggerer of the speech synthesis instruction when the synthesized speech passes the verification.

[0160] In the above embodiments, after the speech is quickly generated, the verified speech is sent to the triggerer of the speech synthesis instruction, thereby ensuring that the synthesized speech is natural and compliant.

[0161] This embodiment achieves a dual improvement in privacy protection and voice quality. Specifically, by using voiceprint obfuscation technology, the similarity of synthesized speech can be ≤2.8, while maintaining a MOS score ≥4.1 (an improvement of 0.9 points compared to traditional desensitization schemes).

[0162] This embodiment also provides resistance to attacks on the watermarking system. Specifically, the quantum watermark achieves an extraction success rate of ≥99.5% and a source tracing accuracy of 100% under attacks such as MP3 compression (64kbps) and resampling (8kHz→16kHz).

[0163] This embodiment also achieves real-time performance and resource optimization. Specifically, the GPU acceleration engine reduces the de-identification latency to 50ms, with a memory footprint of only 38MB, supporting embedded device deployment (CPU utilization ≤15%).

[0164] This embodiment also achieves a breakthrough in cross-scenario applicability. Specifically, the federated learning framework supports simultaneous training in 32 languages, and the adaptation cycle for privacy models in new languages ​​is shortened from 3 months to 2 weeks.

[0165] This embodiment also enables compliance and auditing capabilities. Specifically, the dynamic watermark log can be traced back to a single synthesis request, meeting the auditing requirements of regulations such as GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act), improving the efficiency of data breach liability determination by 90%.

[0166] This embodiment can be applied to data anonymization in TTS speech synthesis tasks in various scenarios such as finance and healthcare.

[0167] For example, in financial scenarios, intelligent customer service systems in banks and voice assistants in securities trading often need to read out information such as account balance and transaction password when responding to users. If the robot reads out this text directly, it will lead to privacy leaks. Therefore, it is necessary to complete the desensitization in the TTS synthesis stage to avoid the broadcast of sensitive information.

[0168] For example, in a healthcare setting, a hospital system's voice medical record query function needs to read out patient diagnosis records to doctors, but it needs to hide sensitive information such as names, ID numbers, and contact information, while ensuring accurate reading of medical terminology. This embodiment allows for data anonymization when the voice medical record query robot reads out patient diagnosis records, thereby avoiding the leakage of patient privacy.

[0169] As can be seen from the above technical solutions, this invention can desensitize the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain the first data, achieving desensitization processing for various modalities; it can also embed the first data with a watermark based on a dynamic quantum watermark embedding mechanism to obtain the second data, facilitating subsequent traceability and privacy verification; based on a federated learning-driven differential privacy protection mechanism, it can train a sub-TTS model at each node using the second data of each node, and aggregate the sub-TTS models of each node into a target TTS model, breaking down data silos and preventing model inversion attacks; it can process the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine, achieving real-time data desensitization and improving processing efficiency; and it can verify the synthesized speech based on a cross-modal privacy consistency verification mechanism, further ensuring the speech synthesis effect.

[0170] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a preferred embodiment of the data desensitization method based on TTS speech synthesis according to the present invention.

[0171] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a data desensitization program based on TTS speech synthesis.

[0172] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0173] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0174] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a data desensitization program based on TTS speech synthesis, but also to temporarily store data that has been output or will be output.

[0175] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a data desensitization program based on TTS speech synthesis) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0176] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various TTS-based data desensitization method embodiments described above, for example... Figure 1 The steps are shown.

[0177] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, a desensitization unit 111, an embedding unit 112, a training unit 113, a synthesis unit 114, a verification unit 115, and a sending unit 116.

[0178] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the data de-identification method based on TTS speech synthesis described in the various embodiments of this invention.

[0179] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0180] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0181] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0182] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0183] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0184] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0185] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0186] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0187] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0188] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0189] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a data desensitization method based on TTS speech synthesis, and the processor 13 can execute the multiple instructions to achieve the following:

[0190] For each node, obtain the initial text and the user's initial voice corresponding to each initial text;

[0191] The initial text and the user's initial speech are desensitized based on a multimodal voiceprint obfuscation and desensitization architecture to obtain the first data;

[0192] The first data is watermarked using a dynamic quantum watermarking embedding mechanism to obtain the second data.

[0193] Based on a federated learning-driven differential privacy protection mechanism, a sub-TTS model is trained at each node using the second data of each node, and the sub-TTS models of each node are aggregated into the target TTS model.

[0194] In response to a speech synthesis instruction triggered by a target text and a target user's speech corresponding to the target text, the target text and the target user's speech are input into the target TTS model for speech synthesis, and the real-time speech data stream output by the target TTS model is processed based on a real-time streaming desensitization acceleration engine to obtain synthesized speech.

[0195] The synthesized speech is verified based on a cross-modal privacy consistency verification mechanism;

[0196] When the synthesized speech passes the verification, the synthesized speech is sent to the triggerer of the speech synthesis instruction.

[0197] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0198] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0199] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0200] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0201] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0203] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0204] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0205] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data desensitization method based on TTS speech synthesis, characterized in that, The data desensitization method based on TTS speech synthesis includes: For each node, obtain the initial text and the user's initial voice corresponding to each initial text; The initial text and the user's initial speech are desensitized based on a multimodal voiceprint obfuscation desensitization architecture to obtain first data. This includes: using a bidirectional long short-term memory sensitive word detection network to identify privacy information in the initial text, and replacing the privacy information based on a local rewriting strategy to obtain replacement text; using a generative adversarial network to generate noise orthogonal to the corresponding user's fundamental frequency envelope in the user's initial speech to obtain noisy speech; and combining the replacement text and the noisy speech to obtain the first data. The process of embedding a watermark into the first data based on a dynamic quantum watermarking embedding mechanism to obtain the second data includes: obtaining the user ID of the user corresponding to the first data in the generator included in the dynamic quantum watermarking embedding mechanism; performing a hash operation on the user ID to obtain a user ID hash value; generating a quantum state watermark based on the user ID hash value; encoding the quantum state watermark according to Qubit entanglement coding to obtain a watermark code; and embedding the watermark code into the zero-crossing rate feature of the speech Mel spectrogram of the first data to obtain the second data. Based on a federated learning-driven differential privacy protection mechanism, a sub-TTS model is trained at each node using the second data of each node, and the sub-TTS models of each node are aggregated into the target TTS model. In response to a speech synthesis instruction triggered by a target text and a target user's speech corresponding to the target text, the target text and the target user's speech are input into the target TTS model for speech synthesis, and the real-time speech data stream output by the target TTS model is processed based on a real-time streaming desensitization acceleration engine to obtain synthesized speech. The synthesized speech is verified based on a cross-modal privacy consistency verification mechanism; When the synthesized speech passes the verification, the synthesized speech is sent to the triggerer of the speech synthesis instruction.

2. The data desensitization method based on TTS speech synthesis as described in claim 1, characterized in that, The dynamic quantum watermark embedding mechanism further includes an extractor; after obtaining the second data, the method further includes: In the extractor, the quantum annealing algorithm is used to decode the second data to obtain the watermark to be verified; When the watermark to be verified is the same as the quantum state watermark, the second data is determined to pass verification; or When the watermark to be verified is different from the quantum state watermark, it is determined that the second data has failed verification.

3. The data desensitization method based on TTS speech synthesis as described in claim 1, characterized in that, The federated learning-driven differential privacy protection mechanism utilizes the second data from each node to train a sub-TTS model at each node, and aggregates the sub-TTS models from each node into a target TTS model, including: Using the second data from each node, train a sub-TTS model at each node and obtain the model parameters for each sub-model; Laplacian noise is added to the model parameters of each sub-model using gradient perturbation; The model parameters of each sub-model after adding noise are aggregated based on the homomorphic encryption algorithm to obtain the global model parameters; The target TTS model is generated based on the global model parameters.

4. The data desensitization method based on TTS speech synthesis as described in claim 1, characterized in that, The process of processing the real-time speech data stream output by the target TTS model using the real-time streaming desensitization acceleration engine to obtain synthesized speech includes: Using a WebGL parallel computing mechanism, a dynamic desensitization rule engine is used to perform real-time regular expression matching on sensitive fields in the real-time speech data stream, and the matched fields are desensitized to obtain the synthesized speech.

5. The data desensitization method based on TTS speech synthesis as described in claim 1, characterized in that, The verification of the synthesized speech based on the cross-modal privacy consistency verification mechanism includes: The semantic consistency between the synthesized speech and the target text is compared based on the Transformer multi-task network. The system detects whether there are sensitive word change events in the synthesized speech. When a sensitive word change event exists, the system adjusts the opening degree parameter of the 3D lip shape model to synchronize the 3D lip shape with the changed speech content.

6. A data desensitization device based on TTS speech synthesis, characterized in that, The data desensitization device based on TTS speech synthesis includes: The acquisition unit is used to acquire the initial text and the user's initial speech corresponding to each initial text for each node; The desensitization unit is used to desensitize the initial text and the user's initial speech based on a multimodal voiceprint obfuscation desensitization architecture to obtain first data. The process includes: using a bidirectional long short-term memory sensitive word detection network to identify privacy information in the initial text, and replacing the privacy information based on a local rewriting strategy to obtain replacement text; using a generative adversarial network to generate noise orthogonal to the corresponding user's fundamental frequency envelope in the user's initial speech to obtain noisy speech; and combining the replacement text and the noisy speech to obtain the first data. An embedding unit is used to embed a watermark into the first data based on a dynamic quantum watermark embedding mechanism to obtain second data. The embedding unit includes: obtaining the user ID of the user corresponding to the first data from the generator included in the dynamic quantum watermark embedding mechanism; performing a hash operation on the user ID to obtain a user ID hash value; generating a quantum state watermark based on the user ID hash value; encoding the quantum state watermark according to Qubit entanglement coding to obtain a watermark code; and embedding the watermark code into the zero-crossing rate feature of the speech Mel spectrogram of the first data to obtain the second data. The training unit is used for a differential privacy protection mechanism driven by federated learning. It uses the second data of each node to train a sub-TTS model at each node, and aggregates the sub-TTS models of each node into the target TTS model. The synthesis unit is configured to respond to a speech synthesis instruction triggered by a target text and a target user's speech corresponding to the target text, input the target text and the target user's speech into the target TTS model for speech synthesis, and process the real-time speech data stream output by the target TTS model based on a real-time streaming desensitization acceleration engine to obtain synthesized speech. The verification unit is used to verify the synthesized speech based on a cross-modal privacy consistency verification mechanism; The sending unit is used to send the synthesized speech to the triggerer of the speech synthesis instruction when the synthesized speech passes the verification.

7. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the data desensitization method based on TTS speech synthesis as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the data desensitization method based on TTS speech synthesis as described in any one of claims 1 to 5.

Images