Traceable speech synthesis method

By collecting sound samples with authorized owners and building sound models, third-party supervision unified encryption technology to add watermarks, solving the problems of inability to trace and easy to crack in the existing technology, realizing traceability of arbitrary voices and high confidentiality watermark detection.

CN119068887APending Publication Date: 2024-12-03WUHU XIAOLESE INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411114547.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, manufacturers can only detect the audio watermarks they produce and cannot trace the audio of unknown manufacturers. The confidentiality and robustness of existing audio watermarks are easy to be cracked, especially when the audio is edited. It is difficult to detect the original watermark later.

Method used

By reaching an agreement with the authorized person, collecting his voice samples and building a sound model, using AI voice manufacturers to generate voices with the authorized person's tone, third-party supervision unified encryption technology adds synthetic human identity information as a watermark to synthesize voice, forming a possible The traceable voice watermark ensures that the watermark can still be detected after the voice has been edited.

Benefits of technology

It realizes traceability of any voice and finds users who synthesize the voice. The watermark framework network has strong confidentiality and robustness. It can still detect the original watermark after various audio clips, resampling and adding background music, avoiding The dilemma of different manufacturers can only detect the audio watermarks they produce.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068887A_ABST
    Figure CN119068887A_ABST
Patent Text Reader

Abstract

The invention is suitable for the technical field of speech synthesis traceability, and provides a traceable speech synthesis method, which comprises the following steps that: all manufacturers provide synthesized speech and synthesized person identity information to a third party for supervision; and then, a third party supervises and adds the identity information of the synthesis person as a watermark to the synthesis voice by using a unified encryption technology to form the synthesis voice (watermark version), the synthesis voice (watermark version) is provided for the synthesis person, and an anti-fraud department can use a decryption mode corresponding to an original encryption mode to detect the identity watermark and check the identity information of the synthesis person. According to the traceable speech synthesis method, the difficulty that different manufacturers can only detect the audio watermarks produced by themselves is avoided by supervising the use of a unified encryption mode to add the watermarks, and a third party supervises that all manufacturers add the watermarks into the speech in a unified manner, so that a fraud person does not need to confirm which manufacturer the speech comes from during traceability; and user real-name identity information of a synthesizer is added during speech synthesis, so that speech traceability is facilitated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of speech synthesis tracing, and in particular relates to a traceable speech synthesis method. Background Art

[0002] Voice cloning and speech synthesis technology are developing day by day, and it only takes a few seconds of recording to imitate the voice of a speaker while maintaining a high level of realism. In addition to the potential benefits, this powerful technology also brings significant risks, including voice fraud and speaker impersonation. Unlike traditional methods that rely solely on passive methods to detect synthetic data, watermarking provides an active and powerful defense mechanism to address these imminent risks.

[0003] In the existing technology, manufacturers can only detect the watermark of audio they produce, so it is impossible to trace the audio of an unknown manufacturer. In addition, the existing audio watermarking method lacks confidentiality and robustness and is easily cracked. In addition, it is difficult to detect the original watermark of the audio after the audio is edited. Summary of the invention

[0004] The present invention provides a traceable speech synthesis method, aiming to solve the problem in the prior art that a manufacturer can only detect the watermark of the audio produced by itself, and thus cannot trace the audio of an unknown manufacturer.

[0005] The present invention is implemented in this way: a traceable speech synthesis method comprises the following steps:

[0006] S1. Voice cloning: The manufacturer first reaches an agreement with the licensor to obtain the right to collect and use its voice samples. The manufacturer clones the licensor's voice timbre. In the recording environment, the licensor reads a series of texts or has a free conversation to ensure that diverse and comprehensive voice data is collected. The AI ​​voice manufacturer uses audio analysis technology to conduct detailed acoustic analysis of the collected voice samples. The manufacturer will build a voice model that can accurately simulate and replicate the licensor's voice. After the voice model is established, the AI ​​voice manufacturer uses synthesis technology to generate voices with similar timbre to the licensor;

[0007] S2. Speech synthesis: The synthesizer provides the audio copy and purchases the sound cloning technology from the manufacturer to generate the voice of the licensee, and the dubbing corresponding to the audio copy forms the synthesized speech;

[0008] S3. Supervisory encryption: All manufacturers will provide the synthesized voice and the identity information of the synthesizer to third-party supervisors. The third-party supervisor will then use unified encryption technology to add the identity information of the synthesizer to the synthesized voice as a watermark to form a synthesized voice (watermark version), which will then be provided to the synthesizer. When fraudulent voice audio appears, the anti-fraud department can use the decryption method corresponding to the original encryption method to detect the identity watermark and check the identity information of the synthesizer.

[0009] Preferably, in S1, acoustic features of the voice are extracted, such as pitch, timbre, volume, speaking speed, etc. These features together constitute the unique voice of the authorized person.

[0010] Preferably, in S1, using TTS technology, the synthesis engine will use the voice model of the authorized person to generate speech instead of using a general voice library.

[0011] Preferably, in S1, the manufacturer needs to ensure that the manufacturer obtains explicit authorization from the licensor to use the generated synthetic speech in various application scenarios.

[0012] Preferably, in S2, the synthesizer reaches an agreement with the selected voice cloning technology provider and pays the corresponding fees and identity information to purchase the dubbing service generated by the voice cloning technology.

[0013] Preferably, in S2, when using the synthesized dubbing, the synthesizer needs to ensure compliance with relevant copyright and authorization regulations, including communicating and negotiating with the voice cloning technology provider and the licensor to ensure the right to legally use the generated synthesized dubbing.

[0014] Preferably, in S3, after completing the production of the synthesized speech, the manufacturer collects the corresponding synthesizer identity information, and the identity information may include the synthesizer's name and authentication information of the unique identifier.

[0015] Preferably, in S3, the synthesized speech is submitted to a third-party supervision process, including online uploading and physical transmission, and ensuring that the data is not tampered with or leaked during the transmission process.

[0016] Preferably, in S3, the encryption process performs encoding operations on the identity information to ensure that only a third-party supervisor can decode and verify the watermark information.

[0017] Preferably, in S3, the victim reports the case to the anti-fraud department and provides the fraudulent voice audio as evidence. After receiving the evidence, the anti-fraud department contacts the third-party supervisor, and the third-party supervisor uses the decryption method corresponding to its own encryption method to attempt to decrypt the fraudulent voice audio. Once the fraudulent voice audio comes from a regulated manufacturer, it can be decrypted normally and the watermark information carried in the voice audio can be extracted. The watermark information will reveal the identity information of the synthesizer and complete the identity tracing of the synthesizer involved in the case.

[0018] Beneficial Effects

[0019] Compared with the prior art, the beneficial effects of the present invention are: a traceable speech synthesis method of the present invention uses a unified encryption method to add watermarks through supervision, avoiding the dilemma that different manufacturers can only detect the audio watermarks they produce. The third-party supervision adds watermarks to the speech for all manufacturers, so the defrauded person does not need to confirm which manufacturer the speech comes from when tracing the source. The real-name identity information of the synthesizer is added during speech synthesis to facilitate speech tracing. The watermark framework network uses an audio editing simulator to simulate various audio editing operation interferences in reality, thereby training a neural network that can still detect watermarks after various audio editing operations. The tracing mechanism can trace any speech and find the user who synthesized the speech. While having extremely strong confidentiality, the speech watermark framework can still detect the original watermark after the speech has undergone various audio editing, resampling, adding background music and other operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the speech synthesis process of the present invention;

[0021] Figure 2 It is a schematic diagram of the speech tracing process of the present invention;

[0022] Figure 3 Schematic diagram of the watermark framework network and its parameter learning method of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] See also Figure 1 and Figure 2 The present invention provides a technical solution: a traceable speech synthesis method, comprising the following steps:

[0025] S1. Voice cloning: The manufacturer first reaches an agreement with the licensor to obtain the right to collect and use its voice samples. The manufacturer clones the licensor's voice timbre. In the recording environment, the licensor reads a series of texts or has a free conversation to ensure that diverse and comprehensive voice data is collected. The AI ​​voice manufacturer uses audio analysis technology to conduct detailed acoustic analysis of the collected voice samples. The manufacturer will build a voice model that can accurately simulate and replicate the licensor's voice. After the voice model is established, the AI ​​voice manufacturer uses synthesis technology to generate voices with similar timbre to the licensor;

[0026] S2. Speech synthesis: The synthesizer provides the audio copy and purchases the sound cloning technology from the manufacturer to generate the voice of the licensee, and the dubbing corresponding to the audio copy forms the synthesized speech;

[0027] S3. Supervisory encryption: All manufacturers will provide the synthesized voice and the identity information of the synthesizer to third-party supervisors. The third-party supervisor will then use unified encryption technology to add the identity information of the synthesizer to the synthesized voice as a watermark to form a synthesized voice (watermark version), which will then be provided to the synthesizer. When fraudulent voice audio appears, the anti-fraud department can use the decryption method corresponding to the original encryption method to detect the identity watermark and check the identity information of the synthesizer.

[0028] Furthermore, in S1, acoustic features of the voice are extracted, such as pitch, timbre, volume, speaking speed, etc. These features together constitute the unique voice of the authorized person.

[0029] Furthermore, in S1, using the TTS technology, the synthesis engine will use the voice model of the authorized person to generate speech instead of using a general voice library.

[0030] Furthermore, in S1, the manufacturer needs to ensure that it obtains explicit authorization from the licensor to use the generated synthetic speech in various application scenarios.

[0031] Furthermore, in S2, the synthesizer reaches an agreement with the selected voice cloning technology provider and pays the corresponding fees and identity information to purchase the dubbing service generated by the voice cloning technology.

[0032] Furthermore, in S2, when using the synthesized dubbing, the synthesizer needs to ensure compliance with relevant copyright and licensing regulations, including communicating and negotiating with the voice cloning technology provider and the licensor to ensure that he has the right to legally use the generated synthesized dubbing.

[0033] Furthermore, in S3, after completing the production of the synthesized speech, the manufacturer collects the corresponding synthesizer's identity information, and the identity information may include the synthesizer's name and authentication information of the unique identifier.

[0034] Furthermore, in S3, the synthesized speech is submitted to a third-party supervision process, including online uploading and physical transmission, and ensuring that the data is not tampered with or leaked during the transmission process.

[0035] Furthermore, in S3, the encryption process performs encoding operations on the identity information to ensure that only a third-party supervisor can decode and verify the watermark information.

[0036] Furthermore, in S3, the victim reports the case to the anti-fraud department and provides fraudulent voice audio as evidence. After receiving the evidence, the anti-fraud department contacts the third-party supervisor. The third-party supervisor uses the decryption method corresponding to its own encryption method to attempt to decrypt the fraudulent voice audio. Once the fraudulent voice audio comes from a regulated manufacturer, it can be decrypted normally and the watermark information carried in the voice audio can be extracted. The watermark information will reveal the identity information of the synthesizer and complete the identity tracing of the synthesizer involved in the case.

[0037] The manufacturer must provide the real-name identity information of the synthesizer. When obtaining fraudulent audio normally, it is not known which manufacturer the audio was synthesized by. This method can directly avoid the "manufacturer confirmation" link.

[0038] Supervision uses a unified encryption method to add watermarks to avoid the dilemma that different manufacturers can only detect the audio watermarks they produce. Third-party supervision adds watermarks to the voice for all manufacturers. Therefore, the victims of deception do not need to confirm which manufacturer the voice comes from when tracing the source. The real-name identity information of the synthesizer is added during voice synthesis to facilitate voice tracing and directly lock down criminals.

[0039] The watermark framework network uses an audio editing simulator to simulate various audio editing operation interferences in reality, thereby training a neural network that can still detect watermarks after various audio editing operations. The tracing mechanism can trace the source of any voice and find the user who synthesized the voice. While the voice watermark framework has extremely strong confidentiality, it can still detect the original watermark after the voice has undergone various audio editing, resampling, adding background music and other operations.

[0040] See also Figure 3 ,Watermark framework network and its parameter learning method,W t represents the billion-level network parameter value of the t-th cycle in learning, γ is the learning rate, usually in the range of 1e-6 to 1e-4, at time 0, initially, W0 is generally randomly generated by normal distribution, Loss is the loss function, usually the difference between the target value and the actual value, is the gradient of the loss function, that is, the partial derivative of the loss function with respect to the billion-level network parameters. t By continuously updating, the loss can be reduced. In the present invention, if the output decrypted synthetic human identity is taken as the actual value and the input synthetic human identity is taken as the target value, through deep learning, the gap between the two can be gradually converged as the training steps increase.

[0041]

[0042] W t0 =W0

[0043] The audio editing simulator achieves extremely strong robustness. Through deep learning, the billion-level network parameters will converge to a parameter combination that makes the decrypted synthesized person identity consistent with the input synthesized person identity. However, in actual use, the watermark can only be correctly extracted if the synthesized speech input in the watermarking step is consistent with the audio input for watermark extraction in the detection and watermark extraction step. However, under normal circumstances, the speech synthesizer will use the synthesized speech for short video production. The edited speech is very different from the original synthesized speech, making it difficult to extract the watermark information.

[0044] In order to achieve the robustness of the watermark framework against various audio clips, the audio clip simulator is used to simulate the changes to the audio caused by performing various random audio clip operations. Through deep learning training, the billion-level network parameters will converge to be able to resist the various audio clip interferences of the audio clip simulator. When there are enough audio data for deep learning training and the audio clip combinations of the audio clip simulator are also sufficient, the billion-level network parameters will be generalized. On the one hand, the audio clip simulator can basically cover the audio clip operation combinations actually used; on the other hand, even if the audio clip operation combination actually used is a combination that has not been generated by the audio clip simulator, due to the generalization of deep learning itself, the billion-level network parameters can also resist the interference of the actual audio clip operation on the audio, thereby extracting the real synthetic human identity watermark information from the speech.

[0045] The use of billions of neural network parameters has great randomness. As long as the neural network model is not leaked, the neural network parameters cannot be reproduced. The speech watermarking framework uses a billion-level parameter network. Each different parameter combination is a password, which has extremely strong confidentiality.

[0046] The parameters of the decoder and the encoder are the same, which is equivalent to the same password for decryption and encryption.

[0047] Using a billion-level parameter network, each different parameter combination is a password, ensuring sufficient randomness and preventing it from being cracked.

[0048] The audio editing simulator is used to simulate various audio editing operations in reality, so as to train network parameters that are sufficiently resistant to various audio editing interferences during watermark framework network training.

[0049] The working principle and use process of the present invention: After adding third-party supervision, all manufacturers need to provide the synthesized voice and synthesizer identity information to the third-party supervision, and then the third-party supervision will uniformly add the synthesizer identity information as a watermark to the synthesized voice to form a synthesized voice (watermark version), and then provide it to the synthesizer. Since there is only one executor, the third-party supervision, who adds the encrypted watermark at this time, rather than multiple manufacturers, there is only one set of encryption methods. When tracing the voice source, the victim contacts the anti-fraud department and provides the fraudulent voice audio to the third-party supervision. Since all synthesized voices (watermark versions) are watermarked through third-party supervision, the decryption method corresponding to the encryption method of the third-party supervision can be used to detect the synthesizer identity information carried in the watermark for the fraudulent voice audio, and the criminals can be directly locked without the need to go through the manufacturer's link.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A traceable speech synthesis method, characterized in that: The following steps are involved: S1. Voice cloning: The manufacturer first reaches an agreement with the licensor to obtain the right to collect and use its voice samples. The manufacturer clones the licensor's voice timbre. In the recording environment, the licensor will read a series of texts or have a free conversation to ensure that diverse and comprehensive voice data is collected. The AI ​​voice manufacturer uses audio analysis technology to conduct detailed acoustic analysis of the collected voice samples. The manufacturer will build a voice model that can accurately simulate and replicate the licensor's voice. After the voice model is established, the AI ​​voice manufacturer uses synthesis technology to generate voices with similar timbre to the licensor; S2. Speech synthesis: The synthesizer provides the audio copy and purchases the sound cloning technology from the manufacturer to generate the voice of the licensee, and the dubbing corresponding to the audio copy forms the synthesized speech; S3. Supervisory encryption: All manufacturers will provide the synthesized voice and the identity information of the synthesizer to the third-party supervisor. The third-party supervisor will then use unified encryption technology to add the identity information of the synthesizer to the synthesized voice as a watermark to form a synthesized voice (watermark version), which will then be provided to the synthesizer. Once the synthesized voice (watermark version) is used by the synthesizer for fraud, the third-party supervisor can use the decryption method corresponding to the original encryption method to detect the identity watermark and view the identity information of the synthesizer.

2. A traceable speech synthesis method as claimed in claim 1, characterized in that: In S1, acoustic features of the voice are extracted, such as pitch, timbre, volume, speaking speed, etc. These features together constitute the unique voice of the authorized person.

3. The traceable speech synthesis method according to claim 1, characterized in that: In S1, using TTS technology, the synthesis engine will use the voice model of the authorized person to generate speech instead of using a general voice library.

4. A traceable speech synthesis method as claimed in claim 1, characterized in that: In S1, the manufacturer needs to ensure that it obtains explicit authorization from the licensor to use the generated synthetic speech in various application scenarios.

5. The traceable speech synthesis method according to claim 1, characterized in that: In S2, the synthesizer reaches an agreement with the selected voice cloning technology provider and pays the corresponding fees and identity information to purchase the dubbing service generated by the voice cloning technology.

6. A traceable speech synthesis method as claimed in claim 1, characterized in that: In S2, when using synthetic dubbing, the synthesizer needs to ensure compliance with relevant copyright and licensing regulations, including communicating and negotiating with the voice cloning technology provider and licensor to ensure that he has the right to legally use the generated synthetic dubbing.

7. The traceable speech synthesis method according to claim 1, characterized in that: In S3, after completing the production of the synthesized speech, the manufacturer collects the corresponding synthesizer's identity information, and the identity information may include the synthesizer's name and authentication information of the unique identifier.

8. The traceable speech synthesis method according to claim 1, characterized in that: In S3, the synthesized speech is submitted to a third-party supervision process, including online uploading and physical transmission, and ensuring that the data is not tampered with or leaked during the transmission process.

9. The traceable speech synthesis method according to claim 1, characterized in that: In S3, the encryption process encodes the identity information to ensure that only a third-party supervisor can decode and verify the watermark information.

10. The traceable speech synthesis method according to claim 1, characterized in that: In S3, the victim reports the case to the anti-fraud department and provides the fraudulent voice audio as evidence. After receiving the evidence, the anti-fraud department contacts the third-party regulator. The third-party regulator uses the decryption method corresponding to its own encryption method to attempt to decrypt the fraudulent voice audio. Once the fraudulent voice audio comes from the regulated manufacturer, it can be decrypted normally and the watermark information carried in the voice audio can be extracted. The watermark information will reveal the identity information of the synthesizer and complete the identity tracing of the synthesizer involved in the case.