A method for adding source speaker watermark in converted speech

Abstract

The present invention discloses a method for adding a source speaker watermark to converted speech in the field of speech signal processing technology. The method aims to address the technical issues of low efficiency, poor quality of reconstructed speech waveforms, and poor adaptability to mainstream end-to-end speech conversion methods. The method includes retaining a latent representation of the converted speech and adding a latent representation of the source speaker's features to the latent representation of the converted speech. This method not only reduces computational complexity and scale, but also does not damage the original converted speech. As a result, the converted speech with the source speaker watermark added, obtained after decoding the encrypted latent representation of the converted speech, is similar to the original converted speech. The watermarked converted speech has high quality and is interpretable. Therefore, the source speaker watermark can be directly embedded in the converted speech generated by mainstream end-to-end speech conversion methods. The method can be directly used as a supplementary component of a speech conversion model, thus possessing wide applicability.

Description

Technical Field

[0001] The invention relates to a method for adding a source speaker watermark in converted speech, belonging to the technical field of speech signal processing. Background Art

[0002] Generative AI technology enables the creation of realistic voice content through voice conversion. Voice conversion involves converting a source voice into a target voice, maintaining the original content while adapting the source speaker's style to that of the target. Currently, various voice conversion methods have been proposed for applications in personalized speech synthesis, singing voice conversion, and voice impersonation. However, the resulting voice is ultimately a fabricated voice and could be used for harmful activities such as online pranks, malicious editing, and fraud.

[0003] Therefore, voice conversion providers should add identification information to the converted voice, that is, implicit watermark identification that cannot be directly perceived by humans but can be extracted from the content through technical means. Actively adding traceable identification information is more convenient and easier than directly performing authentication and tracing methods, and is more in line with the requirements of generated voice.

[0004] The DPAS speech steganography method commonly used in the industry (Reference: Chen K, Zhou H, Zhao H, et al. Distribution-Preserving Steganography Based on Text-to-Speech Generative Models[J]. IEEE Transactions on Dependable and Secure Computing, 2022, 19(5):3343-3356.) can map the source speaker identity into a binary embedded speech method, which can well embed the source speaker identity into the speech, but it will damage the quality of the generated speech, and the identity watermark is not highly recognizable. The industry will also adopt VoxTracer's traceable VC framework (Reference: Ren Y, Zhu H, Zhai L, et al. Who is SpeakingActually? Robust and Versatile Speaker Traceability for Voice Conversion[C] / / Proceedings of the 31st ACM International Conference on Multimedia. New York,NY, USA: Association for Computing Machinery, 2023: 8674-8685.) to embed the source speaker identity into the converted speech. However, the watermarking method currently used in the industry is to embed the source speaker's identity in the speech generation process. However, the mainstream end-to-end speech conversion method directly generates the converted speech. The watermarking method currently used in the industry cannot be directly embedded into the trained end-to-end mainstream speech conversion method. Watermarking can only be achieved by retraining the vocoder and embedding the watermarking method currently used in the industry. Obviously, retraining the vocoder is cumbersome and time-consuming, and the vocoder embedded with the source speaker's identity is different from the vocoder used for speech conversion before adding the watermark. This vocoder mismatch will also reduce the quality of the speech waveform reconstructed by speech conversion.

[0005] Therefore, the existing method of adding watermarks to the reconverted speech has low efficiency and poor quality of the reconstructed speech waveform, and has poor adaptability to the mainstream end-to-end speech conversion method. Summary of the Invention

[0006] The purpose of this application is to overcome the deficiencies in the prior art and to provide a method for adding a source speaker watermark to the converted speech, which is efficient, has high-quality reconstructed speech waveforms, and is highly adaptable to mainstream end-to-end speech conversion methods.

[0007] To achieve the above objectives, this application is implemented using the following technical solutions:

[0008] In a first aspect, the present application provides a method for adding a source speaker watermark to a converted speech, comprising:

[0009] Acquiring a spectrum of the converted speech, and extracting a potential representation of the converted speech from the spectrum of the converted speech;

[0010] Obtaining identity features of a source speaker, and extracting a latent representation of source speaker features from the identity features of the source speaker;

[0011] Using the latent representation of the source speaker's characteristics as a source speaker watermark, and adding it to the latent representation of the converted speech, to obtain a encrypted latent representation of the converted speech;

[0012] The encrypted converted speech latent representation is decoded to obtain the converted speech with the source speaker watermark added.

[0013] In some embodiments of the first aspect, obtaining a spectrum of the converted speech and extracting a potential representation of the converted speech from the spectrum of the converted speech includes:

[0014] Get the converted voice;

[0015] Performing a short-time Fourier transform on the converted speech to obtain a spectrum of the converted speech;

[0016] A carrier coding network module is used to encode the spectrum of the converted speech to obtain a potential representation of the converted speech; wherein the carrier coding network module is mainly composed of a three-layer gated convolutional neural network.

[0017] In some embodiments of the first aspect, obtaining the identity feature of the source speaker and extracting the latent representation of the source speaker feature from the identity feature of the source speaker includes:

[0018] Get the source voice;

[0019] Encoding the source speech using a pre-trained speaker identity encoder to obtain identity features of the source speaker;

[0020] Performing dimension expansion processing on the identity feature of the source speaker to obtain a dimensionally expanded feature vector;

[0021] A source speaker identity encoding network module is used to encode the dimensionally expanded feature vector to obtain a potential representation of the source speaker feature; wherein the source speaker identity encoding network module is mainly composed of a three-layer gated convolutional neural network.

[0022] In some embodiments of the first aspect, the step of adding the latent representation of the source speaker's features as a source speaker watermark to the latent representation of the converted speech to obtain the encrypted latent representation of the converted speech includes:

[0023] The latent representation of the source speaker features is added to the latent representation of the converted speech in the form of a residual to obtain the dense latent representation of the converted speech;

[0024] The decoding of the encrypted converted speech potential representation to obtain the converted speech with the source speaker watermark added includes:

[0025] Decoding the latent representation of the encrypted converted speech through a encrypted converted speech decoding network module to obtain a spectrum of the encrypted converted speech; wherein the encrypted converted speech decoding network module is mainly composed of a four-layer gated convolutional neural network;

[0026] The spectrum of the encrypted converted speech is subjected to an inverse short-time Fourier transform to obtain a waveform of the encrypted converted speech, and the waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added, thereby adding the source speaker watermark to the converted speech.

[0027] In some embodiments of the first aspect, the waveform of the encrypted converted speech is obtained, wherein the waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added thereto, including:

[0028] Performing a short-time Fourier transform on the waveform of the encrypted converted speech to obtain a spectrum of the encrypted converted speech;

[0029] Inputting the encrypted converted speech spectrum into a source speaker identity decoding module for decoding to obtain reconstructed source speaker identity features; wherein the source speaker identity decoding module is mainly composed of a four-layer gated convolutional neural network and a layer of adaptive average pooling;

[0030] Verify and compare the cosine similarity value between the reconstructed source speaker identity feature and the source speaker identity feature, and in response to the cosine similarity value being greater than a first threshold, output the waveform of the encrypted converted speech as the waveform of the converted speech with the source speaker watermark added.

[0031] In some embodiments of the first aspect, the waveform of the encrypted converted speech is obtained, wherein the waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added thereto, including:

[0032] Performing a short-time Fourier transform on the waveform of the encrypted converted speech to obtain a spectrum of the encrypted converted speech;

[0033] Inputting the encrypted converted speech spectrum into a source speaker identity decoding module for decoding to obtain reconstructed source speaker identity features; wherein the source speaker identity decoding module is mainly composed of a four-layer gated convolutional neural network and a layer of adaptive average pooling;

[0034] Verifying and comparing a cosine similarity value between the reconstructed source speaker identity feature and the source speaker identity feature, and in response to the cosine similarity value being greater than a first threshold, inputting the reconstructed source speaker identity feature and the waveform of the encrypted converted speech into a speech conversion module to obtain a reconstructed source speech;

[0035] The source speech is obtained, and a speech similarity value between the reconstructed source speech and the source speech is verified and compared. In response to the speech similarity value being greater than a second threshold, the waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added.

[0036] In some embodiments of the first aspect, after decoding the encrypted converted speech latent representation to obtain the converted speech with the source speaker watermark added,

[0037] Get the converted voice;

[0038] A loss function is constructed by using the spectrum difference between the converted speech and the converted speech with the source speaker watermark added thereto. The loss function is used to optimize the training vector encoding network module, the source speaker identity encoding network module and / or the encrypted converted speech decoding network module.

[0039] In a second aspect, the present application also provides a computer device comprising a processor and a memory connected to the processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the steps of the method for adding a source speaker watermark to the converted speech as described in any embodiment of the first aspect are performed.

[0040] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for adding a source speaker watermark to the converted speech as described in any embodiment of the first aspect.

[0041] In a fourth aspect, the present application further provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method for adding a source speaker watermark to the converted speech as described in any embodiment of the first aspect are implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The method for adding a source speaker watermark to the converted speech provided by the present application retains the latent representation of the converted speech and adds the latent representation of the source speaker features on the basis of the latent representation of the converted speech. This watermark adding method is not only computationally small and lightweight but also does not damage the original converted speech. Therefore, the converted speech with the source speaker watermark added obtained after the latent representation of the encrypted converted speech is decoded is similar to the original converted speech. The converted speech after adding the watermark has high quality and is interpretable. Therefore, the source speaker watermark can be directly embedded in the converted speech generated by the end-to-end mainstream speech conversion method. In addition, the converted speech after adding the watermark has strong robustness against common noise attacks. The method for adding a source speaker watermark to the converted speech can be directly used as a supplementary part of the speech conversion model without the need to retrain an additional speech conversion model, and has wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 is a flowchart of the steps of the method for adding a source speaker watermark to the converted speech provided by this embodiment;

[0046] Figure 2 is a data flow chart of the method for adding a source speaker watermark to converted speech provided by this embodiment;

[0047] Figure 3 yes Figure 2 Schematic diagram of the process of embedding the source speaker identity and reconstructing the source speaker identity;

[0048] Figure 4 The comparative experiment provided in this embodiment is a graph showing the speech quality comparison results of the converted speech with the source speaker watermark added by the DPAS method and the converted speech without adding the watermark;

[0049] Figure 5 This is a graph showing the voice quality comparison results of the converted speech with the source speaker watermark added by VoxTracer and the converted speech without the added watermark in the comparative experiment provided in this embodiment;

[0050] Figure 6 The comparative experiment provided in this embodiment is provided by the method of adding the source speaker watermark in the converted speech provided in this embodiment, and the voice quality comparison result of the converted speech with the source speaker watermark added and the converted speech without adding the source speaker watermark is shown;

[0051] Figure 7 Cross-compatibility score chart based on SNR, PESQ, and MCS tests in the comparative experiment provided in this embodiment;

[0052] Figure 8 In the comparative experiment provided in this embodiment, the regenerated source speech is compared with the source speech and the score graph of the reconstructed source speech;

[0053] Figure 9 This is a schematic block diagram of the principles of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0054] It should be noted that: the source speaker is the speaker of the source speech, the target speaker is the speaker that is to be simulated after speech conversion, and the speech conversion model converts the source speech into a converted speech with the voice characteristics of the target speaker, making the converted speech sound like the target speaker's voice while keeping the speech content information unchanged compared to the source speech; the voice characteristics of the target speaker also include spectral distribution, pitch, timbre, speaking speed, etc., but they are different from those of the source speaker. For "source speaker", "target speaker" and "source speech", there are words with the same meaning but different expressions recorded in other technical documents.

[0055] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. Unless there is a conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0056] The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0057] For FreeVC (Reference: Li J, Tu W, Xiao L. Freevc: Towards High-Quality Text-Free One-Shot Voice Conversion[C] / / ICASSP 2023 - 2023 IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP).2023: 1-5.) and ConsistencyVC (Reference: Guo H, Liu C, Ishi CT, et al. Usingjoint training speaker encoder with consistency loss to achieve cross-lingualvoice conversion and expressive voice conversion[M]. arXiv, 2023.) as the representative of the mainstream end-to-end speech conversion method. The existing mainstream speech watermarking method represented by the DPAS speech steganography method and the VoxTracer traceable VC framework cannot directly add the source speaker's watermark to the converted speech output by the end-to-end speech conversion module. The existing solution is to rebuild a speech conversion module, embed the speech watermarking method module in the new speech conversion module, and then input the converted speech to reconstruct the new converted speech as the converted speech with the added watermark.

[0058] It is not difficult for those skilled in the art to see that, firstly, rebuilding the voice conversion module is time-consuming and labor-intensive, and wastes computing resources. Secondly, the rebuilt voice conversion module is different from the original voice conversion module, resulting in a large gap between the newly converted voice and the original converted voice, and the quality of the newly converted voice is poor.

[0059] Example 1:

[0060] Figure 1 This is a flow chart of a method for adding a source speaker watermark to a converted speech in the first embodiment of the present invention. This flow chart only shows the logical sequence of the method described in this embodiment. In other possible embodiments of the present invention, different methods may be used without conflict. Figure 1 The steps shown or described are accomplished in the order shown.

[0061] The method for adding a source speaker watermark to the converted speech provided in this embodiment can be applied to a terminal and can be executed by a speech conversion device. The device can be implemented by software and / or hardware and can be integrated into a terminal, such as any smart phone, tablet computer or computer device with communication functions. Figure 1 , the method of this implementation specifically includes the following steps:

[0062] Acquire a spectrum of the converted speech, and extract a potential representation of the converted speech from the spectrum of the converted speech; wherein the spectrum of the converted speech can be directly acquired, or the spectrum of the converted speech can be acquired by processing the converted speech into a spectrum through Fourier transform and its variants, and the converted speech can be obtained by performing speech conversion on the source speech through a speech conversion model (also known as a vocoder).

[0063] The identity features of the source speaker are obtained, and a latent representation of the source speaker features is extracted from the identity features of the source speaker. The identity features of the source speaker can be obtained directly, or the identity features of the source speaker can be extracted from the source speech using voiceprint recognition technology, which is mainly used to encode the voice characteristics of the speaker into a vector of fixed length.

[0064] Using the latent representation of the source speaker's characteristics as a source speaker watermark, and adding it to the latent representation of the converted speech, to obtain a encrypted latent representation of the converted speech;

[0065] The encrypted converted speech latent representation is decoded to obtain the converted speech with the source speaker watermark added, thereby completing the embedding of the source speaker identity. Optionally, after decoding the encrypted converted speech latent representation, the waveform of the converted speech with the source speaker watermark added is obtained, and the waveform of the converted speech with the source speaker watermark added can be used to generate the converted speech with the source speaker watermark added.

[0066] Those skilled in the art will readily appreciate that, compared to mainstream methods for adding watermarks to converted speech, the method for adding a source speaker watermark to converted speech provided in this embodiment separately extracts the latent representation of the source speaker's characteristics and the latent representation of the converted speech. This method then obtains the intrinsic representations underlying the source and converted speech signals, which reflect speech characteristics and information, including the underlying essential features and structure of speech. These two latent representations are then fused into a secret latent representation of the converted speech. By decoding the secret latent representation of the converted speech, a secret converted speech waveform is obtained, thereby adding the source speaker's representation to the converted speech. The speech played back based on the secret converted speech waveform primarily utilizes latent features from the source speaker's characteristics as a watermark, and is also added to the converted speech through fusion between the latent representations. These latent representations, when transmitted in speech, are difficult for the human ear to detect and are therefore highly imperceptible. Therefore, the waveform difference between the converted speech with the source speaker watermark added and the original unwatermarked speech is smaller, resulting in higher quality converted speech with the source speaker watermark added.

[0067] Furthermore, because the method for adding a source speaker watermark to converted speech provided by this embodiment achieves watermarking through the extraction and fusion of latent representations, the watermark is added directly to the original output of the speech conversion model, eliminating the need to reconstruct the speech conversion model. This avoids the problem of a significant discrepancy between the converted speech output of the new speech conversion model and the original converted speech waveform. Furthermore, compared to traditional watermarking methods that require reconstructing speech conversion models, the method for adding a source speaker watermark to converted speech provided by this embodiment is more lightweight, requiring no more than 2 Mbs of data in some embodiments.

[0068] It is not difficult to see that for a speech conversion module constructed using an end-to-end mainstream speech conversion method, the method for adding a source speaker watermark to the converted speech provided by this embodiment can be used as a subsequent additional supplementary module of the speech conversion module and has strong adaptability.

[0069] It is worth noting that the method for adding the source speaker watermark to the converted speech provided in this embodiment is not only applicable to the speech conversion module constructed by the end-to-end mainstream speech conversion method, but also to speech conversion modules constructed in other forms. In view of the same principle, it will not be described in detail here.

[0070] It is also worth noting that in the process of adding the latent representation of the source speaker features as the source speaker watermark to the latent representation of the converted speech, the embedding position of the latent representation of the source speaker features is flexibly controllable, and the watermark can be added at a position that is not easily noticed by the listener, thereby improving imperceptibility.

[0071] Example 2:

[0072] This embodiment provides a method for adding a source speaker watermark to converted speech. This embodiment is optimized based on the first embodiment to improve the technical effect and refine the technical solution. For details not fully described in this embodiment, please refer to the first embodiment.

[0073] As mentioned in the first embodiment, the spectrum of the converted speech is obtained, and the potential representation of the converted speech is extracted from the spectrum of the converted speech. Optionally, in a further but non-limiting embodiment of the present application, the converted speech is obtained, i.e., the output result of the speech conversion module on the source speech;

[0074] Perform a short-time Fourier transform on the converted speech to obtain a spectrum of the converted speech; optionally, the short-time Fourier transform formula is as follows:

[0075] ,

[0076] Where, To convert the spectrum of speech, is the short-time Fourier transform operation, Represents the converted speech waveform, which can be directly obtained from the converted speech;

[0077] A carrier coding network module is used to encode the spectrum of the converted speech to obtain a latent representation of the converted speech. The carrier coding network module is mainly composed of a three-layer gated convolutional neural network. Gated convolutional neural networks can capture latent representations in a nonlinear manner, are suitable for audio feature extraction, and have efficient parallel computing capabilities. Optionally, the encoding formula of the carrier coding network module is as follows:

[0078] ,

[0079] Where, Encoding operation for carrier encoding network module, The latent representation for the converted speech.

[0080] The first embodiment mentions obtaining the identity features of the source speaker and extracting the potential representation of the source speaker features from the identity features of the source speaker. As one embodiment, the method includes:

[0081] Obtaining source speech emitted by a source speaker;

[0082] Use the pre-trained speaker identity encoder to encode the source speech and obtain the identity features of the source speaker;

[0083] Perform dimension expansion processing on the identity features of the source speaker to obtain a dimension-expanded feature vector. The dimension-expanded feature vector has strong expressive power and is easy to process and analyze. Optionally, the dimension expansion processing formula is as follows:

[0084] ,

[0085] Where, is the identity characteristic of the source speaker, For dimension expansion operations, is the feature vector after dimension expansion;

[0086] The source speaker identity encoding network module is used to encode the dimensionally expanded feature vector to obtain the potential representation of the source speaker feature; wherein the source speaker identity encoding network module is mainly composed of a three-layer gated convolutional neural network. Optionally, the encoding formula of the source speaker identity encoding network module is as follows:

[0087] ,

[0088] Where, The encoding operation of the source speaker identity encoding network module, is the latent representation of the source speaker's characteristics.

[0089] Optionally, the speaker identity encoder may select a method based on traditional machine learning, or a method based on deep learning such as DeepSpeaker or x-vector.

[0090] In the first embodiment, the latent representation of the source speaker's features is used as a source speaker watermark and added to the latent representation of the converted speech to obtain a encrypted latent representation of the converted speech. Optionally, in a further but non-limiting embodiment of the present application, the following steps are included:

[0091] The latent representation of the source speaker features is added to the latent representation of the converted speech in the form of a residual to obtain the dense latent representation of the converted speech;

[0092] The decoding of the encrypted converted speech potential representation mentioned in the first embodiment to obtain the converted speech with the source speaker watermark added thereto may include, as one embodiment,

[0093] The latent representation of the encrypted speech is decoded through the encrypted speech decoding network module to obtain the spectrum of the encrypted speech; wherein the encrypted speech decoding network module is mainly composed of a four-layer gated convolutional neural network;

[0094] Perform an inverse short-time Fourier transform on the spectrum of the encrypted converted speech to obtain the waveform of the encrypted converted speech. The waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added, thereby adding the source speaker watermark to the converted speech. Optionally, the inverse short-time Fourier transform formula is as follows:

[0095] ,

[0096] Where, is the decoding operation of the secret conversion speech decoding network, is the inverse short-time Fourier transform operation, The waveform of the encrypted converted speech.

[0097] In order to improve the accuracy of watermark identification of the source speaker, the waveform of the encrypted converted speech is obtained as the waveform of the converted speech with the source speaker watermark added, and further includes:

[0098] Performing short-time Fourier transformation on the waveform of the encrypted converted speech to obtain the encrypted converted speech spectrum;

[0099] The encrypted converted speech spectrum is input into the source speaker identity decoding module for decoding to obtain the reconstructed source speaker identity feature. The reconstructed source speaker identity decoding module mainly consists of a four-layer gated convolutional neural network and a layer of adaptive average pooling. Optionally, the reconstruction decoding formula for reconstructing the source speaker identity feature is as follows:

[0100] ,

[0101] Where, To reconstruct the source speaker’s identity, is the decoding operation of the source speaker identity decoding module;

[0102] The cosine similarity between the reconstructed source speaker identity features and the source speaker identity features is verified and compared. In response to the cosine similarity value being greater than a first threshold, the reconstructed source speaker identity features and the source speaker identity features are accurately consistent, and the waveform of the encrypted converted speech is output as the waveform of the converted speech with the source speaker watermark added. Cosine similarity is an intuitive and easy-to-understand similarity metric that does not require in-depth mathematical theoretical knowledge and can effectively measure the directional proximity of two vectors. In the field of voiceprint recognition, this metric is particularly suitable for processing speaker identity features in speech data. The reconstructed source speaker identity features can be used to extract the source speaker's identity.

[0103] Furthermore, in addition to being used to directly extract the source speaker identity and directly verify the accuracy of the watermark identifying the source speaker, the reconstructed source speaker identity features can also be further used to reconstruct the source speech. Specifically, unlike the above method of directly using the reconstructed source speaker identity features for verification and comparison, as another embodiment, the cosine similarity value between the reconstructed source speaker identity features and the source speaker identity features is verified and compared. In response to the cosine similarity value being greater than a first threshold, the reconstructed source speaker identity features and the waveform of the encrypted converted speech are input into the speech conversion module to obtain the reconstructed source speech.

[0104] In addition to extracting the source speaker's identity, reconstructing the source speech can also be used to interpretably verify the accuracy of the source speaker's features in the encrypted converted speech and whether the source speaker can be identified through the added watermark:

[0105] The source speech is obtained, and a speech similarity value between the reconstructed source speech and the source speech is verified and compared. In response to the speech similarity value being greater than a second threshold, the waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added.

[0106] The method for adding the source speaker watermark to the converted speech provided in this embodiment uses multiple encoding network modules and decoding network modules. These network modules can be optimized and trained during use. Therefore, as one embodiment, after decoding the latent representation of the encrypted converted speech and obtaining the converted speech with the source speaker watermark,

[0107] Get the converted voice;

[0108] A loss function is constructed by comparing the spectral differences between the converted speech and the converted speech with the source speaker watermark added. This loss function is used to optimize the training vector encoding network module, the source speaker identity encoding network module, and / or the encrypted converted speech decoding network module. Specifically, the loss function constructed based on spectral differences can replace or be integrated into the existing loss functions in these network modules.

[0109] In order to evaluate the effectiveness of this application, a comparative experiment was conducted by comparing with the existing technology:

[0110] In the converted speech obtained by three speech conversion modules, namely VQMIVC (Reference: Wang D, Deng L, Yeung YT,et al. VQMIVC: Vector Quantization and Mutual Information-Based UnsupervisedSpeech Representation Disentanglement for One-Shot Voice Conversion[C] / / Proc.Interspeech 2021. 2021: 1344-1348.), FreeVC and ConsistencyVC, the source speaker identity watermark is embedded and verified and the source speech is reconstructed. All experiments were conducted on the VCTK corpus (reference: J. Yamagishi, C. Veaux, K. MacDonald, et al., “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019). This corpus contains 109 English speakers. Using a pre-trained model on the official VCTK corpus, we randomly generated 10,000 utterances of converted speech. These were then randomly divided into 9,000, 1,000, and 1,000 utterances for training, validation, and test sets, respectively. It is important to note that there was no overlap between any of the speakers (source or target) in the training and test sets. We use the official pre-trained model on the VCTK corpus (reference: Wan L, Wang Q, Papir A, et al. Generalized End-to-End Loss for Speaker Verification[C] / / 2018 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP). 2018: 4879-4883.) to obtain the source speaker identity.

[0111] The method for adding a source speaker watermark to the converted speech provided in this embodiment is compared with the currently most advanced speech steganography model DPAS, the speech digital watermark model WavMark (Reference: Chen G, Wu Y, Liu S, et al. WavMark: Watermarking for Audio Generation [M]. arXiv, 2023.), and the speaker identity tracing model VoxTracer on the same dataset. Refer to Tables 1 to 3 to show the effectiveness of the method for adding a source speaker watermark to the converted speech provided in this embodiment in actively embedding and extracting the source speaker identity in the converted speech.

[0112] The converted speech and the source speaker's identity are input. The signal-to-noise ratio (SNR), perceptual evaluation of speech quality (PESQ), mean opinion score (MOS), and speaker similarity (MOS-S) are used to characterize the imperceptibility of the watermark in the converted speech, indicating the effectiveness of embedding the source speaker's identity. NR and PESQ serve as objective evaluation metrics for speech quality. PESQ scores range from -0.5 to 4.5, with values ​​greater than 4.0 considered good auditory quality. A larger SNR indicates greater imperceptibility. MOS and MOS-S serve as subjective evaluation metrics for speech quality. Scores range from 0 to 5, with values ​​greater than 3.5 considered good speech quality.

[0113]

[0114] In addition, the MOSNet model (Reference: Lo CC, Fu SW, Huang WC, et al. MOSNet: Deep Learning-Based Objective Assessment for Voice Conversion[C] / / Proc. Interspeech 2019. 2019: 1541-1545.) was used to objectively evaluate the speech quality before and after the source speaker watermark was embedded in the converted speech. The difference and correlation between the two were calculated using the Linear Correlation Coefficient (LCC), Spearman Rank Correlation Coefficient (SRCC), and Mean Square Error (MSE). For DPAS, VoxTracer, and the method of adding source speaker watermarks to the converted speech provided in this embodiment, MOSNet was used to compare the speech quality of the converted speech with and without the source speaker watermark. Figure 4 、 Figure 5 and Figure 6 (from Figures 4 to 6 The following table shows the MOS of the converted speech using DPAS, VoxTracer, and the method for adding a source speaker watermark to the converted speech provided by this embodiment, respectively. The horizontal axis represents the MOS of the converted speech, while the vertical axis represents the MOS of the encrypted converted speech (i.e., the converted speech with the source speaker watermark added). It is easy to see that the method for adding a source speaker watermark to the converted speech provided by this embodiment achieves considerable consistency in speech quality before and after the source speaker watermark is embedded in the converted speech, while the consistency of the speech quality of DPAS and VoxTracer is poor.

[0115] For the source speaker tracking task, the speaker verification system performance EER (the lower the better), the mean cosine similarity MCS between the reconstructed source speaker identity and the original source speaker identity (the higher the better), and the proportion of the number of consistent source speaker identities between the reconstructed source speaker identity and the original source speaker identity ACC (the higher the better) are used to compare the source speaker tracing results of other methods with those of the method provided in this implementation. The speech length of all methods is based on the minimum speech length required for embedding. The experimental results are shown in Table 2.

[0116]

[0117] As can be seen from Tables 1 and 2, the method for adding a source speaker watermark to the converted speech provided by this embodiment can effectively embed the source speaker identity as a watermark in the converted speech, and the converted speech with the added source speaker watermark can also effectively extract the source speaker identity again for verification.

[0118] To measure the robustness of the encrypted converted speech, we used the converted speech within the framework of the ConsistencyVC method and processed it into the converted speech with the source speaker watermark using the method provided in this embodiment. We considered the following common noise attack types and tested the robustness of the converted speech with the source speaker watermark under different attacks:

[0119] 1. Noise: Adds Gaussian noise with a coefficient of 0.001 and speckle noise with coefficients of 0.1 and 0.01 to the speech.

[0120] 2. Resampling (RS): Convert the speech sampling rate containing the source speaker identity information to other common sampling rates (8000, 22050, 44100), and then convert it back to the original sampling rate.

[0121] 3. Low-pass filter (LP): Uses 4kHz and 5kHz cutoff frequencies to remove high-frequency components from speech.

[0122] 4. Amplitude Scaling (AS): Reduces the speech amplitude to 90% of the original amplitude.

[0123] 5. Lossy Compression (LC): compresses voice into MP3 format at 32kbps and 64kbps and then converts back.

[0124]

[0125] According to Table 3 and with reference to the corresponding scores in Tables 1 and 2, it is not difficult to see that the converted speech under the framework of the ConsistencyVC method, after adding the source speaker watermark to the converted speech using the method for adding the source speaker watermark to the converted speech provided by this embodiment, has good robustness against different types of attacks; the robustness tests under the framework of the other speech conversion modules are not repeated here in view of the same principles.

[0126] The method for adding the source speaker watermark to the converted speech provided in this application is combined with different speech conversion modules to measure cross-versatility. The embedding and extraction models trained by FreeVC and ConsistencyVC are cross-applied. That is, the converted speech output by one of the speech conversion modules is used to train the method for adding the source speaker watermark to the converted speech provided in this embodiment, and the trained method for adding the source speaker watermark to the converted speech is used to add the source speaker watermark to the converted speech output by another speech conversion module, and the applicability is tested. In this embodiment, the imperceptibility is determined by the SNR and PESQ test methods, and the effect of extracting the source speaker identity is determined by the MCS test, such as Figure 7 As shown, Figure 7 Here, "original" represents the test score of the converted speech with the source speaker watermark added, which is obtained by training the method for adding the source speaker watermark to the converted speech provided in this embodiment using the converted speech output by FreeVC; "cross" represents the test score of the converted speech with the source speaker watermark added, which is obtained by using the method for adding the source speaker watermark to the converted speech trained by the converted speech output by FreeVC to add the source speaker watermark to the converted speech output by ConsistencyVC.

[0127] It is not difficult for those skilled in the art to see that the method for adding a source speaker watermark to converted speech provided by this embodiment has strong cross-applicability.

[0128] The method of adding the source speaker watermark to the converted speech provided in this embodiment to measure the quality of the reconstructed source speech is helpful for Mosnet to compare the source speech and the reconstructed source speech. It should be noted that, considering that the voice conversion module VC may not be completely decoupled during the speech conversion process, the quality of the reconstructed original speech is not high. Therefore, Figure 8 As shown (from the left "1" horizontal axis is Mosnet's score for the source speech, the vertical axis is Mosnet's score for the regenerated source speech, the middle "2" horizontal axis is Mosnet's score for the source speech, the vertical axis is Mosnet's score for the reconstructed source speech, the right "3" horizontal axis is Mosnet's score for the reconstructed source speech, the vertical axis is Mosnet's score for the regenerated source speech), a full comparative experiment was set up using FreeVC, that is, the regenerated source speech of FreeVC was compared with the source speech and the reconstructed source speech respectively, and the waveform of the encrypted converted speech and the speaker identity features extracted from the source speech were input into the speech conversion module VC to regenerate the source speech. It is worth pointing out that the speech conversion module VC may have the problem of incomplete decoupling, which reduces the quality of the reconstructed source speech, which is related to Figure 8 The performance is consistent, that is, the regenerated source speech is not completely equivalent to the source speech, refer to Figure 8 The scores of the regenerated source speech, the original speech, and the reconstructed source speech all have similar data distributions. This indicates that the quality loss of the reconstructed source speech compared to the original speech is mainly caused by the speech conversion module, rather than a problem with the method of adding the source speaker watermark to the converted speech provided in this embodiment.

[0129] In practice, the method for adding a source speaker watermark to the converted speech provided in this embodiment serves only as a supplement to VC, adding no additional burden to VC. Furthermore, during the tracking process, the source speaker's identity information can be quickly verified and used to restore the original speech. Therefore, this embodiment also compares the method for adding a source speaker watermark to the converted speech with other methods in terms of model size and time consumption. The results are shown in Table 4.

[0130]

[0131] In summary, it's easy to see that both DAPS and VoxTracer use the WaveGlow vocoder, resulting in larger model sizes. While this doesn't impose a significant burden on two-stage VC, requiring only mapping the secret information to a Gaussian distribution, end-to-end VC requires the use of the vocoder again, resulting in a larger model. The method for adding a source speaker watermark to converted speech provided in this embodiment embeds the source speaker identity information in the converted speech, and the model size for both embedding and extracting the source speaker identity information is less than 1MB, adding no additional burden to any VC method.

[0132] Similarly, the DAPS and VoxTracer methods require Mel-transformation and the WaveGlow vocoder to regenerate the speech waveform for end-to-end VC, which adds unnecessary time. However, the method for adding a source speaker watermark to the converted speech provided in this embodiment takes almost negligible time to embed and extract the source speaker identity information, without placing any additional burden on VC. It is undeniable that the Wavmark method also embeds the source speaker directly during the conversion process and only requires a very short carrier speech. However, its carrier speech has a fixed duration of 1 second, while the method for adding a source speaker watermark to the converted speech provided in this embodiment only takes 0.5 seconds.

[0133] In summary, the method for adding a source speaker watermark to the converted speech provided by this embodiment has the advantages of small size, fast speed, high quality, high interpretability, strong robustness, and the intermediate product can be used to reconstruct the source speech, compared with the traditional method of adding a source speaker watermark to the converted speech.

[0134] Example 3:

[0135] This embodiment provides a computer device, including a processor and a memory connected to the processor. A computer program is stored in the memory. When the computer program is executed by the processor, the steps of the method for adding a source speaker watermark to converted speech provided in Embodiment 1 or 2 are performed.

[0136] The computer device may be a server or an electronic terminal. As one embodiment, Figure 9The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data obtained and generated in the method of adding a source speaker watermark to the converted speech. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements the method of adding a source speaker watermark to the converted speech provided in embodiment one or two.

[0137] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0138] The computer device provided in this embodiment has the same technical effects as those in Embodiment 1 or 2, and will not be described in detail here.

[0139] Example 4:

[0140] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method for adding a source speaker watermark to converted speech provided in the first or second embodiment are implemented.

[0141] The computer-readable storage medium provided in this embodiment has the same technical effects as those in Embodiment 1 or 2, and will not be described in detail here.

[0142] Embodiment 5:

[0143] This embodiment provides a computer program product having a computer program stored thereon. When executed by a processor, the computer program implements the steps of the method for adding a source speaker watermark to converted speech provided in Embodiment 1 or Embodiment 2. The computer program product provided in this embodiment can be transmitted, distributed, and downloaded in the form of a signal via the Internet.

[0144] The computer program product provided in this embodiment has the same technical effects as those in Embodiment 1 or 2, and will not be described in detail here.

[0145] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for adding a source speaker watermark to a converted speech, characterized in that: include, Acquiring a spectrum of the converted speech, and extracting a potential representation of the converted speech from the spectrum of the converted speech; Obtaining identity features of a source speaker, and extracting a latent representation of source speaker features from the identity features of the source speaker; Using the latent representation of the source speaker's characteristics as a source speaker watermark, and adding it to the latent representation of the converted speech, to obtain a encrypted latent representation of the converted speech; Decoding the encrypted converted speech latent representation to obtain the converted speech with the source speaker watermark added; The obtaining of the identity features of the source speaker and extracting the potential representation of the source speaker features from the identity features of the source speaker include: Get the source voice; Encoding the source speech using a pre-trained speaker identity encoder to obtain identity features of the source speaker; Performing dimension expansion processing on the identity feature of the source speaker to obtain a dimensionally expanded feature vector; A source speaker identity encoding network module is used to encode the dimensionally expanded feature vector to obtain a potential representation of the source speaker feature; wherein the source speaker identity encoding network module is mainly composed of a three-layer gated convolutional neural network; The method of adding the latent representation of the source speaker's features as a source speaker watermark to the latent representation of the converted speech to obtain the encrypted latent representation of the converted speech includes: The latent representation of the source speaker features is added to the latent representation of the converted speech in the form of a residual to obtain the dense latent representation of the converted speech; The decoding of the encrypted converted speech potential representation to obtain the converted speech with the source speaker watermark added includes: Decoding the latent representation of the encrypted converted speech through a encrypted converted speech decoding network module to obtain a spectrum of the encrypted converted speech; wherein the encrypted converted speech decoding network module is mainly composed of a four-layer gated convolutional neural network; Performing an inverse short-time Fourier transform on the spectrum of the encrypted converted speech to obtain a waveform of the encrypted converted speech, and using the waveform of the encrypted converted speech as the waveform of the converted speech with the source speaker watermark added thereto, thereby adding the source speaker watermark to the converted speech; the obtaining of the waveform of the encrypted converted speech and using the waveform of the encrypted converted speech as the waveform of the converted speech with the source speaker watermark added thereto comprises: Performing a short-time Fourier transform on the waveform of the encrypted converted speech to obtain a spectrum of the encrypted converted speech; Inputting the encrypted converted speech spectrum into a source speaker identity decoding module for decoding to obtain reconstructed source speaker identity features; wherein the source speaker identity decoding module is mainly composed of a four-layer gated convolutional neural network and a layer of adaptive average pooling; verifying and comparing a cosine similarity value between the reconstructed source speaker identity feature and the source speaker identity feature, and in response to the cosine similarity value being greater than a first threshold, inputting the reconstructed source speaker identity feature and the waveform of the encrypted converted speech into a speech conversion module to obtain a reconstructed source speech; The source speech is obtained, and a speech similarity value between the reconstructed source speech and the source speech is verified and compared. In response to the speech similarity value being greater than a second threshold, the waveform of the encrypted converted speech is used as the waveform of the converted speech with the source speaker watermark added.

2. The method for adding a source speaker watermark to a converted speech according to claim 1, wherein: The obtaining of the spectrum of the converted speech and extracting the potential representation of the converted speech from the spectrum of the converted speech include: Get the converted voice; Performing a short-time Fourier transform on the converted speech to obtain a spectrum of the converted speech; A carrier coding network module is used to encode the spectrum of the converted speech to obtain a potential representation of the converted speech; wherein the carrier coding network module is mainly composed of a three-layer gated convolutional neural network.

3. The method for adding a source speaker watermark to a converted speech according to claim 1, wherein: After decoding the encrypted converted speech latent representation to obtain the converted speech with the source speaker watermark added, Get the converted voice; A loss function is constructed by using the spectrum difference between the converted speech and the converted speech with the source speaker watermark added thereto. The loss function is used to optimize the training vector encoding network module, the source speaker identity encoding network module and / or the encrypted converted speech decoding network module.

4. A computer device, characterized in that: The method comprises a processor and a memory connected to the processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the steps of the method for adding a source speaker watermark to the converted speech as claimed in any one of claims 1 to 3 are performed.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for adding a source speaker watermark to converted speech according to any one of claims 1 to 3 are implemented.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for adding a source speaker watermark to converted speech as claimed in any one of claims 1 to 3 are implemented.

Images