Voice conversion speaker diarization method and device
By generating and recovering Mel spectrograms for speech conversion, the source tracing problem of speech conversion technology under lossy compression environment is solved, realizing efficient speaker identity and voice verification, which is suitable for social media and voice calls.
Patent Information
- Application Number
- CN202310236391.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-10
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2043-03-10
AI Technical Summary
Existing speech conversion technologies cannot effectively trace the source of information on social media, especially under lossy compression, resulting in poor accuracy in tracing. Furthermore, existing watermarking technologies are not robust enough to lossy compression.
A pre-trained VC model is used to generate Mel spectrograms. A pre-trained speaker ID extractor and ID encoder are combined to hide the speaker's identity. The speaker embedding is restored after lossy compression by a speech inverter, thus realizing speaker verification and speech verification.
Without compromising the quality of the converted speech, it achieves high robustness and high accuracy in tracing multiple compression standards, and can restore the identity of the original speaker and the original speech, making it suitable for various social media and voice call applications.
Smart Images
Figure CN116665687B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of digital media processing, and in particular to a voice conversion speaker tracing method and device. BACKGROUND
[0002] Voice conversion (VC) is a technique that converts the voice of a source speaker into that of a target speaker while preserving the linguistic content. With the development of deep learning, VC has attracted extensive attention and applications, such as speaker anonymization, movie dubbing, voice customization, and singing conversion. Recently, some network platforms and mobile applications (e.g., MetaVoice, Respeecher, and Resemble AI) have emerged, which release VC as a product or even an online service, making it easy to manipulate voices like editing text. As a result, various types of converted audio or video are widely spread and shared on social media (e.g., TikTok, YouTube, and Twitter), and are gradually applied to voice calls and online meetings (e.g., Voicemod).
[0003] For VC, the identity of the source speaker is a key clue to trace the source of the converted voice. However, all mainstream public VC platforms currently cannot fully meet the requirements of tracing. First, the converted voice uploaded and spread to social media inevitably undergoes lossy compression, which severely affects the accuracy of tracing, especially for low bit rate compression. Second, different audio encoding standards or encoders have different characteristics, and the choice of compression standards and bit rates by social media or disseminators is not predictable, which further exacerbates the difficulty of tracing. Although audio watermarking can be partially used for VC tracing, existing watermarking techniques have poor robustness to lossy compression and are usually limited to specific audio encoding standards. Therefore, it is of great significance and urgency to develop new methods for effective VC tracing. SUMMARY
[0004] The present application provides a voice conversion speaker tracing method and device. To solve or at least partially solve the technical problem of poor robustness and universality in the prior art.
[0005] To solve the above technical problems, the first aspect of the present application provides a voice conversion speaker tracing method, comprising: a voice conversion stage, a hiding stage, a transmission stage, and a tracing stage.
[0006] In the voice conversion stage, a pre-trained VC model is used with the source speech s and the target speech t as input, and the synthesized Mel spectrogram m is output.
[0007] In the hiding phase, the pre-trained speaker ID extractor takes the source speech s as input to generate a speaker embedding v representing the speaker identity, and then the ID encoder maps v to a latent code z following a Gaussian distribution, and z is input to the speech generator conditioned on the Mel-spectrogram m to synthesize the converted speech x.
[0008] In the transmission phase, the converted speech x is transmitted over the Internet and after various lossy compression and audio processing operations, the speech
[0009] In the tracing phase, the speech inverter takes the Mel-spectrogram obtained from as condition to convert the compressed or processed speech back to the latent code and then the ID decoder maps back to the speaker embedding for two types of verification: speaker verification and speech verification; wherein the speaker verification is realized according to the comparison between and the speaker embedding v extracted from the enrollment speech, and the speech verification is realized according to the comparison between input to the speech recovery model to obtain the recovered speech and the source speech s.
[0010] In an embodiment, the pre-trained speaker ID extractor in the hiding phase adopts a classical speaker verification model, which uses the extracted speaker embedding to represent the speaker identity, and the speaker ID extractor includes three LSTM layers and a dense layer, each LSTM layer has 768 units, and the final dense layer is used to output a 256-dimensional speaker embedding.
[0011] In an embodiment, the ID encoder is implemented based on a VAE-based encoder, and the speech generator is implemented based on a generative flow model, wherein the ID encoder is trained using the variational objective of the standard VAE, the VAE is a variational autoencoder, given the speaker of the embedding v, the standard Gaussian prior and the Gaussian posterior wherein μ z and σ z are the learnable mean and variance, and I is the identity matrix, and the loss function of the ID encoder is as follows:
[0012]
[0013] wherein represents the KL divergence between q(z|v) and p(z).
[0014] In an embodiment, the speech inverter shares the same network structure with the speech generator, and is initialized using the parameters of the speech generator, fine-tuned, and uses the reconstruction loss for training of the speech inverter, i.e.,
[0015]
[0016] where ||·||1 denotes the L1 distance.
[0017] In an embodiment, the ID decoder is implemented based on the decoder part of the VAE, and the loss function of the ID decoder is calculated as:
[0018]
[0019] where ||·||1 denotes the L1 distance.
[0020] In an embodiment, the speech inverter and the ID decoder adopt a joint training strategy, and the overall loss function is:
[0021]
[0022] The overall loss function used for the traceability stage is: z and λ v are the weight combinations balancing different loss terms.
[0023] Based on the same inventive concept, the second aspect of the present application provides a speech conversion speaker traceability device, comprising: a speech conversion module, a hidden module, a transmission module, and a traceability module;
[0024] The speech module is configured to use a pre-trained VC model to take the source speech s and the target speech t as input, and output the synthesized Mel spectrogram m.
[0025] The hidden module is configured to take the source speech s as input through a pre-trained speaker ID extractor to generate a speaker embedding v representing the speaker identity, and then map v to a latent code z following a Gaussian distribution through an ID encoder, and input z to a speech generator conditioned on the Mel spectrogram m to synthesize the converted speech x.
[0026] The transmission module is configured to transmit the converted speech x on the Internet, and obtain the compressed or processed speech
[0027] The traceability module is configured to take the Mel spectrogram obtained from the compressed or processed speech as a condition, and convert the compressed or processed speech back to the latent code are then decoded by an ID decoder to re-mapped as a speaker embedding for two types of verification: speaker verification and voice verification; wherein, according to speaker verification is realized by comparing the speaker embedding v extracted from the registered voice with input to a voice recovery model to obtain a recovered voice according to voice verification is realized by comparing the recovered voice with the source voice s.
[0028] Based on the same inventive concept, the third aspect of the present application provides a computer readable storage medium having stored thereon a computer program which, when executed, implements the method of the first aspect.
[0029] Based on the same inventive concept, the fourth aspect of the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of the first aspect when executing the program.
[0030] Compared with the prior art, the present application has the following advantages and beneficial technical effects:
[0031] The present application proposes a robust and universal voice conversion speaker tracing framework, which can be integrated with any VC framework and used in various social media and voice call applications without affecting the quality of the converted voice, and can restore the identity of the source speaker or even the original voice when VC abuse causes serious personal or social harm. The specific advantages include: (1) high voice quality. The converted voice generated by the present application is better than existing audio watermarking and steganography methods, and the quality does not decrease significantly compared with the converted voice without hidden identity information; the original voice recovered by the present application is almost identical to the source voice. (2) Strong robustness and wide application range. The present application has good robustness to various audio and voice compression (such as AAC, MP3, Opus and SILK) with a wide range of compression bit rates (16kbps-128kbps), and can achieve nearly 100% tracing accuracy even in 0.74s of voice length, which is better than existing watermarking and steganography methods. (3) Strong universality. The present application can be used in almost all existing voice conversion frameworks; even for VC that directly generates voice waveforms, the present application can be applied after extracting the Mel spectrogram. BRIEF DESCRIPTION OF DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0033] Figure 1 Flowchart of VC traceability in a lossy environment in the embodiments of the present application;
[0034] Figure 2 Framework diagram of the VC traceability method in the embodiments of the present application. DETAILED DESCRIPTION
[0035] The present application discloses a robust and general voice conversion speaker traceability method. In order to solve the problem that it is difficult to actively trace the source speaker identity when voice conversion technology is misused in practical application, the present application designs a traceability method based on asynchronous VAE-Glow, which can hide the source speaker identity in parallel in the voice conversion process, and can accurately recover the hidden source speaker identity and even the original voice from the compressed converted voice. This method can be integrated with any voice conversion framework and used in various social media and voice call applications, and can realize speaker identity hiding and traceability without affecting the quality of converted voice. When facing mainstream audio and voice compression standards (such as AAC, MP3, Opus and SILK) with a wide range of bit rates (16-128kbps), only 0.74s of voice can achieve nearly 100% traceability accuracy.
[0036] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0037] Embodiment one
[0038] The embodiments of the present application provide a voice conversion speaker traceability method, which comprises a voice conversion stage, a hiding stage, a transmission stage and a traceability stage.
[0039] In the voice conversion stage, a pre-trained VC model is used to take the source voice s and the target voice t as input, and output the synthesized Mel spectrogram m.
[0040] In the hiding phase, the pre-trained speaker ID extractor takes the source speech s as input to generate a speaker embedding v representing the speaker identity, which is then mapped to a latent code z following a Gaussian distribution by the ID encoder. The z is input to the speech generator conditioned on the Mel-spectrogram m to synthesize the converted speech x.
[0041] In the transmission phase, the converted speech x is transmitted over the Internet and subjected to various lossy compression and audio processing operations to obtain the speech
[0042] In the tracing phase, the speech inverter takes the compressed or processed speech obtained from the Internet as input and conditioned on the Mel-spectrogram to convert it back to the latent code which is then remapped to the speaker embedding by the ID decoder. The speaker verification is achieved by comparing with the speaker embedding v extracted from the enrollment speech, while the speech verification is achieved by comparing to the source speech s.
[0043] In the present invention, the tracing of VC in a lossy environment is achieved. Even if the converted speech is subjected to various lossy processing (such as lossy compression, resampling, Gaussian noise, etc.), the source speaker identity or even the source speech can still be accurately recovered from the distorted speech, as shown in Figure 1
[0044] The workflow of VC tracing is as follows. The VC platform provides online VC services, and users must first register with their own speech for speaker verification. When users enjoy VC services, the user identity will be hidden in the converted speech in an imperceptible manner. The converted speech is transmitted over the Internet, during which it may be subjected to lossy compression or processing. If the converted speech is used for illegal purposes, the VC platform can achieve the tracing purpose by recovering the source speaker identity or even the source speech.
[0045] According to the above workflow, the method of the present invention includes four phases: VC phase, hiding phase, transmission phase, and tracing phase. The method framework is shown in Figure 2 , and the related symbols are listed in Table 1.
[0046] Table 1 Symbols and their meanings
[0047]
[0048] In particular, the synthesized Mel-spectrogram m of the VC stage will be used for the following speech waveform generation. The concealment stage conceals the speaker identity in an imperceptible way into the converted speech during the speech generation process. In the transmission stage, the converted speech x is transmitted over the Internet and suffers various lossy compression and audio processing operations, resulting in a degraded quality of the converted speech x, denoted as The provenance stage is the reverse process of the concealment stage, which attempts to recover the speaker identity concealed in the converted speech.
[0049] The proposed method can be applied to any VC method, providing it with a provenance functionality.
[0050] To ensure the generality of the method, the method is preferably independent of the internal implementation of the VC, i.e. the VC is considered as a black-box operation. Considering that the Mel-spectrogram is the output of most VC methods, the method combines the Mel-spectrogram with the speaker identity to generate a provenance-enabled converted speech, which makes the concealment process independent of the VC process.
[0051] To obtain the converted Mel-spectrogram, the proposed method can use any pre-trained VC model. It is worth noting that a few VC methods do not generate a Mel-spectrogram but directly output a converted speech waveform. In this case, the converted speech waveform can be converted into a Mel-spectrogram by using a short-time Fourier transform, which almost does not introduce additional latency.
[0052] In an embodiment, the pre-trained speaker ID extractor of the concealment stage employs a classical speaker verification model, which uses the extracted speaker embedding to represent the speaker identity. The speaker ID extractor comprises three LSTM layers and a dense layer, each LSTM layer has 768 units, and the final dense layer is used to output a 256-dimensional speaker embedding.
[0053] In particular, to uniquely identify the source speaker, the present embodiment employs a classical speaker verification model as the speaker ID extractor, and uses the extracted speaker embedding to represent the speaker identity. The pre-trained speaker ID extractor is pre-trained on a speech search corpus using a GE2E loss function, which contains 36M sentences from 18K American English users.
[0054] In an embodiment, the ID encoder is implemented based on a VAE-based encoder, and the speech generator is implemented based on a generative flow model, wherein the ID encoder is trained using the variational objective of a standard VAE, the VAE being a variational autoencoder, given an embedding v of a speaker, a standard Gaussian prior and a Gaussian posterior where μ z and σ zwhere I is the identity matrix, and ID is the loss function of the ID encoder.
[0055]
[0056] where denotes the KL divergence between q(z|v) and p(z).
[0057] The identity matrix (i.e., a matrix with 1s on the diagonal and 0s elsewhere) is a common mathematical notation for the standard deviation of a multidimensional standard Gaussian distribution. Specifically, a Glow model is used as the speech generator, which can realize the hiding of speaker embedding in the speech generation process. Specifically, the speaker embedding replaces the latent code, the Mel spectrogram is used as the condition to guide the speech generation, and the reversibility of Glow is used for the following backtracking.
[0058] The latent code of Glow always follows a Gaussian prior distribution, while the distribution of the speaker embedding is usually uniform or undefined, so the speaker embedding cannot be directly input into the Glow model for speech generation. The ID encoder maps the speaker embedding into a latent code following a Gaussian distribution, which is then input into the Glow-based speech generator to synthesize speech.
[0059] The implementation of the ID encoder is based on the encoder of VAE. The reasons are as follows: first, VAE imposes a Gaussian prior on the latent code, which follows a Gaussian distribution, which meets the Gaussian requirement of Glow; second, the variational inference adopted in the VAE encoder makes the VAE decoder more robust to input changes. While the decoder part of VAE is used asynchronously in the backtracking stage.
[0060] To ensure robustness in a lossy environment, during the transmission stage, the converted speech is disturbed by lossy compression and audio processing operations to simulate real-world environments. Specifically, for the speech waveform generated by the speech generator, it is first saved in WAV format, which will cause a saving error in the conversion from floating-point to integer data. Then the WAV format speech is compressed with multiple audio encoding standards at different bit rates.
[0061] AAC, MP3, Opus, and SILK are considered as four encoding standards, and noise attacks, resampling, re-quantization, amplitude modification, and filtering are used to test robustness.
[0062] In one embodiment, the speech inverter shares the same network structure as the speech generator, and is initialized using the parameters of the speech generator, and then fine-tuned, and the reconstruction loss is used for the training of the speech inverter, i.e.,
[0063]
[0064] Where ||·||1 represents the L1 distance.
[0065] Specifically, the source speaker's identity is recovered by using two inverse models corresponding to the hidden stages. The first model, the speech inverter, shares the same network structure as the Glow-based speech synthesizer and is initialized using the parameters of the speech generator. Instead of directly utilizing the speech-latent process of the speech generator to reconstruct the latent code, the speech inverter is fine-tuned, which is beneficial for latent code recovery. A reconstruction loss is used for training the speech inverter.
[0066] In one implementation, the ID decoder is based on the decoder portion of the VAE, and the loss function of the ID decoder is calculated as follows:
[0067]
[0068] Where ||·||1 represents the L1 distance.
[0069] Specifically, the second model in the source tracing stage is the ID decoder, which is implemented by the decoder part of the VAE. A normal VAE decoder uses random latent codes as input to improve robustness, thus it can fit our information loss scenario, i.e., the recovered latent code has random perturbations compared to the original latent code. Unlike the separate training strategy used in the hidden stage (where the ID encoder and speech generator are trained separately and then used jointly because the output of the ID encoder matches the input requirements of the speech generator), the speech inverter and ID decoder employ a joint training strategy.
[0070] In one implementation, the speech inverter and ID decoder employ a joint training strategy, with the overall loss function being:
[0071]
[0072] λ is the overall loss function used in the source tracing phase. z and λ v It is a weight combination that balances different loss terms.
[0073] Compared to the normal VAE and Glow models, the VAE-Glow structure proposed in this invention (i.e. Figure 2 The VAE (which includes an ID encoder, speech generator, speech inverter, and ID decoder) employs an asynchronous training strategy. Specifically, the encoder (embedding to latent code) and decoder (latent code to embedding), as well as the forward (latent code to speech) and inverse (speech to latent code) processes of the Glow model, are trained asynchronously. Asynchronous training addresses the mismatch between converted and distorted speech, thereby improving source tracing robustness in lossy environments.
[0074] Furthermore, regarding speaker verification
[0075] The present embodiment adopts the cosine similarity and decision threshold commonly used in speaker verification to determine whether the recovered speaker identity matches the previously registered speaker identity. Specifically, the present embodiment first extracts pairs of speaker embeddings from the paired speech training set belonging to the same speaker or different speakers. Then, the cosine similarity of each pair of speaker embeddings is calculated. Finally, the threshold of the cosine similarity is adjusted until an optimal trade-off between the false rejection rate and the false acceptance rate is achieved. In the inference stage, if the cosine similarity between the recovered speaker embedding and the registered speech is greater than the threshold, the speaker is confirmed as the source speaker of the converted speech.
[0076] Voice verification
[0077] For judicial purposes and to achieve complete traceability, it is necessary to recover the source speech from the converted speech, not just verify the existence of the source speaker identity. VC can synthesize the target speech by fusing the disentangled speaker embedding with the content representation, so it can be used to achieve speech recovery. More specifically, AutoVC is used as a speech recovery model, the converted speech is input into the content encoder of AutoVC to generate the content representation, and the recovered speaker embedding is combined with the content representation, which is input into the decoder of AutoVC to generate the recovered speech. It should be noted that in the field of voice conversion (VC), it is generally believed that speech is composed of two parts, "content" and "speaker identity feature", so the process of separating "content" or "speaker identity feature" (also referred to as "speaker embedding" here) from speech is called "disentanglement".
[0078] The present application discloses a robust and universal voice conversion speaker tracing method. In order to solve the problem that it is difficult to actively trace the source speaker identity when voice conversion technology is misused in practical application, the present application designs a tracing method based on asynchronous VAE-Glow, which can hide the source speaker identity in parallel during voice conversion and accurately recover the hidden source speaker identity or even the original speech from the compressed converted speech. This method can be integrated with any voice conversion framework and used in various social media and voice call applications, and can realize the hiding and tracing of speaker identity without affecting the quality of converted speech. When facing mainstream audio and voice compression standards such as AAC, MP3, Opus and SILK with a wide range of bit rates (16-128kbps), it can achieve nearly 100% tracing accuracy with only 0.74s of speech.
[0079] Embodiment two
[0080] Based on the same inventive concept, the embodiment provides a voice conversion speaker tracing device, comprising a voice conversion module, a hiding module, a transmission module and a tracing module;
[0081] The voice module is configured to input the source voice s and the target voice t into the pre-trained VC model to output a synthesized Mel spectrogram m;
[0082] The hiding module is configured to input the source voice s into the pre-trained speaker ID extractor to generate a speaker embedding v representing the speaker identity, then input the v into the ID encoder to map the v to a latent code z following a Gaussian distribution, and then input the z into a voice generator conditioned on the Mel spectrogram m to synthesize the converted voice x;
[0083] The transmission module is configured to transmit the converted voice x on the Internet, and obtain a compressed or processed voice
[0084] The tracing module is configured to input the compressed or processed voice obtained Mel spectrogram to a voice reverser to convert the compressed or processed voice back to the latent code , and then input the latent code into an ID decoder to map the latent code back to the speaker embedding for two types of verification: speaker verification and voice verification; wherein the speaker verification is realized according to the comparison between the speaker embedding and the speaker embedding v extracted from the registered voice, and the voice verification is realized according to the comparison between the voice input into the voice recovery model to obtain the recovered voice and the source voice s.
[0085] Since the device introduced in the second embodiment of the application is the device used to implement the voice conversion speaker tracing method in the first embodiment of the application, the specific structure and modifications of the device can be understood by those skilled in the art based on the method introduced in the first embodiment of the application, and therefore will not be described here. Any device used in the method in the first embodiment of the application belongs to the scope of protection of the application.
[0086] Embodiment three
[0087] Based on the same inventive concept, the application further provides a computer readable storage medium having a computer program stored thereon, the program being executed to implement the method as described in the first embodiment.
[0088] Since the computer readable storage medium introduced in the embodiment three of the present application is the computer readable storage medium used for implementing the voice conversion speaker tracing method in the embodiment one of the present application, the specific structure and deformation of the computer readable storage medium can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. The computer readable storage medium used by the method in the embodiment one of the present application belongs to the scope of the present application.
[0089] Embodiment four
[0090] Based on the same inventive concept, the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method in the embodiment one when executing the above program.
[0091] Since the computer device introduced in the embodiment four of the present application is the computer device used for implementing the voice conversion speaker tracing method in the embodiment one of the present application, the specific structure and deformation of the computer device can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. The computer device used by the method in the embodiment one of the present application belongs to the scope of the present application.
[0092] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.
[0093] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for carrying out the functions specified in one or more flows and / or blocks. Figure 1 The means for carrying out the functions specified in one or more flows and / or blocks.
[0094] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.
[0095] It is apparent that those skilled in the art can, without departing from the spirit and scope of the application, make various changes and modifications to the embodiments of the application. As such, it is intended that the application cover all such changes and modifications that fall within the scope of the application.
Claims
1. A voice conversion speaker diarization method, characterized by, Comprise: a speech conversion stage, a hiding stage, a transmission stage and a tracing stage; In the speech conversion stage, a pre-trained VC model is used to take the source speech s and the target speech t as input, and output the synthesized Mel spectrogram m; In the hiding stage, a pre-trained speaker ID extractor is used to take the source speech s as input to generate a speaker embedding v representing the speaker identity, and then an ID encoder is used to map v to a latent code z following a Gaussian distribution, and then z is input into a speech generator conditioned on the Mel spectrogram m to synthesize the converted speech x. In the transmission phase, the converted speech x is transmitted over the Internet, and after various lossy compression and audio processing operations, the speech y is obtained During the source tracing phase, a voice inverter is used to... The obtained Mel spectrogram As a condition, the compressed or processed speech Reconvert to latent code Then through the ID decoder Remapping to speaker embedding Used for two types of verification: speaker verification and voice verification; where, according to Speaker verification is achieved by comparing the speaker embedding v extracted from the registered speech. Input into the speech recovery model to obtain the recovered speech according to Voice verification is achieved by comparing the result with the source speech s.
2. The voice conversion speaker diarization method of claim 1, wherein, The pre-trained speaker ID extractor in the hiding stage uses a classic speaker verification model to represent the speaker identity using the extracted speaker embedding. The speaker ID extractor includes three LSTM layers and a density layer, each LSTM layer has 768 units, and the final density layer is used to output a 256-dimensional speaker embedding.
3. The voice conversion speaker diarization method of claim 1, wherein, The ID encoder is implemented based on a VAE-based encoder, and the speech generator is implemented based on a generative flow model, wherein the ID encoder is trained using a variational objective of a standard VAE, the VAE is a variational autoencoder, and a standard Gaussian prior and a Gaussian posterior where μ z and σ z are learnable mean and variance, I is an identity matrix, and a loss function of the ID encoder is as follows: in Let represent the KL divergence between q(z|v) and p(z).
4. The voice conversion speaker diarization method of claim 1, wherein, The speech inverter shares the same network structure as the speech generator, and is initialized using the parameters of the speech generator, and then fine-tuned, and the reconstruction loss is used for training of the speech inverter, i.e. Where ||·||1 represents the L1 distance.
5. The voice conversion speaker diarization method of claim 4, wherein, The ID decoder is implemented based on the decoder part of the VAE, and the loss function of the ID decoder is calculated as: Where ||·||1 represents the L1 distance.
6. The voice conversion speaker diarization method of claim 5, wherein, The speech inverter and the ID decoder use a joint training strategy, and the overall loss function is: The overall loss function employed for the trace phase, λ z and λ v is a combination of weights balancing the different loss terms.
7. A voice conversion speaker diarization apparatus characterized by comprising: Comprise: a speech conversion module, a hiding module, a transmission module and a tracing module; The speech module is used to take the source speech s and the target speech t as input, and output the synthesized Mel spectrogram m using a pre-trained VC model; The hiding module is used to take the source speech s as input to generate a speaker embedding v representing the speaker identity using a pre-trained speaker ID extractor, and then an ID encoder is used to map v to a latent code z following a Gaussian distribution, and then z is input into a speech generator conditioned on the Mel spectrogram m to synthesize the converted speech x. The transmission module is used to convert the speech x for transmission over the Internet, and the speech is obtained after various lossy compression and audio processing operations The traceability module is used to pass through the voice reverser to recover the original speech from the Mel-spectrogram obtained is conditioned, the compressed or processed speech is reconverted into a latent code is then remapped into a speaker embedding by an ID decoder for two types of verification: speaker verification and voice verification; wherein, according to a comparison with the speaker embedding v extracted from the enrollment speech, speaker verification is implemented, and the input to the speech recovery model to obtain the recovered speech according to a comparison with the source speech s, voice verification is implemented. 8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the method of any one of claims 1 to 6.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 6.