Voice conversion method and apparatus

Abstract

The application discloses a voice conversion method and device. The method comprises the following steps: aligning a frame-level acoustic feature sequence with a phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence of attention content information; generating a phoneme-level hidden variable sequence carrying content and acoustic information simultaneously based on the phoneme-level text feature sequence, the phoneme-level acoustic feature sequence and a target object identifier; inputting the phoneme-level hidden variable sequence into a trained duration prediction network to obtain a predicted duration sequence corresponding to the phoneme-level hidden variable sequence; expanding the duration of each phoneme-level hidden variable in the phoneme-level hidden variable sequence based on the predicted duration sequence to obtain a frame-level hidden variable sequence; and generating target audio corresponding to the target object identifier based on the frame-level hidden variable sequence. The application can not only retain the emotion of the source audio and not reveal the timbre, but also generate audio closer to the prosody and timbre of the target speaker.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech conversion method and apparatus. Background Technology

[0002] Voice conversion (VC) technology aims to convert the timbre of a source audio to that of a target speaker while preserving the content and emotion of the source audio. VC systems are typically driven by the source audio, resulting in generated audio that is prosodicly closer to the source speaker, thus reducing its similarity to the target speaker. While existing text-to-speech (TTS) systems can generate the target speaker's timbre, they are less natural in their emotional expression than VC systems. Summary of the Invention

[0003] This application provides a speech conversion method, apparatus, computing device, storage medium, and computer program product to achieve both the preservation of the emotion of the source audio and the generation of audio that is closer to the rhythm and timbre of the target speaker.

[0004] The technical solution adopted in this application is as follows.

[0005] Firstly, this application provides a speech conversion method, including: Obtain the target object identifier and the preprocessed data extracted from the source audio. The preprocessed data includes phoneme-level text feature sequences related to audio content information, frame-level acoustic feature sequences related to audio sound information, and labeled duration sequences corresponding to the phoneme-level text feature sequences. Based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content information of interest. Based on phoneme-level text feature sequences, phoneme-level acoustic feature sequences, and target object identifiers, a phoneme-level latent variable sequence is generated that simultaneously carries content and acoustic information. Inputting the phoneme-level latent variable sequence into the trained duration prediction network yields the predicted duration sequence corresponding to the phoneme-level latent variable sequence. Based on the predicted duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence. Generate target audio corresponding to the target object identifier based on frame-level latent variable sequences; The duration prediction network is trained based on preprocessed data extracted from the target object identifier and its corresponding training audio.

[0006] Thus, this application aligns frame-level acoustic features to phoneme-level acoustic features, strips away the timbre information of the source speaker, ignores timbre details, avoids learning timbre from source acoustic features, avoids timbre leakage, forces focus on content-related acoustic features, and ensures that the generated phoneme-level acoustic features accurately reflect content information, avoiding interference from the prosody of the source audio. By combining the target object identifier to generate a phoneme-level latent variable sequence, the effective fusion of acoustic information of content and target object is achieved. At the same time, when expanding the latent variable sequence to generate the frame-level latent variable sequence, the predicted duration is obtained based on the duration prediction network. Since the input of the duration prediction network is also related to the target object identifier and is pre-trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio, the predicted duration is also related to the target object. The predicted duration sequence is more in line with the natural prosody of the target speaker, significantly improving timbre similarity. Thus, the prosody and timbre quality of the final synthesized target audio are closer to the target object.

[0007] In conjunction with the first aspect, in one possible implementation, based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content of interest. This includes: for each phoneme-level text feature in the phoneme-level text feature sequence, calculating a weight vector reflecting its relevance to each frame-level acoustic feature in the frame-level acoustic feature sequence; weighting the frame-level acoustic feature sequence according to the weight vector to obtain the first phoneme-level acoustic feature corresponding to the current phoneme-level text feature; for each frame interval corresponding to the labeled duration in the phoneme-level text feature sequence, aggregating all frame-level acoustic features belonging to that frame interval to obtain the second phoneme-level acoustic feature; and fusing all the first phoneme-level acoustic features with all the second phoneme-level acoustic features to obtain the phoneme-level acoustic feature sequence.

[0008] Thus, by calculating the weight vector and weighting the calculation, the attention weight tends to retain the frames related to the text, weaken the details related to the timbre, and selectively aggregate the relevant acoustic frames to obtain the local acoustic features related to the content; the aggregation processing directly compresses the frame-level features to the phoneme level, retains the overall acoustic statistical characteristics of the phoneme, that is, obtains the global acoustic statistical features, thereby preventing the attention weight from focusing too much on a few frames; through the complementarity of the attention mechanism and average pooling, the accuracy and completeness of the phoneme-level acoustic features after dimensionality reduction are ensured.

[0009] In conjunction with the first aspect, in one possible implementation, a phoneme-level latent variable sequence carrying both content and acoustic information is generated based on a phoneme-level text feature sequence, a phoneme-level acoustic feature sequence, and a target object identifier. This includes: fusing the phoneme-level text feature sequence and the phoneme-level acoustic feature sequence to obtain a phoneme-level joint feature sequence; embedding the target object identifier according to the dimensions of the phoneme-level text features and expanding it according to the length of the phoneme-level text feature sequence to obtain an identifier sequence; and fusing the identifier sequence with the phoneme-level joint feature sequence and inputting it into a trained phoneme-level latent variable extractor to obtain a phoneme-level latent variable sequence.

[0010] In this way, by integrating phoneme-level text and acoustic features, the latent variables are ensured to encode both content and acoustic information. By expanding the target object identifier and fusing it with joint features, speaker information injection is achieved, enabling the generated latent variable sequence to be deeply embedded in the target timbre features.

[0011] In conjunction with the first aspect, in one possible implementation, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended based on the prediction duration sequence to obtain a frame-level latent variable sequence. This includes: for each phoneme-level latent variable in the phoneme-level latent variable sequence, repeating and concatenating the prediction durations corresponding to the prediction durations in the prediction duration sequence to obtain a frame-level latent variable corresponding to the current phoneme-level latent variable; and arranging the frame-level latent variables corresponding to each phoneme-level latent variable in order according to the order of the phoneme-level latent variables in the phoneme-level latent variable sequence to form a frame-level latent variable sequence.

[0012] Thus, by extending the phoneme-level latent variables with the duration prediction sequence, the generated frame-level latent variable sequence better matches the duration characteristics of the target audio, thereby enhancing the prosodic similarity of the target audio.

[0013] In conjunction with the first aspect, in one possible implementation, the preprocessed data is obtained based on preprocessing operations, which include: acquiring an initial frame-level acoustic feature sequence and multiple initial phoneme-level text feature sequences; vectorizing the phoneme-level text features in each of the initial phoneme-level text feature sequences according to a preset dimension; fusing the vectorized multiple initial phoneme-level text feature sequences to obtain the phoneme-level text feature sequence; and vectorizing the frame-level acoustic features in the initial frame-level acoustic feature sequence according to the preset dimension to obtain the frame-level acoustic feature sequence.

[0014] In this way, multiple phoneme-level text feature sequences are fused into a single phoneme-level text feature sequence, enriching the dimensionality and information content of the input data and improving the quality and diversity of speech synthesis. By processing both sound and text features into vectors of the same dimension, multi-source features are mapped to a unified dimension, eliminating feature scale differences, ensuring the uniformity and processability of the input data, and facilitating subsequent feature processing and audio generation.

[0015] In conjunction with the first aspect, in one possible implementation, the preprocessed data is obtained based on a preprocessing operation, which further includes: extracting the initial frame-level acoustic feature sequence, as well as text feature sequence, phoneme feature sequence, tone feature sequence, and semantic feature sequence from the source audio; and extending the text feature sequence, tone feature sequence, and semantic feature sequence extracted from the source audio to align with the phoneme feature sequence to obtain the multiple initial phoneme-level text feature sequences.

[0016] In conjunction with the first aspect, in one possible implementation, generating target audio corresponding to the target object identifier based on the frame-level latent variable sequence includes: inputting the frame-level latent variable sequence into a trained frame-level acoustic feature generator to obtain a frame-level predicted acoustic feature sequence; inputting the frame-level predicted acoustic feature sequence into a trained distribution transformation module to convert it into a frame-level predicted latent representation sequence; and inputting the frame-level predicted latent representation sequence into a trained audio generator to generate the target audio.

[0017] By generating audio corresponding to the target object identifier through frame-level latent variable sequences, and combining it with a trained acoustic feature generator and decoder, the sound quality and speaker similarity of the generated audio are improved.

[0018] In one possible implementation, the method further includes: Obtain the target object identifier and the phoneme-level text feature sequence, frame-level acoustic feature sequence, and labeled duration sequence extracted from its corresponding training audio; Perform the first training operation: Based on the labeled duration sequence and the phoneme-level text feature sequence, align the frame-level acoustic feature sequence with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence containing content information; based on the phoneme-level text feature sequence, the phoneme-level acoustic feature sequence, and the target object identifier, generate a phoneme-level latent variable sequence that simultaneously carries content and acoustic information; input the phoneme-level latent variable sequence into the duration prediction network to obtain a predicted duration sequence corresponding to the phoneme-level latent variable sequence; train the duration prediction network based on a preset loss, wherein the preset loss includes the loss between the predicted duration sequence and the labeled duration sequence.

[0019] In one possible implementation, the method further includes: Obtain the frame-level standard latent representation sequence and standard spectral information extracted from the training audio corresponding to the target object identifier; The speech conversion model containing the duration prediction network is invoked to perform the conversion operation, which includes: after performing the first training operation, performing duration expansion on each phoneme-level latent variable in the phoneme-level latent variable sequence based on the target duration sequence to obtain a frame-level latent variable sequence; inputting the frame-level latent variable sequence into a frame-level acoustic feature generator to obtain a frame-level predicted acoustic feature sequence; inputting the frame-level standard latent representation sequence into the distribution transformation module to obtain a frame-level standard acoustic feature sequence; and inputting the target acoustic feature sequence into an audio generator to generate the target audio. The speech conversion model is trained based on the preset loss, which further includes: the loss between the target spectral information extracted from the target audio and the standard spectral information extracted from the training audio, and the loss between the frame-level predicted acoustic feature sequence and the frame-level standard acoustic feature sequence. During the training of the speech conversion model: the target duration sequence is the labeled duration sequence, and the target acoustic feature sequence is the frame-level standard latent representation sequence; during the use of the speech conversion model after training: the target duration sequence is the predicted duration sequence, and the target acoustic feature sequence is the frame-level predicted latent representation sequence, which is obtained by inputting the frame-level predicted acoustic feature sequence into the distribution transformation module.

[0020] In conjunction with the first aspect, in one possible implementation, the transformation operation further includes: inputting the frame-level latent variable sequence into a frame-level acoustic feature generator to obtain predicted spectral information; the preset loss further includes: the loss between the predicted spectral information and the standard spectral information; And / or, the training operation further includes: testing the target audio input discriminator to obtain an adversarial loss; the preset loss also includes the adversarial loss.

[0021] Secondly, this application provides a method for training a speech conversion model, the method comprising: The process involves obtaining frame-level standard latent representation sequences extracted from training audio, standard spectral information extracted from training audio, target object identifiers, and preprocessed data extracted from training audio corresponding to the target object identifiers. The preprocessed data includes phoneme-level text feature sequences related to audio content information, frame-level acoustic feature sequences related to audio sound information, and labeled duration sequences corresponding to phoneme-level text feature sequences. The speech conversion model is invoked to perform the conversion operation: Based on the labeled duration sequence and phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence that focuses on the content information; Based on the phoneme-level text feature sequence, phoneme-level acoustic feature sequence, and target object identifier, a phoneme-level latent variable sequence carrying both content and acoustic information is generated; The phoneme-level latent variable sequence is input into the duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence; Based on the target duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence; The frame-level latent variable sequence is input into the frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence; The frame-level standard latent representation sequence is input into the distribution transformation module to obtain the frame-level standard acoustic feature sequence; The target acoustic feature sequence is input into the audio generator to generate the target audio; The speech conversion model is trained based on a preset loss. The trained speech conversion model is used to output target audio based on the input target object identifier, phoneme-level text feature sequence, frame-level acoustic feature sequence, and standard duration sequence. The preset losses include: the loss between the predicted duration sequence and the labeled duration sequence, the loss between the target spectral information extracted from the target audio and the standard spectral information extracted from the training audio, and the loss between the frame-level predicted acoustic feature sequence and the frame-level standard acoustic feature sequence. During the training of the speech conversion model: the target duration sequence is the labeled duration sequence, and the target acoustic feature sequence is the frame-level standard acoustic feature sequence; during the use of the speech conversion model after training: the target duration sequence is the predicted duration sequence, and the target acoustic feature sequence is the frame-level predicted acoustic feature sequence.

[0022] Thus, this application provides an end-to-end training process (from feature alignment to audio generation) to ensure the stability of the model in practical applications. By using the loss between the predicted duration sequence and the labeled duration sequence, the predicted duration is ensured to be closer to the target object; by comparing the spectrum of the target audio with the standard spectrum, the quality of the generated audio is enhanced; the distribution alignment of the frame-level standard latent representation with the predicted latent representation (KL loss) forces the model to learn the acoustic feature distribution of the target speaker, reducing generation bias.

[0023] In conjunction with the second aspect, in one possible implementation, the transformation operation further includes: inputting the frame-level latent variable sequence into the frame-level acoustic feature generator to obtain the predicted spectrum information; the preset loss also includes: the loss between the predicted spectrum information and the standard spectrum information; And / or, the method further includes: examining the target audio input discriminator to obtain an adversarial loss; the preset loss also includes the adversarial loss.

[0024] Thus, by constraining the predicted spectrum with the standard spectrum, the generated frame-level acoustic features are ensured to closely resemble the characteristics of the target speaker in the frequency domain, thereby improving timbre similarity; the discriminator's adversarial loss (such as MSD / MPD) distinguishes between generated audio and real audio, thus enabling the decoder to generate more natural waveforms.

[0025] Thirdly, this application also provides a speech conversion apparatus. The speech conversion apparatus includes modules for performing the speech conversion method in the first aspect or any optional implementation of the first aspect. For example, the speech conversion apparatus includes: The first module is used to obtain the target object identifier and the preprocessed data extracted from the source audio. The preprocessed data includes a phoneme-level text feature sequence related to the audio content information, a frame-level acoustic feature sequence related to the audio sound information, and a labeled duration sequence corresponding to the phoneme-level text feature sequence. The second module is used to align the frame-level acoustic feature sequence with the phoneme-level text feature sequence based on the labeled duration sequence and the phoneme-level text feature sequence, so as to generate a phoneme-level acoustic feature sequence of the content information of interest. The third module is used to generate a phoneme-level latent variable sequence that carries both content and acoustic information based on the phoneme-level text feature sequence, the phoneme-level acoustic feature sequence, and the target object identifier. The fourth module is used to input the phoneme-level latent variable sequence into the trained duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence. The duration prediction network is trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio. The fifth module is used to perform duration extension on each phoneme-level latent variable in the phoneme-level latent variable sequence based on the predicted duration sequence to obtain a frame-level latent variable sequence. The sixth module is used to generate target audio corresponding to the target object identifier based on the frame-level latent variable sequence.

[0026] Fourthly, this application also provides a speech conversion model training apparatus. The speech conversion model training apparatus includes modules for performing the speech conversion model training apparatus in the second aspect or any optional implementation of the second aspect. For example, the speech conversion model training apparatus includes: The data acquisition module is used to acquire frame-level standard latent representation sequences extracted from training audio, standard spectral information extracted from training audio, target object identifiers, and preprocessed data extracted from training audio corresponding to the target object identifiers. The preprocessed data includes phoneme-level text feature sequences related to audio content information, frame-level acoustic feature sequences related to audio sound information, and labeled duration sequences corresponding to phoneme-level text feature sequences. The training module is used to call the speech conversion model to perform conversion operations: Based on the labeled duration sequence and phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence that focuses on the content information; Based on the phoneme-level text feature sequence, phoneme-level acoustic feature sequence, and target object identifier, a phoneme-level latent variable sequence carrying both content and acoustic information is generated; The phoneme-level latent variable sequence is input into the duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence; Based on the target duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence; The frame-level latent variable sequence is input into the frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence; The frame-level standard latent representation sequence is input into the distribution transformation module to obtain the frame-level standard acoustic feature sequence; The target acoustic feature sequence is input into the audio generator to generate the target audio; The training module is also used to train the speech conversion model based on a preset loss. The trained speech conversion model is used to output the target audio based on the input target object identifier, phoneme-level text feature sequence, frame-level acoustic feature sequence, and standard duration sequence. The preset losses include: the loss between the predicted duration sequence and the labeled duration sequence, the loss between the target spectral information extracted from the target audio and the standard spectral information extracted from the training audio, and the loss between the frame-level predicted acoustic feature sequence and the frame-level standard acoustic feature sequence. During the training of the speech conversion model: the target duration sequence is the labeled duration sequence, and the target acoustic feature sequence is the frame-level standard latent representation sequence; during the use of the speech conversion model after training: the target duration sequence is the predicted duration sequence, and the target acoustic feature sequence is the frame-level predicted latent representation sequence, which is obtained by inputting the frame-level predicted acoustic feature sequence into the distribution transformation module.

[0027] Fifthly, this application also provides a computing device. The computing device includes a memory and a processor. The memory stores computer programs or instructions that, when executed by the processor, implement the methods described in the first aspect or any possible implementation thereof, or implement the methods described in the second aspect or any possible implementation thereof.

[0028] Sixthly, this application provides a computer-readable storage medium. The storage medium stores a computer program or instructions that, when executed by a processor, implement the method described in the first aspect or any possible implementation thereof, or implement the method described in the second aspect or any possible implementation thereof.

[0029] Seventhly, this application provides a computer program product. The computer program product includes a computer program or instructions that, when executed by a processor, implement the method in the first aspect or any possible implementation thereof, or implement the method in the second aspect or any possible implementation thereof.

[0030] The beneficial effects of aspects two through seven above can be referred to in aspect one or any possible implementation of aspect one, and will not be elaborated here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

[0031] Other advantages, objectives and features of this application will be partly apparent from the description below, and partly understood by those skilled in the art through study and practice of this application. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0033] Figure 1a This is one of the schematic diagrams of the implementation environment of the speech conversion model training method provided in the embodiments of this application; Figure 1b This is the second schematic diagram of the implementation environment of the speech conversion model training method provided in the embodiments of this application; Figure 1c This is one of the schematic diagrams illustrating the implementation environment of the speech conversion method provided in the embodiments of this application; Figure 1d This is the second schematic diagram of the implementation environment of the speech conversion method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a speech conversion method in an exemplary embodiment of this application; Figure 3 This is one of the flowcharts for preprocessing operations; Figure 4 This is the second flowchart of the preprocessing operation; Figure 5 This is an example diagram showing the alignment of text feature sequences with phoneme feature sequences; Figure 6 This is an example diagram of aligning frame-level acoustic feature sequences with phoneme-level text feature sequences; Figure 7 This is a schematic diagram of the sub-steps of step S203; Figure 8 This is the schematic diagram of step S203; Figure 9 This is a schematic diagram of the sub-steps of step S205; Figure 10 This is a schematic diagram of the sub-steps of step S209; Figure 11 This is an example diagram of duration extension using phoneme-level latent variables; Figure 12 This is a schematic diagram of the sub-steps of step S2011; Figure 13a This is a diagram illustrating the training of a speech conversion model; Figure 13b This is a schematic diagram illustrating the use of the speech conversion model; Figure 14 This is a flowchart illustrating a speech conversion model training method according to an exemplary embodiment of this application; Figure 15 This is a flowchart of the conversion operation; Figure 16 This is a schematic block diagram of the speech conversion device provided in the embodiments of this application; Figure 17 This is a schematic block diagram of the speech conversion model training device provided in the embodiments of this application; Figure 18 This is a schematic block diagram of a computing device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0034] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0035] The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. In this application, "at least one" means one or more, and "more than one" means two or more. The terms "first," "second," and other ordinal terms used in this application may be used to describe various constituent elements, but these constituent elements are not limited by these terms. The purpose of using these terms is solely to distinguish one constituent element from others and should not be construed as indicating or implying relative importance. For example, without departing from the scope of this application, a first constituent element may be named a second constituent element, and similarly, a second constituent element may be named a first constituent element.

[0036] Before introducing the embodiments of this application, the technical terms involved in this application will be introduced first.

[0037] Timbre: refers to the characteristic of a sound, determined by the spectral characteristics of sound waves, and can distinguish the sounds produced by different speakers or musical instruments. In speech conversion, timbre conversion refers to changing the vocal characteristics of the source speaker to make them closer to the vocal characteristics of the target speaker.

[0038] Prosody: It can reflect the speaker's emotional state or speech style. Specifically, it includes features such as intonation, rhythm, stress, and speech rate, and is an important feature dimension of speech besides timbre and content. In speech conversion, prosodic modeling directly affects the naturalness and expressiveness of the generated speech.

[0039] Frame: In speech signal processing, a frame refers to a short segment of a continuous speech signal after it has been segmented into frames. A frame is the basic unit of speech analysis, used to extract short-time spectral features, etc.

[0040] A phoneme is the smallest unit of speech, defined based on the natural properties of sound. It is analyzed according to the articulation actions within a syllable; one action constitutes one phoneme. Phonemes are divided into two main categories: vowels and consonants. For example, the Chinese word "ma" contains two phonemes, / m / and / a / , while the English word "cat" contains three phonemes: / k / , / æ / , and / t / . Phonemes are crucial basic units in the field of speech processing.

[0041] Loss: In machine learning, loss refers to a measure of the difference between the model's predicted output and the true value, used to guide the optimization of model parameters. This application involves various loss functions, including temporal loss, spectral loss, adversarial loss, and KL divergence loss, which together constrain the model to learn the speech features of the target speaker.

[0042] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0043] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large-scale speech processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0044] Key technologies in speech technology include Automatic Speech Recognition (ASR) and Voice Conversion (VC). Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods. Large-scale modeling has revolutionized speech technology; pre-trained models such as WavLM and UniSpeech, which utilize the Transformer architecture, possess strong generalization and versatility, enabling them to excel in various speech processing tasks.

[0045] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0046] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0047] The solutions provided in this application relate to artificial intelligence-based voice conversion (VC) technology, which will be described in detail in subsequent embodiments.

[0048] This application provides an innovative speech conversion scheme that aims to accurately convert the timbre features of the target speaker while preserving the content information and emotional expression of the source audio. The application scenarios for this speech conversion technology include: 1. Dubbing Adjustment for Film and Television Works: In the post-production of film and television works, when it is necessary to adjust or replace the dubbing of certain characters, the solution proposed in this application can generate a voice similar in style to the original character's dubbing based on the target actor's timbre and rhythm, thereby achieving seamless replacement. This is especially useful when actors cannot perform re-dubbing or when it is desired to change certain lines.

[0049] 2. Personalized Voice Assistant: With the increasing popularity of voice assistants, users desire to interact using their own voice or the voice of a specific person. The solution proposed in this application can convert the voice assistant's voice into the user's desired tone based on sample audio provided by the user, offering a more personalized experience.

[0050] 3. Voice Navigation and Public Address: In voice navigation and public address systems, the solution proposed in this application can be used to provide voice prompts that are more in line with local customs for people from different regions and cultural backgrounds. For example, based on the language characteristics and rhythms of different regions, more localized voice navigation information can be generated.

[0051] 4. Music Production: In music creation, lyricists and composers can use the solution provided in this application to convert lyrics into the target singer's timbre for demonstration, helping to evaluate the song's effect. This allows for initial song demonstration and adjustments without the singer's involvement in recording.

[0052] 5. Education and Language Learning: In language learning software, the solution proposed in this application can convert standard language teaching materials into the local accent or intonation of the target language, helping learners better adapt to the language characteristics of different regions and improve learning outcomes.

[0053] 6. Virtual Character Voice Acting: In game and animation production, the voice acting for virtual characters needs to match the character's personality and emotions. The solution proposed in this application can generate voices that match the character's characteristics based on the character settings, thereby enhancing the immersion and expressiveness of the work.

[0054] 7. Medical and Rehabilitation Applications: In the rehabilitation of speech disorders, the solution proposed in this application can help patients generate speech with the target timbre, assist in the speech training and rehabilitation process, and provide patients with training materials that are closer to normal speech.

[0055] Before introducing the embodiments of this application, the relevant technologies involved in this application will be introduced first.

[0056] Voice conversion (VC) technology aims to convert the timbre of the source audio into that of the target speaker while preserving the content and emotion of the source audio. VC systems are typically driven by the source audio, resulting in generated audio that is prosodicly closer to the source speaker, thus reducing the similarity to the target speaker. Furthermore, the acoustic features used in VC systems may suffer from timbre leakage, further affecting the conversion results. While existing text-to-speech (TTS) systems can generate the target speaker's timbre, their emotional expression is not as natural as that of VC systems.

[0057] In summary, some related technologies have the problem that the audio generated by speech conversion is closer to the prosody of the source speaker, thus reducing the similarity to the target speaker, and may also have the technical problem of leaking the timbre of the source speaker.

[0058] The following describes one or more exemplary operating environments to facilitate a clearer understanding of the functions and intentions of the various implementation methods in this application. Figures 1-2 are schematic diagrams of the implementation environment of the method provided in this application embodiment. The method provided in this application embodiment can be applied in the environment of Figure 1, which includes a client 101, a server 102, and a database 103. The client 101 and server 102 establish a communication connection via wired or wireless means, and the database 103 can establish a communication connection with the server 102 via wired or wireless means. The database 103 can be a cloud database or a local database. The device form of the client 101 includes, but is not limited to, smartphones, tablets (such as iOS / Android devices), PCs (such as iMacs), and smart wearable devices. The software carrier of the client 101 can be a native application (App) or its functional sub-modules, or a web browser. The server 102 should be interpreted broadly as an entity capable of responding to external requests and providing data, resources, or services. Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but it is not limited to these. This application does not limit the number of servers.

[0059] This application provides a method for training a speech conversion model, as well as a speech conversion method. The speech conversion method of this application can be implemented based on a trained speech conversion model.

[0060] refer to Figure 1a and Figure 1b The model training method of this application can be implemented by server 102, that is, the code for training the model can be deployed in server 102. Database 103 stores training data (target object identifier, and source audio or preprocessed data of source audio), which can be generated and uploaded by the terminal device or imported locally. When the code runs, based on the preprocessed data of the training audio of the target object and the target object identifier obtained from client 101 or database 103, the model training method of this application is executed, such as... Figure 1a Alternatively, database 103 can be used to store training audio data, which can be generated and uploaded by the terminal device or imported locally. During code execution, based on the training audio and target object identifier obtained from client 101 or database 103, server 102 extracts preprocessed data from the training audio before executing the model training method of this application, such as... Figure 1b .

[0061] The trained model can be deployed on server 102 and provide speech conversion services to users. (See reference...) Figure 1c and Figure 1d The speech conversion method of this application is implemented by server 102, meaning that speech conversion code can be deployed in server 102. When the code runs, based on the preprocessed data of the source audio obtained from client 101 and the target object identifier, the speech conversion method of this application is executed, and the target audio is returned to client 101, such as... Figure 1a Alternatively, during code execution, based on the source audio and target object identifier sent by client 101, server 102 extracts preprocessed data from the source audio and then executes the speech conversion method of this application to obtain the target audio, which is then returned to client 101, such as... Figure 1d .

[0062] It is understood that the speech conversion model can be trained and deployed in both public and private environments. For example, the logic code of the model training method provided in this application can be stored on a private cloud server 102, and a private database 103 stores private data. When training the model, the private cloud server 102 can obtain the corresponding data from the private cloud customer's (the party providing the speech conversion service using the private deployment service) private database 103 to train the model, enabling training and rapid deployment of the model in a private environment. Simultaneously, the private data does not pass through third parties, ensuring its security. The trained model can be deployed on the private cloud server 102 to provide private speech conversion services to the customers' users. Various smart devices (such as smart robots and smartphones) can access the private service. After access, the smart device can send the audio to be converted to the private cloud server. The private cloud server calls the trained model to perform rapid speech conversion and returns the target audio to the user device.

[0063] The technical solutions of this application are described below through several embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0064] refer to Figure 2 In a first aspect, embodiments of this application provide a speech conversion method, the method comprising S201-S2011: S201: Obtain the target object identifier and the preprocessed data extracted from the source audio. The preprocessed data includes phoneme-level text feature sequences related to audio content information, frame-level acoustic feature sequences related to audio sound information, and annotation duration sequences corresponding to the phoneme-level text feature sequences. S203: Based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content of interest. S205: Based on phoneme-level text feature sequences, phoneme-level acoustic feature sequences, and target object identifiers, generate phoneme-level latent variable sequences that simultaneously carry content and acoustic information; S207: Input the phoneme-level latent variable sequence into the trained duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence; The duration prediction network is trained based on preprocessed data extracted from the target object identifier and its corresponding training audio.

[0065] S209: Based on the predicted duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence. S2011: Generate target audio corresponding to the target object identifier based on frame-level latent variable sequences.

[0066] Thus, this application aligns frame-level acoustic features to phoneme-level acoustic features, strips away the timbre information of the source speaker, ignores timbre details, avoids learning timbre from source acoustic features, avoids timbre leakage, forces focus on content-related acoustic features, and ensures that the generated phoneme-level acoustic features accurately reflect content information, avoiding interference from the prosody of the source audio. By combining the target object identifier to generate a phoneme-level latent variable sequence, the effective fusion of acoustic information of content and target object is achieved. At the same time, when expanding the latent variable sequence to generate the frame-level latent variable sequence, the predicted duration is obtained based on the duration prediction network. Since the input of the duration prediction network is also related to the target object identifier and is pre-trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio, the predicted duration is also related to the target object. The predicted duration sequence is more in line with the natural prosody of the target speaker, significantly improving timbre similarity. Thus, the prosody and timbre quality of the final synthesized target audio are closer to the target object.

[0067] The above methods can be implemented based on a trained speech conversion model, such as... Figure 13b As shown, the following is combined Figure 2-12 The steps in steps S201-S2011 and other optional steps are described in detail.

[0068] Regarding step S201 : Obtain the target object identifier and the preprocessed data extracted from the source audio; Users can input source audio into the speech-to-text software provided in the client and select the target audio they wish to convert. For example... Figure 1a The client-side front-end processing program performs preprocessing operations, obtains the preprocessing results, and sends the target object identifier and preprocessed data to the server. Of course, as... Figure 1b Alternatively, the client can send the target object identifier and source audio to the server, which will then preprocess the source audio to obtain preprocessed data.

[0069] The target object identifier can be a unique identifier for each target object, and its specific form is not limited. In this embodiment, the target object identifier is represented by a simple number. For example, Zhang San's target object identifier is 1, Li Si's target object identifier is 2, and Wang Wu's target object identifier is 3. It is understood that the target object identifier is only used for program identification and is invisible to the user. The front-end user sees the specific description of the target object identifier, such as the name "Zhang San" or "Li Si". When the user selects "Zhang San", it means that the target object identifier sent to the server is 1.

[0070] Preprocess the data, including a phoneme-level text feature sequence related to audio content information, a frame-level acoustic feature sequence related to audio sound information, and an annotation duration sequence corresponding to the phoneme-level text feature sequence.

[0071] In some embodiments, the preprocessed data is obtained by performing preprocessing operations on the source audio. The preprocessing operations mainly include two-level processing: 1) The first preprocessing: Extract the initial frame-level acoustic feature sequence, multiple initial phoneme-level text feature sequences from the source audio, and the lengths of the multiple initial phoneme-level text feature sequences are the same. Exemplarily, referring to FIG. 13, the multiple phoneme-level text feature sequences are respectively: a text feature sequence (words), a phoneme feature sequence (phones), a tone feature sequence (tones), and a semantic feature sequence (bert). It should be noted that in this application, the length of a sequence refers to the number of elements that make up the sequence. 2) The second preprocessing: After obtaining the initial frame-level acoustic feature sequence and multiple initial phoneme-level text feature sequences in the first preprocessing, all the features in the sequences are uniformly processed into vectors of the same length, so as to obtain the phoneme-level text feature sequence and the frame-level acoustic feature sequence.

[0072] Refer to Figure 3 , the preprocessing operations include S301-S303, and these steps mainly implement the first preprocessing mentioned above: S301: Extract the initial frame-level acoustic feature sequence, text feature sequence, phoneme feature sequence, tone feature sequence, and semantic feature sequence from the source audio; The type of the audio is not limited, and it can be the speech of at least one language / dialect, such as Mandarin, Cantonese, English, Russian, etc.

[0073] The initial frame-level acoustic feature sequence refers to a sequence composed of acoustic features. Similarly, the text feature sequence, phoneme feature sequence, tone feature sequence, and semantic feature sequence respectively refer to sequences composed of text features, phoneme features, tone features, and semantic features. The following explains each feature.

[0074] The acoustic features can be obtained by frame-dividing the audio, and each frame corresponds to an acoustic feature. A frame is the basic unit of speech analysis and is used to extract short-time spectral features, etc. It can be the decoupled self-supervised learning (SSL) acoustic feature vec extracted from the audio.

[0075] The text feature (word) refers to the representation at the character or word level. Taking Mandarin as an example for illustration. If the source audio is data.wav corresponding to "delicious", the text features can be extracted from the audio through an automatic speech recognition (ASR) model. For example, the text features of data.wav include {"good", "eat"}.

[0076] A phone is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phone. For example, the phones in Mandarin are divided into two major categories: vowels and consonants. The phone features of data.wav include {h, ao, ch, i}.

[0077] Tone features (tone) are used to indicate the pitch changes of syllables during pronunciation. A tone annotation tool can be used to extract the tone features of each syllable. For example, the tone features of data.wav include the tones of Chinese pinyin {3, 1}.

[0078] Semantic features are the feature representations that reflect the deep semantic content of the text. They are used to enhance the expressiveness and naturalness of speech synthesis. For example, after extracting the text features from data.wav, the semantic features bert in the text features can be further extracted through a pre-trained language model (such as BERT). The semantic features of data.wav may include {delicious}. Semantic features not only include the literal meaning but also cover deep information such as context association and emotional tendency. For example, the word-level semantic representation of "I am Zhang San" is {I, am, Zhang San}, and the sentence-level semantic representation is the whole sentence {I am Zhang San}.

[0079] S303: Expand the text feature sequence, tone feature sequence, and semantic feature sequence extracted from the source audio to align with the phone feature sequence, and obtain multiple initial phone-level text feature sequences; The alignment mentioned in this application means that the number of features remains the same. That is, the number of text features, tone features, and semantic features is the same as the number of phone features.

[0080] For example, the W2P (Word-to-Phoneme) can be used to align the text features and phone features. The W2P feature represents the number of phones associated with each word. In Chinese, one word may correspond to multiple phones. Through a forced alignment tool (such as Montreal Forced Aligner), phoneme duration annotations are generated to expand the word-level features (M words) to the phone level (N phonemes, N ≥ M). For example: The initially extracted text features of data.wav include {good, eat}, and the number of features is M = 2. Aligning to the phone features {h, ao, ch, i}, the number of features is N = 4. The text feature of "good" needs to be copied to the two phonemes "h" and "ao", and the text feature of "eat" needs to be copied to the two phonemes "ch" and "i", as Figure 5 shown. The expansion of tone features and semantic features is the same.

[0081] All aligned text features (word) constitute the text feature sequence words, all aligned phoneme features (phone) constitute the phoneme feature sequence phones, all aligned tone features (tone) constitute the tone feature sequence tones, and all aligned semantic features (bert) constitute the semantic feature sequence bert.

[0082] refer to Figure 4 The preprocessing operation includes steps S401-S403, which mainly implement the second preprocessing mentioned above: S401: Vectorize the phoneme-level text features in each initial phoneme-level text feature sequence according to a preset dimension, and fuse the vectorized initial phoneme-level text feature sequences to obtain a phoneme-level text feature sequence. The text feature sequence, phoneme feature sequence, tone feature sequence, and semantic feature sequence aligned in step S303 are then embedded according to a preset dimension D. This step can be performed by a text encoder, which receives four sequences: words, phones, tones, and bert. Specifically, each aligned text feature, phoneme feature, tone feature, and semantic feature can be converted into a vector of the same preset dimension (denoted as D) through independent embedding, ensuring that they can be added or concatenated later. That is, each aligned text feature (word), phoneme feature (phone), tone feature (tone), and semantic feature (bert) (if the original dimension ≠ D) is represented by a 1*D vector through embedding, resulting in a text feature Cword, a phoneme feature Cphone, a tone feature Ctone, and a semantic feature Cbert. Thus, the sequence of text features consisting of N 1*D vectors corresponding to N text features Cword, the sequence of phoneme features consisting of N 1*D vectors corresponding to N phoneme features Cphone, the sequence of tone features consisting of N 1*D vectors corresponding to N tone features Ctone, and the sequence of semantic features consisting of N 1*D vectors corresponding to N semantic features Cbert are all N*D matrices.

[0083] Continuing the example above, assuming the lengths of the text feature sequence, phoneme feature sequence, tone feature sequence, and semantic feature sequence are all N, the text feature sequence Cwords includes N text features Cword, the phoneme feature sequence Cphones includes N phoneme features Cphone, the tone feature sequence Ctones includes N tone features Ctone, and the semantic feature sequence Cbert includes N semantic features Cbert, the fusion of Cwords, Cphones, Ctones, and Cbert features can be achieved by the text encoder in the speech conversion model. The text encoder can be some Transformer layers (without changing the dimensions). The text encoder is a structure within the speech conversion model and is trained synchronously with the training of the speech conversion model. The fusion here can be based on the stacking of weights K1-K4, for example: Ctext=K1*Cword+K2*Cphone+K3*Ctone+K4*Cbert; Each feature in Cword, Cphone, Ctone, and Cbert is an embedding vector with the same dimension, D. Therefore, each text feature, Ctext, is also an embedding vector with dimension D. During model training, the initial values ​​of K1-K4 can be set to 1.

[0084] In this way, the sequences of text, phonemes, tones, and semantic features are fused into phoneme-level text feature sequences, which enriches the dimensions and information content of the input data and improves the quality and diversity of speech synthesis.

[0085] S403: Vectorize the frame-level acoustic features in the frame-level acoustic feature sequence according to a preset dimension.

[0086] The acoustic features (Vec) extracted from the audio in step S301 are generally frame-level low-dimensional features (such as ContentVec, Mel spectrum, etc.), typically a T*F dimensional matrix (T is the number of frames, F is the feature dimension, F≠D). For example, the ContentVec feature might be a T×256 dimensional frame-level vector sequence. For subsequent fusion with text features, each acoustic feature needs to be converted to D dimensions through linear projection / convolution. The resulting frame-level acoustic feature sequence after dimension conversion is then a T*D matrix.

[0087] The labeled duration sequence corresponding to the phoneme-level text feature sequence can be obtained using existing forced alignment models (such as MFA). For example, given the input "I am a person", the forced alignment model will output a time label indicating the time from frame X1 to frame X2 for each phoneme. In this way, the number of frames (i.e., labeled duration) corresponding to each phoneme feature can be obtained, and the number of frames corresponding to all phoneme features constitutes the labeled duration sequence.

[0088] Regarding step S203 Based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content of interest. As mentioned earlier, the alignment referred to in this application means that the number of features remains consistent. Similarly, the alignment of the frame-level acoustic feature sequence and the phoneme-level text feature sequence here means that the number of frame-level acoustic features in the frame-level acoustic feature sequence is consistent with the number of phoneme-level text features in the phoneme-level text feature sequence.

[0089] refer to Figure 6 The phoneme-level acoustic features in the final desired phoneme-level acoustic feature sequence are temporarily denoted as Cssl. Assume the length of the frame-level acoustic feature sequence is T=8, as shown in frames 1-8. The lengths of the phoneme-level text feature sequence and the annotation duration sequence are N=3. The phoneme-level text feature sequence is specifically {Ctext1, Ctext2, Ctext3}, and the annotation duration sequence is specifically {3,3,2}. This means that the first 3 frames are aligned with Ctext1 to obtain the phoneme-level acoustic feature Cssl1; the next 3 frames are aligned with Ctext2 to obtain the phoneme-level acoustic feature Cssl2; and the next 2 frames are aligned with Ctext3 to obtain the phoneme-level acoustic feature Cssl3.

[0090] refer to Figure 7 In some embodiments, step S203 specifically includes steps S701-S705: S701: For each phoneme-level text feature in the phoneme-level text feature sequence, calculate a weight vector that reflects its correlation with each frame-level acoustic feature in the frame-level acoustic feature sequence. Calculate the weighted frame-level acoustic feature sequence based on the weight vector to obtain the first phoneme-level acoustic feature corresponding to the current phoneme-level text feature. Specifically, this can be achieved based on attention mechanisms. For example, see reference... Figure 8 The phoneme-level text feature sequence (N*D matrix) is used as the query, and the frame-level acoustic feature sequence (T*D matrix) is used as the key and value. The QKV attention mechanism calculates the relevance between Q and K for each Q using a dot product, resulting in a 1*T weight vector. Then, for all Qs, the weight vectors are used to weight and sum V, and the average is obtained to obtain the first phoneme-level acoustic feature sequence (N*D matrix) composed of the first phoneme-level acoustic features (Catt).

[0091] S703: For each frame interval corresponding to the labeled duration in the phoneme-level text feature sequence, aggregate all frame-level acoustic features belonging to the same frame interval to obtain the second phoneme-level acoustic feature. Aggregation processing can employ pooling operations, including average pooling and max pooling, or it can use statistical methods (such as mean, median, maximum, minimum, etc.) to merge multiple features into one feature. In this embodiment, averaging processing (specifically pooling averaging) is used. Without departing from the spirit of this invention, the averaging processing can be replaced by any method that can aggregate the frame intervals corresponding to the labeled duration into a phoneme-level representation.

[0092] For example, combining Figure 6 , 8 The specific time-annotated sequence is {3,3,2}, which corresponds to three frame intervals: frame 1-frame 3, frame 4-frame 6, and frame 7-frame 8. For each frame interval, the second phoneme-level acoustic feature Cavg can be obtained by averaging all frame-level acoustic features belonging to that frame interval based on pooling (i.e., averaging the sum of frames 1-frame 3 to obtain a second phoneme-level acoustic feature Cavg1, averaging the sum of frames 4-frame 6 to obtain a second phoneme-level acoustic feature Cavg2, and averaging the sum of frames 7-frame 8 to obtain a second phoneme-level acoustic feature Cavg3). All the second phoneme-level acoustic features Cavg constitute the second phoneme-level acoustic feature sequence (N*D matrix).

[0093] S705: Fuse all first-phoneme-level acoustic features with all second-phoneme-level acoustic features to obtain a phoneme-level acoustic feature sequence.

[0094] refer to Figure 8 The first phoneme-level acoustic feature (Catt) and all the second phoneme-level acoustic features (Cavg) can be weighted and then added together, with the sum of their weights being 1. For example, if the weight of the first phoneme-level acoustic feature (Catt) is α and the weight of the second phoneme-level acoustic feature (Cavg) is β, then the fused phoneme-level acoustic feature is the final required CSSL: CSSL = α * Catt + β * Cavg, where α + β = 1. The proportions of Catt and Cavg can be adjusted by adjusting the values ​​of α and β.

[0095] Thus, step S701 calculates the weighted summation of weight vectors, with attention weights tending to retain text-related frames and weaken timbre-related details, selectively aggregating relevant acoustic frames to obtain content-related local acoustic features; step S703, average pooling, directly compresses frame-level features to the phoneme level, preserving the overall acoustic statistical characteristics of the phonemes, i.e., obtaining global acoustic statistical features, thereby preventing attention weights from over-focusing on a few frames; through the complementarity of the attention mechanism and average pooling, the accuracy and completeness of the dimensionality-reduced phoneme-level acoustic features are ensured.

[0096] Combination Figure 13bThis step can be achieved using the SSL encoder within the speech conversion model. The SSL encoder is an internal structure of the speech conversion model that is trained synchronously with the training of the speech conversion model.

[0097] Regarding step S205 Based on phoneme-level text feature sequences, phoneme-level acoustic feature sequences, and target object identifiers, a phoneme-level latent variable sequence carrying both content and acoustic information is generated. refer to Figure 13b , Figure 9 Step S205 specifically includes steps S901-S905: S901: Based on phoneme-level text feature sequences and phoneme-level acoustic feature sequences, a phoneme-level joint feature sequence is obtained by fusing them. refer to Figure 8 Phoneme-level joint features are denoted by c, phoneme-level text features by Ctext, and phoneme-level acoustic features by Cssl. Then, phoneme-level text feature sequences and phoneme-level acoustic feature sequences can be superimposed based on weights K5-K6, for example: c = K5 * Ctext + K6 * Cssl; During model training, the initial values ​​of K5 and K6 can be set to 1.

[0098] Both phoneme-level text feature sequences and phoneme-level acoustic feature sequences are N*D matrices, therefore the phoneme-level joint feature sequence is also an N*D matrix.

[0099] S903: Embed the target object identifier according to the dimension of phoneme-level text features, and expand it according to the length of the phoneme-level text feature sequence to obtain the identifier sequence; For example, if the user selects Wang Wu as the audio object to be synthesized, and the target object identifier of Wang Wu is 3, then the dimension of 3 can be expanded to D dimensions through an embedding layer to become a 1*D vector. Then, it is repeated according to the length (i.e. N) of the phoneme-level text feature sequence to obtain the matrix, i.e. the identifier sequence. The matrix, i.e. the identifier sequence, is composed of N 1*D vectors, which is actually an N*D matrix.

[0100] S905: After fusing the identifier sequence with the phoneme-level joint feature sequence, input the fused sequence into the trained phoneme-level latent variable extractor to obtain the phoneme-level latent variable sequence.

[0101] refer to Figure 8 Let c represent the phoneme-level joint features, spk represent the identifiers, and xd represent the phoneme-level latent variables. Then, the identifier sequence and the phoneme-level joint features can be superimposed based on weights K7-K8, for example: xd = K7*c + K8*spk; The phoneme-level latent variable extractor is an internal structure of the speech conversion model, which is trained synchronously with the training of the speech conversion model. During model training, the initial values ​​of K7-K8 can be set to 1.

[0102] Since both the phoneme-level text feature sequence and the phoneme-level acoustic feature sequence are N*D matrices, and the phoneme-level latent variable extractor is just some Transformer layers (without changing the dimensions), the phoneme-level latent variable sequence is also an N*D matrix.

[0103] In this way, by integrating phoneme-level text and acoustic features, the latent variables are ensured to encode both content and acoustic information. By expanding the target object identifier and fusing it with joint features, speaker information injection is achieved, enabling the generated latent variable sequence to be deeply embedded in the target timbre features.

[0104] Regarding step S207 Input the phoneme-level latent variable sequence into the trained duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence; The duration prediction network is trained based on preprocessed data extracted from the target object identifier and its corresponding training audio. More specifically, it is trained by using the phoneme-level latent variable sequence obtained from the preprocessed data corresponding to the target object identifier as the input to the duration prediction network and the labeled duration sequence corresponding to the target object identifier as the output.

[0105] In some embodiments, the duration prediction network can be trained separately, and the training process is as follows: 1) Obtain the target object identifier and the phoneme-level text feature sequence, frame-level acoustic feature sequence, and labeled duration sequence extracted from its corresponding training audio; 2) Perform the first training operation, including: aligning the frame-level acoustic feature sequence with the phoneme-level text feature sequence based on the labeled duration sequence and the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence containing content information; generating a phoneme-level latent variable sequence that carries both content and acoustic information based on the phoneme-level text feature sequence, the phoneme-level acoustic feature sequence, and the target object identifier; inputting the phoneme-level latent variable sequence into the duration prediction network to obtain a predicted duration sequence corresponding to the phoneme-level latent variable sequence; and training the duration prediction network based on a preset loss, wherein the preset loss includes the loss between the predicted duration sequence and the labeled duration sequence.

[0106] More specific details about obtaining the phoneme-level latent variable sequence in the first training operation can be found in the aforementioned steps S201-S205. The only difference is that the preprocessed data is the preprocessed data corresponding to the target object identifier.

[0107] In some embodiments, the duration prediction network is actually a structure within the speech conversion model, trained synchronously with the speech conversion model. The specific training process is detailed in subsequent model training method embodiments.

[0108] The length of the predicted duration sequence is consistent with the length of the phoneme-level latent variable sequence.

[0109] Regarding step S209 Based on the predicted duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence. This step can be done Figure 13b The length adjuster is implemented in the code.

[0110] refer to Figure 10 Step S209 specifically includes S1001-S1003: S1001: For each phoneme-level latent variable in the phoneme-level latent variable sequence, repeat and concatenate the prediction durations in the prediction duration sequence to obtain the frame-level latent variable corresponding to the current phoneme-level latent variable. S1003: Based on the order of each phoneme-level latent variable in the phoneme-level latent variable sequence, arrange the frame-level latent variables corresponding to each phoneme-level latent variable in order to form a frame-level latent variable sequence.

[0111] like Figure 8 In this context, xf represents a frame-level latent variable, see reference [link / reference] Figure 11 Assuming the length N of the phoneme-level latent variable sequence is 3, specifically {xd1, xd2, xd3}, and the length N of the prediction duration sequence is 3, specifically {2, 3, 4}, then xd1 is repeated according to the corresponding prediction duration 2 and concatenated to obtain the frame-level latent variable xf1; xd2 is repeated according to the corresponding prediction duration 3 and concatenated to obtain the frame-level latent variable xf2; and xd3 is repeated according to the corresponding prediction duration 4 and concatenated to obtain the frame-level latent variable xf3. It should be noted that the repetition of phoneme-level latent variables can be processed in parallel, that is, multiple phoneme-level latent variables can be repeated simultaneously. After parallel repetition, they are arranged sequentially according to the order of the phoneme-level latent variables in the corresponding phoneme-level latent variable sequence to form the frame-level latent variable sequence.

[0112] Thus, by extending the phoneme-level latent variables with the duration prediction sequence, the generated frame-level latent variable sequence better matches the duration characteristics of the target audio, thereby enhancing the prosodic similarity of the target audio.

[0113] Regarding step S2011 Generate target audio corresponding to the target object identifier based on frame-level latent variable sequences.

[0114] refer to Figure 12Step S2011 specifically includes S1201-S1205: S1201: Input the frame-level latent variable sequence into the trained frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence; S1203: Input the frame-level predicted acoustic feature sequence into the trained distribution transformation module and convert it into a frame-level predicted latent representation sequence; S1205: Input the frame-level predicted latent representation sequence into the trained audio generator to generate the target audio.

[0115] refer to Figure 13b The frame-level acoustic feature generator can specifically be a MEL encoder, and the distribution transformation module can specifically be a FLOW normalized stream module. The MEL encoder, FLOW normalized stream module, and audio generator are all internal structures of the speech conversion model, trained synchronously with the speech conversion model. The specific training process will be detailed in subsequent model training method embodiments.

[0116] In summary, this application embodiment aligns frame-level acoustic features to phoneme-level acoustic features, strips away the timbre information of the source speaker, ignores timbre details, avoids learning timbre from source acoustic features, avoids timbre leakage, forces focus on content-related acoustic features, ensures that the generated phoneme-level acoustic features accurately reflect content information, and avoids interference from the prosody of the source audio. By combining the target object identifier to generate a phoneme-level latent variable sequence, the effective fusion of acoustic information of content and target object is achieved. At the same time, when expanding the latent variable sequence to generate a frame-level latent variable sequence, the predicted duration is obtained based on a duration prediction network. Since the input of the duration prediction network is also related to the target object identifier and is pre-trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio, the predicted duration is also related to the target object. The predicted duration sequence is more in line with the natural prosody of the target speaker, significantly improving timbre similarity. Thus, the prosody and timbre quality of the final synthesized target audio are closer to the target object.

[0117] It should be noted that this specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In practice, when the method program is executed, it can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0118] As mentioned earlier, in some embodiments, the duration prediction network is actually a structure within the speech conversion model, trained synchronously with the speech conversion model. The following describes a training method for the speech conversion model. (Reference) Figure 14Secondly, embodiments of this application provide a speech conversion model training method, which can train the speech conversion model using an end-to-end training approach, as described above. Figure 13a In some embodiments, the training speech conversion model during training is as follows: Figure 13a As shown in dashed box 1301, the speech conversion model obtained after training is shown in dashed box 1302. Figure 13a The training of the speech conversion model includes a text encoder, an acoustic encoder (SSL encoder), a phoneme and variable extractor, a duration prediction network, a length adjuster, a frame-level acoustic feature generator (e.g., a MEL encoder), a distribution transformation module (Normalized Flow module), and a decoder (including an audio generator and a discriminator). Training methods include S1401-S1405: S1401: Obtain the frame-level standard latent representation sequence extracted from the training audio, the standard spectral information extracted from the training audio, the target object identifier, and the preprocessed data extracted from the training audio corresponding to the target object identifier. The preprocessed data includes the phoneme-level text feature sequence related to the audio content information, the frame-level acoustic feature sequence related to the audio sound information, and the annotation duration sequence corresponding to the phoneme-level text feature sequence. The preprocessed data is obtained by preprocessing the training audio, as described in the aforementioned speech conversion method implementation section, and will not be repeated here.

[0119] The frame-level standard latent representation sequence z0 can be extracted using a posterior encoder. The posterior encoder consists of a linear spectrum encoder and a latent representation encoder. The linear spectrum encoder converts the input training audio into a linear spectrum (lin), and the latent representation encoder uses this linear spectrum to compress and transform the audio to obtain the frame-level standard latent representation sequence z0. The posterior encoder is no longer needed when using this speech-to-text model subsequently; it is only used during the model training phase.

[0120] Standard spectral information can be Mel information (mel0) extracted from the training audio.

[0121] It should be noted that the data obtained in this step can be divided into multiple parts according to the different target object identifiers, that is, each target object identifier corresponds to one set of data, and multiple batches can be used for joint training.

[0122] S1403: Call the speech conversion model to perform the conversion operation; Combination Figure 13a ,refer to Figure 15 The conversion operation specifically includes: S1501: Based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content information of interest. This step can be achieved using an acoustic encoder.

[0123] S1503: Based on the phoneme-level text feature sequence output by the text encoder, the phoneme-level acoustic feature sequence output by the acoustic encoder, and the target object identifier, generate a phoneme-level latent variable sequence that carries both content and acoustic information. This step can be implemented using a phoneme-level latent variable extractor.

[0124] S1505: Input the phoneme-level latent variable sequence into the duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence; S1507: Based on the target duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence. This step can be achieved using a length adjuster.

[0125] S1509: Input the frame-level latent variable sequence into the frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence; S1511: Input the frame-level standard latent representation sequence into the distribution transformation module to obtain the frame-level standard acoustic feature sequence; S1513: Input the target acoustic feature sequence into the audio generator to generate the target audio.

[0126] Most of the content of the first operation above is the same as the aforementioned speech conversion method; only the differences are described below: (1) Output processing of the duration prediction network and input processing of the length regulator: contrast Figure 13b , Figure 13a In step S1507, the target duration sequence input to the length regulator during the training process of the speech conversion model is the labeled duration sequence dur0, and the target duration sequence input to the length regulator during the use process after the speech conversion model is trained is the predicted duration sequence dur1.

[0127] (2) Output of the frame-level acoustic feature generator: In some embodiments, the conversion operation S1509 specifically includes: inputting the frame-level latent variable sequence into the frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence and predicted spectral information. (See reference) Figure 13aDuring training, the speech conversion model's mel encoder simultaneously outputs the frame-level predicted acoustic feature sequence x and the predicted spectral information mel1. In contrast, after training, during actual use, the mel encoder only outputs the frame-level predicted acoustic feature sequence x and no longer predicts the spectral information mel1. In this case, refer to... Figure 13b .

[0128] (3) Calculation direction processing of the distribution transformation module: The distributed transformation module is a normalized flow module that supports bidirectional computation. During the training of the speech conversion model, the distributed transformation module performs the second transformation operation, achieving forward computation (frame-level standard latent representation sequence z0 → frame-level standard acoustic feature sequence x0). After the speech conversion model is trained and put into use, the distributed transformation module performs the opposite operation of the second transformation operation, achieving reverse computation (frame-level predicted acoustic feature sequence x1 → frame-level predicted latent representation sequence z1).

[0129] (4) The target acoustic feature sequence required for the audio generator to generate the target audio: refer to Figure 13a During the training process of the speech conversion model, the target acoustic feature sequence in step S1513 is the frame-level standard acoustic feature sequence z0; reference Figure 13b In the process of using the speech conversion model after training, the target acoustic feature sequence is the frame-level predicted acoustic feature sequence z1, and the frame-level predicted latent representation sequence z1 is obtained by the frame-level predicted acoustic feature sequence xf input distribution transformation module.

[0130] The decoder in this application employs a HiFi-GAN architecture, which consists of a generator and two discriminators. The generator accepts z0 as input and progressively high-samples the data until the generated sequence matches the temporal resolution of the original waveform. The two discriminators are a multi-period discriminator (MPD) and a multi-scale discriminator (MSD), which evaluate the audio at different periodic windows and scales, respectively. GAN-based training is used to improve the quality of the reconstructed speech.

[0131] In some embodiments, the method further includes: testing the target audio input discriminator generated by the generator to obtain an adversarial loss loss5; S1405: The speech conversion model is trained based on a preset loss. The trained speech conversion model is used to output target audio based on the input target object identifier, phoneme-level text feature sequence, frame-level acoustic feature sequence, and standard duration sequence. The preset losses include: (1) The loss 1 between the predicted duration sequence dur1 and the labeled duration sequence dur0; (2) The loss 2 between the target spectral information mel2 extracted from the target audio and the standard spectral information mel0 extracted from the training audio; (3) The loss between the frame-level predicted acoustic feature sequence x1 and the frame-level standard acoustic feature sequence x0 is loss3; (4) The loss 4 between the predicted spectrum information mel1 and the standard spectrum information mel0; (5) Countermeasure loss5.

[0132] Thus, this application provides an end-to-end training process (from feature alignment to audio generation) to ensure the stability of the model in practical applications. By using the loss between the predicted duration sequence and the labeled duration sequence, the predicted duration is ensured to be closer to the target object; by comparing the spectrum of the target audio with the standard spectrum, the quality of the generated audio is enhanced; the distribution alignment of the frame-level standard latent representation with the predicted latent representation (KL loss) forces the model to learn the acoustic feature distribution of the target speaker, reducing generation bias; by using the loss constraint between the predicted spectrum and the standard spectrum, the generated frame-level acoustic features are ensured to closely resemble the characteristics of the target speaker in the frequency domain, improving timbre similarity; the adversarial loss of the discriminator (such as MSD / MPD) distinguishes between generated audio and real audio, thereby enabling the decoder to generate more natural waveforms.

[0133] The speech conversion technology involved in this application strictly complies with relevant laws and regulations when deployed and used in various application scenarios. All data used in the training and inference process is obtained through legal and public channels or with explicit authorization from the data owner, and has undergone strict desensitization and anonymization processing to ensure that no sensitive personal information is involved. In specific applications (such as film and television dubbing, personalized speech synthesis, virtual human voice, etc.), if it is necessary to use the voice of a specific natural person, explicit permission must be obtained in advance from the person or the legal rights holder.

[0134] Based on the same technical concept, in the third aspect, refer to Figure 16 This application also provides a voice conversion device 1600. The voice conversion device 1600 includes modules for performing the voice chat processing method of the first aspect described above. For example, the voice conversion device 1600 includes: The first module 1601 is used to obtain the target object identifier and preprocessed data extracted from the source audio. The preprocessed data includes a phoneme-level text feature sequence related to audio content information, a frame-level acoustic feature sequence related to audio sound information, and a labeled duration sequence corresponding to the phoneme-level text feature sequence. The second module 1602 is used to align the frame-level acoustic feature sequence with the phoneme-level text feature sequence based on the labeled duration sequence and the phoneme-level text feature sequence, so as to generate a phoneme-level acoustic feature sequence of the content information of interest. The third module 1603 is used to generate a phoneme-level latent variable sequence that carries both content and acoustic information based on the phoneme-level text feature sequence, the phoneme-level acoustic feature sequence, and the target object identifier. The fourth module 1604 is used to input the phoneme-level latent variable sequence into the trained duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence. The duration prediction network is trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio. The fifth module 1605 is used to perform duration extension on each phoneme-level latent variable in the phoneme-level latent variable sequence based on the predicted duration sequence to obtain a frame-level latent variable sequence. The sixth module 1606 is used to generate target audio corresponding to the target object identifier based on the frame-level latent variable sequence.

[0135] For more details, please refer to the method implementation examples in the first aspect, which will not be repeated here.

[0136] The speech conversion device 1600 of this application embodiment can execute the method provided in this application embodiment. The implementation principle is similar. The actions performed by each module of the speech conversion device 1600 in each embodiment of this application correspond to the steps in the method of the first aspect of this application embodiment. For detailed functional descriptions of each module of the speech conversion device 1600, please refer to the description in the method embodiment of the first aspect above, which will not be repeated here.

[0137] Based on the same technical concept, in the third aspect, refer to Figure 17 This application also provides a speech conversion model training apparatus 1700. The speech conversion model training apparatus 1700 includes modules for performing the voice chat processing method described in the first aspect above. For example, the speech conversion model training apparatus 1700 includes: The data acquisition module 1701 is used to acquire the frame-level standard latent representation sequence extracted from the training audio, the standard spectral information extracted from the training audio, the target object identifier, and the preprocessed data extracted from the training audio corresponding to the target object identifier. The preprocessed data includes the phoneme-level text feature sequence related to the audio content information, the frame-level acoustic feature sequence related to the audio sound information, and the annotation duration sequence corresponding to the phoneme-level text feature sequence. Training module 1702 is used to call the speech conversion model to perform conversion operations: Based on the labeled duration sequence and phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence that focuses on the content information; Based on the phoneme-level text feature sequence, phoneme-level acoustic feature sequence, and target object identifier, a phoneme-level latent variable sequence carrying both content and acoustic information is generated; The phoneme-level latent variable sequence is input into the duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence; Based on the target duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence; The frame-level latent variable sequence is input into the frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence; The frame-level standard latent representation sequence is input into the distribution transformation module to obtain the frame-level standard acoustic feature sequence; The target acoustic feature sequence is input into the audio generator to generate the target audio; Training module 1702 is also used to train the speech conversion model based on a preset loss. The trained speech conversion model is used to output target audio based on the input target object identifier, phoneme-level text feature sequence, frame-level acoustic feature sequence, and standard duration sequence. The preset losses include: the loss between the predicted duration sequence and the labeled duration sequence, the loss between the target spectral information extracted from the target audio and the standard spectral information extracted from the training audio, and the loss between the frame-level predicted acoustic feature sequence and the frame-level standard acoustic feature sequence. During the training of the speech conversion model: the target duration sequence is the labeled duration sequence, and the target acoustic feature sequence is the frame-level standard latent representation sequence; during the use of the speech conversion model after training: the target duration sequence is the predicted duration sequence, and the target acoustic feature sequence is the frame-level predicted latent representation sequence, which is obtained by inputting the frame-level predicted acoustic feature sequence into the distribution transformation module.

[0138] For more details, please refer to the method implementation examples in the second aspect, which will not be repeated here.

[0139] It should be noted that the various modules described herein are divided into modules for clarity. However, in actual implementation, the boundaries between modules may be blurred. For example, any or all functional modules in this application may share various hardware and / or software elements. As another example, any and / or all functional modules in this application may be wholly or partially implemented by a shared processor executing software instructions. Furthermore, various software sub-modules executed by one or more processors may be shared among various software modules. Accordingly, unless expressly required, the scope of this application is not limited by mandatory boundaries between various hardware and / or software elements.

[0140] Based on the same technical concept, the fifth aspect, refer to Figure 18 This application embodiment also provides a computing device 1800, including components such as a memory 1802, a processor 1801, a communication module 1803, and an input / output interface 1804. Optionally, the components can be connected and communicate with each other via a bus 1805. The memory 1802 is used to store computer programs or instructions; when the processor 1801 executes the computer program or instructions, it implements the method steps in any of the method embodiments of the first or second aspect. It should be noted that... Figure 17 The structure of the device 1800 shown is merely illustrative and does not constitute a limitation on the device to which the method provided in the embodiments of this application applies.

[0141] The specific entity of the computing device may be a server, used to implement the method steps in any of the method embodiments of the first or second aspect.

[0142] The memory 1802 can be used to store operating systems and computer programs or instructions, which, when invoked by the processor 1801, implement the methods shown in the embodiments of the first or second aspect of the present invention. The memory 1802 can also store programs for implementing other functions or services. The memory 1802 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computing device, such as the hard disk or memory of the computing device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computing device, such as a plug-in hard disk, a secure digital card (SD card), a flash memory card, etc., provided on the computing device. Of course, the computer-readable storage medium can also include both internal storage units and external storage devices of the computing device. In this embodiment, the computer-readable storage medium is typically used to store software installed on the computing device, such as program code of the methods of the embodiments of the first or second aspect. Furthermore, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0143] The processor 1801 is connected to the memory 1802 via bus 1805 and performs corresponding functions by calling the application programs stored in the memory 1802. In some embodiments, it may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 1801 is typically used to control the overall operation of the processing device, such as performing control and processing related to data interaction or communication with other entities. In this embodiment, the processor 1801 is used to run program code stored in the memory 1802 or process data.

[0144] The computing device 1800 can connect to a network via a communication module 1803 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. The communication module 1803 may include a wired network interface and / or a wireless network interface, meaning the communication module may include at least one of a wired communication module or a wireless communication module.

[0145] The computing device 1800 can connect to necessary input / output devices, such as a keyboard and display device, via the input / output interface 1804. The device 1800 itself can have a display device, and other display devices can also be connected externally via interface 1804. It is understood that the input / output interface 1804 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 1804 can be a component of the device 1800, or it can be an external device connected to the device 1800 when needed.

[0146] The bus 1805 used to connect the various components may include a pathway for transmitting information between the components. The bus 1805 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 1805 can be divided into address bus, data bus, control bus, etc.

[0147] Based on the same technical concept, this application also provides a computer-readable storage medium storing a computer program or instructions. When the computer program or instructions are executed by a processing device, they implement the method steps in any of the method embodiments of the first or second aspect. Further details can be found in the method embodiments, which will not be repeated here. In this embodiment, the computer-readable storage medium can be non-volatile or volatile. Computer-readable storage media include flash memory, hard disks, multimedia cards, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), magnetic disks, optical disks, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computing device, such as the hard disk or memory of the computing device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computing device, such as a plug-in hard disk, a secure digital card (SD card), a flash memory card, etc., equipped on the computing device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computing device. In this embodiment, the computer-readable storage medium is typically used to store software installed on the computing device, such as program code of the methods of the embodiments of the first or second aspect. In addition, computer-readable storage media can also be used to temporarily store various types of data that have been output or will be output.

[0148] Based on the same technical concept, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the first or second aspect of the above-described method embodiments.

[0149] It should be noted that the order in which the embodiments are described in this application is not intended to limit the priority of the embodiments. The reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0150] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application and in its specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0151] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many forms under the guidance of this application without departing from the spirit and scope of protection of the claims. All equivalent transformations made under the inventive concept of this application using the content of this application's specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of this application.

Claims

1. A speech conversion method, characterized in that, include: Obtain the target object identifier and the preprocessed data extracted from the source audio. The preprocessed data includes a phoneme-level text feature sequence related to the audio content information, a frame-level acoustic feature sequence related to the audio sound information, and a labeled duration sequence corresponding to the phoneme-level text feature sequence. Based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content of interest. Based on the phoneme-level text feature sequence, phoneme-level acoustic feature sequence, and target object identifier, a phoneme-level latent variable sequence carrying both content and acoustic information is generated. The phoneme-level latent variable sequence is input into the trained duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence. The duration prediction network is trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio. Based on the predicted duration sequence, the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence is extended to obtain the frame-level latent variable sequence. The target audio corresponding to the target object identifier is generated based on the frame-level latent variable sequence.

2. The method according to claim 1, characterized in that, The step of aligning the frame-level acoustic feature sequence with the phoneme-level text feature sequence based on the labeled duration sequence and the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence for the content of interest includes: For each phoneme-level text feature in the phoneme-level text feature sequence, a weight vector reflecting its correlation with each frame-level acoustic feature in the frame-level acoustic feature sequence is calculated. The frame-level acoustic feature sequence is weighted and calculated according to the weight vector to obtain the first phoneme-level acoustic feature corresponding to the current phoneme-level text feature; For each frame interval corresponding to the labeled duration in the phoneme-level text feature sequence, all frame-level acoustic features belonging to the same frame interval are aggregated to obtain the second phoneme-level acoustic features. The phoneme-level acoustic feature sequence is obtained by fusing all the first phoneme-level acoustic features with all the second phoneme-level acoustic features.

3. The method according to claim 1, characterized in that, The process of generating a phoneme-level latent variable sequence that simultaneously carries content and acoustic information based on phoneme-level text feature sequences, phoneme-level acoustic feature sequences, and target object identifiers includes: Based on the phoneme-level text feature sequence and the phoneme-level acoustic feature sequence, a phoneme-level joint feature sequence is obtained by fusing them. The target object identifier is embedded according to the dimensions of the phoneme-level text features, and then expanded according to the length of the phoneme-level text feature sequence to obtain the identifier sequence; The identifier sequence is fused with the phoneme-level joint feature sequence and then input into the trained phoneme-level latent variable extractor to obtain the phoneme-level latent variable sequence.

4. The method according to claim 1, characterized in that, The step of extending the duration of each phoneme-level latent variable in the phoneme-level latent variable sequence based on the predicted duration sequence to obtain a frame-level latent variable sequence includes: For each phoneme-level latent variable in the phoneme-level latent variable sequence, repeat and concatenate the prediction durations in the prediction duration sequence to obtain the frame-level latent variable corresponding to the current phoneme-level latent variable. Based on the order of each phoneme-level latent variable in the phoneme-level latent variable sequence, the frame-level latent variables corresponding to each phoneme-level latent variable are arranged in order to form the frame-level latent variable sequence.

5. The method according to claim 1, characterized in that, The preprocessed data is obtained based on preprocessing operations, which include: Acquire initial frame-level acoustic feature sequences and multiple initial phoneme-level text feature sequences; Each initial phoneme-level text feature sequence is vectorized according to a preset dimension, and the vectorized initial phoneme-level text feature sequences are fused to obtain the phoneme-level text feature sequence. The frame-level acoustic features in the initial frame-level acoustic feature sequence are vectorized according to the preset dimension to obtain the frame-level acoustic feature sequence.

6. The method according to claim 5, characterized in that, The preprocessing operation also includes: Extract the initial frame-level acoustic feature sequence, as well as the text feature sequence, phoneme feature sequence, tone feature sequence, and semantic feature sequence from the source audio. The text feature sequence, tone feature sequence, and semantic feature sequence extracted from the source audio are extended to be aligned with the phoneme feature sequence to obtain the various initial phoneme-level text feature sequences.

7. The method according to claim 1, characterized in that, The step of generating the target audio corresponding to the target object identifier based on the frame-level latent variable sequence includes: The frame-level latent variable sequence is input into the trained frame-level acoustic feature generator to obtain the frame-level predicted acoustic feature sequence. The frame-level predicted acoustic feature sequence is input into the trained distribution transformation module and converted into a frame-level predicted latent representation sequence. The frame-level predicted latent representation sequence is input into the trained audio generator to generate the target audio.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Obtain the target object identifier and the phoneme-level text feature sequence, frame-level acoustic feature sequence, and labeled duration sequence extracted from its corresponding training audio; Based on the labeled duration sequence and the phoneme-level text feature sequence, the frame-level acoustic feature sequence is aligned with the phoneme-level text feature sequence to generate a phoneme-level acoustic feature sequence. Based on phoneme-level text feature sequences, phoneme-level acoustic feature sequences, and target object identifiers, a phoneme-level latent variable sequence is generated. Inputting the phoneme-level latent variable sequence into the duration prediction network yields the predicted duration sequence corresponding to the phoneme-level latent variable sequence. The duration prediction network is trained based on a preset loss, which includes the loss between the predicted duration sequence and the labeled duration sequence.

9. A voice conversion device, characterized in that, include: The first module is used to obtain the target object identifier and the preprocessed data extracted from the source audio. The preprocessed data includes a phoneme-level text feature sequence related to the audio content information, a frame-level acoustic feature sequence related to the audio sound information, and a labeled duration sequence corresponding to the phoneme-level text feature sequence. The second module is used to align the frame-level acoustic feature sequence with the phoneme-level text feature sequence based on the labeled duration sequence and the phoneme-level text feature sequence, so as to generate a phoneme-level acoustic feature sequence of the content information of interest. The third module is used to generate a phoneme-level latent variable sequence that carries both content and acoustic information based on the phoneme-level text feature sequence, the phoneme-level acoustic feature sequence, and the target object identifier. The fourth module is used to input the phoneme-level latent variable sequence into the trained duration prediction network to obtain the predicted duration sequence corresponding to the phoneme-level latent variable sequence. The duration prediction network is trained based on the preprocessed data extracted from the target object identifier and its corresponding training audio. The fifth module is used to perform duration extension on each phoneme-level latent variable in the phoneme-level latent variable sequence based on the predicted duration sequence to obtain a frame-level latent variable sequence. The sixth module is used to generate target audio corresponding to the target object identifier based on the frame-level latent variable sequence.

10. A computing device, characterized in that, It includes a memory and a processor, the memory being used to store computer programs or instructions; when the computer programs or instructions are executed by the processor, the method of any one of claims 1-8 is implemented.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a processor, implement the method of any one of claims 1-8.

12. A computer program product, characterized in that, The computing program product includes a computer program or instructions that, when executed by a processor, implement the method of any one of claims 1-8.

Images