Speech processing method and apparatus, computer device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a speech processing method and device, computer equipment and a computer readable storage medium, which can construct a speech synthesis model and a non-parallel speech conversion model in advance, synthesize target text into intermediate speech with a specified voice through the speech synthesis model, and then convert the specified voice of the intermediate speech into the voice of the user speech through a parallel speech conversion model to obtain target synthesized speech after obtaining the user speech of a target user, so that the speech cloning operation can be quickly performed, the operation of the user in the speech cloning is simple, the operation efficiency of the speech cloning can be effectively improved, and the embodiments of the present application can generate a corresponding parallel conversion model for the user speech, multiple users can share one speech synthesis model and non-parallel speech conversion model, the speech conversion model structure can be simplified, the speech conversion model can be lightened, and the storage consumption of the speech conversion model on the computer equipment can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of information processing, and in particular, to a speech processing method and device, computer equipment and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of information technology, the massive popularization and application of computer equipment such as smart phones, tablet computers and notebook computers, computer equipment is developing in the direction of diversification and personalization, and computer equipment can already synthesize speech comparable to that of a real person, enriching the experience of human-computer interaction. For example, current common speech processing technologies include speech synthesis, speech conversion and speech cloning technologies. Voice cloning refers to a technology in which a machine extracts timbre information from a user-provided voice and uses the user's timbre to synthesize speech. Voice cloning is an extension of speech synthesis technology. Traditional speech synthesis implements text-to-speech conversion on a fixed speaker, while voice cloning further specifies the speaker's timbre. Currently, there are many practical scenarios for voice cloning, such as in voice navigation and audio novels, where users can customize their own voice packs by uploading voices to navigate or read novels with their own voices, thereby enhancing the interest of using the application program.

[0003] In the prior art, when a user uses voice cloning technology for personalized customization, the user usually needs to provide a segment of their own voice and the text corresponding to the voice in order to implement voice cloning. However, in the use scenario of voice cloning, the recorded voice provided by the user and the reading content of the voice may not be consistent, which results in the need for cleaning and correction operations before voice model training. Therefore, it is difficult to obtain recorded voices that are consistent with the reading content, and the requirements for the user during voice recording are high, which affects the user experience. SUMMARY

[0004] Embodiments of the present application provide a speech processing method, device, computer equipment and computer readable storage medium, by synthesizing a target text into an intermediate voice of a specified timbre, after obtaining a user voice of a target user, directly converting the specified timbre of the intermediate voice into the timbre of the user voice to obtain a target synthesized voice, thereby being able to quickly perform voice cloning operations, making the operation simple when the user performs voice cloning, and being able to effectively improve the operation efficiency of voice cloning. Furthermore, embodiments of the present application can also simplify the structure of the speech conversion model, making the speech conversion model lightweight, thereby reducing the storage consumption of the speech conversion model on the computer equipment.

[0005] Embodiments of the present application provide a speech processing method, which comprises:

[0006] The voice conversion processing is performed based on a user voice of a target user and specified timbre information, to obtain specified converted voice of a specified timbre, wherein the specified timbre information is timbre information determined from a plurality of preset timbre information, and the specified converted voice is a user voice with the specified timbre;

[0007] The voice conversion model is trained according to the user voice and the specified converted voice, to obtain a target voice conversion model;

[0008] The target text of the speech to be synthesized and the specified timbre information are input into a speech synthesis model, to generate intermediate speech of the specified timbre;

[0009] The intermediate speech is subjected to voice conversion processing through the target voice conversion model, to generate target synthesized speech matching the timbre of the target user.

[0010] Correspondingly, the embodiment of the application further provides a speech processing device, which comprises:

[0011] The first processing unit is configured to perform voice conversion processing based on a user voice of a target user and specified timbre information, to obtain specified converted voice of a specified timbre, wherein the specified timbre information is timbre information determined from a plurality of preset timbre information, and the specified converted voice is a user voice with the specified timbre;

[0012] The training unit is configured to train a voice conversion model according to the user voice and the specified converted voice, to obtain a target voice conversion model;

[0013] The generation unit is configured to input the target text of the speech to be synthesized and the specified timbre information into a speech synthesis model, to generate intermediate speech of the specified timbre;

[0014] The second processing unit is configured to perform voice conversion processing on the intermediate speech through the target voice conversion model, to generate target synthesized speech matching the timbre of the target user.

[0015] In some embodiments, the device further comprises:

[0016] The first acquisition subunit is configured to acquire language content features and prosody features from a user voice of a target user;

[0017] The first processing subunit is configured to perform voice conversion processing based on the language content features, the prosody features and specified timbre information, to obtain specified converted voice of a specified timbre.

[0018] In some embodiments, the device further comprises:

[0019] The second acquisition subunit is used to acquire sample speech, sample speech text, and sample timbre information;

[0020] The first adjustment unit is used to adjust the model parameters of the preset speech model based on the sample speech, the text of the sample speech, and the sample timbre information to obtain the adjusted preset speech model.

[0021] The second processing subunit is used to continue to acquire the next sample speech, the text of the next sample speech, and the sample timbre information from the training sample speech set, and to execute the step of adjusting the model parameters of the preset speech synthesis model based on the sample speech, the text of the sample speech, and the sample timbre information, until the training status of the adjusted speech model meets the model training termination condition, and a trained preset speech model is obtained as the speech synthesis model.

[0022] In some embodiments, the device further includes:

[0023] The second adjustment unit is used to adjust the model parameters of the parallel speech conversion model based on the user's speech and the specified converted speech, until the model training termination condition of the parallel speech conversion model is met, so as to obtain the trained parallel speech conversion model as the target speech conversion model.

[0024] In some embodiments, the device further includes:

[0025] The third acquisition subunit is used to acquire training speech pairs and preset timbre information corresponding to the training speech, wherein the training speech pair includes original speech and output speech, the original speech and the output speech are the same speech, and all speech in the training speech pair is speech in the training sample speech set;

[0026] The third adjustment unit is used to adjust the model parameters of the non-parallel speech conversion model based on the original speech, the output speech, and the preset timbre information until the model training termination condition of the non-parallel speech conversion model is met, so as to obtain the trained non-parallel speech conversion model as the target non-parallel speech conversion model.

[0027] In some embodiments, the device further includes:

[0028] The third processing subunit is used to perform language content extraction processing on the original speech through the language feature processor of the non-parallel speech conversion model to obtain the language content features of the original speech.

[0029] The fourth processing subunit is used to perform prosodic extraction processing on the original speech through the prosodic feature processor of the non-parallel speech conversion model to obtain the prosodic features of the original speech.

[0030] The fourth adjustment unit is used to adjust the model parameters of the non-parallel speech conversion model based on the language content features of the original speech, the prosodic features of the original speech, the preset timbre information, and the output speech.

[0031] In some embodiments, the device further includes:

[0032] The first generation subunit is used to perform language information filtering processing on the original speech, determine the language information corresponding to the original speech, generate a first specified length vector based on the language information, and use the first specified length vector as language content features.

[0033] In some embodiments, the device further includes:

[0034] The second generation subunit is used to perform prosodic information filtering processing on the original speech, determine the prosodic information corresponding to the original speech, generate a second specified length vector based on the prosodic information, and use the second specified length vector as the prosodic feature.

[0035] In some embodiments, the device further includes:

[0036] The fifth processing subunit is used to extract the language content of the user's speech through the language feature processor of the target non-parallel speech conversion model to obtain the language content features of the user's speech.

[0037] The sixth processing subunit is used to perform prosodic extraction processing on the user speech through the prosodic feature processor of the target non-parallel speech conversion model to obtain the prosodic features of the user speech.

[0038] In some embodiments, the device further includes:

[0039] The input subunit is used to input the language content features of the user's speech, the prosodic features of the user's speech, and the specified timbre information into the target non-parallel speech conversion model to generate the specified converted speech with the specified timbre.

[0040] Accordingly, this application also provides a computer device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of any one of the speech processing methods.

[0041] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any one of the speech processing methods.

[0042] This application provides a speech processing method, apparatus, computer device, and computer-readable storage medium. By constructing a speech synthesis model and a non-parallel speech conversion model, the speech synthesis model synthesizes target text into intermediate speech with a specified timbre. After obtaining the user's speech, the parallel speech conversion model directly converts the specified timbre of the intermediate speech into the timbre of the user's speech to obtain the target synthesized speech. This enables rapid speech cloning operations, simplifying the user's speech cloning process and effectively improving the efficiency of speech cloning. Furthermore, this application can generate corresponding parallel conversion models for user speech, allowing multiple users to share a single speech synthesis model and non-parallel speech conversion model. This simplifies the speech conversion model structure, making the model lightweight and reducing the storage consumption of the computer device. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a scenario for the voice processing system provided in an embodiment of this application.

[0045] Figure 2 This is a schematic flowchart of a speech processing method provided in an embodiment of this application.

[0046] Figure 3 This is a schematic diagram illustrating the training of the speech synthesis model provided in an embodiment of this application.

[0047] Figure 4 This is a schematic diagram illustrating the training of a non-parallel speech conversion model provided in an embodiment of this application.

[0048] Figure 5 This is a schematic diagram illustrating the application of the non-parallel speech conversion model provided in the embodiments of this application.

[0049] Figure 6 This is a schematic diagram illustrating the training of the parallel speech conversion model provided in an embodiment of this application.

[0050] Figure 7 This is a schematic diagram illustrating the application of the speech synthesis model provided in the embodiments of this application.

[0051] Figure 8 This is a schematic diagram illustrating the application of the parallel speech conversion model provided in the embodiments of this application.

[0052] Figure 9This is a schematic diagram of the structure of the voice processing device provided in the embodiments of this application.

[0053] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0055] This application provides a voice processing method, apparatus, computer device, and computer-readable storage medium. Specifically, the voice processing method of this application can be executed by a computer device, which can be a terminal. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. The terminal may also include a client, which can be a video application client, music application client, game application client, browser client carrying a game program, or instant messaging client, etc.

[0056] Please see Figure 1 , Figure 1 This is a schematic diagram of a voice processing system provided in an embodiment of this application. The system includes a computer device and may include at least one terminal, at least one server, and a network. The user's terminal can connect to servers of different games via the network. The terminal is any device with computing hardware capable of supporting and executing software products corresponding to the game. Additionally, the terminal has one or more multi-touch screens for sensing and obtaining input from touch or swipe operations performed by the user at multiple points on one or more touch displays. Furthermore, when the system includes multiple terminals, multiple servers, and multiple networks, different terminals can connect to each other through different networks and different servers. The network can be a wireless network or a wired network, such as a wireless local area network (WLAN), local area network (LAN), cellular network, 2G network, 3G network, 4G network, 5G network, etc. Additionally, different terminals can also connect to other terminals or servers using their own Bluetooth networks or hotspot networks.

[0057] Specifically, the computer device can acquire language content features and prosodic features from the user's speech; perform speech conversion processing based on the language content features, the prosodic features, and specified timbre information to obtain specified converted speech with a specified timbre; train the speech conversion model based on the user's speech and the specified converted speech to obtain a target speech conversion model; input the target text to be synthesized speech and the specified timbre information into the speech synthesis model to generate intermediate speech with a specified timbre; and perform speech conversion processing on the intermediate speech through the target speech conversion model to generate target synthesized speech that matches the timbre of the target user.

[0058] It should be noted that, Figure 1 The schematic diagram of the speech processing system shown is merely an example. The speech processing system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of speech processing systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0059] This invention provides a voice processing method, apparatus, computer device, and computer-readable storage medium. The voice processing method can be used with a terminal, such as a smartphone, tablet computer, laptop computer, or personal computer. The voice processing method, apparatus, terminal, and storage medium are described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0060] Please see Figure 2 , Figure 2 This is a schematic flowchart of a speech processing method provided in an embodiment of this application. The specific process can be summarized in steps 101 to 104 as follows:

[0061] 101. Based on the user's voice and specified timbre information of the target user, perform voice conversion processing to obtain specified converted voice with specified timbre, wherein the specified timbre information is timbre information determined from multiple preset timbre information, and the specified converted voice is the user's voice with the specified timbre.

[0062] Prior to the step "performing speech conversion based on the target user's voice and specified timbre information", the method includes:

[0063] Extract language content features and prosodic features from the user's speech of the target user;

[0064] The speech conversion processing based on the target user's voice and specified timbre information includes:

[0065] Based on the language content features, the prosodic features, and the specified timbre information, speech conversion processing is performed to obtain the specified converted speech with the specified timbre.

[0066] In one embodiment, before the step of "performing speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre", the method may include:

[0067] Acquire training speech pairs and preset timbre information corresponding to the training speech, wherein the training speech pair includes original speech and output speech, the original speech and the output speech are the same speech, and all speech in the training speech pair is speech in the training sample speech set;

[0068] The model parameters of the non-parallel speech conversion model are adjusted based on the original speech, the output speech, and the preset timbre information until the model training termination condition of the non-parallel speech conversion model is met, resulting in a trained non-parallel speech conversion model, which serves as the target non-parallel speech conversion model.

[0069] Optionally, the method for step "adjusting the model parameters of the non-parallel speech conversion model based on the original speech, the preset timbre information, and the output speech" may include:

[0070] The language feature processor of the non-parallel speech conversion model extracts the language content of the original speech to obtain the language content features of the original speech.

[0071] The prosodic feature processor of the non-parallel speech conversion model is used to extract the prosodic features of the original speech.

[0072] The model parameters of the non-parallel speech conversion model are adjusted based on the language content features of the original speech, the prosodic features of the original speech, the preset timbre information, and the output speech.

[0073] Specifically, the method for the step "extracting language content features from the original speech using the language feature processor of the non-parallel speech conversion model" may include:

[0074] The original speech is subjected to language information filtering processing to determine the language information corresponding to the original speech, and a first specified length vector is generated based on the language information, and the first specified length vector is used as the language content feature.

[0075] In another specific embodiment, the method for the step "performing prosodic extraction processing on the original speech using the prosodic feature processor of the non-parallel speech conversion model to obtain the prosodic features of the original speech" may include:

[0076] The original speech is subjected to prosodic information filtering processing to determine the prosodic information corresponding to the original speech, and a second specified length vector is generated based on the prosodic information, and the second specified length vector is used as the prosodic feature.

[0077] In this embodiment of the application, the method for the step "obtaining language content features and prosodic features from the user's speech of the target user" may include:

[0078] The language feature processor of the target non-parallel speech conversion model extracts the language content of the user's speech to obtain the language content features of the user's speech.

[0079] The prosodic feature processor of the target non-parallel speech conversion model is used to extract the prosodic features of the user's speech.

[0080] To obtain a specified converted speech with a specified timbre, the step "performing speech conversion based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre" can include the following methods:

[0081] The language content features, prosodic features, and specified timbre information of the user's speech are input into the target non-parallel speech conversion model to generate the specified converted speech with the specified timbre.

[0082] 102. The speech conversion model is trained based on the user's speech and the specified conversion speech to obtain the target speech conversion model.

[0083] Specifically, the method for the step "training the speech conversion model based on the user's speech and the specified converted speech to obtain the target speech conversion model" may include:

[0084] The model parameters of the parallel speech conversion model are adjusted based on the user's speech and the specified converted speech until the model training termination condition of the parallel speech conversion model is met, thus obtaining the trained parallel speech conversion model, which serves as the target speech conversion model.

[0085] 103. Input the target text to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre.

[0086] To obtain the speech synthesis model, before the step "inputting the target text to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre", the method may include:

[0087] Obtain sample speech, sample speech text, and sample timbre information;

[0088] The model parameters of the preset speech model are adjusted based on the sample speech, the text of the sample speech, and the sample timbre information to obtain the adjusted preset speech model.

[0089] Continue to acquire the next sample speech, the text of the next sample speech, and the sample timbre information from the training sample speech set, and execute the step of adjusting the model parameters of the preset speech synthesis model based on the sample speech, the text of the sample speech, and the sample timbre information, until the training status of the adjusted speech model meets the model training termination condition, and obtain the trained preset speech model as the speech synthesis model.

[0090] 104. The intermediate speech is processed by the target speech conversion model to generate target synthesized speech that matches the timbre of the target user.

[0091] To further illustrate the speech processing method provided in this application embodiment, the following will use the application of the speech processing method in a specific implementation scenario as an example. The specific application scenario is as follows:

[0092] (1) The embodiments of this application include a pre-training stage, in which the speech synthesis model and the non-parallel speech conversion model can be trained.

[0093] Please see Figure 3 , Figure 3 This diagram illustrates the training process of a speech synthesis model. During training, the model can utilize existing multi-person speech data, corresponding text data, and preset timbres from a database. After training, the model is saved for use in the application phase. Specifically, the pre-training stage involves inputting a large amount of text-to-speech and timbre labeling data into a neural network model for training. This is typically based on an end-to-end deep neural network model, with various model structures available, including but not limited to popular ones like Tacotron and FastSpeech.

[0094] Please see Figure 4 , Figure 4 This diagram illustrates the training of a non-parallel speech conversion model. When training the non-parallel speech conversion model, a pre-trained language feature extraction module can be used to extract language-related feature representations of the original speech, and a prosodic feature module can be used to extract prosodic feature representations of the original speech. The language-related feature representations and prosodic feature representations, along with timbre markings and the output speech, are then input into the non-parallel speech conversion model for training.

[0095] The language feature extraction module aims to obtain timbre-independent language feature representations from the input speech. This module removes information irrelevant to the language content, extracting only the language information and converting it into a fixed-length vector representation. The extracted language information should accurately reflect the content of the original speech without errors or omissions. It should be noted that this language feature extraction module requires a neural network model. There are several possible implementation methods. One method is to train a speech recognition model with a large amount of speech and text data, selecting the output of a specific hidden layer as the language feature representation. Another method is to use unsupervised training, such as using a VQVAE model, to compress and quantize the speech into several speech units, and then restore these speech units to the original speech. In this self-reconstruction training process, the quantized units gradually learn to become timbre-independent speech units, which are the language feature representations. Other methods can also be used, not limited to the two mentioned above. In one embodiment, the prosodic feature extraction module aims to obtain prosodic feature representations from the input speech and convert them into vector representations. This prosodic feature extraction module aims to ensure that the converted speech maintains the same prosodic style as the original speech, making the data before and after conversion completely parallel except for timbre, thus facilitating the modeling of a parallel conversion model. Technically, there are various implementation methods, primarily using signal processing tools and algorithms for extraction, such as using common speech features like fundamental frequency and energy, or features related to speech emotion classification.

[0096] In this embodiment, the purpose of the non-parallel speech conversion model is to generate converted speech with corresponding timbre and semantic content based on the language feature representation, prosodic feature representation, and specified timbre marker extracted from the user's speech, thus constructing parallel speech data for training the parallel conversion model. This non-parallel speech conversion model requires that the timbre of the converted speech is similar to that of the target user's speech, while the semantic content, prosody, etc., are completely consistent with the original speech. During the pre-training stage, the language feature representation extracted by the language feature extraction model, the prosodic feature representation obtained by the prosodic feature module, the timbre marker, and the corresponding output speech are input into the neural network model for training. Generally, a deep neural network model is used, and the specific model structure can be constructed in various ways, such as convolutional networks, recurrent neural networks, Transformers, or any combination thereof.

[0097] (2) The embodiments of this application include a parallel speech conversion model training stage, which can be used to train the parallel speech conversion model.

[0098] Please see Figure 5 , Figure 5This is a schematic diagram of the application of a non-parallel speech conversion model. After determining the user's speech of the target user who needs to be cloned, the non-parallel speech conversion model trained in the pre-training stage can be used to convert the user's speech into a speech with a specified timbre. The text content and prosodic information of the user's speech remain unchanged. That is, after conversion, the text content and prosodic information of the speech with the specified timbre are the same as the text content and prosodic information of the user's speech, thus constructing parallel speech data.

[0099] Please see Figure 6 , Figure 6 This is a training diagram of a parallel speech conversion model. After obtaining a speech with a specified timbre, a speech pair can be formed based on the specified timbre speech and the user's speech. The specified timbre speech and the user's speech are then input into the parallel speech conversion model to train the model. This parallel speech conversion model can use a simple neural network model, such as a one-layer recurrent neural network, or other model structures that can meet the above conditions.

[0100] (3) The embodiments of this application include a model application stage for a speech synthesis model and a parallel speech conversion model. The specific model applications of the speech synthesis model and the parallel speech conversion model are described below.

[0101] Please see Figure 7 , Figure 7 This is a schematic diagram of the application of a speech synthesis model. When detection requires sound cloning of target text based on the timbre of the user's speech, the speech synthesis model can determine any text selected by the user as the target text and convert the target text into intermediate speech with a specified timbre.

[0102] Please see Figure 8 , Figure 8 This is a schematic diagram of the application of a parallel speech conversion model, which can convert intermediate speech with a specified timbre into the timbre corresponding to the user's speech, thereby obtaining the target synthesized speech.

[0103] In summary, the embodiments of this application provide a speech processing method that constructs a speech synthesis model, a non-parallel speech conversion model, and a parallel speech conversion model. The method uses a speech synthesis model shared by multiple users to synthesize target text into intermediate speech with a specified timbre. After obtaining the user's speech, the parallel speech conversion model directly converts the specified timbre of the intermediate speech into the timbre of the user's speech to obtain the target synthesized speech. This method enables rapid speech cloning operations, simplifies the user's speech cloning process, and improves the efficiency of speech cloning.

[0104] Please see Figure 9 , Figure 9 This is a schematic diagram of a voice processing device provided in an embodiment of this application. The device includes:

[0105] The first processing unit 201 is used to perform speech conversion processing based on the user's speech and specified timbre information of the target user to obtain specified converted speech with specified timbre, wherein the specified timbre information is timbre information determined from multiple preset timbre information, and the specified converted speech is user speech with the specified timbre.

[0106] Training unit 202 is used to train the speech conversion model based on the user's speech and the specified converted speech to obtain the target speech conversion model;

[0107] The generation unit 203 is used to input the target text of the speech to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre.

[0108] The second processing unit 204 is used to perform speech conversion processing on the intermediate speech through the target speech conversion model to generate target synthesized speech that matches the timbre of the target user.

[0109] In some embodiments, the device further includes:

[0110] The first acquisition subunit is used to acquire language content features and prosodic features from the user's speech of the target user;

[0111] The first processing subunit is used to perform speech conversion processing based on the language content features, the prosodic features and the specified timbre information to obtain the specified converted speech with the specified timbre.

[0112] In some embodiments, the device further includes:

[0113] The second acquisition subunit is used to acquire sample speech, sample speech text, and sample timbre information;

[0114] The first adjustment unit is used to adjust the model parameters of the preset speech model based on the sample speech, the text of the sample speech, and the sample timbre information to obtain the adjusted preset speech model.

[0115] The second processing subunit is used to continue to acquire the next sample speech, the text of the next sample speech, and the sample timbre information from the training sample speech set, and to execute the step of adjusting the model parameters of the preset speech synthesis model based on the sample speech, the text of the sample speech, and the sample timbre information, until the training status of the adjusted speech model meets the model training termination condition, and a trained preset speech model is obtained as the speech synthesis model.

[0116] In some embodiments, the device further includes:

[0117] The second adjustment unit is used to adjust the model parameters of the parallel speech conversion model based on the user's speech and the specified converted speech, until the model training termination condition of the parallel speech conversion model is met, so as to obtain the trained parallel speech conversion model as the target speech conversion model.

[0118] In some embodiments, the device further includes:

[0119] The third acquisition subunit is used to acquire training speech pairs and preset timbre information corresponding to the training speech, wherein the training speech pair includes original speech and output speech, the original speech and the output speech are the same speech, and all speech in the training speech pair is speech in the training sample speech set;

[0120] The third adjustment unit is used to adjust the model parameters of the non-parallel speech conversion model based on the original speech, the output speech, and the preset timbre information until the model training termination condition of the non-parallel speech conversion model is met, so as to obtain the trained non-parallel speech conversion model as the target non-parallel speech conversion model.

[0121] In some embodiments, the device further includes:

[0122] The third processing subunit is used to perform language content extraction processing on the original speech through the language feature processor of the non-parallel speech conversion model to obtain the language content features of the original speech.

[0123] The fourth processing subunit is used to perform prosodic extraction processing on the original speech through the prosodic feature processor of the non-parallel speech conversion model to obtain the prosodic features of the original speech.

[0124] The fourth adjustment unit is used to adjust the model parameters of the non-parallel speech conversion model based on the language content features of the original speech, the prosodic features of the original speech, the preset timbre information, and the output speech.

[0125] In some embodiments, the device further includes:

[0126] The first generation subunit is used to perform language information filtering processing on the original speech, determine the language information corresponding to the original speech, generate a first specified length vector based on the language information, and use the first specified length vector as language content features.

[0127] In some embodiments, the device further includes:

[0128] The second generation subunit is used to perform prosodic information filtering processing on the original speech, determine the prosodic information corresponding to the original speech, generate a second specified length vector based on the prosodic information, and use the second specified length vector as the prosodic feature.

[0129] In some embodiments, the device further includes:

[0130] The fifth processing subunit is used to extract the language content of the user's speech through the language feature processor of the target non-parallel speech conversion model to obtain the language content features of the user's speech.

[0131] The sixth processing subunit is used to perform prosodic extraction processing on the user speech through the prosodic feature processor of the target non-parallel speech conversion model to obtain the prosodic features of the user speech.

[0132] In some embodiments, the device further includes:

[0133] The input subunit is used to input the language content features of the user's speech, the prosodic features of the user's speech, and the specified timbre information into the target non-parallel speech conversion model to generate the specified converted speech with the specified timbre.

[0134] This application provides a speech processing device. A first processing unit 201 performs speech conversion processing based on a target user's speech and specified timbre information to obtain a specified converted speech with a specified timbre. The specified timbre information is determined from multiple preset timbre information, and the specified converted speech is the user's speech with the specified timbre. A training unit 202 trains a speech conversion model based on the user's speech and the specified converted speech to obtain a target speech conversion model. A generation unit 203 inputs the target text of the speech to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre. A second processing unit 204 performs speech conversion processing on the intermediate speech using the target speech conversion model to generate target synthesized speech that matches the timbre of the target user. This application embodiment constructs a speech synthesis model, a non-parallel speech conversion model, and a parallel speech conversion model. The speech synthesis model synthesizes the target text into intermediate speech with a specified timbre. After acquiring the target user's speech, the parallel speech conversion model directly converts the specified timbre of the intermediate speech into the timbre of the user's speech to obtain the target synthesized speech. This enables rapid speech cloning operations, simplifying the user's voice cloning process and effectively improving efficiency. Furthermore, this application embodiment can generate corresponding parallel conversion models for user speech, allowing multiple users to share a single non-parallel speech conversion model. This simplifies the speech conversion model structure, making it lightweight and reducing the storage consumption of computer equipment.

[0135] Accordingly, this application also provides a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 300 includes a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, and a computer program stored in the memory 302 and executable on the processor. The processor 301 and the memory 302 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0136] The processor 301 is the control center of the computer device 300. It connects various parts of the computer device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it performs various functions of the computer device 300 and processes data, thereby monitoring the computer device 300 as a whole.

[0137] In this embodiment, the processor 301 in the computer device 300 loads the instructions corresponding to the processes of one or more applications into the memory 302 according to the following steps, and the processor 301 runs the applications stored in the memory 302 to achieve various functions:

[0138] Speech conversion processing is performed based on the user's voice and specified timbre information of the target user to obtain specified converted speech with specified timbre. The specified timbre information is timbre information determined from multiple preset timbre information, and the specified converted speech is the user's voice with the specified timbre.

[0139] The speech conversion model is trained based on the user's speech and the specified converted speech to obtain the target speech conversion model;

[0140] The target text to be synthesized and the specified timbre information are input into the speech synthesis model to generate intermediate speech with the specified timbre.

[0141] The intermediate speech is processed by the target speech conversion model to generate target synthesized speech that matches the voice of the target user.

[0142] In one embodiment, before performing speech conversion processing based on the target user's voice and specified timbre information, the method further includes:

[0143] Extract language content features and prosodic features from the user's speech of the target user;

[0144] The speech conversion processing based on the target user's voice and specified timbre information includes:

[0145] Based on the language content features, the prosodic features, and the specified timbre information, speech conversion processing is performed to obtain the specified converted speech with the specified timbre.

[0146] In one embodiment, before inputting the target text to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre, the method further includes:

[0147] Obtain sample speech, sample speech text, and sample timbre information;

[0148] The model parameters of the preset speech model are adjusted based on the sample speech, the text of the sample speech, and the sample timbre information to obtain the adjusted preset speech model.

[0149] Continue to acquire the next sample speech, the text of the next sample speech, and the sample timbre information from the training sample speech set, and execute the step of adjusting the model parameters of the preset speech synthesis model based on the sample speech, the text of the sample speech, and the sample timbre information, until the training status of the adjusted speech model meets the model training termination condition, and obtain the trained preset speech model as the speech synthesis model.

[0150] In one embodiment, training the speech conversion model based on the user's speech and the specified converted speech to obtain the target speech conversion model includes:

[0151] The model parameters of the parallel speech conversion model are adjusted based on the user's speech and the specified converted speech until the model training termination condition of the parallel speech conversion model is met, thus obtaining the trained parallel speech conversion model, which serves as the target speech conversion model.

[0152] In one embodiment, before performing speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre, the method further includes:

[0153] Acquire training speech pairs and preset timbre information, wherein the training speech pairs include original speech and output speech, and the original speech and the output speech are the same speech;

[0154] The model parameters of the non-parallel speech conversion model are adjusted based on the original speech, the output speech, and the preset timbre information until the model training termination condition of the non-parallel speech conversion model is met, resulting in a trained non-parallel speech conversion model, which serves as the target non-parallel speech conversion model.

[0155] In one embodiment, adjusting the model parameters of the non-parallel speech conversion model based on the original speech, the preset timbre information, and the output speech includes:

[0156] The language feature processor of the non-parallel speech conversion model extracts the language content of the original speech to obtain the language content features of the original speech.

[0157] The prosodic feature processor of the non-parallel speech conversion model is used to extract the prosodic features of the original speech.

[0158] The model parameters of the non-parallel speech conversion model are adjusted based on the language content features of the original speech, the prosodic features of the original speech, the preset timbre information, and the output speech.

[0159] In one embodiment, the step of extracting language content features from the original speech using the language feature processor of the non-parallel speech conversion model to obtain language content features of the original speech includes:

[0160] The original speech is subjected to language information filtering processing to determine the language information corresponding to the original speech;

[0161] A first specified length vector is generated based on the language information, and the first specified length vector is used as a language content feature.

[0162] In one embodiment, the prosodic feature processor of the non-parallel speech conversion model performs prosodic extraction processing on the original speech to obtain the prosodic features of the original speech, including:

[0163] The original speech is subjected to prosodic information filtering processing to determine the prosodic information corresponding to the original speech, and a second specified length vector is generated based on the prosodic information, and the second specified length vector is used as the prosodic feature.

[0164] In one embodiment, obtaining language content features and prosodic features from the user's speech of the target user includes:

[0165] The language feature processor of the target non-parallel speech conversion model extracts the language content of the user's speech to obtain the language content features of the user's speech.

[0166] The prosodic feature processor of the target non-parallel speech conversion model is used to extract the prosodic features of the user's speech.

[0167] In one embodiment, the step of performing speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre includes:

[0168] The language content features, prosodic features, and specified timbre information of the user's speech are input into the target non-parallel speech conversion model to generate the specified converted speech with the specified timbre.

[0169] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0170] Optional, such as Figure 10As shown, the computer device 300 also includes: a touch screen display 303, a radio frequency circuit 304, an audio circuit 305, an input unit 306, and a power supply 307. The processor 301 is electrically connected to the touch screen display 303, the radio frequency circuit 304, the audio circuit 305, the input unit 306, and the power supply 307. Those skilled in the art will understand that... Figure 10 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0171] The touch display screen 303 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 303 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 301. It can also receive and execute commands from the processor 301. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 301 to determine the type of touch event. Subsequently, the processor 301 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 303 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 303 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 303 can also be used as part of the input unit 306 to achieve input functions.

[0172] In this embodiment, a processor 301 executes an application to generate a graphical interface on a touch display screen 303. The touch display screen 303 is used to present the graphical interface and receive user commands generated by the graphical interface.

[0173] The radio frequency circuit 304 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0174] Audio circuitry 305 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 305 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 305, converted back into audio data, and output to processor 301 for processing. The audio data is then transmitted via radio frequency circuitry 304 to, for example, another computer device, or output to memory 302 for further processing. Audio circuitry 305 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0175] The input unit 306 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0176] Power supply 307 is used to supply power to various components of computer device 300. Optionally, power supply 307 can be logically connected to processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 307 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0177] although Figure 10 As not shown in the diagram, the computer device 300 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0178] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0179] As can be seen from the above, the computer device provided in this embodiment performs speech conversion processing based on the user's speech and specified timbre information to obtain specified converted speech with a specified timbre. The specified timbre information is determined from multiple preset timbre information, and the specified converted speech is the user's speech with the specified timbre. A speech conversion model is trained based on the user's speech and the specified converted speech to obtain a target speech conversion model. The target text to be synthesized and the specified timbre information are input into the speech synthesis model to generate intermediate speech with the specified timbre. The intermediate speech is then processed by the target speech conversion model to generate target synthesized speech that matches the timbre of the target user. This application embodiment constructs a speech synthesis model, a non-parallel speech conversion model, and a parallel speech conversion model. The speech synthesis model synthesizes the target text into intermediate speech with a specified timbre. After acquiring the target user's speech, the parallel speech conversion model directly converts the specified timbre of the intermediate speech into the timbre of the user's speech to obtain the target synthesized speech. This enables rapid speech cloning operations, simplifying the user's voice cloning process and effectively improving efficiency. Furthermore, this application embodiment can generate corresponding parallel conversion models for user speech, allowing multiple users to share a single non-parallel speech conversion model. This simplifies the speech conversion model structure, making it lightweight and reducing the storage consumption of computer equipment.

[0180] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0181] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute the steps of any of the speech processing methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0182] Speech conversion processing is performed based on the user's voice and specified timbre information of the target user to obtain specified converted speech with specified timbre. The specified timbre information is timbre information determined from multiple preset timbre information, and the specified converted speech is the user's voice with the specified timbre.

[0183] The speech conversion model is trained based on the user's speech and the specified converted speech to obtain the target speech conversion model;

[0184] The target text to be synthesized and the specified timbre information are input into the speech synthesis model to generate intermediate speech with the specified timbre.

[0185] The intermediate speech is processed by the target speech conversion model to generate target synthesized speech that matches the voice of the target user.

[0186] In one embodiment, before performing speech conversion processing based on the target user's voice and specified timbre information, the method further includes:

[0187] Extract language content features and prosodic features from the user's speech of the target user;

[0188] The speech conversion processing based on the target user's voice and specified timbre information includes:

[0189] Based on the language content features, the prosodic features, and the specified timbre information, speech conversion processing is performed to obtain the specified converted speech with the specified timbre.

[0190] In one embodiment, before inputting the target text to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre, the method further includes:

[0191] Obtain sample speech, sample speech text, and sample timbre information;

[0192] The model parameters of the preset speech model are adjusted based on the sample speech, the text of the sample speech, and the sample timbre information to obtain the adjusted preset speech model.

[0193] Continue to acquire the next sample speech, the text of the next sample speech, and the sample timbre information from the training sample speech set, and execute the step of adjusting the model parameters of the preset speech synthesis model based on the sample speech, the text of the sample speech, and the sample timbre information, until the training status of the adjusted speech model meets the model training termination condition, and obtain the trained preset speech model as the speech synthesis model.

[0194] In one embodiment, training the speech conversion model based on the user's speech and the specified converted speech to obtain the target speech conversion model includes:

[0195] The model parameters of the parallel speech conversion model are adjusted based on the user's speech and the specified converted speech until the model training termination condition of the parallel speech conversion model is met, thus obtaining the trained parallel speech conversion model, which serves as the target speech conversion model.

[0196] In one embodiment, before performing speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre, the method further includes:

[0197] Acquire training speech pairs and preset timbre information, wherein the training speech pairs include original speech and output speech, and the original speech and the output speech are the same speech;

[0198] The model parameters of the non-parallel speech conversion model are adjusted based on the original speech, the output speech, and the preset timbre information until the model training termination condition of the non-parallel speech conversion model is met, resulting in a trained non-parallel speech conversion model, which serves as the target non-parallel speech conversion model.

[0199] In one embodiment, adjusting the model parameters of the non-parallel speech conversion model based on the original speech, the preset timbre information, and the output speech includes:

[0200] The language feature processor of the non-parallel speech conversion model extracts the language content of the original speech to obtain the language content features of the original speech.

[0201] The prosodic feature processor of the non-parallel speech conversion model is used to extract the prosodic features of the original speech.

[0202] The model parameters of the non-parallel speech conversion model are adjusted based on the language content features of the original speech, the prosodic features of the original speech, the preset timbre information, and the output speech.

[0203] In one embodiment, the step of extracting language content features from the original speech using the language feature processor of the non-parallel speech conversion model to obtain language content features of the original speech includes:

[0204] The original speech is subjected to language information filtering processing to determine the language information corresponding to the original speech;

[0205] A first specified length vector is generated based on the language information, and the first specified length vector is used as a language content feature.

[0206] In one embodiment, the prosodic feature processor of the non-parallel speech conversion model performs prosodic extraction processing on the original speech to obtain the prosodic features of the original speech, including:

[0207] The original speech is subjected to prosodic information filtering processing to determine the prosodic information corresponding to the original speech, and a second specified length vector is generated based on the prosodic information, and the second specified length vector is used as the prosodic feature.

[0208] In one embodiment, obtaining language content features and prosodic features from the user's speech of the target user includes:

[0209] The language feature processor of the target non-parallel speech conversion model extracts the language content of the user's speech to obtain the language content features of the user's speech.

[0210] The prosodic feature processor of the target non-parallel speech conversion model is used to extract the prosodic features of the user's speech.

[0211] In one embodiment, the step of performing speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre includes:

[0212] The language content features, prosodic features, and specified timbre information of the user's speech are input into the target non-parallel speech conversion model to generate the specified converted speech with the specified timbre.

[0213] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0214] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0215] Since the computer program stored in the storage medium can execute the steps of any of the speech processing methods provided in the embodiments of this application, the embodiments of this application perform speech conversion processing based on the user's speech and specified timbre information to obtain specified converted speech with a specified timbre. The specified timbre information is timbre information determined from multiple preset timbre information, and the specified converted speech is user speech with the specified timbre. The speech conversion model is trained according to the user's speech and the specified converted speech to obtain a target speech conversion model. The target text to be synthesized and the specified timbre information are input into the speech synthesis model to generate intermediate speech with the specified timbre. The intermediate speech is processed by the target speech conversion model to generate target synthesized speech that matches the timbre of the target user. This application embodiment constructs a speech synthesis model, a non-parallel speech conversion model, and a parallel speech conversion model. The speech synthesis model synthesizes the target text into intermediate speech with a specified timbre. After acquiring the target user's speech, the parallel speech conversion model directly converts the specified timbre of the intermediate speech into the timbre of the user's speech to obtain the target synthesized speech. This enables rapid speech cloning operations, simplifying the user's voice cloning process and effectively improving efficiency. Furthermore, this application embodiment can generate corresponding parallel conversion models for user speech, allowing multiple users to share a single non-parallel speech conversion model. This simplifies the speech conversion model structure, making it lightweight and reducing the storage consumption of computer equipment.

[0216] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0217] The above provides a detailed description of a voice processing method, apparatus, computer device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A speech processing method, characterized in that, include: A non-parallel speech conversion model is used to extract language content features and prosodic features from the user's speech. The prosodic features are vectors representing the prosodic features obtained from the user's speech by a prosodic feature extraction module. The non-parallel speech conversion model is used to generate a specified converted speech with a specified timbre for training the speech conversion model, based on the user's speech and specified timbre information. The specified converted speech with the specified timbre is a speech with the specified timbre and whose semantic content and prosodic are consistent with the user's speech. The non-parallel speech conversion model performs speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre, wherein the specified timbre information is timbre information determined from multiple preset timbre information; The speech conversion model is trained based on the user's speech and the specified converted speech to obtain the target speech conversion model; The target text to be synthesized and the specified timbre information are input into the speech synthesis model to generate intermediate speech with the specified timbre. The intermediate speech is processed by the target speech conversion model to generate target synthesized speech that matches the voice of the target user.

2. The speech processing method according to claim 1, characterized in that, Before inputting the target text to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre, the process also includes: Obtain sample speech, sample speech text, and sample timbre information; The model parameters of the preset speech model are adjusted based on the sample speech, the text of the sample speech, and the sample timbre information to obtain the adjusted preset speech model. Continue to acquire the next sample speech, the text of the next sample speech, and the sample timbre information from the training sample speech set, and execute the step of adjusting the model parameters of the preset speech synthesis model based on the sample speech, the text of the sample speech, and the sample timbre information, until the training status of the adjusted speech model meets the model training termination condition, and obtain the trained preset speech model as the speech synthesis model.

3. The speech processing method according to claim 1, characterized in that, The step of training the speech conversion model based on the user's speech and the specified converted speech to obtain the target speech conversion model includes: The model parameters of the parallel speech conversion model are adjusted based on the user's speech and the specified converted speech until the model training termination condition of the parallel speech conversion model is met, thus obtaining the trained parallel speech conversion model, which serves as the target speech conversion model.

4. The speech processing method according to claim 1, characterized in that, Before performing speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre, the process further includes: Acquire training speech pairs and preset timbre information corresponding to the training speech, wherein the training speech pair includes original speech and output speech, the original speech and the output speech are the same speech, and all speech in the training speech pair is speech from the training sample speech set; The model parameters of the non-parallel speech conversion model are adjusted based on the original speech, the output speech, and the preset timbre information until the model training termination condition of the non-parallel speech conversion model is met, and the trained non-parallel speech conversion model is obtained as the target non-parallel speech conversion model.

5. The speech processing method according to claim 4, characterized in that, The adjustment of model parameters for the non-parallel speech conversion model based on the original speech, the preset timbre information, and the output speech includes: The language feature processor of the non-parallel speech conversion model extracts the language content of the original speech to obtain the language content features of the original speech. The prosodic feature processor of the non-parallel speech conversion model is used to extract the prosodic features of the original speech. The model parameters of the non-parallel speech conversion model are adjusted based on the language content features of the original speech, the prosodic features of the original speech, the preset timbre information, and the output speech.

6. The speech processing method according to claim 4, characterized in that, The language feature processor of the non-parallel speech conversion model extracts language content features from the original speech to obtain language content features of the original speech, including: The original speech is subjected to language information filtering processing to determine the language information corresponding to the original speech, and a first specified length vector is generated based on the language information, and the first specified length vector is used as the language content feature.

7. The speech processing method according to claim 4, characterized in that, The prosodic feature processor of the non-parallel speech conversion model performs prosodic extraction processing on the original speech to obtain the prosodic features of the original speech, including: The original speech is subjected to prosodic information filtering processing to determine the prosodic information corresponding to the original speech, and a second specified length vector is generated based on the prosodic information, and the second specified length vector is used as the prosodic feature.

8. The speech processing method according to claim 4, characterized in that, The process of obtaining language content features and prosodic features from the target user's speech includes: The language feature processor of the target non-parallel speech conversion model extracts the language content of the user's speech to obtain the language content features of the user's speech. The prosodic feature processor of the target non-parallel speech conversion model is used to extract the prosodic features of the user's speech.

9. The speech processing method according to claim 8, characterized in that, The speech conversion processing based on the language content features, the prosodic features, and the specified timbre information to obtain the specified converted speech with the specified timbre includes: The language content features, prosodic features, and specified timbre information of the user's speech are input into the target non-parallel speech conversion model to generate the specified converted speech with the specified timbre.

10. A voice processing device, characterized in that, include: The first acquisition subunit is used to acquire language content features and prosodic features from the user's speech of the target user through a non-parallel speech conversion model. The prosodic features are vectors obtained by the prosodic feature extraction module based on the user's speech. The non-parallel speech conversion model is used to generate a specified converted speech with a specified timbre for training the speech conversion model based on the user's speech and specified timbre information. The specified converted speech with the specified timbre is a speech with the specified timbre and whose semantic content and prosodic are consistent with the user's speech. The first processing unit is configured to perform speech conversion processing based on the language content features, the prosodic features, and the specified timbre information through the non-parallel speech conversion model to obtain a specified converted speech with a specified timbre. The specified timbre information is timbre information determined from multiple preset timbre information, and the specified converted speech is speech with language content and prosodic information corresponding to the specified timbre and the user's speech. The training unit is used to train the speech conversion model based on the user's speech and the specified converted speech to obtain the target speech conversion model; The generation unit is used to input the target text of the speech to be synthesized and the specified timbre information into the speech synthesis model to generate intermediate speech with the specified timbre. The second processing unit is used to perform speech conversion processing on the intermediate speech through the target speech conversion model to generate target synthesized speech that matches the timbre of the target user.

11. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the steps of the speech processing method as described in any one of claims 1 to 9 by calling the computer program stored in the memory.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the speech processing method as described in any one of claims 1 to 9.

Images