A voice conversion method and related equipment

Abstract

This application relates to speech processing in artificial intelligence. In scenarios where any source speech needs to be converted into a target speech with a target timbre and source speech content specified for a target sound source, speech recognition is performed on the source speech. After obtaining the speech recognition result, the speech recognition result and the target object identifier of the target sound source can be input into a pre-trained speech conversion model. Because this speech conversion model is obtained by synchronously training an acoustic model and a vocoder, the acoustic features of the trained vocoder are the predicted acoustic features output by the acoustic model that was trained simultaneously with it. This ensures the speech synthesis effect of the trained vocoder and improves the accuracy of the target speech output by the vocoder. If necessary, this application may also involve blockchain technology. The pre-trained speech conversion model and related data generated during its training process can be stored in blockchain nodes.

Description

Technical Field

[0001] The present application mainly relates to the field of speech processing technology, and more specifically to a speech conversion method and related equipment. Background Art

[0002] With the development of multimedia communication technology and artificial intelligence (AI), speech synthesis and speech recognition have become key technologies for human-computer voice communication. To meet the needs of personalized voice applications or to maintain personal voice privacy, voice conversion (VC) technology, which has been established as a result of this, can be used to convert one person's voice into another's without changing the language content.

[0003] Currently, a speech recognition model can be used to perform speech recognition on sample speech. After obtaining the sample speech recognition results, an acoustic model is trained based on them. The vocoder is then trained using the actual acoustic features of the sample speech. The trained acoustic model and vocoder form a speech conversion model. In this way, in speech conversion applications, the pre-trained acoustic model extracts features from the speech recognition results of the source speech. After obtaining the predicted acoustic features of the target speaker, these features are input into the vocoder to synthesize the target speech that matches the timbre of the target speaker.

[0004] However, in the current speech conversion method, the error of the target speech synthesized and output by the pre-trained vocoder is large, resulting in a low similarity between the target speech and the speech actually output by the target sounding object. Summary of the Invention

[0005] In view of this, the present application provides a voice conversion method, the method comprising:

[0006] Obtain the source voice of any sound-producing object and the target object identifier of the target sound-producing object;

[0007] Performing speech recognition on the source speech to obtain a speech recognition result;

[0008] Inputting the speech recognition result and the target object identifier into a speech conversion model, and outputting a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech;

[0009] The speech conversion model includes an acoustic model and a vocoder obtained through synchronous training, and the input information for training the vocoder includes output information of the acoustic model.

[0010] In some embodiments, the speech conversion model is pre-trained, and the training method includes:

[0011] Obtaining a training speech recognition result of a training speech generated by a training object; wherein the training object includes at least one sound-producing object, and the sound-producing object is configured with a corresponding object identifier; and the training speech is from the speech generated by the corresponding sound-producing object in the training data set;

[0012] Inputting the training speech recognition result and the object identifier into an acoustic model to obtain predicted acoustic features of the training object, and recording the timbre features of the sound-producing object corresponding to the object identifier;

[0013] Inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object;

[0014] Obtaining a first error between the predicted acoustic feature and a reference acoustic feature of the training object, and a second error between the predicted speech and the training speech;

[0015] During the back propagation process, the first parameter of the acoustic model is updated according to the first error, the second parameter of the vocoder is updated according to the second error, and the updated acoustic model and vocoder are trained to obtain a speech conversion model.

[0016] In some embodiments, inputting the training speech recognition result and the object identifier into an acoustic model to obtain the predicted acoustic features of the training object includes:

[0017] Performing feature extraction on the training speech recognition result to obtain speech coding features and timbre features corresponding to the object identifier;

[0018] fusing the speech coding features and the timbre features to obtain predicted acoustic features of the training object;

[0019] The step of inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object comprises:

[0020] The speech coding features and the predicted acoustic features are input into a vocoder to obtain the predicted speech of the training object.

[0021] In some embodiments, the training method further comprises:

[0022] The training object is a designated sound-producing object, and the correspondence between the object identifier of the designated sound-producing object and the timbre features extracted by the acoustic model is recorded; the target sound-producing object includes any designated sound-producing object;

[0023] The corresponding relationship is used to update the embedded timbre table represented by the encoding embedding layer of the acoustic model, and the target timbre feature corresponding to the target object identifier is obtained by querying the embedded timbre table.

[0024] In some embodiments, the training method further comprises:

[0025] Retrieving the reference fundamental frequency of the training speech;

[0026] Inputting the reference fundamental frequency of the training speech into a fundamental frequency processing model to obtain fundamental frequency characteristics of the corresponding sound object;

[0027] During the back-propagation process, updating a third parameter of the baseband processing model according to the second error;

[0028] The step of inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object comprises:

[0029] The fundamental frequency feature and the predicted acoustic feature are input into a vocoder to obtain the predicted speech of the training object.

[0030] In some embodiments, the training method further comprises:

[0031] Inputting the prosody information contained in the training speech recognition result into a fundamental frequency prediction model to obtain the predicted fundamental frequency of each of the vocalization objects contained in the training object;

[0032] Retrieve a predicted target value of the fundamental frequency of the sound object corresponding to the object identifier;

[0033] Obtaining a third error between the predicted fundamental frequency and the predicted target fundamental frequency value of the same sound object;

[0034] During the reverse transmission process, a fourth parameter of the fundamental frequency prediction model is updated according to the third error.

[0035] In some embodiments, training the updated acoustic model and the vocoder to obtain a speech conversion model includes:

[0036] Training the updated acoustic model and the vocoder until a first training constraint condition is satisfied, and stopping training the acoustic model;

[0037] Inputting the predicted speech output by the vocoder in the next training and the corresponding training speech into the discriminator, updating the second parameter of the vocoder according to the discrimination result, and training the vocoder with the updated second parameter until the second training constraint condition is satisfied, thereby obtaining a speech conversion model;

[0038] The first training constraint condition is configured for the training process of the acoustic model; and the second training constraint condition is configured for the training process of the vocoder.

[0039] In some embodiments, the method for obtaining the fundamental frequency prediction target value of the sound object includes:

[0040] Obtain multiple voices produced by the same sound-producing object in the training data set;

[0041] Extracting a reference fundamental frequency of each of the plurality of speech sounds;

[0042] Normalization processing is performed on the reference fundamental frequencies of the multiple voices of the same sound-producing object to obtain a predicted target value of the fundamental frequency of the corresponding sound-producing object.

[0043] In some embodiments, inputting the speech recognition result and the target object identifier into a speech conversion model, and outputting a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech, comprises:

[0044] Inputting the speech recognition result and the target object identifier into the pre-trained acoustic model to obtain target speech coding features and target acoustic features;

[0045] Inputting the prosody information contained in the speech recognition result into the pre-trained fundamental frequency prediction model to obtain a target predicted fundamental frequency;

[0046] Retrieving normalized fundamental frequency information corresponding to the target object identifier, and performing denormalization processing on the target predicted fundamental frequency using the normalized fundamental frequency information of the target sounding object to obtain a predicted reference fundamental frequency of the target sounding object;

[0047] Inputting the predicted reference fundamental frequency into the pre-trained fundamental frequency processing model to obtain a target fundamental frequency feature of the target sounding object;

[0048] The target speech coding feature, the target acoustic feature and the target fundamental frequency feature are input into a pre-trained vocoder, and a target speech of the target sounding object is output; the target speech has a target timbre feature corresponding to the target object identifier and the content of the source speech.

[0049] In some embodiments, inputting the target speech coding feature, the target acoustic feature, and the target fundamental frequency feature into a pre-trained vocoder to output the target speech of the target sounding object includes:

[0050] Inputting the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature and target prosody feature in the target speech coding feature into a pre-trained vocoder to output the target speech of the target sounding object; or

[0051] Inputting the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature in the target speech coding feature into a pre-trained vocoder to output the target speech of the target sounding object; or

[0052] The target speech coding feature, the target acoustic feature, the target fundamental frequency feature and the preset speech energy feature are input into a pre-trained vocoder, and the target speech of the target sounding object is output.

[0053] In another aspect, the present application further provides a speech conversion device, comprising:

[0054] A data acquisition module, used to acquire the source voice of the source sounding object and the target object identifier of the target sounding object;

[0055] A speech recognition module is used to extract features from the source speech to obtain a speech recognition result;

[0056] a speech conversion module, configured to input the speech recognition result and the target object identifier into a speech conversion model, and output a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech;

[0057] The speech conversion model includes an acoustic model and a vocoder obtained through synchronous training, and the input information for training the vocoder includes output information of the acoustic model.

[0058] In another aspect, the present application further provides a computer device, comprising: at least one memory and at least one processor, wherein:

[0059] The memory is used to store a program for implementing the above-mentioned voice conversion method;

[0060] The processor is used to load and execute the program stored in the memory to implement the voice conversion method as described above.

[0061] On the other hand, the present application also proposes a computer-readable storage medium on which a computer program is stored. The computer program is loaded and executed by a processor to implement the voice conversion method as described above.

[0062] On the other hand, the present application also proposes a computer program product, including computer instructions, which are read and executed by a processor to implement the voice conversion method as described above.

[0063] It can be seen that the present application relates to speech processing in artificial intelligence, such as a speech conversion method. In a scenario where any source speech needs to be converted into a target speech that conforms to a specified target sound-producing object and retains the content of the source speech, speech recognition is performed on the source speech. After obtaining the speech recognition result, the speech recognition result and the target object identifier of the target sound-producing object can be input into a pre-trained speech conversion model. Since the speech conversion model is obtained by synchronously training the acoustic model and the vocoder, the acoustic features of the trained vocoder are the predicted acoustic models output by the acoustic model that is trained synchronously with it, thereby ensuring the speech synthesis effect of the vocoder trained in this way and improving the accuracy of the output target speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0065] Figure 1 A schematic diagram showing an optional example of an architecture of a speech conversion system applicable to the speech conversion method proposed in this application;

[0066] Figure 2 A schematic diagram of the hardware structure of an optional example of a computer device suitable for the speech conversion method proposed in this application is shown;

[0067] Figure 3 A schematic diagram of the hardware structure of another optional example of a computer device suitable for the voice conversion method proposed in this application is shown;

[0068] Figure 4 A flowchart of an optional example of the voice conversion method proposed in this application is shown;

[0069] Figure 5 A flowchart illustrating another optional example of the voice conversion method proposed in this application is shown;

[0070] Figure 6 A flowchart illustrating another optional example of the voice conversion method proposed in this application is shown;

[0071] Figure 7 A schematic diagram showing an optional training method for a speech conversion model in the speech conversion method proposed in this application is shown;

[0072] Figure 8 A flowchart illustrating another optional example of the voice conversion method proposed in this application is shown;

[0073] Figure 9 A flowchart illustrating an optional example of implementing the voice conversion method by calling a pre-trained voice conversion model in the voice conversion method proposed in this application is shown;

[0074] Figure 10 A schematic structural diagram of an optional example of a speech conversion device proposed in this application is shown;

[0075] Figure 11 A structural diagram of another optional example of the speech conversion device proposed in this application is shown. DETAILED DESCRIPTION

[0076] Regarding the description of the background technology, after research, it is known that this is because the voice changing model and vocoder in the speech conversion system are trained independently, resulting in the real acoustic features input into the voice changer during the training phase (acoustic features extracted directly from the voice of the target voice-producing object) and the predicted acoustic features input during the use phase (that is, the acoustic model output by the pre-trained voice changing model) being inconsistent. In this way, in the vocoder application, the prediction error of the vocoder will increase, so that the acoustic feature error predicted by the voice changing model is superimposed on the prediction error of the vocoder itself, resulting in a decrease in the synthesis effect of the vocoder, that is, the output result of the vocoder model trained based on the real acoustic features in the voice changing application has low accuracy, affecting the overall voice changing effect.

[0077] In order to solve the above problems, the present application proposes to jointly train the voice changing model and the vocoder, that is, in the pre-training stage of the speech conversion model, the voice changing model and the vocoder are trained synchronously, so that the input information of the training vocoder comes from the output information of the voice changing model, rather than the acoustic features of the training speech. In this way, during the training process, the prediction error of the acoustic model that starts training synchronously and the prediction error of the vocoder itself can be considered to adjust the model parameters of the vocoder, and ensure that the trained vocoder fuses the input information containing the acoustic features output by the acoustic model during use to obtain a high-precision target speech, that is, a target speech with a high similarity to the speech output by the target sound-producing object, thereby improving the overall voice changing effect.

[0078] Among them, the above-mentioned speech conversion model can use artificial intelligence (AI) technology to train and process input information. During the processing, speech technologies such as speech recognition technology and speech synthesis technology, as well as machine learning and deep learning algorithms such as artificial neural networks, belief networks, and reinforcement learning can be used. This application does not limit the artificial technology involved in the speech conversion method and its application, which can be determined according to the situation.

[0079] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0080] It should be understood that the terms "system," "device," "unit," and / or "module" used in this application are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0081] As used in this application and the claims, unless the context clearly indicates an exception, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular and may include the plural, unless the context clearly indicates otherwise. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements. The phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus that includes the elements.

[0082] Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two. The following terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0083] In addition, flow charts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0084] Reference Figure 1, is an architectural diagram of an optional example of a voice conversion system applicable to the voice conversion method proposed in this application. The application scenario of this voice conversion system can refer to an application scenario in which the source voice (i.e., the voice produced by any sound-producing object) is converted into a target voice that conforms to the target timbre of the target sound-producing object (i.e., any specified sound-producing object) while keeping the semantic content of the source voice (i.e., the voice produced by any sound-producing object) unchanged, such as a short video production scenario, a live voice-changing scenario, an audio novel broadcast scenario, etc. This application does not limit the voice conversion scenario, which can be determined according to the circumstances. Figure 1 As shown, the speech conversion system proposed in this application may include but is not limited to: a client 11 and a server 12, wherein:

[0085] The client 11 may be an application running on a terminal device that supports the voice conversion function, such as a voice assistant, a speech synthesis engine, or a dedicated application installed on the terminal device, or a web application on a website that supports the voice conversion function. This application does not limit the type of the client 11. The terminal device may include, but is not limited to, a smartphone, a tablet computer, a desktop computer, a wearable device (such as a smartwatch, an augmented reality (AR) device, a virtual reality (VR) device, etc.), a netbook, an e-book reader, an audio player, a vehicle-mounted terminal, a smart home device, a smart medical device, a smart transportation device, a robot, and other electronic devices.

[0086] In some embodiments, the client 11 can be an application that supports multiple voice processing functions such as voice recognition and voice changing. In this case, the terminal device can directly send any source voice collected or received to the client 11, and the client executes the voice conversion method proposed in this application, converts it into the target voice of the specified target sound object, and then sends it to the audio and video player for playback. Optionally, the client 11 can also include an APP (Application) that supports different voice processing functions, such as a voice recognition APP, a voice conversion APP / voice changing processing APP, and other applications. This application does not limit the composition structure of the client 11, which can be determined according to the circumstances.

[0087] In some other embodiments, especially for terminal devices with low data processing capabilities, the client 11 can also send the received source voice and the target object identifier of the target sound-making object to the server 12, and the server 12 executes the voice conversion method proposed in this application, obtains the required target voice, and then feeds it back to the client 11 for output. This application does not limit the execution subject of the voice conversion method, which can be determined according to the situation.

[0088] Server 12 can be a service device that provides corresponding services for the voice conversion function implemented by client 11. It can be a standalone physical server, a server cluster consisting of multiple physical servers, or a cloud server with cloud computing capabilities. Based on the above analysis, server 12 can achieve data communication with terminal device 11 via a wired or wireless network to meet application requirements. The implementation process varies depending on the application and is not described in detail in this embodiment.

[0089] Following the above analysis, for the speech recognition model, speech conversion model and other models in the speech conversion method proposed in this application, they are usually pre-trained based on the corresponding training samples. After obtaining the model that meets the corresponding training constraints, the trained model is stored. In this way, in the speech conversion application scenario, the required model can be directly called to process the speech or speech sequence features in the application scenario to obtain the required processing results.

[0090] In the examples of this application, for the speech conversion model that implements speech conversion processing, the server can execute a corresponding training method so that the speech conversion model can have the ability to filter out the timbre characteristics of the source sound object, integrate the timbre characteristics of the target sound object, and convert the source speech into the target speech. Regarding the training implementation process of the speech conversion model, please refer to the description of the corresponding part of the method embodiment below. The examples of this application are not described in detail here. It should be noted that the above-mentioned speech conversion model can be integrated with the above-mentioned speech recognition model. Of course, the speech recognition model can also be an independent model different from the speech conversion model. This application does not limit this.

[0091] It should be understood that the above Figure 1 The system architecture in the voice conversion scenario shown does not constitute a limitation on the voice conversion system in the embodiment of the present application. In actual applications, the system may include Figure 1 More devices or combined devices shown, such as database servers, or other communication servers of third-party platforms, etc., are not listed one by one in this application.

[0092] Combined with the above analysis, refer to Figure 2 , is a hardware structure diagram of an optional example of a computer device suitable for the voice conversion method proposed in this application. The computer device can be the above-mentioned server 12 or terminal device. Thus, in actual application, the voice conversion method proposed in this application can be executed by the client 11 running on the server 12 or the terminal device, or the server 12 and the client 11 can cooperate to implement the voice conversion method, which can be determined according to actual application requirements. Taking the computer device as a server as an example, Figure 2 As shown, the computer device may include but is not limited to: at least one memory 21 and at least one processor 22, wherein:

[0093] The memory 21 can be used to store a program for implementing the speech conversion method described in the embodiments of the present application; the processor 22 can be used to load and execute the program stored in the memory 21 to implement the various steps of the speech conversion method described in the corresponding method embodiment. The specific implementation process can refer to but is not limited to the description of the corresponding part of the embodiment below.

[0094] In the embodiment of the present application, the memory 21 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. The processor 22 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices. This application does not limit the number and type of memory 21 and processor 22 included in the computer device, which may be determined as appropriate.

[0095] In some embodiments, the computer device proposed in this application may also include: a communication interface for realizing data transmission between the computer device and an external device (such as a terminal device, a storage server or other communication server, etc.), and realizing data transmission between the internal components of the computer device. The communication interface may include a network interface suitable for a wireless communication network and / or a wired communication network, specifically a data interface of a communication module such as a GSM module, a WIFI module, 4G / 5G / 6G (fourth generation / fifth generation / sixth generation mobile communication network), near field communication (NFC), etc., and may also include interfaces such as USB interfaces, serial / parallel ports, I / O, etc. to meet the communication needs between different components within the computing device, as well as the communication needs between the computer device and other local devices. This application does not limit the number and type of communication interfaces included, which can be determined based on the functional requirements of the computer device. This embodiment does not list them one by one.

[0096] Based on this, in an embodiment of the present application, the terminal device running the client 11 can realize data communication between the two through a communication interface matching the server 12, that is, build a data communication link between the two, so that the client 11 can send information such as the source voice and the target object identification of the target sound-making object to the server 12 through the data communication link, and the server 12 executes the voice conversion processing method proposed in this application, and feeds back the obtained target voice to the client 11 through the data communication link for playback. This application does not impose any restrictions on the communication protocol followed by the data communication link, the data transmission implemented according to the communication protocol, and other implementation methods, which can be determined as appropriate.

[0097] In some other embodiments proposed in this application, for example, when the computer device is a terminal device running the client 11, Figure 3 As shown, the computer device may also include: at least one input device such as a touch sensing unit for sensing touch events on a touch display panel, a camera, a microphone, etc.; at least one output device such as a display, a speaker, etc.; a power supply component for supplying power to various components in the computer device, which may include a power management module, at least one power supply, and associated components for implementing functions such as power distribution and management; a sensor module containing one or more sensors for providing different aspects of status information to the computer device, such as input status, motion posture, sensing the distance to an approaching object, etc.; a communication module for supporting wired or wireless communication between the computer device and other devices, such as a communication module corresponding to the communication interface listed above, an antenna, etc. It can be determined based on the application function requirements of the terminal device, and may include but is not limited to the components listed in this application. The components included in different types of terminal devices may be different, and this application does not list them all, and Figure 3 The components included in the computer device listed in this application are not shown.

[0098] therefore, Figure 2 and Figure 3 The structure of the computer device shown does not constitute a limitation on the computer device in the embodiment of the present application. In actual applications, the computer device may include Figure 2 or Figure 3 More or fewer components as shown, or combinations of certain components, are not listed here one by one in this application.

[0099] It can be understood that for the voice conversion method or device proposed in this application, the server therein can be a node on the blockchain, and the synthesized target data, pre-trained voice conversion model and other related data can be saved on the blockchain for users to call at any time. The implementation process is not described in detail in this application.

[0100] Based on the above description of the application scenario and its system architecture, the solution proposed in the embodiment of the present application involves artificial intelligence speech recognition, speech synthesis, machine learning / deep learning and other technologies, which can be illustrated by the following but not limited to the following embodiments:

[0101] Reference Figure 4, is a flow chart of an optional example of the voice conversion method proposed in this application. In any voice conversion scenario, facing the need to convert the source voice of any sound-producing object into the target voice of a specified target sound-producing object, the voice conversion method proposed in this application can be executed by the server or the client or both. This application does not limit the execution subject of the voice conversion method, which can be determined according to the situation. This embodiment of the application is explained by taking the server execution as an example. Figure 4 As shown, the voice conversion method may include but is not limited to the following steps:

[0102] Step S41, obtaining the source voice of any sound-producing object and the target object identifier of the target sound-producing object;

[0103] The speech conversion application described in the embodiment of the present application may refer to a voice-changing application scenario in which the source speech of any sound-producing object is converted into the target speech of a target sound-producing object without changing the audio content, rhythm, and other irrelevant information of the sound-producing object. The present application does not impose any restrictions on the source of the source speech and its sound-producing object, and can be determined according to application requirements. The target sound-producing object can be a designated sound-producing object specified by the user or assigned by the system. In order to distinguish different sound-producing objects, unique object identifiers can be configured for different sound-producing objects. In this way, when the user notifies the server of the selected or uploaded audio to be converted, i.e., the source speech, through the client, the target object identifier of the target sound-producing object can be uploaded at the same time, so that the server knows which sound-producing object's audio the audio to be converted needs to be converted.

[0104] Based on the above analysis, in some embodiments, a server may receive a voice conversion request from a client. This voice conversion request may include the source voice and the target object identifier. The server may then parse the voice conversion request to obtain the source voice and target object identifier. Of course, the source voice and target object identifiers are not limited to being obtained simultaneously. They may also be obtained sequentially based on actual application requirements. This application does not provide a detailed example.

[0105] Among them, the above-mentioned object identifier is a unique identifier of the sound-making object, which can be a number, personal biometric information, identity information, etc., used to distinguish the different timbre characteristics of different sound-making objects. This application does not limit the content of the object identifier.

[0106] It should be understood that in different speech conversion scenarios, the implementation method of the server obtaining the source speech and / or target object identifier may be different. For example, when the source speech is sent from the server to the client, when the client requests to convert it into the target speech of the target sounding object, the client can directly send a speech identifier that can characterize the source speech or its information source to the server, so that the server can determine the source speech to be converted and execute the speech conversion method accordingly; if the source speech is collected by a terminal device running the client or sent by an external device, the client can directly upload it to the server for conversion processing in the manner described above. Of course, the client can also directly execute the speech conversion method to obtain the target speech. This often depends on the subject executing the speech conversion method, and this application does not impose any restrictions.

[0107] Step S42: performing recognition processing on the source speech to obtain a speech recognition result;

[0108] According to the working principle of speech conversion technology, for the source speech output by the source sound object (i.e., the sound object of the source speech, which can be the original sound object, or the sound object of the speech obtained after one or more speech conversion processes, etc.), this application can use automatic speech recognition technology (Automatic Speech Recognition, ASR) to realize speech recognition of the source speech, and obtain information related to pronunciation, rhythm, semantic text, etc. in the source speech. The speech recognition results obtained may include but are not limited to: discrete variable labels and continuous variable PPGs (Phonetic Posteriorgrams phoneme posterior probability). The discrete variable labels can be the pronunciation information of one or more consecutive speech frames contained in the source speech, and PPGs can include the rhythm information of the source speech. It can be seen that the speech recognition results such as the pronunciation information and rhythm information are feature information of the time series extracted from different speech frames. This application does not limit the speech recognition processing method.

[0109] Optionally, the present application can pre-train a speech recognition model for implementing speech recognition processing, which can be a neural network structure obtained by training sample audio based on ASR technology, or it can use an unsupervised pre-trained convolutional neural network such as wav2vec to learn the structure of speech from the original audio to obtain a speech recognition model, etc. The present application does not limit the training implementation method of the speech recognition model and its model network structure.

[0110] Step S43: Input the speech recognition result and the target object identifier into the speech conversion model, and output the target speech having the target timbre features corresponding to the target object identifier and the content of the source speech.

[0111] In combination with the above description of the technical solution of the present application, in order to solve the technical problem that the acoustic model and the vocoder are trained separately, resulting in the input information source of the vocoder training stage being inconsistent with the input information source of the use stage, reducing the accuracy of the vocoder output and affecting the voice changing effect, the present application proposes to jointly train the acoustic model and the vocoder, that is, to synchronously train the acoustic model and the vocoder included in the speech conversion model, and the input information used to train the vocoder includes the output information of the acoustic model.

[0112] Therefore, during the model training phase, the vocoder and the acoustic model can start training synchronously. During each training process, the predicted acoustic features output by the acoustic model of this training can be directly input into the vocoder to complete one training of the vocoder. After several training sessions, the vocoder trained in this way can effectively reduce the prediction error of the vocoder itself in actual speech conversion applications, compared with the vocoder trained based on the real acoustic model, thereby improving the overall voice changing effect.

[0113] Based on this, in the implementation process of the above-mentioned step S43, the speech recognition result and the target object identifier are sent to the pre-trained acoustic model, and the acoustic model determines the target timbre feature corresponding to the target object identifier, so as to combine the target timbre feature and perform encoding and decoding processing on the speech recognition result such as pronunciation information, rhythmic information, etc., so as to obtain the target acoustic feature that integrates the target timbre feature of the target sound object, and input it into the pre-trained voice coder for synthesis processing to obtain the target speech content that retains the source speech and conforms to the timbre of the target sound object, which is equivalent to the content of the source speech directly spoken by the target sound object, meeting the user's voice changing application needs. The implementation process can refer to the description of the corresponding part of the detailed embodiment below, and this embodiment will not be described in detail here.

[0114] Afterwards, when the server executes the voice conversion method, the server can feed back the target voice to the client output that provides the source voice or the preset terminal operation, or the client can perform other application processing based on the obtained target voice, such as fusing it with other multimedia information to obtain the required target video, etc. This implementation process can also be implemented by the server. This application does not limit the application processing method after the server or client executes the voice conversion method and obtains the target voice, which can be determined according to the situation.

[0115] For example, in a live voice-changing scenario, the live server can perform voice conversion on the recorded host's voice to obtain a live voice with the timbre of a specified target object (such as the timbre of a celebrity, the timbre of a virtual object personalized by the host, etc.), which can be integrated with the live image content, and then the obtained live video can be sent to the live client of each viewer who enters the host's live broadcast room for playback, so that when the audience watches the live broadcast, what they hear is the voice of the target voice object rather than the host's own voice, which helps to improve the fun of the live broadcast and can also meet the needs of host voice protection.

[0116] In a short video production scenario, for example, the characteristics of each sound-producing object contained in the short video and the personal production requirements of the short video producer can be used to extract the voice of any sound-producing object in the short video as the source voice, and specify the voice of the sound-producing object to be converted into. That is, after specifying the target sound-producing object, the source voice is converted into a target voice that conforms to the timbre of the target sound-producing object according to the voice conversion method proposed in this application. Then, the source short video is updated using the target voice corresponding to the source voice of each object in the short video, and the produced target short video can be obtained for publication. It can be understood that the voice conversion implementation process in other voice conversion application scenarios is similar, and this application will not give detailed examples one by one here.

[0117] In scenarios such as making phone calls and voice calls on social software, users can also configure in advance on the local client the timbre of the target sound object they want to convert to. In this way, the client can use the voice received in real time (such as the audio collected by the audio collector) as the source voice, and then convert the source voice into the target voice according to the voice conversion method described above and send it to the other party of the call. It should be understood that the voice change processing process in this voice call scenario can also be performed by the communication server that provides voice call services, executing the voice conversion method described above, converting the source voice sent by any call client, and sending the obtained target voice to the other party of the call for output, etc. In this implementation process, the target sound object representation can be pre-specified by the communication party or specified by the communication server, and this application does not impose any restrictions on this.

[0118] It should be noted that the voice changing scenarios applicable to the voice conversion method proposed in this application include but are not limited to the application scenarios listed above. When facing the need to change the source voice of any sound-making object into the target voice of the timbre of a specified target sound-making object, it can be implemented according to the voice conversion method described above. This application does not give detailed examples one by one.

[0119] In summary, this application starts training the acoustic model and the vocoder simultaneously, so that the input during the vocoder training process is the predicted acoustic model output by the acoustic model. In this way, when using the speech conversion model constructed in this way to process the speech recognition results of the source speech, the vocoder can retain the content of the source speech and perform synthesis processing based on the acoustic features output by the acoustic model. It can more accurately and reliably predict the target speech of the target sounding object, improve the overall voice changing effect, and better meet the needs of voice changing applications.

[0120] Reference Figure 5 , which is a flow chart of another optional example of the voice conversion method proposed in this application. The embodiment of this application can describe the training process of the voice conversion model in the voice conversion method, but is not limited to the model training implementation method described in this embodiment, and the model training method can be executed by the above-mentioned computer device. This embodiment takes the scenario of the server implementing the model training method as an example to illustrate, such as Figure 5 As shown, the method may include:

[0121] Step S51, obtaining the training speech generated by the training subject;

[0122] In an embodiment of the present application, the above-mentioned training object may include at least one sound-producing object, and the sound-producing object is configured with a corresponding object identifier. The training speech may come from the speech produced by the sound-producing objects included in the training object in the training data set (such as a collection of multiple speech sounds produced by different sound-producing objects, which can be extracted from a corpus database. This application does not limit the content and source of the training data set).

[0123] Among them, when the training object contains multiple sound objects, the training voice obtained can include one or more voices of each of the multiple sound objects, and they are combined into training voice in a certain order. It can be seen that the training voice obtained at this time can be a mixed voice of multiple people, but there is no limit on the number of sound objects it contains.

[0124] It should be noted that, to facilitate subsequent processing, this application may record the order of the sound objects corresponding to the various speech sounds that constitute the training sentence, such as by utilizing the object identifiers of multiple sound objects to achieve the recording of the ordering results, but is not limited to this implementation method. In addition, when obtaining the above-mentioned training speech, one or more speech sounds may be randomly selected from the training data set to constitute the training speech. This application does not limit the method for obtaining the training speech.

[0125] Step S52: input the training speech into the pre-trained speech recognition model to obtain the training speech recognition result;

[0126] For the relevant description of the speech recognition model, please refer to the description of the corresponding part of the above embodiment, and the embodiments of this application will not be described in detail here.

[0127] In the embodiment of the present application, since the training processes of the speech recognition model and the speech conversion model are independent of each other, after the speech recognition model is trained separately, the present application directly calls the pre-trained speech recognition model to perform feature extraction on the training speech to meet the training requirements of the speech conversion model, thereby ensuring the consistency of the method for obtaining the speech recognition results input in the training phase and the use phase of the speech conversion model, so as to improve the training efficiency and accuracy of the speech conversion model.

[0128] Step S53: Input the training speech recognition result and the object identifier into the acoustic model to obtain the predicted acoustic features of the training object, and record the timbre features of the sound-producing object corresponding to the object identifier;

[0129] In an embodiment of the present application, the acoustic model adopts a neural network structure composed of an acoustic encoder (Voice Conversion Encoder) and an acoustic decoder (Voice Conversion Decoder), and uses the acoustic encoder to realize feature extraction of the training speech recognition results, converts the input speech recognition results of variable length into feature vectors of corresponding length, and extracts the hidden features of the input information during the conversion process to obtain information related to the speech content that is unrelated to the source sound object.

[0130] Exemplarily, the above-mentioned acoustic encoder can be an encoding network obtained by training a convolutional neural network. Under this network structure, the input information (such as the pronunciation information, rhythmic information and other identification features contained in the training speech recognition results) can be mapped to the hidden feature space through multiple convolutional layers of the convolutional neural network, and then the distributed hidden features can be mapped to the sample label space through the fully connected layer to realize the classification and integration of the hidden features. The implementation process is not described in detail in this application. It should be noted that the network structure of the acoustic encoder is not limited to this convolutional neural network. Other types of neural networks such as recurrent neural networks can also be used. According to actual application requirements, feature extraction can also be achieved by combining spatial / semantic attention mechanisms to improve the reliability and accuracy of the obtained audio coding features. This application does not limit the network structure of the acoustic encoder.

[0131] In addition, during the feature extraction process of the above-mentioned acoustic encoder, the extracted timbre features of the sound-emitting object can also be associated with the object identifier. The embedding layer of the acoustic encoder represents an embedded timbre table (such as an embedding table) for recording the correspondence between the timbre features and the object identifier. The present application does not limit the form of the embedded timbre table. The initial timbre features in the embedded timbre table can be directly extracted from a certain speech of the corresponding sound-emitting object, but are not limited to this.

[0132] It should be noted that in the process of continuous training of the acoustic encoder, the timbre features extracted from the speech of the same sound-making object input in each training can be used to update the timbre features corresponding to the object identifier of the sound-making object in the embedded timbre table, so as to achieve learning optimization of the timbre features of different sound-making objects in the embedded timbre table and improve the reliability and accuracy of the recorded timbre features.

[0133] Afterwards, the acoustic decoder can be used to fuse the encoding processing results output by the acoustic encoder to predict acoustic features that have the timbre of the training object and are consistent with the training speech content, which are recorded as predicted acoustic features. The feature fusion processing method implemented by the acoustic decoder can be determined based on the network working principle of the decoder, and this application will not elaborate. Optionally, the acoustic decoder can also adopt a neural network structure to achieve fusion processing of features of different dimensions based on the feature fusion processing capabilities of the neural network to obtain the required predicted acoustic features. This application does not limit the network structure of the acoustic decoder.

[0134] Step S54, inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object;

[0135] A vocoder is a speech signal encoder / decoder that utilizes model parameter prediction and speech synthesis technology. In speech conversion technology applications, the vocoder can synthesize a target speech that matches the timbre of the target sound-producing object based on the predicted acoustic characteristics of the target sound-producing object. This application does not detail the speech synthesis processing method of the vocoder in speech conversion applications, as it can be determined based on the working principle of the vocoder.

[0136] It can be seen that in the process of training the speech conversion model, the present application will jointly train the acoustic model and vocoder it contains. In this way, each time the vocoder is trained, the predicted acoustic features output by the acoustic model trained at the same time are directly input into the vocoder for training, so that the input of the vocoder in the training stage is consistent with the input in the use stage, that is, both are output results from the acoustic model. Compared with the method of independently training the vocoder and inputting the real acoustic model for training, the vocoder obtained by this joint training method of the present application participates in speech conversion in actual voice change scenarios, which can effectively improve the accuracy of the target speech output by the vocoder, that is, increase the similarity between the predicted target speech and the real speech of the target sounding object.

[0137] Step S55, obtaining a first error between the predicted acoustic feature and the reference acoustic feature of the training object, and a second error between the predicted speech and the training speech;

[0138] The embodiment of the present application can adopt a multi-task training method to realize the training of the speech conversion model, improve the model training efficiency and prediction accuracy. Therefore, during the training process of the entire speech conversion model, the present application can realize parameter adjustment of different parts of the model based on the prediction target, which can be determined according to the training requirements of each part.

[0139] Among them, combined with the training implementation process of the acoustic model and vocoder described above, since these two parts adopt a joint training method, feature association is realized in the entire model forward propagation calculation process. The present application can obtain the prediction errors of each part of the speech conversion model according to this forward propagation processing method, such as step S55, to obtain the first error of the acoustic model prediction result and the second error of the vocoder prediction result, thereby characterizing the accuracy of the corresponding part prediction result. The present application does not limit the method for obtaining the error and its representation method, which can be determined according to the situation.

[0140] Based on this, in an embodiment of the present application, a suitable preset loss function, such as a cross-entropy loss function, can be called to obtain a first loss between the predicted acoustic features and the reference acoustic features, and a second loss between the predicted speech and the training speech, so as to characterize the error of the prediction result of the corresponding part based on the obtained loss, so as to subsequently adjust the model parameters of the corresponding part accordingly. This application does not limit the type of loss function based on which the error of the prediction results of the above-mentioned acoustic model and vocoder are based, and is not limited to the loss calculation method described in this embodiment. In addition, for the prediction results of different components in the speech conversion model, the same or different error acquisition methods can be used to obtain the error of the corresponding prediction results.

[0141] In some embodiments, in the process of obtaining the above-mentioned second error, it is often impossible to directly calculate the loss of the two speech signals, the predicted speech and the training speech. The present application proposes that a unified signal processing method can be used to pre-process the predicted speech and the training speech to obtain the corresponding predicted speech features and training speech features. After that, the loss function can be used to calculate the loss between the predicted speech features and the training speech features, that is, the loss between the predicted speech and the training speech, thereby obtaining the above-mentioned second error.

[0142] In one possible implementation, the above-mentioned preprocessing method may include but is not limited to a short-time Fourier transform (STFT) processing method, which is used to determine the frequency and phase of the local area sine wave in the corresponding speech signal (such as predicted speech or training speech), thereby obtaining the speech characteristics of the speech signal so that it can be suitable for loss calculation requirements.

[0143] In some other embodiments, the present application can use a GAN (Generative Adversarial Network) training method to realize the error calculation of the prediction results of the speech conversion model. In this case, the above-mentioned acoustic model and vocoder can both be used as generators, and the corresponding discriminator can be configured. The prediction results output by the corresponding generator and the prediction targets corresponding to the prediction results are input into the discriminator for prediction scoring, and the size of the obtained prediction score is used to represent the accuracy or error of the prediction result.

[0144] Therefore, in some other embodiments, the present application may input the predicted acoustic features and the reference acoustic features of the corresponding sound-producing object (i.e., the real acoustic features of the sound-producing object obtained by extracting features from the speech of the sound-producing object in the training speech) into the first discriminator, score the accuracy of the predicted acoustic features, and obtain a first discrimination result for the predicted acoustic features, thereby determining the above-mentioned first error based on the obtained first discrimination result. It can be seen that the first error can be the score obtained by the scoring operation, or the error value determined based on the score, etc. The present application does not limit the content of the first error. Similarly, the predicted speech and the training speech can be input into the second discriminator for scoring, and then the above-mentioned second error is determined based on the obtained second discrimination result. The present application does not elaborate on the implementation process of this adversarial learning training method.

[0145] It should be noted that when the above-mentioned training objects include multiple sound objects and the training speech includes the speech of each of these multiple sound objects, since these multiple speech can form a training speech in a certain order, in the process of obtaining the predicted acoustic features, the speech recognition results of each sound object in the training speech and the timbre characteristics of the sound object can be fused to obtain the predicted acoustic features of the sound object. According to this processing method, the present application can obtain the predicted acoustic features of each sound object that constitutes the training object, and then obtain the first error of the predicted acoustic features of the corresponding sound object according to but not limited to the method described above.

[0146] As can be seen, in this mixed speech training scenario, the above-mentioned first error can include multiple prediction errors for different sound objects. The process of obtaining the prediction error of the acoustic characteristics of different sound objects is similar and will not be described in detail in this application. It is understood that if the training object refers to a sound object, the above-mentioned first error refers to the prediction error of the acoustic characteristics of this training object.

[0147] Step S56, updating a first parameter of the acoustic model based on the first error, and updating a second parameter of the vocoder based on the second error, training the updated acoustic model and vocoder until the first training constraint is satisfied, and then stopping training the acoustic model;

[0148] Since the first error and the second error above respectively characterize the prediction accuracy of the corresponding parts in the speech conversion model, in the process of performing gradient backpropagation based on the error and updating the model parameters, in order to eliminate the problem of model instability caused by the simultaneous update of the same model parameters by multiple prediction targets, the embodiment of the present application proposes that the errors of the respective prediction targets of the acoustic model and the vocoder are only used to implement the parameter update of the corresponding parts, that is, the acoustic feature prediction target of the acoustic model will be responsible for training the acoustic model. Therefore, in the backpropagation process, the first parameter of the acoustic model will be updated based on the first error, such as updating the network parameters of the neural network constituting the acoustic model. The first parameter may include but is not limited to the weight matrix, which may be determined according to the circumstances.

[0149] Similarly, the speech prediction target of the vocoder will be responsible for training the vocoder. Thus, during the above-mentioned reverse transmission process, the second parameters of the vocoder will be updated based on the second error. The second parameters may include, but are not limited to, the weight matrix constituting the vocoder, etc., and can be determined based on the network structure of the vocoder and its network parameters that affect the accuracy of the vocoder output. This embodiment will not be described in detail.

[0150] Therefore, to improve model training efficiency and reliability, the acoustic model and vocoder can be trained simultaneously. During this training phase, the two models simultaneously update their own model parameters based on the errors in their respective prediction results. This synchronizes the training time of the acoustic model and vocoder, thus saving training time. However, it should be noted that if model training efficiency is not a consideration, the parameter updates of the two models can also be performed sequentially and are not limited to synchronous updates.

[0151] It is understood that since each training process of the speech conversion model is similar, after the model parameters of the corresponding part of the speech conversion model are updated once using the obtained errors according to the method described above, the method described above can be repeated to continue training the acoustic model with the updated first parameters and the vocoder with the updated second parameters until the preset training constraints are met. The acoustic model and vocoder obtained by the final training constitute the desired speech conversion model. The training constraints may include reaching a preset number of model training times, reaching the minimum error, or achieving stable convergence, etc., at which point training can be terminated. This application does not limit the content of the training constraints, which can be determined as appropriate.

[0152] In addition, in the actual application of this application, the acoustic model and vocoder training can be implemented according to the method described in the steps above to obtain a speech conversion model, which can be used to call the computer device in the voice change scenario to convert any source speech into a target speech that conforms to the target sounding object, and retain the content information of the source speech. According to the voice change requirements, adaptive adjustments can also be made based on the training method described above. The implementation process is similar to the training process described above and will not be described in detail in this application.

[0153] Based on this, in order to further improve the synthesis stability and synthesis effect of the vocoder, in some embodiments, the entire training process of the speech conversion model can be divided into two stages. The above steps can belong to the first training stage of the model. In this training stage, as described above, the acoustic feature prediction target and the speech prediction target can be used to achieve synchronous training of the acoustic model and the vocoder, thereby obtaining a stable acoustic model and a relatively stable vocoder. As described above, the first training stage can determine whether to terminate the training of this stage based on the training effect of the acoustic model, and obtain an acoustic model that meets the first training constraints. Subsequently, this acoustic model will no longer be trained, and the trained acoustic model will be used as the final acoustic model in the speech conversion model.

[0154] Among them, the above-mentioned first training constraint condition can be determined based on the training requirements of the acoustic model, which may include whether the number of training times of the acoustic model reaches a preset number, whether the prediction error of the acoustic model for the acoustic characteristics of the sound-emitting object in the training object is converged and stable, etc., which can be determined according to actual needs. This application does not limit the content of the first training constraint condition.

[0155] It should be noted that, in the case where the training object includes multiple sound-producing objects, the first parameters of the acoustic model can be updated based on the first errors obtained for each of the multiple sound-producing objects, so that the trained acoustic model can be applied to the acoustic feature prediction of different sound-producing objects. The implementation process can refer to the implementation process described above, and this embodiment will not be described in detail. Similarly, for the training process of the vocoder, the second errors of each of the multiple sound-producing objects can also be used to sequentially update the second parameters of the vocoder to obtain a vocoder suitable for speech synthesis of different sound-producing objects. The implementation process will not be described in detail.

[0156] Step S57: The predicted speech output by the next training vocoder and the corresponding training speech are input into the discriminator, and the second parameter of the vocoder is updated according to the discrimination result. The vocoder with the updated second parameter is trained until the second training constraint condition is met, thereby obtaining a speech conversion model.

[0157] Following the above analysis, after a stable acoustic model is obtained through training, the present application can introduce a discriminator to continue training the vocoder. In this case, the vocoder acts as a generator, and the discriminator calculates the probability that the input predicted speech is a non-synthesized speech (such as the corresponding training speech). This can represent the synthesis accuracy of the predicted speech output by the vocoder. The larger the probability value, the higher the accuracy of the predicted speech. The probability or the score given based on it can be determined as the error between the predicted speech of the vocoder in this training and the corresponding training speech.

[0158] The goal of this CAN training method is to ensure that the discriminator identifies the predicted speech output by the vocoder as non-synthesized speech, thereby improving the vocoder's synthesis performance. It should be noted that the discriminator's discrimination result (which represents the error between the predicted speech output by the vocoder during this training and the training speech participating in this model training) will be used to continuously update the vocoder's second parameter during the reverse transmission process. At this time, the first parameter of the acoustic model does not need to be updated, ensuring that the final trained vocoder and acoustic model are more stable and more suitable for speech conversion applications.

[0159] It should be noted that the above-mentioned second training constraint condition may also include that the number of training times for the vocoder model reaches a preset number of training times, the prediction error of the vocoder converges to a minimum value, etc. This application does not limit the content of the second training constraint condition, which can be determined according to the circumstances.

[0160] In summary, in the training process of the speech conversion model described in the embodiment of the present application, in the first training stage, during each training process, the acoustic model and the vocoder are trained synchronously, so that the acoustic features of the vocoder input are the predicted acoustic features of the vocoder output of this training, so that the training input and the use input of the vocoder are both from the acoustic model output, thereby improving the speech synthesis effect of the vocoder. After completing the first stage of synchronous training and obtaining a stable acoustic model and a relatively stable vocoder, this embodiment will adopt an adversarial training method and introduce a discriminator to continue training the vocoder, thereby improving the stability and output accuracy of the vocoder, thereby improving the voice conversion model's voice change effect of converting any source speech into the target speech of a specified target sounding object, and better meeting the needs of voice change applications.

[0161] Reference Figure 6 , is a flow chart of another optional example of the speech conversion method proposed in this application. This embodiment can be another optional training implementation method of the speech conversion model in the speech conversion method. It can be an optional optimization implementation method of the model training implementation method described in the above embodiment, but is not limited to the training implementation method described in this embodiment. The method can still be executed by a computer device, combined with Figure 7An optional network structure diagram of the speech conversion model shown in FIG. Figure 6 As shown, the training method of the speech conversion model may include:

[0162] Step S61, obtaining the training speech generated by the training subject;

[0163] Step S62, inputting the training speech into the pre-trained speech recognition model to obtain the training speech recognition result;

[0164] Regarding the implementation process of step S61 and step S62, reference may be made to the description of the corresponding parts of the above embodiment, which will not be elaborated in this embodiment.

[0165] Step S63: Input the training speech recognition result and the object identifier into an acoustic encoder to obtain speech coding features and timbre features corresponding to the object identifier;

[0166] In the embodiment of the present application, combined with the above description of the network structure of the acoustic model, the acoustic encoder included therein can be used to realize feature extraction of the training speech recognition results, and obtain speech coding features such as pronunciation features and rhythmic features that are not related to the training object. At the same time, the extracted timbre features can be associated with the object identifier of the corresponding sound-making object, and the timbre features of the object identifier can be recorded, that is, the predicted speech of what timbre the training speech is to be converted into. The implementation process will not be described in detail in the embodiment of the present application.

[0167] During model training, the embedding layer of the acoustic encoder can maintain an embedded timbre table. Combined with the description of the corresponding portion of the embodiment above, the timbre features corresponding to the corresponding object identifiers recorded in the embedded timbre table can be continuously updated during the continuous training of the acoustic encoder. Thus, in the final stage of model training, when the speech conversion model is trained specifically using the speech of a designated vocalizing object as training speech, the correspondence between the object identifier of the designated vocalizing object and the timbre features extracted by the acoustic model can be recorded. This correspondence can then be used to update and maintain the embedded timbre table represented by the encoding embedding layer of the acoustic encoder.

[0168] Based on this, in actual speech conversion applications, the selected target sound object can be the specified sound object. During the speech conversion process, during the acoustic encoder processing process, it can directly query the embedded timbre table to obtain the target timbre characteristics corresponding to the target object identifier, so that the acoustic decoder can predict the acoustic characteristics of the target sound object based on this. The implementation process can be combined with the description of the corresponding part of the context.

[0169] Step S64: input the timbre feature and the speech coding feature into an acoustic decoder to obtain a predicted acoustic feature of the training object;

[0170] The acoustic decoder in the acoustic model is used to implement fusion processing of the input speech coding features and the timbre features, and predict the timbre of the sound object to be converted (in this embodiment, it may refer to the sound object included in the training object) and the acoustic features consistent with the content of the training speech. The fusion processing implementation process is not described in detail in this embodiment.

[0171] In some other embodiments, combined with the model training implementation method described in the above embodiments, the acoustic model can output speech coding features and predicted acoustic features. In this way, the speech coding features and predicted acoustic features can be input into the vocoder to obtain the predicted speech of the training object. Compared with only inputting the predicted acoustic features into the vocoder, this embodiment considers the original features of the training speech contained in the speech coding features during the speech synthesis process, thereby improving the speech synthesis effect.

[0172] Step S65, inputting the prosody information contained in the training speech recognition result into a fundamental frequency prediction model to obtain the predicted fundamental frequency of each of the vocalization objects included in the training object;

[0173] In speech conversion applications, although the fundamental frequencies (such as the lowest frequency of a speech signal wave, which usually affects the tone of speech) of different sound-producing objects are known, such as 50Hz (unit: Hertz) to 400Hz, the actual fundamental frequencies of different sound-producing objects often vary. For unknown sound-producing objects (such as source sound-producing objects), their fundamental frequencies cannot be directly extracted. In this regard, the present application has analyzed and found that the prosodic information identified from the source speech has a certain correlation with the fluctuation of the fundamental frequency of the source sound-producing object. The embodiment of the present application can construct a fundamental frequency prediction model based on this, so that the normalized fundamental frequency of the sound-producing object can be predicted by the prosodic information of the input speech, that is, the fundamental frequency obtained by normalizing the fundamental frequencies of the multiple speech sounds generated by the sound-producing object. The present application does not limit the structure of the fundamental frequency prediction model, and it can be determined by analyzing the correlation between the prosodic information and the fundamental frequency of different speech sounds.

[0174] After training the fundamental frequency prediction model according to the training method described in the corresponding section below, the computer device can directly call the fundamental frequency prediction model in a speech conversion application for any source speech of an unknown speaker to predict the fundamental frequency under the standard release, so that it can be subsequently combined with the actual fundamental frequency information of the target speaker to perform denormalization processing to obtain the target fundamental frequency of the target speaker. However, during the model training process, it can participate in training but is not used to obtain the predicted fundamental frequency characteristics of the training object.

[0175] Step S66, retrieve the reference fundamental frequency of the training speech;

[0176] Step S67, inputting the reference fundamental frequency of the training speech into the fundamental frequency processing model to obtain the fundamental frequency characteristics of the corresponding sound object;

[0177] In the training process of the speech conversion model proposed in this application, the training object serves as both the source sound object and the sound object to be synthesized, and the training speech serves as both the source speech and the speech prediction target. Therefore, in order to obtain the fundamental frequency characteristics of the training object, the real fundamental frequency of the training speech can be directly retrieved as the reference fundamental frequency, which can be input into the fundamental frequency processing model to obtain the hidden layer feature representation and obtain the fundamental frequency characteristics of the corresponding sound object. It can be understood that in the case where the training speech contains the speech of multiple sound objects, the reference fundamental frequency retrieved can be the real fundamental frequency of the speech of each of these multiple sound objects. The method for obtaining the fundamental frequency in the speech signal is not described in detail in this application.

[0178] Step S68, inputting the fundamental frequency feature, speech coding feature and predicted acoustic feature into a vocoder to obtain the predicted speech of the training object;

[0179] It can be seen that during the speech synthesis processing, the vocoder of the present application will comprehensively consider the fundamental frequency characteristics of the sound-producing object of the speech to be converted, the predicted acoustic characteristics, and the speech coding characteristics of the training speech to be converted, so as to achieve a speech synthesis result with high reliability and accuracy, that is, to improve the reliability and accuracy of the predicted speech output by the vocoder.

[0180] Step S69, obtaining a first error between the predicted acoustic feature and the reference acoustic feature of the training object, a second error between the predicted speech and the corresponding training speech, and a third error between the predicted fundamental frequency of the same sound object and the predicted fundamental frequency target value;

[0181] Among them, the fundamental frequency prediction target value of the sound object can be pre-calculated and stored, such as by obtaining multiple voices produced by the same sound object in the training data set, extracting the reference fundamental frequencies of each of the multiple voices, and then normalizing the reference fundamental frequencies of each of the multiple voices of the same sound object to obtain the fundamental frequency prediction target value of the corresponding sound object. Optionally, the present application can adopt a variance normalization processing method to realize the calculation of the fundamental frequency prediction target value, such as calculating the mean and standard deviation of the fundamental frequencies of the multiple voices of the same sound object to obtain the corresponding fundamental frequency mean and fundamental frequency standard deviation, and combining the reference fundamental frequency of the corresponding sound object in the training object, according to the calculation method of fundamental frequency prediction target value = (reference fundamental frequency - fundamental frequency mean) / fundamental frequency standard deviation, to obtain the fundamental frequency prediction target value of the sound object, but is not limited to this calculation method.

[0182] For the method of obtaining the above-mentioned first error and second error, reference may be made to the description of the corresponding parts of the above embodiment, which will not be elaborated in this embodiment; and in the process of obtaining the third error, a preset loss function may also be called to calculate the loss between the predicted fundamental frequency of the same sound object and the fundamental frequency prediction target value, thereby obtaining the prediction error of the fundamental frequency of the sound object by the fundamental frequency prediction model, namely the third error. The implementation process will not be described in detail in this application.

[0183] Step S610: Update the first parameter of the acoustic model based on the first error, update the second parameter of the vocoder and the third parameter of the fundamental frequency processing model based on the second error, update the fourth parameter of the fundamental frequency prediction model based on the third error, train the updated models, and obtain a speech conversion model.

[0184] Thus, it can be seen that in the multi-task training method proposed in this application, in order to avoid overfitting of the fundamental frequency prediction model, the fundamental frequency prediction model will be trained independently according to the fundamental frequency prediction target. During its training process, if it is detected that the third error of this training meets the third training constraint for the fundamental frequency prediction model, the training of the fundamental frequency prediction model can be stopped. Therefore, the training constraints in the speech conversion model training process can also include a third training constraint pre-configured for the fundamental frequency prediction model. This application does not limit its content and it can be determined as needed. The embodiment of the present application can use the speech prediction target to implement the training of the vocoder and the fundamental frequency processing model. The training implementation process can refer to the description of the corresponding part of the above embodiment, and this embodiment will not be repeated.

[0185] In summary, each component of the speech conversion model in the embodiments of the present application has its own prediction target. During model training, the parameters of the corresponding component of the model can be stably updated based on the prediction target. During vocoder training, the method described above can still be followed. After acoustic model training is completed, a discriminator can be introduced to continue stabilizing vocoder training. This implementation process is not described in detail here.

[0186] In some embodiments proposed in the present application, for the model training method described in each embodiment above, the acoustic model and the vocoder in the speech conversion model can be integrated into a network, which can more conveniently transmit the encoding result of the acoustic encoder to the vocoder, so that the vocoder has multi-dimensional reference information, improves the speech synthesis effect, increases the user's acceptance and satisfaction with the voice changing function, and is more conducive to the promotion of the product. Moreover, compared with the data transmission between two independent models, the present application can realize data transmission within a network, reducing the first packet delay of data transmission, so that it can better meet the low-latency requirements of real-time voice changing applications. In addition, through structural links, the number of calls to the model in the voice changing application is reduced, and the voice changing efficiency is improved.

[0187] It should be noted that the speech recognition model in the above embodiment can also be integrated into the system architecture of the entire speech conversion model. The technology such as wav2vec can be used to implement the training of the speech recognition model to complete the speech recognition task, which can further reduce the first packet delay. However, this processing method may reduce the pronunciation accuracy after voice change, resulting in unstable overall audio quality. Therefore, in actual applications, the composition of the system structure and the structural relationship between the components can be flexibly and reasonably determined based on actual application requirements, whether they need to be integrated together, whether the network structure of the acoustic model and / or the vocoder needs to be adjusted, etc. The embodiments of this application do not give detailed examples one by one.

[0188] In another embodiment, as described above, in the vocoder training implementation method, the present application will input the speech coding features output by the acoustic model into the vocoder. The speech coding features are obtained by the acoustic encoder through feature extraction of the training speech recognition results of the training speech. The training speech recognition results often include pronunciation information and prosody information, etc. Therefore, if the speech coding features input to the vocoder include pronunciation features and prosody features, the output predicted speech can clearly feel the rhythmic fluctuations of the training speech. In this way, in actual applications, if the timbre of the target sound object itself has another expression method, inputting prosody features into the vocoder during the training stage will affect the audience's judgment of the voice change effect of the synthesized output target speech. In this case, the present application considers inputting the pronunciation features of the speech coding features instead of the prosody features when training the vocoder, so as to obtain a target speech that better meets the actual voice change requirements. The training implementation process of this vocoder is the same as the training implementation process described above and will not be described in detail in this application.

[0189] Based on this, during the above-mentioned model training process, the speech coding features of the above-mentioned input vocoder may include pronunciation features and prosodic features, thereby obtaining a speech conversion model suitable for the first type of application scenario; in another implementation process, the speech coding features of the input vocoder may include pronunciation features, but no longer include prosodic features, thereby obtaining a speech conversion model suitable for the second type of application scenario, so that in the corresponding application scenario, the corresponding speech conversion model can be called to meet the voice change requirements of the application scenario, and obtain a target speech that meets the timbre characteristics of the target sound-producing object and the requirements of the special application scenario. This application does not limit the respective contents of the first and second types of application scenarios, and can be pre-configured according to actual conditions.

[0190] In addition, in order to meet the voice changing needs of the third type of application scenarios, such as the live broadcast field, other dimensional features such as speech energy features can be added during the training of the vocoder to further improve the overall effect after voice changing. This application does not restrict the content of the dimensional features added to the vocoder, which can be determined according to the situation.

[0191] Based on the description of the above embodiments, in the training process of the above-mentioned speech conversion model, when the training data set contains a large number of sound objects and the speech generated by each sound object is sufficient, the present application can directly use multiple groups of training speech composed of multiple sound object speech to train the speech conversion model, so that it can learn the common characteristics of multiple sound objects, so that the trained speech conversion model can be applied to the acoustic feature prediction of any sound object, and then synthesize the speech of the sound object. In this case, the trained speech conversion model can also convert the source speech of any sound object into the target speech of the specified target sound object with high accuracy.

[0192] However, when the training data set contains a small number of sound objects, the present application can first train the speech conversion model in the manner described above, and then obtain the speech of the specified target sound object (i.e., the training object) as the training speech. Using at least one training speech of this target sound object, continue to train the speech conversion model obtained by training the speech of multiple sound objects, and finally obtain a speech conversion model suitable for converting the source speech of any sound object into the target speech of the specified target sound object with high precision, that is, a speech conversion model for the target sound object. Afterwards, when faced with the need to convert any source speech into the target speech of the target sound object, the speech conversion model finally trained can be called to convert any source speech into the target speech.

[0193] Among them, the present application can also use multiple training sentences of different target sound objects in sequence, continue to train the speech conversion model according to the training method described above, and record the object identification of the target sound object used in each training, and the correspondence between the timbre characteristics of the target sound object obtained through training, and record them in the embedded timbre table. In this way, when it is necessary to convert any source speech into a target speech with the timbre of any of the above-mentioned target sound objects, the trained speech conversion model can be directly called to meet the voice change requirements. Regarding how to use the training sentences of a sound object to implement the training method of the speech conversion model, you can refer to the description of the corresponding part of the above embodiment, which will not be repeated in this embodiment.

[0194] Reference Figure 8 , is a flowchart of another optional example of the speech conversion method proposed in this application. This embodiment can be based on the pre-trained speech conversion model described in the above embodiment, and provides a detailed description of how a computer device calls the speech conversion model to implement the process of converting any source speech into the target speech of a specified target sounding object, but is not limited to the detailed implementation method proposed in the embodiment of this application. Figure 9 The schematic diagram of the voice conversion method implemented by calling the voice conversion model is shown in FIG. Figure 8As shown, the method may include but is not limited to the following steps:

[0195] Step S81, obtaining the source voice of the source sounding object and the target object identifier of the target sounding object;

[0196] Step S82: inputting the source speech into the pre-trained speech recognition model to obtain a speech recognition result;

[0197] Step S83: Input the speech recognition result and the target object identifier into a pre-trained acoustic encoder for encoding processing to obtain target speech coding features and target timbre features corresponding to the target object identifier;

[0198] Regarding the implementation process of steps S81 to S83, please refer to the description of the corresponding parts of the above embodiment, which will not be repeated in this embodiment.

[0199] In this embodiment, combined with the description of the acoustic encoder training process, it can maintain an embedded timbre table. After obtaining the target object identifier, it can directly look up the table to obtain the target timbre characteristics. It is understood that the acoustic encoder can also update the timbre characteristics of the corresponding sounding object in the embedded timbre table based on feedback information from the speech conversion application to improve the accuracy of the timbre characteristics. This implementation process is similar to the implementation process of learning timbre characteristics during model training and is not described in detail in this application.

[0200] Step S84: input the target speech coding features and the target timbre features into a pre-trained acoustic decoder for fusion processing to obtain the target acoustic features;

[0201] Step S85, inputting the prosody information contained in the speech recognition result into a pre-trained fundamental frequency prediction model to obtain a target predicted fundamental frequency;

[0202] Step S86, retrieve the normalized fundamental frequency information corresponding to the target object identifier, and use the normalized fundamental frequency information of the target sounding object to perform denormalization processing on the target predicted fundamental frequency to obtain the predicted reference fundamental frequency of the target sounding object;

[0203] Combined with the relevant description of the fundamental frequency prediction model in the model training process above, the fundamental frequency prediction model is to predict the fundamental frequency under the standard distribution, which is actually to predict the normalized fundamental frequency. Therefore, Figure 9 As shown, the present application can call pre-stored normalized fundamental frequency information such as the fundamental frequency mean and fundamental frequency standard deviation of the target sound object to perform denormalization processing on the target predicted fundamental frequency to obtain the predicted reference fundamental frequency of the target sound object. Wherein, the predicted reference fundamental frequency = target predicted fundamental frequency * fundamental frequency standard deviation + fundamental frequency mean, but the processing method is not limited to this.

[0204] Step S87, inputting the predicted reference fundamental frequency into the pre-trained fundamental frequency processing model to obtain the target fundamental frequency feature of the target sound object;

[0205] Step S88: input the target speech coding features, target timbre features and target fundamental frequency features into the vocoder to obtain the target speech with the target timbre features and the content of the source speech.

[0206] In combination with the above analysis, in different application scenarios, the above step S88 may include but is not limited to the following implementation methods:

[0207] The target timbre features, target fundamental frequency features, and target pronunciation features and target prosody features in the target speech coding features are input into a pre-trained vocoder to output the target speech of the target sound-producing object; alternatively, the target timbre features, target fundamental frequency features, and target pronunciation features in the target speech coding features are input into a pre-trained vocoder to output the target speech of the target sound-producing object; alternatively, the target speech coding features, target acoustic features, target fundamental frequency features, and preset speech energy features are input into a pre-trained vocoder to output the target speech of the target sound-producing object.

[0208] Regarding the processing process of the input information by the above-mentioned models, please refer to the description of the corresponding part in the above model training process, which will not be repeated in this embodiment.

[0209] In summary, since the acoustic features input to the vocoder during the training of the speech conversion model are the predicted acoustic features output by the acoustic model, the vocoder obtained based on this training is used in the voice changing scenario, and the acoustic features of the predicted target sound object output by the acoustic model are input into the vocoder, which improves the speech synthesis effect of the vocoder, makes the target speech and the real speech of the target sound object more similar, better meets the voice changing requirements of converting any source speech into the target speech of a specified target sound object, and improves the user experience.

[0210] It can be understood that in the above-mentioned speech conversion process, the present application can also call the discriminator in the manner described above to perform similarity judgment on the obtained target speech and the reference speech of the target sound-producing object, further optimize the vocoder based on the judgment result, and then use the optimized vocoder to re-synthesize the input information to update the above-mentioned target speech. The implementation process can refer to the description of the corresponding part of the vocoder training process above, and will not be repeated in this embodiment.

[0211] Reference Figure 10 , which is a schematic diagram of an optional example of a speech conversion device proposed in this application, and the device may include but is not limited to:

[0212] The data acquisition module 101 is used to acquire the source speech of the source sounding object and the target object identifier of the target sounding object;

[0213] The speech recognition module 102 is used to extract features from the source speech and obtain a speech recognition result;

[0214] The speech conversion module 103 is configured to input the speech recognition result and the target object identifier into a speech conversion model, and output a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech;

[0215] The speech conversion model includes an acoustic model and a vocoder obtained through synchronous training, and the input information for training the vocoder includes output information of the acoustic model.

[0216] In some other embodiments, in order to implement the training of the speech conversion model, the above apparatus may include a speech conversion model training module, such as Figure 11 As shown, the speech conversion model training module may include:

[0217] The training speech recognition result acquisition module 104 is used to obtain the training speech recognition result of the training speech generated by the training subject;

[0218] The training object includes at least one sound-generating object, and the sound-generating object is configured with a corresponding object identifier; the training speech comes from the speech generated by the corresponding sound-generating object in the training data set.

[0219] The feature extraction module 105 is configured to input the training speech recognition result and the object identifier into an acoustic model to obtain the predicted acoustic features of the training object and record the timbre features of the sound-producing object corresponding to the object identifier;

[0220] The speech synthesis module 106 is configured to input the predicted acoustic features into a vocoder to obtain the predicted speech of the training object;

[0221] an error acquisition module 107, configured to acquire a first error between the predicted acoustic feature and the reference acoustic feature of the training object, and a second error between the predicted speech and the training speech;

[0222] The model parameter updating module 108 is used to update the first parameter of the acoustic model according to the first error, update the second parameter of the vocoder according to the second error, and train the updated acoustic model and vocoder to obtain a speech conversion model during the back propagation process.

[0223] Optionally, the feature extraction module 105 may include:

[0224] A feature extraction unit, configured to extract features from the training speech recognition result to obtain speech coding features and timbre features corresponding to the object identifier;

[0225] A feature fusion unit is used to fuse the speech coding feature and the timbre feature to obtain the predicted acoustic feature of the training object

[0226] Based on this, the speech synthesis module 106 may include:

[0227] The first speech synthesis unit is configured to input the speech coding feature and the predicted acoustic feature into a vocoder to obtain the predicted speech of the training object.

[0228] In some other embodiments, the above-mentioned voice conversion model training module may also include

[0229] a timbre feature recording module for recording, when the training object is a specified sound-producing object, a correspondence between the object identifier of the specified sound-producing object and the timbre feature extracted by the acoustic model; the target sound-producing object includes any specified sound-producing object;

[0230] The embedded timbre table updating module is used to update the embedded timbre table represented by the coding embedding layer of the acoustic model by using the corresponding relationship, and obtain the target timbre feature corresponding to the target object identifier by querying the embedded timbre table.

[0231] In some other embodiments, the above-mentioned voice conversion model training module may also include

[0232] A reference fundamental frequency calling module, used to call the reference fundamental frequency of the training speech;

[0233] A fundamental frequency feature acquisition module, configured to input the reference fundamental frequency of the training speech into a fundamental frequency processing model to obtain the fundamental frequency feature of the corresponding sound object;

[0234] a baseband processing model parameter updating module, configured to update a third parameter of the baseband processing model according to the second error during the back propagation process;

[0235] Based on this, the speech synthesis module 106 may include:

[0236] The second speech synthesis unit is used to input the fundamental frequency feature and the predicted acoustic feature into a vocoder to obtain the predicted speech of the training object.

[0237] In some other embodiments, the above-mentioned voice conversion model training module may also include

[0238] A predicted fundamental frequency obtaining module, configured to input the prosody information contained in the training speech recognition result into a fundamental frequency prediction model to obtain the predicted fundamental frequency of each of the vocalization objects contained in the training object;

[0239] A fundamental frequency prediction target value retrieving module, configured to retrieve a fundamental frequency prediction target value of a sounding object corresponding to the object identifier;

[0240] a fundamental frequency prediction error obtaining module, configured to obtain a third error between the predicted fundamental frequency of the same sound object and the fundamental frequency prediction target value;

[0241] The fundamental frequency prediction model parameter updating module is configured to update a fourth parameter of the fundamental frequency prediction model according to the third error during the reverse transmission process.

[0242] Optionally, in order to obtain a target value for the fundamental frequency prediction of the sound object, the speech conversion model training module may further include:

[0243] The speech acquisition module is used to obtain multiple speech sounds produced by the same sound-producing object in the training data set;

[0244] A reference fundamental frequency extraction module, configured to extract the reference fundamental frequencies of the plurality of speech sounds;

[0245] The normalization processing module is used to perform normalization processing on the reference fundamental frequencies of the multiple voices of the same sound object to obtain a predicted target value of the fundamental frequency of the corresponding sound object.

[0246] Based on the description of the above embodiments, the model parameter updating module 108 may include:

[0247] a model training monitoring unit, configured to train the updated acoustic model and the vocoder until a first training constraint condition is satisfied, and then stop training the acoustic model;

[0248] a vocoder training unit, configured to input the predicted speech output by the vocoder in the next training and the corresponding training speech into a discriminator, update the second parameter of the vocoder according to the discrimination result, and train the vocoder with the updated second parameter until the second training constraint condition is satisfied, thereby obtaining a speech conversion model;

[0249] The first training constraint condition is configured for the training process of the acoustic model; and the second training constraint condition is configured for the training process of the vocoder.

[0250] In some other embodiments, combined with the above analysis, the voice conversion module 103 may include

[0251] A feature extraction unit, configured to input the speech recognition result and the target object identifier into the pre-trained acoustic model to obtain target speech coding features and target acoustic features;

[0252] A fundamental frequency prediction unit, configured to input the prosody information contained in the speech recognition result into the pre-trained fundamental frequency prediction model to obtain a target predicted fundamental frequency;

[0253] A predicted reference fundamental frequency acquisition unit is configured to retrieve the normalized fundamental frequency information corresponding to the target object identifier, and perform denormalization processing on the target predicted fundamental frequency using the normalized fundamental frequency information of the target sounding object to obtain the predicted reference fundamental frequency of the target sounding object;

[0254] a fundamental frequency processing unit, configured to input the predicted reference fundamental frequency into the pre-trained fundamental frequency processing model to obtain a target fundamental frequency feature of the target sounding object;

[0255] The speech synthesis unit is used to input the above-mentioned target speech coding features, target acoustic features and target fundamental frequency features into a pre-trained vocoder, and output the target speech of the target sounding object; the target speech has the target timbre features corresponding to the target object identifier and the content of the source speech.

[0256] Optionally, the speech synthesis unit may include at least one of the following synthesis units:

[0257] a first synthesis unit, configured to input the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature and target prosody feature of the target speech coding feature into a pre-trained vocoder, and output a target speech of the target sounding object;

[0258] A second synthesis unit is configured to input the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature in the target speech coding feature into a pre-trained vocoder, and output the target speech of the target sounding object; or

[0259] The third synthesis unit is used to input the target speech coding feature, the target acoustic feature, the target fundamental frequency feature and the preset speech energy feature into a pre-trained vocoder, and output the target speech of the target sounding object.

[0260] It should be noted that the various modules, units, etc. in the above-mentioned device embodiments can be stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to implement the corresponding functions. Regarding the functions implemented by each program module and its combination, as well as the technical effects achieved, please refer to the description of the corresponding parts of the above-mentioned method embodiments, which will not be repeated in this embodiment.

[0261] The present application also proposes a computer-readable storage medium on which a computer program can be stored. The computer program can be called and loaded by a processor to implement the various steps of the speech conversion method described in the above embodiment.

[0262] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various optional embodiments of the speech conversion method and apparatus. The specific implementation process can be referred to the description of the corresponding embodiments above and is not repeated here.

[0263] Finally, the various embodiments in this specification are described in a progressive or parallel manner. Each embodiment focuses on the differences from other embodiments, and reference can be made to the common and similar parts between the various embodiments. The devices, computer equipment, and systems disclosed in the embodiments correspond to the methods disclosed in the embodiments, so the descriptions are relatively simple. For relevant details, refer to the descriptions of the methods.

[0264] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A voice conversion method, characterized in that: include: Obtain the source voice of any sound-producing object and the target object identifier of the target sound-producing object; Performing speech recognition on the source speech to obtain a speech recognition result; Inputting the speech recognition result and the target object identifier into a speech conversion model, and outputting a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech; The speech conversion model includes an acoustic model and a vocoder obtained through synchronous training, and the input information for training the vocoder includes the output information of the acoustic model; the training method of the speech conversion model includes: Obtaining a training speech recognition result of a training speech generated by a training object; wherein the training object includes at least one sound-producing object, and the sound-producing object is configured with a corresponding object identifier; and the training speech is from the speech generated by the corresponding sound-producing object in the training data set; Inputting the training speech recognition result and the object identifier into an acoustic model to obtain predicted acoustic features of the training object, and recording the timbre features of the sound-producing object corresponding to the object identifier; Inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object; Obtaining a first error between the predicted acoustic feature and a reference acoustic feature of the training object, and a second error between the predicted speech and the training speech; During the back propagation process, the first parameter of the acoustic model is updated according to the first error, the second parameter of the vocoder is updated according to the second error, and the updated acoustic model and vocoder are trained to obtain a speech conversion model.

2. The method according to claim 1, characterized in that Inputting the training speech recognition result and the object identifier into an acoustic model to obtain predicted acoustic features of the training object includes: Performing feature extraction on the training speech recognition result to obtain speech coding features and timbre features corresponding to the object identifier; fusing the speech coding features and the timbre features to obtain predicted acoustic features of the training object; The step of inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object comprises: The speech coding features and the predicted acoustic features are input into a vocoder to obtain the predicted speech of the training object.

3. The method according to claim 2, characterized in that The training method further comprises: The training object is a designated sound-producing object, and the correspondence between the object identifier of the designated sound-producing object and the timbre features extracted by the acoustic model is recorded; the target sound-producing object includes any designated sound-producing object; The corresponding relationship is used to update the embedded timbre table represented by the encoding embedding layer of the acoustic model, and the target timbre feature corresponding to the target object identifier is obtained by querying the embedded timbre table.

4. The method according to claim 1, wherein The training method further comprises: Retrieving the reference fundamental frequency of the training speech; Inputting the reference fundamental frequency of the training speech into a fundamental frequency processing model to obtain fundamental frequency characteristics of the corresponding sound object; During the back-propagation process, updating a third parameter of the baseband processing model according to the second error; The step of inputting the predicted acoustic features into a vocoder to obtain the predicted speech of the training object comprises: The fundamental frequency feature and the predicted acoustic feature are input into a vocoder to obtain the predicted speech of the training object.

5. The method according to claim 4, characterized in that The training method further comprises: Inputting the prosody information contained in the training speech recognition result into a fundamental frequency prediction model to obtain the predicted fundamental frequency of each of the vocalization objects contained in the training object; Retrieve a predicted target value of the fundamental frequency of the sound object corresponding to the object identifier; Obtaining a third error between the predicted fundamental frequency and the predicted target fundamental frequency value of the same sound object; During the back propagation process, a fourth parameter of the fundamental frequency prediction model is updated according to the third error.

6. The method according to any one of claims 1 to 5, characterized in that The step of training the updated acoustic model and the vocoder to obtain a speech conversion model includes: Training the updated acoustic model and the vocoder until a first training constraint condition is satisfied, and stopping the training of the acoustic model; Inputting the predicted speech output by the vocoder in the next training and the corresponding training speech into the discriminator, updating the second parameter of the vocoder according to the discrimination result, and training the vocoder with the updated second parameter until the second training constraint condition is satisfied, thereby obtaining a speech conversion model; The first training constraint condition is configured for the training process of the acoustic model; and the second training constraint condition is configured for the training process of the vocoder.

7. The method according to claim 5, characterized in that The method for obtaining the fundamental frequency prediction target value of the sound object includes: Obtain multiple voices produced by the same sound-producing object in the training data set; Extracting a reference fundamental frequency of each of the plurality of speech sounds; Normalization processing is performed on the reference fundamental frequencies of the multiple voices of the same sound-producing object to obtain a predicted target value of the fundamental frequency of the corresponding sound-producing object.

8. The method according to claim 5, characterized in that The step of inputting the speech recognition result and the target object identifier into a speech conversion model and outputting a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech comprises: Inputting the speech recognition result and the target object identifier into the pre-trained acoustic model to obtain target speech coding features and target acoustic features; Inputting the prosody information contained in the speech recognition result into the pre-trained fundamental frequency prediction model to obtain a target predicted fundamental frequency; Retrieving normalized fundamental frequency information corresponding to the target object identifier, and performing denormalization processing on the target predicted fundamental frequency using the normalized fundamental frequency information of the target sounding object to obtain a predicted reference fundamental frequency of the target sounding object; Inputting the predicted reference fundamental frequency into the pre-trained fundamental frequency processing model to obtain a target fundamental frequency feature of the target sounding object; The target speech coding feature, the target acoustic feature and the target fundamental frequency feature are input into a pre-trained vocoder, and a target speech of the target sounding object is output; the target speech has a target timbre feature corresponding to the target object identifier and the content of the source speech.

9. The method according to claim 8, characterized in that The step of inputting the target speech coding feature, the target acoustic feature, and the target fundamental frequency feature into a pre-trained vocoder and outputting the target speech of the target sounding object comprises: Inputting the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature and target prosody feature in the target speech coding feature into a pre-trained vocoder to output the target speech of the target sounding object; or Inputting the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature in the target speech coding feature into a pre-trained vocoder to output the target speech of the target sounding object; or The target speech coding feature, the target acoustic feature, the target fundamental frequency feature and the preset speech energy feature are input into a pre-trained vocoder, and the target speech of the target sounding object is output.

10. A voice conversion device, characterized in that: The device comprises: A data acquisition module, used to acquire the source voice of the source sounding object and the target object identifier of the target sounding object; A speech recognition module is used to extract features from the source speech to obtain a speech recognition result; a speech conversion module, configured to input the speech recognition result and the target object identifier into a speech conversion model, and output a target speech having a target timbre feature corresponding to the target object identifier and the content of the source speech; The speech conversion model includes an acoustic model and a vocoder obtained by synchronous training, and the input information for training the vocoder includes output information of the acoustic model; The device further includes a speech conversion model training module, wherein the speech conversion model training module includes: A training speech recognition result acquisition module is used to obtain a training speech recognition result of a training speech generated by a training object; wherein the training object includes at least one sound-producing object, and the sound-producing object is configured with a corresponding object identifier; the training speech is the speech generated by the corresponding sound-producing object in the training data set; A feature extraction module is used to input the training speech recognition result and the object identifier into the acoustic model to obtain the predicted acoustic features of the training object and record the timbre features of the sound-producing object corresponding to the object identifier; a speech synthesis module, configured to input the predicted acoustic features into a vocoder to obtain the predicted speech of the training object; an error acquisition module, configured to acquire a first error between the predicted acoustic feature and the reference acoustic feature of the training object, and a second error between the predicted speech and the training speech; The model parameter updating module is used to update the first parameter of the acoustic model according to the first error, update the second parameter of the vocoder according to the second error, and train the updated acoustic model and vocoder to obtain a speech conversion model during back propagation.

11. The device according to claim 10, characterized in that The feature extraction module includes: A feature extraction unit, configured to extract features from the training speech recognition result to obtain speech coding features and timbre features corresponding to the object identifier; a feature fusion unit, configured to fuse the speech coding feature and the timbre feature to obtain a predicted acoustic feature of the training object; The speech synthesis module includes: The first speech synthesis unit is configured to input the speech coding feature and the predicted acoustic feature into a vocoder to obtain the predicted speech of the training object.

12. The device according to claim 11, characterized in that The speech conversion model training module also includes: a timbre feature recording module for recording, when the training object is a specified sound-producing object, a correspondence between the object identifier of the specified sound-producing object and the timbre feature extracted by the acoustic model; the target sound-producing object includes any specified sound-producing object; The embedded timbre table updating module is used to update the embedded timbre table represented by the coding embedding layer of the acoustic model by using the corresponding relationship, and obtain the target timbre feature corresponding to the target object identifier by querying the embedded timbre table.

13. The device according to claim 10, characterized in that The speech conversion model training module also includes: A reference fundamental frequency calling module, used to call the reference fundamental frequency of the training speech; A fundamental frequency feature acquisition module, configured to input the reference fundamental frequency of the training speech into a fundamental frequency processing model to obtain the fundamental frequency feature of the corresponding sound object; a baseband processing model parameter updating module, configured to update a third parameter of the baseband processing model according to the second error during the back propagation process; The speech synthesis module includes: The second speech synthesis unit is used to input the fundamental frequency feature and the predicted acoustic feature into a vocoder to obtain the predicted speech of the training object.

14. The device according to claim 13, characterized in that The speech conversion model training module also includes: A predicted fundamental frequency obtaining module, configured to input the prosody information contained in the training speech recognition result into a fundamental frequency prediction model to obtain the predicted fundamental frequency of each of the vocalization objects contained in the training object; A fundamental frequency prediction target value retrieving module, configured to retrieve a fundamental frequency prediction target value of a sounding object corresponding to the object identifier; a fundamental frequency prediction error obtaining module, configured to obtain a third error between the predicted fundamental frequency of the same sound object and the fundamental frequency prediction target value; The fundamental frequency prediction model parameter updating module is used to update the fourth parameter of the fundamental frequency prediction model according to the third error during the back propagation process.

15. The device according to any one of claims 10 to 14, characterized in that The model parameter updating module includes: a model training monitoring unit, configured to train the updated acoustic model and the vocoder until a first training constraint condition is satisfied, and then stop training the acoustic model; a vocoder training unit, configured to input the predicted speech output by the vocoder in the next training and the corresponding training speech into a discriminator, update the second parameter of the vocoder according to the discrimination result, and train the vocoder with the updated second parameter until the second training constraint condition is satisfied, thereby obtaining a speech conversion model; The first training constraint condition is configured for the training process of the acoustic model; and the second training constraint condition is configured for the training process of the vocoder.

16. The device according to claim 14, characterized in that The speech conversion model training module includes: The speech acquisition module is used to obtain multiple speech sounds produced by the same sound-producing object in the training data set; A reference fundamental frequency extraction module, configured to extract the reference fundamental frequencies of the plurality of speech sounds; The normalization processing module is used to perform normalization processing on the reference fundamental frequencies of the multiple voices of the same sound object to obtain a predicted target value of the fundamental frequency of the corresponding sound object.

17. The device according to claim 14, characterized in that The voice conversion module includes: A feature extraction unit, configured to input the speech recognition result and the target object identifier into the pre-trained acoustic model to obtain target speech coding features and target acoustic features; A fundamental frequency prediction unit, configured to input the prosody information contained in the speech recognition result into the pre-trained fundamental frequency prediction model to obtain a target predicted fundamental frequency; A predicted reference fundamental frequency acquisition unit is configured to retrieve the normalized fundamental frequency information corresponding to the target object identifier, and perform denormalization processing on the target predicted fundamental frequency using the normalized fundamental frequency information of the target sounding object to obtain the predicted reference fundamental frequency of the target sounding object; a fundamental frequency processing unit, configured to input the predicted reference fundamental frequency into the pre-trained fundamental frequency processing model to obtain a target fundamental frequency feature of the target sounding object; The speech synthesis unit is used to input the target speech coding feature, the target acoustic feature and the target fundamental frequency feature into a pre-trained vocoder, and output the target speech of the target sounding object; the target speech has the target timbre feature corresponding to the target object identifier and the content of the source speech.

18. The device according to claim 17, characterized in that The speech synthesis unit comprises: a first synthesis unit, configured to input the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature and target prosody feature of the target speech coding feature into a pre-trained vocoder, and output the target speech of the target sounding object; or A second synthesis unit is configured to input the target timbre feature, the target fundamental frequency feature, and the target pronunciation feature in the target speech coding feature into a pre-trained vocoder, and output the target speech of the target sounding object; or The third synthesis unit is used to input the target speech coding feature, the target acoustic feature, the target fundamental frequency feature and the preset speech energy feature into a pre-trained vocoder, and output the target speech of the target sounding object.

19. A computer device, characterized in that: The computer device comprises: at least one memory and at least one processor, wherein: The memory is used to store a program for implementing the voice conversion method according to any one of claims 1 to 9; The processor is configured to load and execute the program stored in the memory to implement the speech conversion method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is loaded and executed by a processor to implement the speech conversion method according to any one of claims 1 to 9.

21. A computer program product comprising computer instructions, characterized in that The computer instructions are read and executed by a processor to implement the speech conversion method according to any one of claims 1 to 9.

Images