Speech conversion method, device, electronic device and medium based on coding model

Abstract

The embodiment of the present application provides a method, device, electronic device and medium for speech conversion based on a coding model, which belongs to the field of speech conversion technology. The method includes: obtaining a sample speech set, the sample speech set includes a source speaker's speech and a reference speech; performing speech synthesis on the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain a target speech; training the coding model according to the target speech, determining a loss function corresponding to the coding model; updating the coding model according to the loss function to obtain a target coding model; receiving a user's speech request, the speech request carries timbre information; inputting the speech request into the target coding model, and converting the speech in the speech request into a corresponding synthesized speech according to the timbre information. The embodiment of the present application can enhance the generalization of the coding model for any speaker and improve the accuracy and authenticity of speech conversion.

Description

Technical Field

[0001] The present application relates to the field of speech conversion technology, and in particular to a speech conversion method, device, electronic device and medium based on a coding model. Background Art

[0002] With the development of speech signal processing technology, speech conversion has gradually become an important research branch in the field of speech signal processing. The task of speech conversion is to generate speech with the source speaker's speech content and the target speaker's personality characteristics under the premise of the source speaker's speech and the target speaker's speech to be converted. At present, the technical categories of speech conversion mainly include the following three categories: based on autoencoders, based on TTS (Text To Speech), and based on generative adversarial networks. The research focus of these technologies is mainly on speech conversion between speakers in the training data set, which can achieve high speech naturalness and timbre similarity. However, little attention is paid to the problem of speech conversion of speakers outside the training data set. In the existing technology, a massive data set containing a large number of speakers is used to pre-train the speaker encoder, and the speaker's timbre and speech content are decoupled to achieve speech conversion of any speaker. However, under the condition of limited data and computing resources, the speaker encoder is difficult to generalize to any speaker, thereby reducing the accuracy of speech conversion. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a speech conversion method, device, electronic device and medium based on a coding model, which can enhance the generalization of the coding model for any speaker and improve the accuracy and authenticity of speech conversion.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a speech conversion method based on a coding model, the method comprising:

[0005] Acquire a sample speech set, wherein the sample speech set includes a source speaker speech and a reference speech;

[0006] Performing speech synthesis on the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain a target speech;

[0007] Training the coding model according to the target speech and determining a loss function corresponding to the coding model;

[0008] Update the encoding model according to the loss function to obtain a target encoding model;

[0009] receiving a voice request from a user, wherein the voice request carries timbre information;

[0010] The speech request is input into the target coding model, and the speech in the speech request is converted into corresponding synthesized speech according to the timbre information.

[0011] In some embodiments, the encoding model includes a content encoder and a vector encoder; performing speech synthesis on the source speaker speech and the reference speech in the sample speech set based on the encoding model to obtain the target speech includes:

[0012] Inputting the source speaker's speech into the content encoder in the coding model for mapping to obtain the speech content of the source speaker's speech;

[0013] Inputting the reference speech into the vector encoder in the coding model to perform vector extraction to obtain a reference speech vector of the reference speech;

[0014] Speech synthesis is performed according to the reference speech vector and the speech content to obtain the target speech.

[0015] In some embodiments, performing speech synthesis according to the reference speech vector and the speech content to obtain the target speech includes:

[0016] Inputting the source speaker's speech into the vector encoder in the coding model to perform vector extraction to obtain a source speaker vector of the source speaker's speech;

[0017] Replacing the source speaker vector in the source speaker's speech according to the reference speech vector to obtain a replacement result;

[0018] The replacement result and the speech content are input into the generator in the coding model for synthesis to generate the target speech.

[0019] In some embodiments, the loss function includes a coding loss function, a generation loss function, and a discrimination loss function; and the step of training the coding model according to the target speech to determine the loss function corresponding to the coding model includes:

[0020] Training the encoder and the generator in the coding model according to the target speech to generate training speech;

[0021] Calculating the training speech according to the discriminator in the encoding model to obtain a discrimination prediction value;

[0022] Update the discrimination loss function of the discriminator and the generation loss function of the generator according to the discrimination prediction value;

[0023] The encoding loss function of the content encoder is updated according to the speech content.

[0024] In some embodiments, the step of calculating the training speech according to the discriminator in the encoding model to obtain a discriminant prediction value includes:

[0025] Acquire a real speech set, wherein the real speech set consists of real speech of multiple speakers;

[0026] Inputting the training speech into the discriminator in the encoding model to compare the training speech with the real speech in the real speech set to obtain a comparison result;

[0027] The discrimination prediction value is obtained according to the comparison result.

[0028] In some embodiments, the target coding model includes a target encoder and a target generator, the speech request includes a request speech vector; and the step of inputting the speech request into the target coding model and generating a synthesized speech corresponding to the speech request according to the timbre information includes:

[0029] Inputting the speech request into a target encoder in the target coding model to perform vector extraction to obtain the request speech vector of the speech request;

[0030] Inputting the requested speech vector into the target coding model to perform relative entropy calculation to obtain a calculation result;

[0031] The calculation result and the timbre information are input into the target generator of the target coding model to generate the synthesized speech corresponding to the speech request.

[0032] In some embodiments, the target encoder includes a residual layer; the step of inputting the speech request into the target encoder of the target coding model to perform vector extraction to obtain the request speech vector of the speech request includes:

[0033] Inputting the speech request into a target encoder in the target coding model for extraction to obtain a Mel-frequency cepstrum of the speech request;

[0034] The Mel-frequency cepstrum is calculated according to the residual layer of the target encoder to obtain the requested speech vector.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a speech conversion device based on a coding model, the device comprising:

[0036] A sample acquisition module, used to acquire a sample speech set, wherein the sample speech set includes a source speaker's speech and a reference speech;

[0037] A speech synthesis module, configured to perform speech synthesis on the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain a target speech;

[0038] A model training module, used to train the coding model according to the target speech and determine a loss function corresponding to the coding model;

[0039] A model updating module, used for updating the encoding model according to the loss function to obtain a target encoding model;

[0040] A request receiving module, used for receiving a user's voice request, wherein the voice request carries timbre information;

[0041] The speech conversion module is used to input the speech request into the target coding model, and convert the speech request into corresponding synthesized speech according to the timbre information.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the coding model-based speech conversion method described in the first aspect is realized.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the coding model-based speech conversion method described in the first aspect above.

[0044] The speech conversion method and device based on the coding model, electronic device and storage medium proposed in the present application first obtain a sample speech set, and synthesize the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain the target speech, which is convenient for training the coding model. Then, the coding model is trained with the target speech to obtain the loss function of the coding model, and the coding model is updated according to the loss function to obtain the target coding model, thereby enhancing the generalization ability of the coding model and improving the efficiency and accuracy of speech conversion. Finally, the user's speech request is received, and the speech request is input into the target coding model, so that the target coding model converts the speech according to the timbre information in the speech request to obtain the final synthesized speech, thereby enhancing the generalization of the coding model for any speaker, improving the accuracy of speech conversion, and enhancing the authenticity of the synthesized speech by improving the timbre information. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a speech conversion method based on a coding model provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0047] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0048] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0049] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0050] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0051] Figure 7 yes Figure 6 Flow chart of step S601 in FIG.

[0052] Figure 8 It is a structural diagram of a speech conversion device based on a coding model provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0060] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0061] Mapping: Given two data models, data mapping establishes a correspondence between data elements between the models. This process is called data mapping. Data mapping is the first step in many data integration tasks, such as data migration, data cleaning, data integration, semantic web construction, and p2p information systems. There are two ways of data mapping: hand-coded and graphical manual. Hand-coded is to define the data correspondence directly using programming languages ​​such as XSLT, JAVA, and C++. Graphical operations usually support users to draw a line between data items to define the correspondence between data items. Some tools that support visual operations can automatically establish this correspondence. This automatically established correspondence generally requires that the data items have the same name. Regardless of whether the operation is manual or the relationship is automatically established, the tool is ultimately required to automatically convert the graphically represented correspondence into an executable program such as XSLT, JAVA, and C++.

[0062] Robustness: Robust is the transliteration of Robust, which means strong and strong. It is also the ability of the system to survive abnormal and dangerous situations. For example, whether computer software can survive input errors, disk failures, network overloads or intentional attacks without freezing or crashing is the robustness of the software. The so-called "robustness" also refers to the characteristics of the control system maintaining certain other performance under certain (structure, size) parameter perturbations. According to different definitions of performance, it can be divided into stability robustness and performance robustness. A fixed controller designed with the robustness of a closed-loop system as the goal is called a robust controller.

[0063] Loss Function: A loss function or cost function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, it is used for model parameter estimation in statistics and machine learning, for risk management and decision-making in macroeconomics, and for optimal control theory in control theory.

[0064] Relative Entropy: Also known as Kullback-Leibler divergence or information divergence, it is an asymmetric measure of the difference between two probability distributions. In information theory, relative entropy is equivalent to the difference between the Shannon entropy of two probability distributions.

[0065] Mel-Frequency Spectrum (MFC): A spectrum that can be used to represent short-term audio. Its principle is based on the log spectrum represented by the nonlinear Mel scale and its linear cosine transform. Mel-Frequency Cipstal Coefficients (MFCC) are a set of key coefficients used to create the Mel-Frequency Cipstrum. From the fragments of the music signal, we can get a set of cepstrum that can represent the music signal, and the Mel-Frequency Cipstrum coefficients are the cepstrum (the spectrum of the spectrum) derived from this cepstrum. Different from the general cepstrum, the biggest feature of the Mel-Frequency Cipstrum is that the frequency bands on the Mel-Frequency Cipstrum are evenly distributed on the Mel scale. In other words, such frequency bands are closer to the nonlinear human auditory system than the linear cepstrum representation method we usually see. For example, in the technology of audio compression, we often use Mel-Frequency Cipstrum to process.

[0066] Based on this, the embodiments of the present application provide a speech conversion method and device, an electronic device and a storage medium based on a coding model, aiming to enhance the generalization of the coding model for any speaker and improve the accuracy of speech conversion.

[0067] The coding model-based speech conversion method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the coding model-based speech conversion method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The speech conversion method based on the coding model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech conversion method based on the coding model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a speech conversion method based on a coding model, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] Figure 1 is an optional flowchart of the speech conversion method based on the coding model provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0073] Step S101, obtaining a sample speech set;

[0074] It should be noted that the sample speech set includes the source speaker speech and the reference speech.

[0075] It is understandable that the reference voice may be the voice of another speaker, the voice of a cartoon character, or a synthesized voice, and the reference voice may be manually input or directly called from a voice library, and this embodiment does not impose any specific limitation.

[0076] Step S102, performing speech synthesis on the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain a target speech;

[0077] Step S103, training the coding model according to the target speech, and determining a loss function corresponding to the coding model;

[0078] Step S104, updating the coding model according to the loss function to obtain a target coding model;

[0079] Step S105, receiving a user's voice request;

[0080] It should be noted that the voice request carries timbre information.

[0081] Step S106, inputting the speech request into the target coding model, and converting the speech in the speech request into corresponding synthesized speech according to the timbre information.

[0082] In steps S101 to S106 shown in the embodiment of the present application, first, a sample speech set is obtained, and the source speaker speech and the reference speech in the sample speech set are synthesized based on the coding model to obtain the target speech, which is convenient for training the coding model. Then, the coding model is trained with the target speech to obtain the loss function of the coding model, and the coding model is updated according to the loss function to obtain the target coding model, thereby enhancing the generalization ability of the coding model and improving the efficiency and accuracy of speech conversion. Finally, the user's voice request is received, and the voice request is input into the target coding model, so that the target coding model converts the speech according to the timbre information in the voice request to obtain the final synthesized speech, thereby enhancing the generalization of the coding model for any speaker, improving the accuracy of speech conversion, and enhancing the authenticity of the synthesized speech by improving the timbre information.

[0083] In step S102 of some embodiments, speech synthesis is performed on the source speaker's speech and the reference speech in the sample speech set based on the encoding model to obtain the target speech, so as to facilitate the subsequent training of the encoding model through the target speech.

[0084] It should be noted that the encoding model includes an encoder, a generator and a discriminator, wherein the encoder includes two parts: a content encoder and a vector encoder.

[0085] In step S103 of some embodiments, the coding model is trained according to the target speech, and a loss function corresponding to the coding model is determined, thereby improving the speech conversion and speech synthesis efficiency of the coding model.

[0086] In step S104 of some embodiments, the coding model is updated according to the loss function to obtain a target coding model, thereby enhancing the generalization ability of the coding model and facilitating the conversion of the input speech.

[0087] In step S105 of some embodiments, a voice request from a user carrying timbre information is received, wherein the voice request is voice information that the user wants to convert.

[0088] In step S106 of some embodiments, the speech request is input into the target coding model, and the speech in the speech request is converted into corresponding synthesized speech according to the timbre information, thereby realizing the conversion of the speech and improving the authenticity of the speech conversion.

[0089] It should be noted that inputting the voice request into the target coding model can enable the target coding model to convert the voice in the voice request according to the timbre information, thereby improving the authenticity of the voice conversion and avoiding situations such as excessive timbre differences and inaccurate voice conversion.

[0090] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0091] It should be noted that the encoding model includes a content encoder and a vector encoder.

[0092] Step S201, inputting the source speaker's speech into the content encoder in the coding model for mapping to obtain the speech content of the source speaker's speech;

[0093] Step S202, inputting the reference speech into a vector encoder in the coding model to perform vector extraction to obtain a reference speech vector of the reference speech;

[0094] Step S203, performing speech synthesis according to the reference speech vector and the speech content to obtain the target speech.

[0095] In step S201 of some embodiments, the source speaker's speech is input into a content encoder in a coding model for mapping to obtain speech content of the source speaker's speech, thereby avoiding speech content errors, speech content missing, and the like.

[0096] It should be noted that the speech content includes the speaker's speech content, speech phonemes, and speech rhythm.

[0097] In step S202 of some embodiments, the reference speech is input into a vector encoder in a coding model for vector extraction to obtain a reference speech vector of the reference speech to facilitate subsequent speech synthesis.

[0098] In step S203 of some embodiments, speech synthesis is performed based on the reference speech vector and speech content to obtain a target speech. At this time, the target speech includes the speech content of the source speaker's speech and the reference speech vector of the reference speech, thereby retaining the content of the source speaker's speech. When the speech content does not change, it is synthesized with the reference speech vector to obtain the target speech, thereby avoiding the loss of speech content in the target speech.

[0099] See also Figure 3 In some embodiments, step S203 may include but is not limited to steps S301 to S303:

[0100] Step S301, inputting the source speaker's speech into a vector encoder in a coding model to perform vector extraction, and obtaining a source speaker vector of the source speaker's speech;

[0101] Step S302, replacing the source speaker vector in the source speaker's speech according to the reference speech vector to obtain a replacement result;

[0102] Step S303: Input the replacement result and the speech content into the generator in the coding model for synthesis to generate the target speech.

[0103] In step S301 of some embodiments, the source speaker's speech is input into a vector encoder in a coding model for vector extraction to obtain a source speaker vector of the source speaker's speech, so as to facilitate subsequent speech synthesis.

[0104] In step S302 of some embodiments, the source speaker vector in the source speaker's speech is replaced according to the reference speech vector to obtain a replacement result, thereby achieving conversion of the speech information and avoiding problems such as discontinuity of the converted speech due to discrete speech vectors.

[0105] In step S303 of some embodiments, the replacement result and the speech content are input into a generator in the coding model for synthesis to generate the target speech, thereby completing the conversion and synthesis of the speech.

[0106] See also Figure 4 In some embodiments, step S103 may include but is not limited to steps S401 to S404:

[0107] It should be noted that the loss function includes encoding loss function, generation loss function and discrimination loss function.

[0108] Step S401, training the encoder and generator in the encoding model according to the target speech to generate training speech;

[0109] Step S402, calculating the training speech according to the discriminator in the coding model to obtain a discriminant prediction value;

[0110] Step S403, updating the discrimination loss function of the discriminator and the generation loss function of the generator according to the discrimination prediction value;

[0111] Step S404: updating the encoding loss function of the content encoder according to the speech content.

[0112] In step S401 of some embodiments, the encoder and the generator in the encoding model are trained according to the target speech to generate training speech, wherein the training speech is the synthesized speech generated by the encoder and the generator to facilitate the subsequent training of the discriminator.

[0113] In step S402 of some embodiments, the training speech is calculated according to the discriminator in the coding model to obtain a discrimination prediction value to facilitate subsequent updating of the coding loss function.

[0114] It should be noted that the encoder and the generator are first trained with the target speech to obtain the training speech, so that the discriminator can be trained based on the training speech, so that the discriminator determines whether the training speech is synthesized speech, obtains the difference between the training speech and the real speech, and obtains the discrimination prediction value based on the difference between the training speech and the real speech.

[0115] In step S403 of some embodiments, the discrimination loss function of the discriminator and the generation loss function of the generator are updated according to the discrimination prediction value, so as to facilitate updating the discriminator and the generator through the discrimination loss function and the generation loss function, thereby improving the efficiency of model conversion.

[0116] It should be noted that both the discriminative loss function and the generative loss function are adversarial loss functions.

[0117] In some embodiments, the formulas (1) of the discrimination loss function and (2) of the generation loss function are expressed as follows:

[0118]

[0119]

[0120] It can be understood that in formula (1) and formula (2), D is the discriminator, G is the generator, and c is the speech content of the source speaker. is the speech vector of the training speech, E c is the content encoder, The discrimination prediction value calculated by the discriminator on the training speech.

[0121] In step S404 of some embodiments, the encoding loss function of the content encoder is updated according to the speech content, so that the converted synthesized speech maintains the speech content of the source speaker and avoids changes or loss of the speech content.

[0122] In some embodiments, the encoding loss function formula (3) is expressed as follows:

[0123]

[0124] It can be understood that E in formula (3) c (x) is the result obtained by inputting the source speaker's speech into the content encoder.

[0125] See also Figure 5 In some embodiments, step S402 may also include but is not limited to steps S501 to S502:

[0126] Step S501, obtaining a real speech set;

[0127] It should be noted that the real speech set consists of the real speech of multiple speakers.

[0128] Step S502, inputting the training speech into the discriminator in the coding model, so as to compare the training speech with the real speech in the real speech set, and obtain a comparison result;

[0129] Step S503: obtaining a discrimination prediction value according to the comparison result.

[0130] In step S501 of some embodiments, a real speech set is obtained, and the real speech set consists of real speech of multiple speakers, and the real speech includes real speech content and real speech vector, etc., so that the subsequent discriminator can compare the real speech with the training speech.

[0131] In step S502 of some embodiments, the training speech is input into the discriminator in the coding model to compare the training speech with the real speech in the real speech set to obtain a comparison result, so as to accurately obtain the difference between the training speech and the real speech, thereby facilitating the accurate determination of the discrimination prediction value.

[0132] It should be noted that the comparison between the training speech and the real speech includes comparing the timbre information, audio information, phoneme information, etc. of the two, thereby obtaining multiple comparison results.

[0133] In step S503 of some embodiments, the multiple comparison results obtained in step S502 are integrated to obtain a recognition prediction value, thereby achieving an update of the coding model.

[0134] See also Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S602:

[0135] It should be noted that the target coding model includes a target encoder and a target generator.

[0136] Step S601, inputting the speech request into the target encoder in the target coding model to perform vector extraction to obtain a request speech vector of the speech request;

[0137] Step S602, inputting the requested speech vector into the target coding model to perform relative entropy calculation to obtain a calculation result;

[0138] Step S603, inputting the calculation result and timbre information into the target generator of the target coding model to generate synthesized speech corresponding to the speech request.

[0139] In step S601 of some embodiments, the speech request is input into a target encoder in a target coding model for vector extraction to obtain a request speech vector of the speech request, so as to facilitate subsequent speech conversion according to the request speech vector.

[0140] In step S602 of some embodiments, the requested speech vector is input into the target coding model for relative entropy calculation to obtain a calculation result, thereby implementing constraints on the requested speech vector, improving the robustness of the target coding model to the input speech request, and making the requested speech vector smoother and more continuous.

[0141] In step S603 of some embodiments, the calculation result and timbre information are input into the target generator of the target coding model to generate synthesized speech corresponding to the speech request, so that the synthesized speech is consistent with the timbre information in the requested speech, thereby improving the timbre similarity of the speech conversion.

[0142] In some embodiments, the target generator may also be trained using timbre information, thereby increasing a reconstruction loss function corresponding to the target generator. The reconstruction loss function (4) is as follows:

[0143] L fm (G,E c )=||D(x)-D(G(c,z))|| (4)

[0144] It can be understood that in formula (4), D is the discriminator, G is the generator, c is the speech content of the source speaker, z is the request speech vector, and E c For content encoders.

[0145] See also Figure 7 In some embodiments, step S601 may include but is not limited to steps S701 to S702:

[0146] It should be noted that the target encoder includes a residual layer.

[0147] In some embodiments, the target encoder in the target encoder includes four upsampling layers and four residual layers, and the target generator includes four residual layers and four downsampling layers.

[0148] Step S701, inputting the speech request into the target encoder in the target encoding model for extraction to obtain the Mel-frequency cepstrum of the speech request;

[0149] Step S702, calculating the Mel-frequency cepstrum according to the residual layer of the target encoder to obtain the requested speech vector.

[0150] In step S701 of some embodiments, the speech request is input into a target encoder in a target coding model for extraction to obtain a Mel-frequency cepstrum of the speech request, thereby reducing the requirements of the target coding model on the corpus.

[0151] In step S702 of some embodiments, the Mel-frequency cepstrum is calculated according to the residual layer of the target encoder to obtain the requested speech vector, so as to facilitate subsequent speech conversion according to the requested speech vector.

[0152] In some embodiments, the Mel-cepstrum in the target speech request is first extracted, and then the Mel-cepstrum is input into the residual layer of the target encoder for calculation to obtain the requested speech vector, and finally the requested speech vector is input into the target coding model for relative entropy calculation to obtain the calculation result.

[0153] It should be noted that the relative entropy function (5) for inputting the request speech vector into the target coding model for relative entropy calculation is as follows:

[0154] L kl (E s )=log(p(z;μ,σ))-log(N(z;0,1)) (5)

[0155] It can be understood that N(z; 0, 1) represents the standard normal distribution. p(z; μ, σ) means that the speaker vector z follows a normal distribution with a mean of μ and a standard deviation of σ. The mean μ and the standard deviation σ are the encoder E s The output, mean μ and standard deviation σ are the results of inputting the Mel-frequency inverse spectrum into the residual layer of the target encoder for calculation.

[0156] In some embodiments, the loss function (6) of the target encoding network is obtained by formulas (1)-(5), and the specific formula is as follows:

[0157] L(G,E c , E s )=L adv (G,E c )+L fm (G,E c )+L con (G,E c )+L kl (E s ) (6)

[0158] See also Figure 8 The embodiment of the present application further provides a speech conversion device based on a coding model, which can implement the above-mentioned speech conversion method based on a coding model, and the device includes:

[0159] A sample acquisition module 801 is used to acquire a sample speech set, where the sample speech set includes a source speaker's speech and a reference speech;

[0160] The speech synthesis module 802 is used to perform speech synthesis on the source speaker speech and the reference speech in the sample speech set based on the coding model to obtain the target speech;

[0161] A model training module 803 is used to train the coding model according to the target speech and determine the loss function corresponding to the coding model;

[0162] A model updating module 804 is used to update the coding model according to the loss function to obtain a target coding model;

[0163] The request receiving module 805 is used to receive a voice request from a user, where the voice request carries timbre information;

[0164] The speech conversion module 806 is used to input the speech request into the target coding model and convert the speech request into corresponding synthesized speech according to the timbre information.

[0165] The specific implementation of the speech conversion device based on the coding model is basically the same as the specific implementation of the speech conversion method based on the coding model mentioned above, and will not be repeated here.

[0166] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein the program is executed by the processor to realize the above-mentioned speech conversion method based on the coding model. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0167] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0168] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0169] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech conversion method based on the coding model in the embodiment of the present application;

[0170] Input / output interface 903, used to implement information input and output;

[0171] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0172] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0173] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0174] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned coding model-based speech conversion method.

[0175] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The speech conversion method based on a coding model, the speech conversion device based on a coding model, the electronic device and the storage medium provided in the embodiments of the present application first obtain a sample speech set, and synthesize the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain a target speech, which is convenient for training the coding model. Then, the coding model is trained with the target speech to obtain a loss function of the coding model. The coding model is updated according to the loss function to obtain a target coding model, thereby enhancing the generalization ability of the coding model and improving the efficiency and accuracy of speech conversion. Finally, a user's speech request is received, and the speech request is input into the target coding model, so that the target coding model converts the speech according to the timbre information in the speech request to obtain the final synthesized speech, thereby enhancing the generalization of the coding model for any speaker, improving the accuracy of speech conversion, and enhancing the authenticity of the synthesized speech by improving the timbre information.

[0177] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0178] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0179] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0180] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0181] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0182] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many improvements to method flows today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using hardware entity modules.

[0183] For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logic function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used when writing program development. The original code before compilation must also be written in a specific programming language, which is called Hardware Description Language (HDL). There is not only one type of HDL, but many types, such as:

[0184] ABEL (Advanced Boolean Expression Language); AHDL (Altera Hardware Description Language); Confluence; CUPL (Cornell University Programming Language); HDCal; and JHDL (Java Hardware Description Language); Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. At present, in the field of technology, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and language Verilog are more commonly used. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logical method flow.

[0185] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers:

[0186] ARC 625D, Atmel AT91SAM, MicrochIP address PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code, it is entirely possible to implement the same function in the form of a logic gate, a switch, an ASIC, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0187] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0188] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0190] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0194] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0195] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0196] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0197] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0198] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0199] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0200] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A speech conversion method based on a coding model, characterized in that: The encoding model includes a content encoder and a vector encoder; the method includes: Acquire a sample speech set, wherein the sample speech set includes a source speaker speech and a reference speech; Performing speech synthesis on the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain a target speech; Training the coding model according to the target speech to determine a loss function corresponding to the coding model, wherein the loss function includes a coding loss function, a generation loss function, and a discrimination loss function; Update the encoding model according to the loss function to obtain a target encoding model; receiving a voice request from a user, wherein the voice request carries timbre information; Inputting the speech request into the target coding model, and converting the speech in the speech request into corresponding synthesized speech according to the timbre information; The step of performing speech synthesis on the source speaker's speech and the reference speech in the sample speech set based on the coding model to obtain the target speech includes: Inputting the source speaker's speech into the content encoder in the coding model for mapping to obtain the speech content of the source speaker's speech; Inputting the reference speech into the vector encoder in the coding model to perform vector extraction to obtain a reference speech vector of the reference speech; Inputting the source speaker's speech into the vector encoder in the coding model to perform vector extraction to obtain a source speaker vector of the source speaker's speech; Replacing the source speaker vector in the source speaker's speech according to the reference speech vector to obtain a replacement result; Inputting the replacement result and the speech content into a generator in the coding model for synthesis to generate the target speech; The step of training the coding model according to the target speech and determining a loss function corresponding to the coding model includes: Training the encoder and the generator in the coding model according to the target speech to generate training speech; Calculating the training speech according to the discriminator in the encoding model to obtain a discrimination prediction value; Update the discrimination loss function of the discriminator and the generation loss function of the generator according to the discrimination prediction value; The encoding loss function of the content encoder is updated according to the speech content.

2. The method for voice conversion based on a coding model according to claim 1, characterized in that: The step of calculating the training speech according to the discriminator in the coding model to obtain a discriminant prediction value includes: Acquire a real speech set, wherein the real speech set consists of real speech of multiple speakers; Inputting the training speech into the discriminator in the encoding model to compare the training speech with the real speech in the real speech set to obtain a comparison result; The discrimination prediction value is obtained according to the comparison result.

3. The method for voice conversion based on a coding model according to claim 1, characterized in that: The target coding model includes a target encoder and a target generator; The step of inputting the speech request into the target coding model and generating a synthesized speech corresponding to the speech request according to the timbre information comprises: Inputting the speech request into a target encoder in the target coding model to perform vector extraction to obtain a request speech vector of the speech request; Inputting the requested speech vector into the target coding model to perform relative entropy calculation to obtain a calculation result; The calculation result and the timbre information are input into the target generator of the target coding model to generate the synthesized speech corresponding to the speech request.

4. The method for voice conversion based on a coding model according to claim 3, characterized in that: The target encoder includes a residual layer; the target encoder inputting the speech request into the target coding model performs vector extraction to obtain the request speech vector of the speech request, including: Inputting the speech request into a target encoder in the target coding model for extraction to obtain a Mel-frequency cepstrum of the speech request; The Mel-frequency cepstrum is calculated according to the residual layer of the target encoder to obtain the requested speech vector.

5. A speech conversion device based on a coding model, characterized in that: The coding model includes a content encoder and a vector encoder; the device includes: A sample acquisition module, used to acquire a sample speech set, wherein the sample speech set includes a source speaker's speech and a reference speech; A speech synthesis module, configured to perform speech synthesis on the source speaker speech and the reference speech in the sample speech set based on the coding model to obtain a target speech; wherein the speech synthesis module is further configured to input the source speaker speech into the content encoder in the coding model for mapping to obtain the speech content of the source speaker speech; input the reference speech into the vector encoder in the coding model for vector extraction to obtain a reference speech vector of the reference speech; input the source speaker speech into the vector encoder in the coding model for vector extraction to obtain a source speaker vector of the source speaker speech; replace the source speaker vector in the source speaker speech according to the reference speech vector to obtain a replacement result; and input the replacement result and the speech content into the generator in the coding model for synthesis to generate the target speech; A model training module is used to train the coding model according to the target speech and determine the loss function corresponding to the coding model, wherein the loss function includes a coding loss function, a generation loss function and a discrimination loss function; the model training module is also used to train the encoder and the generator in the coding model according to the target speech to generate training speech; calculate the training speech according to the discriminator in the coding model to obtain a discrimination prediction value; update the discrimination loss function of the discriminator and the generation loss function of the generator according to the discrimination prediction value; and update the coding loss function of the content encoder according to the speech content; A model updating module, used for updating the encoding model according to the loss function to obtain a target encoding model; A request receiving module, used for receiving a user's voice request, wherein the voice request carries timbre information; The speech conversion module is used to input the speech request into the target coding model, and convert the speech request into corresponding synthesized speech according to the timbre information.

6. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the coding model-based speech conversion method as described in any one of claims 1 to 4 are realized.

7. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the coding model-based speech conversion method described in any one of claims 1 to 4.

Images