Speech conversion model training, speech conversion method, system and device

Abstract

This application discloses a speech conversion model training and speech conversion method, system, and device. The method comprises obtaining a training dataset comprising first speech data of a first subject and second speech data of a target subject; extracting first spectrum data and first fundamental frequency data of the first speech data, and extracting second fundamental frequency data of the second speech data; generating predicted speech data of the first subject based on the first spectrum data and first fundamental frequency data, and performing discrimination processing on the first speech data and the predicted speech data to obtain a first discrimination result; generating predicted fundamental frequency data of the target subject based on the first fundamental frequency data, and performing discrimination processing on the second fundamental frequency data and the predicted fundamental frequency data to obtain a second discrimination result; and updating parameters of the speech conversion model based on the first and second discrimination results. This method can improve the speech conversion performance of the trained speech conversion model. This application can be widely applied in the field of speech processing technology.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech conversion model training, speech conversion method, system and device. Background Art

[0002] In recent years, with the development of artificial intelligence (AI) technology, various related applications have gradually become part of people's lives and work. Among them, voice conversion technology is an important and widely used technology that facilitates interpersonal communication. Voice conversion technology involves converting the personality characteristics of an original speaker into those of a target speaker, making the converted speech sound consistent with the target speaker's voice while preserving the content of the original speech during the conversion process.

[0003] In related technologies, when performing voice conversion, the voice tone is often simply converted. The converted voice data cannot well reflect the characteristics of the target speaker, resulting in the voice sounding neither like the original speaker nor the target speaker. Therefore, the conversion effect is poor and cannot provide users with a good application experience.

[0004] In summary, the problems existing in related technologies need to be solved urgently. Summary of the Invention

[0005] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, one purpose of an embodiment of the present application is to provide a method for training a speech conversion model, which can improve the speech conversion effect of the trained speech conversion model.

[0007] In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present application include:

[0008] In one aspect, an embodiment of the present application provides a method for training a speech conversion model, wherein the speech conversion model includes a first generator, a second generator, a first discriminator, and a second discriminator; the method for training the speech conversion model includes:

[0009] Acquire a training data set; the training data set includes first speech data of a first subject and second speech data of a target subject;

[0010] extracting first spectrum data and first fundamental frequency data from the first speech data, and extracting second fundamental frequency data from the second speech data;

[0011] generating predicted speech data of the first object by the first generator based on the first spectrum data and the first fundamental frequency data, and inputting the first speech data and the predicted speech data into the first discriminator for discrimination processing to obtain a first discrimination result;

[0012] inputting the first fundamental frequency data into the second generator to generate predicted fundamental frequency data of the target object, and inputting the second fundamental frequency data and the predicted fundamental frequency data into the second discriminator for identification processing to obtain a second identification result;

[0013] The parameters of the speech conversion model are updated according to the first identification result and the second identification result.

[0014] In addition, the training method of a speech conversion model according to the above embodiment of the present application may also have the following additional technical features:

[0015] Furthermore, in one embodiment of the present application, generating the predicted speech data of the first object by the first generator according to the first spectrum data and the first fundamental frequency data includes:

[0016] performing vectorization processing on the first spectrum data to obtain first vector data;

[0017] The first vector data and the first fundamental frequency data are input into the first generator to generate predicted speech data of the first object.

[0018] Furthermore, in one embodiment of the present application, vectorizing the first spectrum data to obtain first vector data includes:

[0019] performing encoding processing on the first spectrum data to obtain a continuous variable;

[0020] Performing vectorization processing on the continuous variable to obtain a discrete variable;

[0021] The difference between the continuous variable and the discrete variable is calculated to obtain the first vector data.

[0022] Furthermore, in one embodiment of the present application, updating the parameters of the speech conversion model according to the first identification result and the second identification result includes:

[0023] Determine a first loss value for training according to the first identification result, and update parameters of the first discriminator and the first generator according to the first loss value;

[0024] According to the second identification result, a second loss value of training is determined, and the parameters of the second discriminator and the parameters of the second generator are updated according to the second loss value.

[0025] Furthermore, in one embodiment of the present application, determining a first loss value for training according to the first identification result, and updating parameters of the first discriminator and the first generator according to the first loss value, includes:

[0026] Detecting the current training stage of the speech conversion model;

[0027] When the speech conversion model is in the training phase of the first discriminator, determining the first loss value according to the deviation between the first identification result and the true result, and updating the parameters of the first discriminator according to the first loss value; or

[0028] When the speech conversion model is in the training stage of the first generator, the first loss value is determined according to the degree of consistency between the first identification result and the true result, and the parameters of the first generator are updated according to the first loss value.

[0029] Furthermore, in one embodiment of the present application, the first loss value or the second loss value is determined by a cross entropy loss function.

[0030] On the other hand, an embodiment of the present application provides a voice conversion method, including:

[0031] Acquiring third voice data to be converted;

[0032] Inputting the third speech data into a speech conversion model; the speech conversion model is trained by the speech conversion model training method as described above;

[0033] The third voice data is converted using the voice conversion model to obtain converted fourth voice data of the target object.

[0034] On the other hand, an embodiment of the present application provides a training system for a speech conversion model, wherein the speech conversion model includes a first generator, a second generator, a first discriminator, and a second discriminator; a training method for the speech conversion model, and the training system for the speech conversion model includes:

[0035] An acquiring unit, configured to acquire a training data set; the training data set includes first speech data of a first subject and second speech data of a target subject;

[0036] an extraction unit, configured to extract first spectrum data and first fundamental frequency data from the first speech data, and extract second fundamental frequency data from the second speech data;

[0037] a first processing unit, configured to generate predicted speech data of the first object by the first generator based on the first spectrum data and the first fundamental frequency data, and input the first speech data and the predicted speech data into the first discriminator for discrimination processing to obtain a first discrimination result;

[0038] a second processing unit, configured to input the first fundamental frequency data into the second generator to generate predicted fundamental frequency data of the target object, and input the second fundamental frequency data and the predicted fundamental frequency data into the second discriminator for discrimination processing to obtain a second discrimination result;

[0039] An updating unit is configured to update parameters of the speech conversion model according to the first identification result and the second identification result.

[0040] In another aspect, an embodiment of the present application provides a computer device, including:

[0041] at least one processor;

[0042] at least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned method for training a speech conversion model or the above-mentioned speech conversion method.

[0044] On the other hand, an embodiment of the present application also provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the training method of the above-mentioned speech conversion model or implement the above-mentioned speech conversion method.

[0045] The advantages and benefits of this application will be partially given in the following description, and partially become apparent from the following description, or learned through practice of this application:

[0046] The embodiment of the present application discloses a training method for a speech conversion model, which includes a first generator, a second generator, a first discriminator, and a second discriminator. The method obtains a training data set; the training data set includes first speech data of a first object and second speech data of a target object; first spectrum data and first fundamental frequency data are extracted from the first speech data, and second fundamental frequency data are extracted from the second speech data; based on the first spectrum data and the first fundamental frequency data, predicted speech data of the first object is generated by the first generator, and the first speech data and the predicted speech data are input into the first discriminator for identification processing to obtain a first identification result; the first fundamental frequency data is input into the second generator to generate predicted fundamental frequency data of the target object, and the second fundamental frequency data and the predicted fundamental frequency data are input into the second discriminator for identification processing to obtain a second identification result; and the parameters of the speech conversion model are updated according to the first identification result and the second identification result. This method constructs a speech conversion model based on two sets of adversarial generative networks through adversarial generation, and updates the parameters of the first generator responsible for speech data generation and the second generator responsible for fundamental frequency conversion in the model. This can make the speech data output by the trained speech conversion model closer to the speaking intonation and rhythm of the target object, and improve the effect of speech conversion in related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A schematic diagram of an implementation environment for a training method for a speech conversion model provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of performing speech conversion using a speech conversion model provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of a method for training a speech conversion model provided in an embodiment of the present application;

[0051] Figure 4 This is a flow chart of step 130 provided in an embodiment of the present application;

[0052] Figure 5This is a flow chart of step 1301 provided in an embodiment of the present application;

[0053] Figure 6 This is a flow chart of step 150 provided in an embodiment of the present application;

[0054] Figure 7 This is a flow chart of step 1501 provided in an embodiment of the present application;

[0055] Figure 8 A flowchart of a voice conversion method provided in an embodiment of the present application;

[0056] Figure 9 A schematic diagram of the structure of a speech conversion model training system provided in an embodiment of the present application;

[0057] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] First, let’s analyze some of the terms used in this application:

[0062] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0063] Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0064] Deep learning (DL) is a new research direction in machine learning, introduced to bring it closer to its original goal: artificial intelligence. Deep learning studies the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound.

[0065] Generative Adversarial Networks (GANs), also known as generative adversarial networks, are a type of deep learning model that has recently become one of the most promising approaches for unsupervised learning on complex distributions. The model produces highly accurate outputs through a learning process involving (at least) two modules: a generator and a discriminator.

[0066] The training of the speech conversion model, speech conversion method, system and device provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the training method of the speech conversion model in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0068] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] In recent years, with the development of artificial intelligence (AI) technology, various related applications have gradually become part of people's lives and work. Among them, voice conversion technology is an important and widely used technology that facilitates interpersonal communication. Voice conversion technology involves converting the personality characteristics of an original speaker into those of a target speaker, making the converted speech sound consistent with the target speaker's voice while preserving the content of the original speech during the conversion process.

[0070] In related technologies, when performing voice conversion, the voice tone is often simply converted. The converted voice data cannot well reflect the characteristics of the target speaker, resulting in the voice sounding neither like the original speaker nor the target speaker. Therefore, the conversion effect is poor and cannot provide users with a good application experience.

[0071] In order to solve the problems existing in the related art, the embodiments of the present application provide a training and speech conversion method, system and device for a speech conversion model, which includes a first generator, a second generator, a first discriminator and a second discriminator; the method obtains a training data set; the training data set includes first speech data of a first object and second speech data of a target object; first spectrum data and first fundamental frequency data are extracted from the first speech data, and second fundamental frequency data are extracted from the second speech data; based on the first spectrum data and the first fundamental frequency data, predicted speech data of the first object is generated by the first generator, and the first speech data and the predicted speech data are input into the first discriminator for identification processing to obtain a first identification result; the first fundamental frequency data is input into the second generator to generate predicted fundamental frequency data of the target object, and the second fundamental frequency data and the predicted fundamental frequency data are input into the second discriminator for identification processing to obtain a second identification result; the parameters of the speech conversion model are updated according to the first identification result and the second identification result. This method constructs a speech conversion model through adversarial generation and updates the parameters of the first generator responsible for speech data generation and the second generator responsible for fundamental frequency conversion in the model. This can make the speech data output by the trained speech conversion model closer to the speaking intonation and rhythm of the target object, thereby improving the effect of speech conversion in related applications.

[0072] Figure 1 Schematic diagram of the implementation environment of a training method for a speech conversion model provided in an embodiment of the present application. Figure 1 The hardware and software components of this implementation environment primarily include a user terminal 101 and a server 102, with the user terminal 101 communicating with the server 102. The speech conversion model training method can be configured on the server 102 side. The server 102 trains the speech conversion model using a large amount of training data and then sends files related to the trained speech conversion model to the user terminal 101. In this way, the user terminal 101 can perform related speech conversion tasks based on the trained speech conversion model.

[0073] Specifically, the user terminal 101 in the present application may include but is not limited to any one or more of a smart watch, a smart phone, a computer, a personal digital assistant (PDA), an intelligent voice interaction device, a smart home appliance or a vehicle-mounted terminal. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The user terminal 101 and the server 102 can establish a communication connection through a wireless network or a wired network, which uses standard communication technology and / or protocols. The network can be set to the Internet or any other network, such as but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), any combination of mobile, wired or wireless networks, private networks or virtual private networks.

[0074] Of course, it is understandable that Figure 1 The implementation environment in the embodiment of the present application is only an optional application scenario of the training method of the speech conversion model provided in the embodiment of the present application. The actual application is not fixed to Figure 1 The hardware and software environment shown.

[0075] The following combination Figure 1 The implementation environment shown is used to describe in detail the training method of the speech conversion model provided in the embodiments of the present application.

[0076] First, the model structure and working principle of the speech conversion model provided in this application are introduced and explained.

[0077] Reference Figure 2In an embodiment of the present application, a speech conversion model is provided that includes two groups of adversarial generative networks, wherein one group of adversarial generative networks includes a first generator and a first discriminator. The first generator is used to generate converted speech data corresponding to the target speaker based on the content information of the extracted original speaker's speech data and the fundamental frequency data converted to the target speaker. The first discriminator is used to be trained together with the first generator through adversarial generation. In the actual application of the speech conversion model for speech conversion, the first discriminator may not be used. The other group of adversarial generative networks includes a second generator and a second discriminator. The second generator is used to generate converted fundamental frequency data corresponding to the target speaker based on the fundamental frequency data of the extracted original speaker's speech data. Similarly, the second discriminator is used to be trained together with the second generator through adversarial generation. In the actual application of the speech conversion model for speech conversion, the second discriminator may not be used.

[0078] In the actual application of the speech conversion model for speech conversion, first, a set of original speech data, that is, the speech data of the original speaker, is input, and then the speech data is processed, its spectrum data is extracted to obtain content information, and its fundamental frequency data is extracted; then, the extracted fundamental frequency data of the original speaker is input into the second generator, and the second generator generates the converted fundamental frequency data corresponding to the target speaker; then, the converted fundamental frequency data and the extracted spectrum data are input into the first generator together, and the first generator generates the converted speech data corresponding to the target speaker.

[0079] For further information, please refer to Figure 3 , Figure 3 is a flow chart of a method for training a speech conversion model provided in an embodiment of the present application, Figure 3 The training method of the speech conversion model shown is mainly executed on the server 102. The method can be applied to the relevant computer equipment in the server 102, but is not limited to the above form. Figure 3 The method includes but is not limited to steps 110 to 150.

[0080] Step 110: Acquire a training data set; the training data set includes first speech data of a first subject and second speech data of a target subject;

[0081] In this step, when training the speech conversion model, batches of training data can be obtained, which constitute a training dataset. Specifically, the training dataset here mainly includes speech data of a first subject and a target subject, where the first subject is the original speaker and the target subject is the target speaker. In this embodiment of the application, the speech data of the first subject is recorded as the first speech data, and the speech data of the target subject is recorded as the second speech data.

[0082] It should be noted that in the embodiment of the present application, there is no restriction on the channels for obtaining the first voice data and the second voice data. They can be collected directly from the relevant environment through the sound receiving device, or they can be obtained from other electronic devices and computer systems through a data transmission interface or remote communication transmission.

[0083] Furthermore, it should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user voice statements, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0084] Step 120: extracting first spectrum data and first fundamental frequency data from the first speech data, and extracting second fundamental frequency data from the second speech data;

[0085] In this step, after obtaining the training data set, the spectrum data and fundamental frequency data of each first speech data in the training data set are extracted. Spectral data is the representation of time-domain speech data in the frequency domain, which can be obtained by performing a Fourier transform on the speech data and contains the content information of the speech data; fundamental frequency data is the frequency of vocal cord vibration when a person speaks. This feature is related to the rhythm and rhythm of the speech and can reflect the speaker's unique speaking characteristics. In this embodiment of the present application, the spectrum data of the first speech data is recorded as the first spectrum data, and the fundamental frequency data of the first speech data is recorded as the first fundamental frequency data. Similarly, the fundamental frequency data of the second speech data is also extracted and recorded as the second fundamental frequency data.

[0086] Specifically, in an embodiment of the present application, when extracting spectrum data, the voice data can be framed and windowed according to a certain frame stack to obtain multiple audio frames, and each audio frame is processed separately. Here, framing refers to cutting the entire segment of voice data into multiple segments according to a preset rule, each segment being an audio frame, and windowing refers to intercepting each audio frame using a preset window function, so that the voice data after framing is more coherent and exhibits better periodic function characteristics. The window function can be any one of a rectangular window, a Hamming window, or a Gaussian window. The acquired audio frames are then subjected to a fast Fourier transform (FFT) to obtain spectrum data for each audio frame, which is then synthesized to obtain the spectrum data for the voice data. When extracting fundamental frequency data, an extraction algorithm based on the time domain, an extraction algorithm based on the frequency domain, or a statistical algorithm in the relevant technology can be adopted, and this application does not impose any restrictions on this.

[0087] Step 130: Generate predicted speech data of the first object by the first generator based on the first spectrum data and the first fundamental frequency data, and input the first speech data and the predicted speech data into the first discriminator for identification processing to obtain a first identification result;

[0088] In this step, after extracting the first spectral data and the first fundamental frequency data, the first generator can reconstruct the speech data of the first object based on the first spectral data and the first fundamental frequency data. The speech data of the first object generated by the first generator is recorded as predicted speech data. The predicted speech data is generated by the first generator using a relevant synthesis algorithm, specifically based on the speech content information contained in the spectral data and the speaker's speech prosody contained in the fundamental frequency data.

[0089] Then, in embodiments of the present application, the generation effect of the first generator can be identified by a first discriminator. Specifically, the purpose of this identification process is to determine the degree of proximity and similarity between the speech data generated by the first generator and the actual speech data output by the first object, that is, the degree of indistinguishability. Therefore, in embodiments of the present application, when performing the identification process through the first discriminator, the first speech data and the predicted speech data can be input into the first discriminator together, and the first discriminator is allowed to identify and output whether each speech data is speech data output by a real speaker or speech data generated by the first generator, thereby obtaining a corresponding identification result, which is recorded as the first identification result. For example, in some embodiments, the first identification result can include two categories: true and false. When certain speech data is input into the first discriminator and the first identification result output is judged to be true, it means that the first discriminator has determined that the speech data is speech data output by a real speaker; conversely, when certain speech data is input into the first discriminator and the first identification result output is judged to be false, it means that the first discriminator has determined that the speech data is speech data generated by the first generator. Here, the data form corresponding to the true and false first identification results can be flexibly set as needed. For example, in some embodiments, the data form of the first identification result can be a numerical value, such as a numerical value 0 and a numerical value 1. When the first identification result is a numerical value 0, it can be indicated that the first identification result output is determined to be false. When the first identification result is a numerical value 1, it can be indicated that the first identification result output is determined to be true. In other embodiments, the data form of the first identification result can be a vector, such as a vector (0, 1) and a vector (1, 0). When the first identification result is a vector (0, 1), it can be indicated that the first identification result output is determined to be false. When the first identification result is a vector (1, 0), it can be indicated that the first identification result output is determined to be true.

[0090] It should be noted that the data format of the above-mentioned first identification result and the correspondence between the data and the actual meaning can be flexibly set as needed, and this application does not impose any specific restrictions on this.

[0091] Step 140: Input the first fundamental frequency data into the second generator to generate predicted fundamental frequency data of the target object, and input the second fundamental frequency data and the predicted fundamental frequency data into the second discriminator for identification processing to obtain a second identification result;

[0092] In this step, the first fundamental frequency data is also input into the second generator, and the fundamental frequency data of the target object is generated by the second generator. The fundamental frequency data of the target object generated by the second generator is recorded as predicted fundamental frequency data.

[0093] Then, in an embodiment of the present application, the generation effect of the second generator can be identified by a second discriminator. Specifically, the purpose of the identification process is to determine the degree of similarity between the fundamental frequency data generated by the second generator and the actual fundamental frequency data of the target user. Therefore, in an embodiment of the present application, when the second discriminator performs the identification process, the second fundamental frequency data and the predicted fundamental frequency data can be input into the second discriminator together, and the second discriminator is allowed to identify and output whether each fundamental frequency data is the fundamental frequency data of the speech data spoken by the real target speaker (i.e., the target object) or the fundamental frequency data generated by the second generator, thereby obtaining a corresponding identification result, which is recorded as the second identification result. For example, in some embodiments, the second identification result can also include two categories: true and false. When a certain fundamental frequency data is input into the second discriminator, the second identification result output is judged to be true, indicating that the second discriminator has determined that the fundamental frequency data is the fundamental frequency data of the speech data output by the real target object; conversely, when a certain fundamental frequency data is input into the second discriminator, the second identification result output is judged to be false, indicating that the second discriminator has determined that the fundamental frequency data is the fundamental frequency data generated by the second generator. Here, the data format corresponding to the true and false second identification results can also be flexibly set according to needs. The specific implementation method is similar to the aforementioned first identification result and will not be repeated here.

[0094] Step 150: Update the parameters of the speech conversion model according to the first identification result and the second identification result.

[0095] In this step, as previously described, the first identification result can reflect the degree of proximity and similarity between the voice data generated by the first generator and the voice data actually output by the first object; the second identification result can reflect the degree of similarity between the fundamental frequency data generated by the second generator and the fundamental frequency data actually output by the target user. Therefore, the first and second identification results can reflect the conversion effect of the voice conversion model as a whole. Therefore, in this embodiment of the application, the parameters of the voice conversion model can be updated based on the first and second identification results to obtain a trained voice conversion model.

[0096] In some embodiments, reference Figure 4 In the embodiment of the present application, step 130 may include the following steps:

[0097] Step 1301: Perform vectorization processing on the first spectrum data to obtain first vector data;

[0098] Step 1302: Input the first vector data and the first fundamental frequency data into the first generator to generate predicted speech data of the first object.

[0099] In an embodiment of the present application, when generating predicted speech data based on the first spectrum data and the first fundamental frequency data, VQ (vector quantization technology) can be used to vectorize the first spectrum data, and the obtained vector data is recorded as the first vector data. Here, vector quantization technology is a data compression technology that can greatly reduce the amount of data processing and help improve the overall data processing efficiency. Then, after obtaining the first vector data, the first vector data and the first fundamental frequency data can be input into the first generator together to obtain the predicted speech data output by the first generator.

[0100] Specifically, in some embodiments, referring to Figure 5 In the embodiment of the present application, step 1301 may include the following steps:

[0101] Step 13011: Encode the first spectrum data to obtain a continuous variable;

[0102] Step 13012: vectorize the continuous variable to obtain a discrete variable;

[0103] Step 13013: Calculate the difference between the continuous variable and the discrete variable to obtain the first vector data.

[0104] In an embodiment of the present application, when the first spectrum data is vectorized, the content information (s) therein can be considered as the difference between the original continuous variable (v) and the quantized discrete variable (c). In this way, the first spectrum data is vectorized into content information in vector form. Starting from the original first spectrum data processing, the first spectrum data can be encoded to obtain v=encoder(x), where x represents the original first spectrum data, encoder() represents the encoding operation, and v represents the obtained continuous variable; then the continuous variable is vectorized to obtain c=quantize(v), where v represents the continuous variable, quantize() represents the vectorization operation, and c represents the obtained discrete variable. The discrete variable c is required to be as close as possible to the continuous variable v. Then, the difference between the continuous variable and the discrete variable can be calculated to obtain the first vector data. In some embodiments, the operation can be repeated multiple times to calculate the average of the difference between the continuous variable and the discrete variable obtained each time, thereby improving the accuracy of the obtained first vector data.

[0105] In some embodiments, reference Figure 6 In the embodiment of the present application, step 150 may include the following steps:

[0106] Step 1501: Determine a first loss value for training based on the first identification result, and update parameters of the first discriminator and the first generator using the first loss value.

[0107] Step 1502: Determine a second loss value for training based on the second identification result, and update the parameters of the second discriminator and the parameters of the second generator using the second loss value.

[0108] In an embodiment of the present application, when updating the parameters of the speech conversion model, it is necessary to update the parameters of the two adversarial generative networks of the speech conversion model respectively according to different identification results. Specifically, since the first identification result itself can reflect the closeness and similarity between the speech data generated by the first generator and the actual speech data output by the first object, the loss value of the training can be determined by the first identification result, recorded as the first loss value, and then the parameters of the first group of adversarial generative networks, that is, the first discriminator and the first generator, are updated by the first loss value. Similarly, the second identification result can reflect the similarity between the fundamental frequency data generated by the second generator and the actual fundamental frequency data of the target user. Therefore, the loss value of the training can be determined by the second identification result, recorded as the second loss value, and then the parameters of the second group of adversarial generative networks, that is, the second discriminator and the second generator, are updated by the second loss value.

[0109] More specifically, taking the parameter update process of the first discriminator and the first generator as an example, refer to Figure 7 In some embodiments, step 1501 may include:

[0110] Step 15011: Detect the current training phase of the speech conversion model;

[0111] Step 15012: When the speech conversion model is in the training phase of the first discriminator, determine the first loss value according to the deviation between the first identification result and the true result, and update the parameters of the first discriminator according to the first loss value; or

[0112] Step 15013: When the speech conversion model is in the training stage of the first generator, the first loss value is determined based on the degree of consistency between the first identification result and the true result, and the parameters of the first generator are updated using the first loss value.

[0113] In the embodiment of the present application, when training the speech conversion model, since it includes a generative adversarial network, which is generally trained in an alternating cycle, taking the generative adversarial network composed of the first discriminator and the first generator as an example, during its training, in some stages, the parameters of the first generator will remain unchanged, and the parameters of the first discriminator will be mainly updated. At this time, the first identification result itself can reflect the identification effect of the first discriminator. The higher the degree of agreement between the first identification result and the true result, the better the identification effect of the first discriminator. The higher the deviation between the first identification result and the true result, the worse the identification effect of the first discriminator. Therefore, at this time, the first loss value is negatively correlated with the degree of agreement between the first identification result and the true result, and positively correlated with the deviation between the first identification result and the true result. For convenience, the first loss value can be directly determined based on the deviation between the first identification result and the true result. The parameters of the first discriminator are then backpropagated and updated using the first loss value.

[0114] In some stages, the parameters of the first discriminator remain unchanged, and the parameters of the first generator are primarily updated. In this case, the better the first discriminator's identification performance, that is, the higher the degree of agreement between the first identification result and the true result, the easier it is to recognize the speech data generated by the first generator, and the worse the generation performance. Therefore, in this stage, the degree of agreement between the first identification result and the true result can be used to determine the first loss value, which is then used to backpropagate and update the parameters of the first generator.

[0115] Therefore, in an embodiment of the present application, the training stage of the current speech conversion model can be detected, and the loss value in the process of training the first generator and the first discriminator can be determined in a corresponding manner, so as to update the parameters.

[0116] It should be noted that in the embodiment of the present application, the number of training stages of the first generator and the first discriminator, as well as the number of iteration rounds of parameters in each training stage, can be flexibly set as needed, and this application does not impose any restrictions on this.

[0117] Similarly, in the embodiment of the present application, when training the second generator and the second discriminator, the training process of the aforementioned first generator and the first discriminator can be referred to, and this application will not elaborate on this.

[0118] It should be noted that for models in the field of machine learning, the accuracy of the results output by the model (such as the identification results in the embodiments of the present application) can be measured by a loss function. The loss function is defined on a single training data and is used to measure the prediction error of a training data. Specifically, the loss value of the training data is determined by the actual result of a single training data and the prediction result of the model for the training data. During actual training, a training data set has a lot of training data, so a cost function is generally used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average value of the prediction error of all training data, which can better measure the prediction effect of the model. For general machine learning models, based on the aforementioned cost function, a regularization term that measures the complexity of the model can be used as the objective function of the training, and the loss value of the entire training data set can be calculated based on the objective function. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions of machine learning models, which will not be elaborated one by one here. In the embodiment of the present application, any loss function can be selected to determine the training loss value, such as the cross entropy loss function. Based on the training loss value, the back propagation algorithm is used to update the model parameters. After several rounds of iteration, a trained speech conversion model can be obtained.

[0119] In addition, please refer to Figure 8 , Figure 8 is a flow chart of a voice conversion method provided in an embodiment of the present application, Figure 8 The voice conversion method shown can be based on Figure 1 The implementation environment shown is executed, for example, can be executed on the server 102. Specifically, the method can be applied to relevant computer devices in the server 102, but is not limited to the above form. Figure 8 The method includes but is not limited to steps 810 to 830.

[0120] Step 810: Acquire third voice data to be converted;

[0121] Step 820: Input the third speech data into a speech conversion model trained by the aforementioned speech conversion model training method; the speech conversion model includes a first generator, a second generator, a first discriminator, and a second discriminator;

[0122] Step 830: Convert the third voice data using the voice conversion model to obtain converted fourth voice data of the target object.

[0123] In the embodiment of the present application, after obtaining a trained speech conversion model, the speech data can be converted using the model to produce speech data with a similar intonation and rhythm to the target subject. Specifically, the acquired speech data is recorded as the third speech data, and the converted speech data is recorded as the fourth speech data.

[0124] The third speech data is input into the speech conversion model, the third speech data is processed, its spectrum data is extracted to obtain content information, and its fundamental frequency data is extracted; then, the extracted fundamental frequency data of the third speech data is input into the second generator, and the converted fundamental frequency data corresponding to the target object is generated by the second generator; then, the converted fundamental frequency data and the extracted spectrum data are input into the first generator together, and the converted fourth speech data corresponding to the target object is generated by the first generator.

[0125] The following describes a training system for a speech conversion model proposed in accordance with an embodiment of the present application with reference to the accompanying drawings.

[0126] Reference Figure 9 The training system of the speech conversion model proposed in the embodiments of the present application includes a first generator, a second generator, a first discriminator, and a second discriminator; the training method of the speech conversion model, and the training system of the speech conversion model include:

[0127] An acquiring unit 910 is configured to acquire a training data set, wherein the training data set includes first speech data of a first subject and second speech data of a target subject;

[0128] an extraction unit 920, configured to extract first spectrum data and first fundamental frequency data from the first speech data, and extract second fundamental frequency data from the second speech data;

[0129] a first processing unit 930 configured to generate predicted speech data of the first object using the first generator based on the first spectrum data and the first fundamental frequency data, and input the first speech data and the predicted speech data into the first discriminator for discrimination processing to obtain a first discrimination result;

[0130] A second processing unit 940 is configured to input the first fundamental frequency data into the second generator to generate predicted fundamental frequency data of the target object, and input the second fundamental frequency data and the predicted fundamental frequency data into the second discriminator for discrimination processing to obtain a second discrimination result;

[0131] The updating unit 950 is configured to update the parameters of the speech conversion model according to the first identification result and the second identification result.

[0132] It can be understood that the contents of the above-mentioned training method embodiments are all applicable to the present training system embodiments, the functions specifically implemented by the present training system embodiments are the same as those of the above-mentioned training method embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned training method embodiments.

[0133] Reference Figure 10 , the embodiment of the present application further discloses a computer device, including:

[0134] at least one processor 1010;

[0135] at least one memory 1020, configured to store at least one program;

[0136] When at least one program is executed by at least one processor 1010, the at least one processor 1010 implements the following Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The embodiment of the speech conversion method is shown.

[0137] It is understandable that if Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The contents of the speech conversion method embodiment shown in FIG. 1 are all applicable to the computer device embodiment. The functions specifically implemented by the computer device embodiment are similar to those in FIG. Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The voice conversion method embodiment shown is the same as that shown in FIG. Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The beneficial effects achieved by the illustrated voice conversion method embodiment are also the same.

[0138] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program is executed by the processor, it is used to implement the following Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The embodiment of the speech conversion method is shown.

[0139] It is understandable that if Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The contents of the speech conversion method embodiment shown in FIG. 1 are applicable to the embodiment of the computer-readable storage medium. The functions specifically implemented by the embodiment of the computer-readable storage medium are similar to those in FIG. Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The voice conversion method embodiment shown is the same as that shown in FIG. Figure 3 The training method embodiment of the speech conversion model shown or Figure 8 The beneficial effects achieved by the illustrated voice conversion method embodiment are also the same.

[0140] In some optional embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, the two boxes shown in succession may actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flow chart of the present application are provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logic flows presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0141] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present application as set forth in the claims using ordinary techniques without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0142] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0143] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0144] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0145] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0146] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0147] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0148] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

[0149] In the description of this specification, reference to the terms "one embodiment," "another embodiment," or "certain embodiments" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0150] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for training a speech conversion model, characterized in that: The speech conversion model includes a first generator, a second generator, a first discriminator, and a second discriminator; the training method of the speech conversion model includes: Acquire a training data set; the training data set includes first speech data of a first subject and second speech data of a target subject; extracting first spectrum data and first fundamental frequency data from the first speech data, and extracting second fundamental frequency data from the second speech data; generating predicted speech data of the first object by the first generator based on the first spectrum data and the first fundamental frequency data, and inputting the first speech data and the predicted speech data into the first discriminator for discrimination processing to obtain a first discrimination result; inputting the first fundamental frequency data into the second generator to generate predicted fundamental frequency data of the target object, and inputting the second fundamental frequency data and the predicted fundamental frequency data into the second discriminator for identification processing to obtain a second identification result; updating parameters of the speech conversion model according to the first identification result and the second identification result; The step of generating the predicted speech data of the first object by the first generator according to the first spectrum data and the first fundamental frequency data includes: performing encoding processing on the first spectrum data to obtain a continuous variable; Performing vectorization processing on the continuous variable to obtain a discrete variable; Calculating a difference between the continuous variable and the discrete variable to obtain first vector data; The first vector data and the first fundamental frequency data are input into the first generator to generate predicted speech data of the first object.

2. The method for training a speech conversion model according to claim 1, wherein: The updating of the parameters of the speech conversion model according to the first identification result and the second identification result includes: Determine a first loss value for training according to the first identification result, and update parameters of the first discriminator and the first generator according to the first loss value; According to the second identification result, a second loss value of training is determined, and the parameters of the second discriminator and the parameters of the second generator are updated according to the second loss value.

3. The method for training a speech conversion model according to claim 2, wherein: The determining a first loss value for training according to the first identification result, and updating parameters of the first discriminator and the first generator according to the first loss value, includes: Detecting the current training stage of the speech conversion model; When the speech conversion model is in the training phase of the first discriminator, determining the first loss value according to the deviation between the first identification result and the true result, and updating the parameters of the first discriminator according to the first loss value; or When the speech conversion model is in the training stage of the first generator, the first loss value is determined according to the degree of consistency between the first identification result and the true result, and the parameters of the first generator are updated according to the first loss value.

4. The method for training a speech conversion model according to claim 2, wherein: The first loss value or the second loss value is determined by a cross entropy loss function.

5. A voice conversion method, characterized in that: include: Acquiring third voice data to be converted; inputting the third speech data into a speech conversion model; The speech conversion model is obtained by training the speech conversion model according to any one of claims 1 to 4; The third voice data is converted using the voice conversion model to obtain converted fourth voice data of the target object.

6. A training system for a speech conversion model, characterized in that: The speech conversion model includes a first generator, a second generator, a first discriminator, and a second discriminator; the training system of the speech conversion model includes: An acquisition unit, configured to acquire a training data set; the training data set includes first speech data of a first subject and second speech data of a target subject; an extraction unit, configured to extract first spectrum data and first fundamental frequency data from the first speech data, and extract second fundamental frequency data from the second speech data; a first processing unit, configured to generate predicted speech data of the first object by the first generator based on the first spectrum data and the first fundamental frequency data, and input the first speech data and the predicted speech data into the first discriminator for discrimination processing to obtain a first discrimination result; a second processing unit, configured to input the first fundamental frequency data into the second generator to generate predicted fundamental frequency data of the target object, and input the second fundamental frequency data and the predicted fundamental frequency data into the second discriminator for discrimination processing to obtain a second discrimination result; an updating unit, configured to update parameters of the speech conversion model according to the first identification result and the second identification result; The step of generating the predicted speech data of the first object by the first generator according to the first spectrum data and the first fundamental frequency data includes: performing encoding processing on the first spectrum data to obtain a continuous variable; Performing vectorization processing on the continuous variable to obtain a discrete variable; Calculating a difference between the continuous variable and the discrete variable to obtain first vector data; The first vector data and the first fundamental frequency data are input into the first generator to generate predicted speech data of the first object.

7. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the training method of the speech conversion model according to any one of claims 1 to 4 or implements the speech conversion method according to claim 5.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to implement the training method of the speech conversion model according to any one of claims 1 to 4 or the speech conversion method according to claim 5 when executed by the processor.

Images