Speech conversion method and apparatus based on improved encoder, electronic device

Abstract

Embodiments of the present application provide a speech conversion method and device based on an improved encoder, and an electronic device, belonging to the field of financial technology. The method comprises: obtaining a target mel-frequency spectrum feature of a target speaker corpus to be converted; preprocessing the target mel-frequency spectrum feature based on a first preset neural network to obtain an intermediate feature result; obtaining a Gaussian distribution mean and a Gaussian distribution variance corresponding to the intermediate feature result based on a second preset neural network; determining a feature distribution space according to the Gaussian distribution mean and the Gaussian distribution variance; randomly sampling in the feature distribution space to obtain a target speaker embedding vector; inputting the target speaker embedding vector into a pre-trained speech conversion network to obtain a reconstructed conversion mel-frequency spectrum feature; and generating a target conversion corpus according to the conversion mel-frequency spectrum feature. Embodiments of the present application are beneficial to expanding the modeling capability and application scenarios of the speech conversion model and improving the stability of the speech conversion process.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech conversion method and apparatus, electronic device and storage medium based on an improved encoder. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology. For example, in the insurance industry, it is necessary to analyze the communication content between insurance institutions and customers or the voice content provided by customers in order to distinguish different customers and classify them for management. This involves speech conversion technology, which is a technology that converts source speech into target speech while keeping the speech content unchanged. Currently, mainstream speech conversion methods primarily utilize Generative Adversarial Networks (GANs) and their variants. For example, researchers have developed various speech conversion models for non-parallel data, including CycleGAN-VC, StarGAN-VC, and StarGANv2-VC. During speech conversion, the input speech typically needs to be decoupled to extract corresponding speaker content vectors and speaker style vectors, and subsequent speech conversion is performed based on these speaker vectors. However, the style vectors output by these speech conversion models are relatively singular, and the conversion process requires simultaneous input of the target speaker's domain information. This significantly limits the modeling capabilities and application scenarios of speech conversion models. Taking insurance business as an example, the speech conversion performed using the aforementioned models is relatively fixed, while different customers have different insurance types, such as direct insurance and reinsurance. Therefore, if the stability of speaker vector extraction is not high, it is impossible to reliably obtain the target speech for conversion. Thus, improving the stability of the speech conversion process has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to propose a speech conversion method, apparatus, electronic device, and storage medium based on an improved encoder, aiming to improve the stability of the speech conversion process.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech conversion method based on an improved encoder, the method comprising:

[0005] Obtain the target Mel-spectral features of the target speaker's speech corpus to be converted;

[0006] The target Mel spectrum features are preprocessed based on the first preset neural network to obtain intermediate feature results;

[0007] Based on the second preset neural network, the mean and variance of the Gaussian distribution corresponding to the intermediate feature results are obtained;

[0008] The feature distribution space corresponding to the target speaker's corpus is determined based on the mean and variance of the Gaussian distribution.

[0009] Random sampling is performed in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectrum feature;

[0010] The target speaker embedding vector is input into a pre-trained speech conversion network to obtain the reconstructed converted Mel spectrum features;

[0011] The target transformed corpus is generated based on the transformed Mel spectrum features.

[0012] In some embodiments, the first preset neural network includes a preset convolutional layer and a downsampling module, wherein the downsampling module is obtained by connecting a one-dimensional convolutional layer and a pooling layer; the preprocessing of the target Mel-spectral features based on the first preset neural network to obtain intermediate feature results includes:

[0013] The target Mel-spectral features are input into the first preset neural network;

[0014] Based on the preset convolutional layer, the feature dimension of the target Mel spectrum feature is reduced to a first preset value to obtain the intermediate Mel spectrum feature;

[0015] The downsampling module reduces the time-domain length of the intermediate Mel-frequency spectral features to a second preset value to obtain intermediate feature results.

[0016] In some embodiments, before inputting the target Mel-frequency spectral features into the first preset neural network, the method further includes:

[0017] The time domain of the target Mel spectral features is reordered according to frames.

[0018] In some embodiments, the method further includes:

[0019] During the random sampling process in the feature distribution space, the target KL divergence is optimized, wherein the target KL divergence represents the deviation between the target speaker embedding vector corresponding to the target Mel spectral feature and the Gaussian distribution.

[0020] In some embodiments, the second preset neural network includes a first fully connected layer and a second fully connected layer, and the step of obtaining the Gaussian distribution mean and Gaussian distribution variance corresponding to the intermediate feature results based on the second preset neural network includes:

[0021] The intermediate feature results are input into the second preset neural network;

[0022] The mean of the Gaussian distribution corresponding to the intermediate feature result is predicted based on the first fully connected layer, and the variance of the Gaussian distribution corresponding to the intermediate feature result is predicted based on the second fully connected layer.

[0023] In some embodiments, the step of randomly sampling in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectral features includes:

[0024] Random sampling is performed in the feature distribution space using a reparameterization method to obtain a target speaker embedding vector that uniquely corresponds to the target Mel spectrum feature.

[0025] In some embodiments, obtaining the target Mel spectral features of the target speaker corpus to be converted includes:

[0026] Obtain the speech waveform of the target speaker's corpus;

[0027] The target frequency is obtained by performing a Fourier transform on the speech waveform of the target speaker's corpus.

[0028] The target frequency is converted into the target Mel spectrum feature of the target speaker's corpus based on a preset conversion function.

[0029] To achieve the above objectives, a second aspect of this application provides a speech conversion device based on an improved encoder, wherein the improved encoder is configured with a first preset neural network and a second preset neural network, and the device includes:

[0030] The spectral feature acquisition module is used to acquire the target Mel spectral features of the target speaker's corpus to be converted;

[0031] The preprocessing module is used to preprocess the target Mel spectrum features based on the first preset neural network to obtain intermediate feature results;

[0032] The parameter calculation module is used to obtain the Gaussian distribution mean and Gaussian distribution variance corresponding to the intermediate feature results based on the second preset neural network;

[0033] The distribution space determination module is used to determine the feature distribution space corresponding to the target speaker's corpus based on the mean of the Gaussian distribution and the variance of the Gaussian distribution.

[0034] The sampling module is used to perform random sampling in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectrum feature;

[0035] The first conversion module is used to input the target speaker embedding vector into a pre-trained speech conversion network to obtain the reconstructed converted Mel spectrum features;

[0036] The second conversion module is used to generate the target conversion corpus based on the converted Mel spectrum features.

[0037] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0038] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0039] The speech conversion method, apparatus, electronic device, and storage medium proposed in this application based on an improved encoder input the target Mel-spectral features of the target speaker's speech corpus to be converted into an improved encoder. Then, a first preset neural network and a second preset neural network are configured to determine the feature distribution space corresponding to the target speaker's speech corpus. Random sampling is then performed in the determined feature distribution space to obtain the target speaker's embedding vector. The target speaker's embedding vector is then input into a pre-trained speech conversion network to obtain stably reconstructed converted Mel-spectral features. The target converted speech corpus is stably generated based on the converted Mel-spectral features. Compared with related technologies, it does not require explicit input of the target speaker's specific domain information. At the same time, sampling by fitting the feature distribution space enables the output speech results to have higher adaptability and diversity, which is beneficial to expanding the modeling capability and application scenarios of the speech conversion model, improving the stability of the speech conversion process, and facilitating the stable acquisition of the target speech to be converted. Attached Figure Description

[0040] Figure 1 This is a flowchart of a speech conversion method based on an improved encoder provided in one embodiment of this application;

[0041] Figure 2 yes Figure 1 The flowchart of step S101 in the text;

[0042] Figure 3 yes Figure 1 The flowchart of step S102 in the document;

[0043] Figure 4 yes Figure 1 The flowchart of step S103 in the process;

[0044] Figure 5 yes Figure 1 The flowchart of step S105 in the process;

[0045] Figure 6 yes Figure 1 The flowchart preceding step S102;

[0046] Figure 7 This is a schematic diagram of the structure of an improved encoder provided in one embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the structure of a speech conversion device based on an improved encoder provided in one embodiment of this application;

[0048] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0052] First, let's analyze some of the terms used in this application:

[0053] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0054] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0055] Speech conversion is an important branch of research in speech signal processing, developed and extended from research on speech synthesis and speaker recognition. The task of speech conversion is to change the distinctive features of the source speaker's speech to resemble those of the target speaker, while preserving the source speaker's semantic information. In short, it aims to make the source speaker's speech sound like the target speaker's speech after conversion, while retaining the original semantics.

[0056] Currently, in the insurance industry, insurance business is divided into direct business and reinsurance business based on the insurance stage. Direct business refers to insurance where the insurer and the insured directly enter into a contract. Reinsurance, on the other hand, refers to an insurance business where, after underwriting direct business, an insurance company enters into a contract with one or more other insurance companies to transfer the portion of risk exceeding its own capacity. In this case, the recipient is the insurer, and the party transferring the risk is the insured. In specific scenarios, it is necessary to distinguish the insurance types of different customers, requiring analysis of the communication content between the insurance institution and the customer, or the voice content provided by the customer, to achieve the purpose of classifying and managing different customers. Early speech conversion required training using parallel data. One drawback of this method is its high sensitivity to data alignment. Time alignment usually requires a lot of manpower and time, and in some tasks, non-parallel data is not even available. This limitation has led researchers to turn their attention to other methods. For example, by using GANs and their variants, researchers have developed various speech conversion methods on non-parallel data, including CycleGAN-VC, StarGAN-VC, and StarGANv2-VC.

[0057] However, these methods typically require decoupling the input speech to extract the corresponding speaker content vector and speaker style vector. The ability to extract the speaker style vector largely determines the style similarity between the final output audio and the target speaker. In the StarGANv2-VC model, only a simple style encoder is used to extract the style, and this process requires simultaneous input of speaker domain information and target Mel spectrum, which greatly limits its modeling capabilities and application scenarios.

[0058] Based on this, embodiments of this application provide a speech conversion method and apparatus, electronic device and storage medium based on an improved encoder, aiming to improve the stability of the speech conversion process.

[0059] The speech conversion method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech conversion method in this application is described.

[0060] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0061] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0062] The speech conversion method provided in this application relates to the field of financial technology. The speech conversion method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech conversion method, but is not limited to the above forms.

[0063] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0064] It should be noted that in the various specific embodiments of this application, when processing is required based on user information, user behavior data, user historical data, and user location information, such as obtaining user corpus data, the user's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require obtaining sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0065] Figure 1 This is an optional flowchart of the speech conversion method based on an improved encoder provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0066] Step S101: Obtain the target Mel spectrum features of the target speaker's speech corpus to be converted;

[0067] Step S102: Preprocess the target Mel spectrum features based on the first preset neural network to obtain intermediate feature results;

[0068] Step S103: Obtain the Gaussian distribution mean and Gaussian distribution variance corresponding to the intermediate feature results based on the second preset neural network;

[0069] Step S104: Determine the feature distribution space corresponding to the target speaker's corpus based on the Gaussian distribution mean and Gaussian distribution variance;

[0070] Step S105: Random sampling is performed in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectrum features;

[0071] Step S106: Input the target speaker embedding vector into the pre-trained speech conversion network to obtain the reconstructed converted Mel spectrum features;

[0072] Step S107: Generate the target transformed corpus based on the transformed Mel spectrum features.

[0073] Steps S101 to S106 of this embodiment involve inputting the target Mel-spectral features of the target speaker's speech corpus to be converted into an improved encoder. Then, the first and second preset neural networks are configured to determine the feature distribution space corresponding to the target speaker's speech corpus. Random sampling is performed in the determined feature distribution space to obtain the target speaker's embedding vector. This target speaker embedding vector is then input into a pre-trained speech conversion network to obtain stably reconstructed converted Mel-spectral features. The target converted speech corpus is then stably generated based on these features. Compared to related technologies, this approach eliminates the need for explicit input of the target speaker's specific domain information. Furthermore, sampling by fitting the feature distribution space allows for higher adaptability and diversity in the output speech results, which is beneficial for expanding the modeling capabilities and application scenarios of the speech conversion model, improving the stability of the speech conversion process, and ensuring stable acquisition of the desired target speech.

[0074] For example, a customer can remotely send their needs to an insurance agency. The insurance agency then applies the aforementioned speech-to-text conversion method based on the improved encoder to the customer's speech data, ultimately generating a target conversion corpus. This target conversion corpus allows the agency to directly and clearly determine the type of insurance service the customer requires.

[0075] In some embodiments, the improved encoder can be seen as an improved encoder for the encoder corresponding to StarGANv2-VC, in order to solve the related problems of the encoder corresponding to StarGANv2-VC. However, it should be noted that in different application scenarios, the improved encoder can also be specifically applied to improve the encoders of other related speech conversion models, etc., and is not limited thereto. To avoid redundancy, the following embodiments are mainly described with reference to the improvement of the encoder corresponding to StarGANv2-VC.

[0076] In steps S106 and S107 of some embodiments, the speech conversion network, the pre-training method of the speech conversion network, and the method of generating the target conversion corpus can be selected and set according to the specific scenario. For example, the target conversion corpus can be generated by using ordinary conversion neural networks, Gaussian mixture speech conversion models, etc., and the conversion Mel spectrum features can be reconstructed and extracted by using vector quantization, speaker interpolation, linear multivariate regression, etc., etc., but are not limited here.

[0077] In step S101 of some embodiments, the target Mel-spectral features of the target speaker corpus to be converted can be those currently acquired. They can also be acquired in other ways, such as in advance or at fixed times, and are not limited thereto.

[0078] Please see Figure 2 In some embodiments, step S101 may include, but is not limited to, steps S201 to S203:

[0079] Step S201: Obtain the speech waveform of the target speaker's speech corpus;

[0080] Step S202: Perform Fourier transform on the speech waveform of the target speaker's corpus to obtain the target frequency;

[0081] Step S203: Based on a preset conversion function, the target frequency is converted into the target Mel spectrum features of the target speaker's corpus.

[0082] In this step, the speech waveform of the target speaker's corpus is acquired, and then a Fourier transform is performed on the speech waveform, such as calculating the windowed short-time Fourier transform, so that the target frequency corresponding to the target speaker's corpus can be accurately calculated, so that the target frequency can be converted into the target Mel-spectral features of the target speaker's corpus through a preset transformation function.

[0083] In step S203 of some embodiments, the preset conversion function can be set accordingly in different application scenarios, and is not limited here.

[0084] In step S102 of some embodiments, the first preset neural network may be configured according to the different specific methods of preprocessing the target Mel spectrum features, which is not limited here.

[0085] Please see Figure 3 In some embodiments, when the first preset neural network includes a preset convolutional layer and a downsampling module, and the downsampling module is obtained by connecting a one-dimensional convolutional layer and a pooling layer, step S102 may include, but is not limited to, steps S301 to S303:

[0086] Step S301: Input the target Mel spectrum features into the first preset neural network;

[0087] Step S302: Based on a preset convolutional layer, reduce the feature dimension of the target Mel spectrum feature to a first preset value to obtain the intermediate Mel spectrum feature;

[0088] Step S303: Based on the downsampling module, the time domain length of the intermediate Mel spectrum feature is reduced to a second preset value to obtain the intermediate feature result.

[0089] In this step, when the target Mel spectrum features are input into the first preset neural network, feature dimensionality reduction of the target Mel spectrum features is achieved based on the preset convolutional layer, so that the feature dimension of the target Mel spectrum features is reduced to meet the feature dimension requirements. Furthermore, the temporal length of the intermediate Mel spectrum features is reduced based on the downsampling module, so as to obtain stable and reliable intermediate feature results.

[0090] In steps S302 and S303 of some embodiments, the preset convolutional layer may be, but is not limited to, a one-dimensional convolutional layer, and the pooling layer included in the downsampling module may be a max pooling layer or may be used to implement average pooling, which is not limited here; the first preset value and the second preset value may be set accordingly in different application scenarios, and are not limited thereto.

[0091] In step S103 of some embodiments, the second preset neural network can also be set according to the different specific methods of obtaining the mean and variance corresponding to the intermediate feature results. For example, if the determined feature distribution space is not a Gaussian distribution space, then the obtained mean and variance can be the values ​​under the corresponding distribution space, which is not limited here.

[0092] Please see Figure 4 In some embodiments, when the second preset neural network includes a first fully connected layer and a second fully connected layer, step S103 may include, but is not limited to, steps S401 to S402:

[0093] Step S401: Input the intermediate feature results into the second preset neural network;

[0094] Step S402: Based on the prediction of the first fully connected layer, the mean of the Gaussian distribution corresponding to the intermediate feature results is obtained, and based on the prediction of the second fully connected layer, the variance of the Gaussian distribution corresponding to the intermediate feature results is obtained.

[0095] In this step, the distribution space of the target speaker is fitted by the first fully connected layer and the second fully connected layer. That is, the mean and variance are solved to approximate the latent space model under the premise that the prior is a Gaussian distribution, so that the target speaker embedding vector can be accurately and reliably obtained by sampling based on the feature latent space distribution in subsequent steps.

[0096] In step S402 of some embodiments, the first fully connected layer and the second fully connected layer can be configured accordingly in different application scenarios to achieve separate parameter prediction functions, without making a specific distinction.

[0097] In step S104 of some embodiments, there are various ways to determine the feature distribution space corresponding to the target speaker's corpus based on the Gaussian distribution mean and Gaussian distribution variance, which are not limited here. For example, the feature distribution space of the target speaker's corpus can be obtained by calculating the Gaussian distribution mean and Gaussian distribution variance based on a preset space fitting function.

[0098] In step S105 of some embodiments, random sampling is performed in a predetermined feature distribution space to transform the random Gaussian noise in the target speaker corpus to obtain the final output target speaker embedding vector.

[0099] Please see Figure 5 In some embodiments, step S105 may include, but is not limited to, step S501:

[0100] Step S501: Random sampling is performed in the feature distribution space using a reparameterization method to obtain a target speaker embedding vector that uniquely corresponds to the target Mel spectrum feature.

[0101] In this step, considering that random sampling may result in multiple target speaker embedding vectors, in order to avoid the non-differentiability of the sampling process, a reparameterization method can be used to perform random sampling in the feature distribution space, thereby obtaining a target speaker embedding vector that uniquely corresponds to the target Mel spectrum feature. In other words, a target speaker embedding vector corresponding to an accuracy coefficient can be obtained, which helps to ensure the stability of the sampling process, reduce the amount of computation and parameters, and improve the extraction performance and application speed of the improved encoder.

[0102] In step S501 of some embodiments, reparameterization can be, but is not limited to, parallel merging, serial merging, etc., that is, the merging module is convolution, and the specific synthesis method can be set according to the specific application scenario.

[0103] In some embodiments, the speech conversion method based on the improved encoder of this application may also include, but is not limited to, step S601:

[0104] In step S601, during the random sampling process in the feature distribution space, the target KL divergence is optimized, whereby the target KL divergence represents the deviation between the target speaker embedding vector corresponding to the target Mel spectrum feature and the Gaussian distribution.

[0105] In this step, by introducing the target KL divergence, the modeling capability of the encoder can be further improved and random sampling of the latent space can be achieved. That is, since the target KL divergence represents the deviation between the target speaker embedding vector corresponding to the target Mel spectral feature and the standard distribution, the sampling results can be optimized by optimizing the target KL divergence, thereby further improving the extraction performance of the improved encoder.

[0106] In step S601 of some embodiments, the specific form of the Gaussian distribution can be various. The application method of the Gaussian distribution can be selected accordingly in different situations. For example, when the mean is zero and the variance is taken as unit variance, it is specifically in the form of a standard distribution, which is not limited here.

[0107] In step S601 of some embodiments, in order to randomly sample from the speaker space and optimize the deviation between the output distribution p(z|x) and the zero-mean, unit-variance Gaussian distribution, an additional objective KL divergence loss function is added to StarGANv2-VC, defined as follows:

[0108]

[0109] in, Let represent the target KL divergence, and p(z|x) represent the output distribution of the improved encoder. The maximum log-likelihood of p(x) is indirectly obtained by optimizing the target KL divergence. Assuming the encoder output of the improved encoder is c, the output x can be written as: x = G(z,c). Assuming q(c,z|x) = q(c|x)q(z|x), and since the output c is a deterministic variable, q(c|x) = 1, we can obtain the following formula:

[0110]

[0111] This shows that optimizing the target KL divergence can maximize the optimization of the output results.

[0112] Please see Figure 6 In some embodiments, step S102 may include, but is not limited to, step S701:

[0113] Step S701: Reorder the time domain of the target Mel spectrum features according to frames.

[0114] In step S701 of some embodiments, the time-domain data of the target Mel-frequency spectral features are reordered according to frames, that is, after the target Mel-frequency spectral features are shuffled in the time domain, they are input into the first preset neural network. Since the target speaker embedding vector is not significantly related to the speech content, the order of speech, etc., the output target speaker embedding vector is not sensitive to the time-domain shuffling operation of the target Mel-frequency spectral features. That is, the distribution of the output before and after the time-domain shuffling needs to be as similar as possible. In this case, it can also be achieved by optimizing another target KL divergence to further improve the robustness of the improved encoder. Specifically, the target KL divergence in this case can be, but is not limited to:

[0115]

[0116] This shows that optimizing the target KL divergence under this condition can maximize the resolution of potential temporal domain disruptions.

[0117] To better illustrate the working principle and content of the above embodiments, a specific example is given below.

[0118] Example 1:

[0119] Please see Figure 7 This diagram illustrates a structural schematic of an improved encoder according to an embodiment of this application, used to implement the speech conversion methods in the above embodiments. Specifically:

[0120] In the improved encoder, instead of directly outputting the speaker embedding vector, a neural network is used to fit the speaker's distribution space. The mean and variance vectors are solved to approximate the latent space model under the assumption that the prior is a Gaussian distribution. The embedding vector for a specific speaker is obtained by sampling in the feature latent space distribution.

[0121] like Figure 7 As shown, the input is a Mel spectrum feature, assuming its dimension is 80xM, where 80 represents the number of Mel spectrum filter banks and M represents the number of frames. First, a one-dimensional convolution layer is used to reduce the feature dimension to 32. Then, an N-layer downsampling module is used to downsample the input from the previous step to reduce the temporal length. The downsampling module can be obtained by one-dimensional convolution followed by max pooling. The one-dimensional convolution is used for feature extraction and dimensionality reduction, and average pooling is used to reduce the temporal length (i.e., downsampling). Then, an average pooling layer is used to reduce the temporal dimension to 1. Finally, two different fully connected layers (i.e., fully connected layer one and fully connected layer two) are used to predict the mean μ and variance σ of the Gaussian distribution. At the same time, in order to avoid the non-differentiability of the sampling process, the case of multidimensional Gaussian distribution is considered. The reparameter technique is used to transform the random Gaussian noise ε to obtain the final output embedding vector z.

[0122] In order to randomly sample from the speaker space and optimize the deviation between the output distribution p(z|x) and the zero-mean, unit-variance Gaussian distribution, an additional objective KL divergence loss function is added to StarGANv2-VC, defined as follows:

[0123]

[0124] in, Let represent the target KL divergence, and p(z|x) represent the output distribution of the improved encoder. The maximum log-likelihood of p(x) is indirectly obtained by optimizing the target KL divergence. Assuming the encoder output of the improved encoder is c, the output x can be written as: x = G(z,c). Assuming q(c,z|x) = q(c|x)q(z|x), and since the output c is a deterministic variable, q(c|x) = 1, we can obtain the following formula:

[0125]

[0126] This shows that optimizing the target KL divergence can maximize the optimization of the output results.

[0127] Since the target speaker embedding vector is not significantly correlated with the speech content or the order of speech, the output target speaker embedding vector is insensitive to the temporal scrambling operation of the target Mel-spectral features. Therefore, the distribution of the output before and after temporal scrambling needs to be as similar as possible. In this case, it can also be achieved by optimizing another target KL divergence to further improve the robustness of the improved encoder. Specifically, the target KL divergence in this case can be, but is not limited to, as follows:

[0128]

[0129] This shows that optimizing the target KL divergence under this condition can maximize the resolution of potential temporal domain disruptions.

[0130] In summary, the improved encoder in this application replaces the style encoder in the original StarGANv2-VC model to perform speech conversion. It first convolves and downsamples the input Mel spectrum to extract speaker-related features. Then, it uses a fully connected layer to calculate the mean and variance vectors to approximate the distribution of the latent variable space. Finally, it obtains a specific speaker embedding vector through sampling. Compared to the original style extraction method, this application does not require explicit input of specific domain information, and the sampling process allows for greater diversity in the output synthesized audio. Furthermore, this application provides two KL divergence objective functions. Optimizing the KL divergence of p(x|z) and the N(0,1) Gaussian distribution indirectly optimizes log p(x). Optimizing the KL divergence between the distributions defined by the output mean and variance after two shuffles improves the encoder's stability, thereby enhancing its extraction performance and improving the feature similarity of the entire speech conversion model.

[0131] Please see Figure 8 This application also provides a speech conversion device based on an improved encoder, which can implement the above-mentioned speech conversion method. The improved encoder is configured with a first preset neural network and a second preset neural network. The device includes:

[0132] The spectral feature acquisition module is used to acquire the target Mel spectral features of the target speaker's corpus to be converted;

[0133] The preprocessing module is used to preprocess the target Mel spectrum features based on the first preset neural network to obtain intermediate feature results;

[0134] The parameter calculation module is used to obtain the Gaussian distribution mean and Gaussian distribution variance corresponding to the intermediate feature results based on the second preset neural network.

[0135] The distribution space determination module is used to determine the feature distribution space corresponding to the target speaker's corpus based on the Gaussian distribution mean and Gaussian distribution variance;

[0136] The sampling module is used to perform random sampling in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectrum features;

[0137] The first conversion module is used to input the target speaker embedding vector into a pre-trained speech conversion network to obtain the reconstructed converted Mel spectrum features;

[0138] The second conversion module is used to generate the target conversion corpus based on the converted Mel spectrum features.

[0139] The specific implementation of this voice conversion device is basically the same as the specific embodiment of the above-mentioned voice conversion method, and belongs to the same inventive concept, so it will not be described again here.

[0140] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech conversion method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0141] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0142] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0143] The memory 902 can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech conversion method of the embodiments of this application.

[0144] The input / output interface 903 is used to implement information input and output;

[0145] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0146] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0147] The processor 901, memory 902, input / output interface 903, and communication interface 904 communicate with each other within the device via bus 905.

[0148] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech conversion method.

[0149] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state memory device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0150] The speech conversion method, apparatus, electronic device, and storage medium based on an improved encoder provided in this application embodiment input the target Mel-spectral features of the target speaker's speech corpus to be converted into an improved encoder. Then, a first preset neural network and a second preset neural network are configured to determine the feature distribution space corresponding to the target speaker's speech corpus. Random sampling is then performed in the determined feature distribution space to finally obtain the target speaker embedding vector. The target speaker embedding vector is then input into a pre-trained speech conversion network to obtain stably reconstructed converted Mel-spectral features. The target converted speech corpus is stably generated based on the converted Mel-spectral features. Compared with related technologies, it does not require explicit input of the target speaker's specific domain information. At the same time, sampling by fitting the feature distribution space can make the output speech results more adaptable and diverse, which is beneficial to expanding the modeling capability and application scenarios of the speech conversion model, improving the stability of the speech conversion process, and facilitating the stable acquisition of the target speech to be converted.

[0151] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0152] The foregoing has described specific embodiments of this application; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0153] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0154] The apparatus, device, computer-readable storage medium and method provided in the embodiments of this application are corresponding. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be described again here.

[0155] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules.

[0156] For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system onto a PLD themselves, eliminating the need for chip manufacturers to design and fabricate dedicated integrated circuit chips. Furthermore, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented using "logic compiler" software, similar to the software compiler used in program development. The source code before compilation must be written in a specific programming language called a Hardware Description Language (HDL). There is not just one type of HDL, but many, such as:

[0157] ABEL (Advanced Boolean Expression Language); AHDL (Altera Hardware Description Language); Confluence; CUPL (Cornell University Programming Language); HDCal; and JHDL (Java Hardware Description Language); Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, among the technologies in this field, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog are more commonly used. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using the aforementioned hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0158] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers:

[0159] The memory controller, including the ARC 625D, Atmel AT91SAM, Microchip IP address PIC18F26K20, and Silicon Labs C8051F320, can also be implemented as part of the memory's control logic. Those skilled in the art will also recognize that, in addition to implementing the controller as purely computer-readable program code, the same functionality can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0160] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0161] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing the embodiments of this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0162] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0168] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0169] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0170] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0171] The embodiments of this application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0172] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0173] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A speech conversion method based on an improved encoder, characterized in that, The improved encoder is configured with a first preset neural network and a second preset neural network, and the method includes: Obtain the target Mel-spectral features of the target speaker's speech corpus to be converted; The target Mel spectrum features are preprocessed based on the first preset neural network to obtain intermediate feature results; Based on the second preset neural network, the mean and variance of the Gaussian distribution corresponding to the intermediate feature results are obtained; The feature distribution space corresponding to the target speaker's corpus is determined based on the mean and variance of the Gaussian distribution. Random sampling is performed in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectrum feature; The target speaker embedding vector is input into a pre-trained speech conversion network to obtain the reconstructed converted Mel spectrum features; Generate the target transformed corpus based on the transformed Mel spectrum features; The first preset neural network includes a preset convolutional layer and a downsampling module, wherein the downsampling module is obtained by connecting a one-dimensional convolutional layer and a pooling layer; the preprocessing of the target Mel-frequency spectral features based on the first preset neural network to obtain intermediate feature results includes: The target Mel-spectral features are input into the first preset neural network; Based on the preset convolutional layer, the feature dimension of the target Mel spectrum feature is reduced to a first preset value to obtain the intermediate Mel spectrum feature; The downsampling module reduces the time-domain length of the intermediate Mel-frequency spectral features to a second preset value to obtain intermediate feature results.

2. The speech conversion method based on the improved encoder according to claim 1, characterized in that, Before inputting the target Mel-spectral features into the first preset neural network, the method further includes: The time domain of the target Mel spectral features is reordered according to frames.

3. The speech conversion method based on an improved encoder according to claim 1 or 2, characterized in that, The method further includes: During the random sampling process in the feature distribution space, the target KL divergence is optimized, wherein the target KL divergence represents the deviation between the target speaker embedding vector corresponding to the target Mel spectral feature and the Gaussian distribution.

4. The speech conversion method based on the improved encoder according to claim 1, characterized in that, The second preset neural network includes a first fully connected layer and a second fully connected layer. The step of obtaining the Gaussian distribution mean and Gaussian distribution variance corresponding to the intermediate feature results based on the second preset neural network includes: The intermediate feature results are input into the second preset neural network; The mean of the Gaussian distribution corresponding to the intermediate feature result is predicted based on the first fully connected layer, and the variance of the Gaussian distribution corresponding to the intermediate feature result is predicted based on the second fully connected layer.

5. The speech conversion method based on the improved encoder according to claim 1, characterized in that, The step of randomly sampling in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectral features includes: Random sampling is performed in the feature distribution space using a reparameterization method to obtain a target speaker embedding vector that uniquely corresponds to the target Mel spectrum feature.

6. The speech conversion method based on the improved encoder according to claim 1, characterized in that, The acquisition of the target Mel-spectral features of the target speaker's speech corpus to be converted includes: Obtain the speech waveform of the target speaker's corpus; The target frequency is obtained by performing a Fourier transform on the speech waveform of the target speaker's corpus. The target frequency is converted into the target Mel spectrum feature of the target speaker's corpus based on a preset conversion function.

7. A speech conversion device based on an improved encoder, characterized in that, The improved encoder is configured with a first preset neural network and a second preset neural network, and the device includes: The spectral feature acquisition module is used to acquire the target Mel spectral features of the target speaker's corpus to be converted; The preprocessing module is used to preprocess the target Mel spectrum features based on the first preset neural network to obtain intermediate feature results; The parameter calculation module is used to obtain the Gaussian distribution mean and Gaussian distribution variance corresponding to the intermediate feature results based on the second preset neural network; The distribution space determination module is used to determine the feature distribution space corresponding to the target speaker's corpus based on the mean of the Gaussian distribution and the variance of the Gaussian distribution. The sampling module is used to perform random sampling in the feature distribution space to obtain the target speaker embedding vector corresponding to the target Mel spectrum feature; The first conversion module is used to input the target speaker embedding vector into a pre-trained speech conversion network to obtain the reconstructed converted Mel spectrum features; The second conversion module is used to generate the target conversion corpus based on the converted Mel spectrum features; The first preset neural network includes a preset convolutional layer and a downsampling module, wherein the downsampling module is obtained by connecting a one-dimensional convolutional layer and a pooling layer; the preprocessing of the target Mel-frequency spectral features based on the first preset neural network to obtain intermediate feature results includes: The target Mel-spectral features are input into the first preset neural network; Based on the preset convolutional layer, the feature dimension of the target Mel spectrum feature is reduced to a first preset value to obtain the intermediate Mel spectrum feature; The downsampling module reduces the time-domain length of the intermediate Mel-frequency spectral features to a second preset value to obtain intermediate feature results.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech conversion method based on the improved encoder as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech conversion method based on the improved encoder as described in any one of claims 1 to 6.

Images