Model training method, speech conversion method and device, equipment and storage medium

Abstract

The application provides a model training method, a speech conversion method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining sample audio data of a sample speaker object; inputting the sample audio data into a neural network model comprising an encoding network and a decoding network; performing reconstruction processing on the sample audio data by using the encoding network to obtain initial audio data; performing speech alignment on the initial audio data to obtain a sample audio embedding vector; performing decoupling processing on the sample audio embedding vector, pre-acquired sample pitch parameters and a sample timbre feature vector by using the decoding network to obtain synthesized audio data; performing loss calculation on the synthesized audio data and sample speech data by using a loss function to obtain a model loss value; and performing parameter updating on the neural network model according to the model loss value to train the neural network model and obtain a speech conversion model. The application can improve the speech conversion effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, speech conversion method and apparatus, device and storage medium. Background Technology

[0002] Speech conversion typically refers to replacing the speaker with another speaker without altering the content of the speech. Common speech conversion models often fail to accurately represent the actual speech content and the speaker's stylistic characteristics, resulting in poor conversion quality. Therefore, improving speech conversion performance has become a pressing technical problem. Summary of the Invention

[0003] The main objective of this application is to propose a model training method, a speech conversion method and apparatus, device and storage medium, which aim to improve the speech conversion effect.

[0004] To achieve the above objectives, a first aspect of this application proposes a model training method, the training method comprising:

[0005] Obtain sample audio data of the speaking object; wherein, the sample audio data includes sample audio content and sample acoustic features, and the sample acoustic features include sample timbre information and sample pitch information;

[0006] The sample audio data is input into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0007] The sample audio data is reconstructed using the encoding network to obtain initial audio data, wherein the initial audio data includes the sample audio content and the sample timbre information, and the initial audio data does not include the sample pitch information;

[0008] The initial audio data is aligned to obtain sample audio embedding vectors;

[0009] The decoding network decouples the sample audio embedding vector, the pre-acquired sample pitch parameters, and the sample timbre feature vector to obtain synthesized audio data, wherein the sample timbre feature vector is used to characterize the speaking style of the sample speaker.

[0010] The model loss value is obtained by calculating the loss of the synthesized audio data and the sample speech data using a preset loss function.

[0011] The parameters of the neural network model are updated based on the model loss value to train the neural network model and obtain a speech conversion model.

[0012] In some embodiments, the process of reconstructing the sample audio data through the coding network to obtain initial audio data includes:

[0013] The initial fundamental frequency parameters, aperiodic parameters, and spectral envelope parameters of the sample audio data are obtained by extracting parameters from the encoded network.

[0014] The target fundamental frequency parameters are obtained by averaging the initial fundamental frequency parameters.

[0015] The initial audio data is obtained by reconstructing the speech using the target fundamental frequency parameters, the aperiodic parameters, and the spectral envelope parameters through the coding network.

[0016] In some embodiments, performing speech alignment on the initial audio data to obtain a sample audio embedding vector includes:

[0017] Phoneme feature recognition is performed on the initial audio data to obtain phoneme feature data, and the duration sequence of the initial audio data is obtained based on the phoneme feature data;

[0018] The initial audio data is speech aligned according to the duration sequence to obtain the sample audio embedding vector.

[0019] In some embodiments, the phoneme feature data includes phoneme category and phoneme number. The step of performing phoneme feature recognition on the initial audio data to obtain phoneme feature data, and obtaining the duration sequence of the initial audio data based on the phoneme feature data, includes:

[0020] The initial audio data is segmented into frames to obtain multiple audio segments;

[0021] The audio segment is identified and processed according to a preset phoneme lookup table to obtain the phoneme category of the initial audio data and the number of phonemes in each phoneme category;

[0022] The duration sequence is obtained based on the phoneme category and the number of phonemes.

[0023] In some embodiments, the step of performing speech alignment on the initial audio data based on the duration sequence to obtain the sample audio embedding vector includes:

[0024] The initial audio data is embedded to obtain an audio text embedding vector;

[0025] The audio text embedding vector is segmented according to the duration sequence to obtain an intermediate embedding vector corresponding to each phoneme category, wherein the number of intermediate embedding vectors is the same as the number of phonemes in the audio category;

[0026] The mean of the intermediate embedding vectors for each phoneme category is calculated to obtain the candidate embedding vectors corresponding to each phoneme category;

[0027] The candidate embedding vectors are copied according to the number of phonemes to obtain the target embedding vectors corresponding to each phoneme category, wherein the number of target embedding vectors is the same as the number of phonemes in the audio category;

[0028] All the target embedding vectors are concatenated to obtain the sample audio embedding vector.

[0029] In some embodiments, before decoupling the sample audio embedding vector, pre-acquired sample pitch parameters, and sample timbre feature vector through the decoding network to obtain synthesized audio data, the training method further acquires the sample timbre feature vector, specifically including:

[0030] The sample audio data is input into a preset voiceprint recognition model, wherein the voiceprint recognition model includes an LSTM layer and a linear layer;

[0031] The LSTM layer is used to extract features from the sample audio data to obtain the sample audio feature latent vector;

[0032] The sample timbre feature vector is obtained by predicting the latent vector of the sample audio features through the linear layer.

[0033] To achieve the above objectives, a second aspect of this application provides a speech conversion method, the method comprising:

[0034] Obtain the raw audio data to be processed

[0035] The original audio data, the pre-acquired target tone features and target timbre features of the target speaker are input into the speech conversion model for speech conversion to obtain the target audio data, wherein the speech conversion model is trained according to the training method described in the first aspect.

[0036] To achieve the above objectives, a third aspect of this application provides a model training apparatus, the training apparatus comprising:

[0037] An audio data acquisition module is used to acquire sample audio data of a sample speaking object; wherein, the sample audio data includes sample audio content and sample acoustic features, and the sample acoustic features include sample timbre information and sample pitch information;

[0038] A data input module is used to input the sample audio data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0039] The reconstruction module is used to reconstruct the sample audio data through the encoding network to obtain initial audio data, wherein the initial audio data includes the sample audio content and the sample timbre information, and the initial audio data does not include the sample pitch information;

[0040] The speech alignment module is used to perform speech alignment on the initial audio data to obtain sample audio embedding vectors;

[0041] The decoupling module is used to decouple the sample audio embedding vector, the pre-acquired sample pitch parameters and the sample timbre feature vector through the decoding network to obtain synthesized audio data, wherein the sample timbre feature vector is used to characterize the speaking style characteristics of the sample speaking object;

[0042] The loss calculation module is used to calculate the loss of the synthesized audio data and the sample speech data using a preset loss function to obtain the model loss value;

[0043] The parameter update module is used to update the parameters of the neural network model based on the model loss value in order to train the neural network model and obtain a speech conversion model.

[0044] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect or the method described in the second aspect.

[0045] To achieve the above objectives, a fifth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0046] The training method, speech conversion method, training device, electronic device, and storage medium proposed in this application acquire sample audio data of sample speaking objects. The sample audio data includes sample audio content and sample acoustic features, with the acoustic features including sample timbre information and sample pitch information. The sample audio data is input into a pre-defined neural network model, which includes an encoding network and a decoding network. The encoding network reconstructs the sample audio data to obtain initial audio data, which includes sample audio content and sample timbre information but excludes sample pitch information. This eliminates pitch information in the sample audio data without altering its audio content and timbre, preventing differences in pitch features between different speaking objects from affecting model training and thus improving training effectiveness. Furthermore, speech alignment is performed on the initial audio data to obtain sample audio embedding vectors. This ensures the audio length of the sample audio embedding vectors matches that of the initial audio data, constraining model training and strengthening feature decoupling of the neural network model from speech features. Furthermore, the sample audio embedding vector, pre-acquired sample pitch parameters, and sample timbre feature vector are decoupled through a decoding network to obtain synthetic audio data. The sample timbre feature vector characterizes the speaking style of the sample speaker. This approach ensures that the synthesized audio data contains speech content, timbre information, and pitch information closely similar to the sample audio data, resulting in better audio quality. Finally, a loss function is used to calculate the loss between the synthesized audio data and the sample speech data to obtain the model loss value. Based on this loss value, the parameters of the neural network model are updated, thus training the neural network model and obtaining a speech conversion model. This effectively improves the model's training performance and enhances its speech conversion accuracy on the input audio data. Attached Figure Description

[0047] Figure 1 This is a flowchart of the training method for the model provided in the embodiments of this application;

[0048] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0050] Figure 4 yes Figure 3 The flowchart of step S301 in the process;

[0051] Figure 5 yes Figure 3The flowchart of step S302 in the text;

[0052] Figure 6 This is another flowchart of the training method for the model provided in the embodiments of this application;

[0053] Figure 7 This is a flowchart of the speech conversion method provided in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram of the structure of the training device for the model provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0062] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0063] Phoneme: The smallest unit of speech based on the natural properties of speech. It is analyzed based on the articulation of a syllable, and one articulation constitutes one phoneme.

[0064] Fundamental frequency (or baseband): In sound, the fundamental frequency refers to the frequency of the fundamental tone in a complex tone. Among the several tones that make up a complex tone, the fundamental tone has the lowest frequency and the highest intensity. The level of the fundamental frequency determines the pitch of a sound. The frequency of speech, as commonly referred to, refers to the frequency of the fundamental tone.

[0065] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0066] Pooling is essentially a sampling process that selects a certain method to perform dimensionality reduction and compression on the input feature map in order to speed up the operation. The pooling process that uses more pooling is called max pooling.

[0067] Activation function: This is a function that runs on the neurons of an artificial neural network and is responsible for mapping the input of the neuron to its output.

[0068] Encoder: Transforms an input sequence into a fixed-length vector.

[0069] Decoding: This involves transforming a previously generated fixed vector into an output sequence; the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0070] Softmax function: The Softmax function is a normalization exponential function that can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), such that each element is in the range (0,1) and the sum of all elements is 1. This function is often used in multi-class classification problems.

[0071] Speech conversion typically refers to replacing the speaker with another speaker without altering the content of the speech. Common speech conversion models often fail to accurately represent the actual speech content and the speaker's stylistic characteristics, resulting in poor conversion quality. Therefore, improving speech conversion performance has become a pressing technical problem.

[0072] Based on this, embodiments of this application provide a model training method, a speech conversion method and apparatus, device and storage medium, aimed at improving speech conversion performance.

[0073] The training method, speech conversion method, apparatus, device, and storage medium of the model provided in this application are specifically described through the following embodiments. First, the training method of the model in the embodiments of this application is described.

[0074] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The model training method provided in this application relates to the field of artificial intelligence technology. The model training method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model training method, but is not limited to the above forms.

[0077] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0078] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0079] Figure 1 This is an optional flowchart of the model training method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0080] Step S101: Obtain sample audio data of the sample speaking object; wherein, the sample audio data includes sample audio content and sample acoustic features, and the sample acoustic features include sample timbre information and sample pitch information;

[0081] Step S102: Input the sample audio data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0082] Step S103: The sample audio data is reconstructed through an encoding network to obtain initial audio data, wherein the initial audio data includes sample audio content and sample timbre information, but does not include sample pitch information.

[0083] Step S104: Perform speech alignment on the initial audio data to obtain the sample audio embedding vector;

[0084] Step S105: The sample audio embedding vector, the pre-acquired sample pitch parameters and the sample timbre feature vector are decoupled by the decoding network to obtain synthesized audio data. The sample timbre feature vector is used to characterize the speaking style of the sample speaker.

[0085] Step S106: Calculate the loss of the synthesized audio data and sample speech data using a preset loss function to obtain the model loss value;

[0086] Step S107: Update the parameters of the neural network model based on the model loss value to train the neural network model and obtain the speech conversion model.

[0087] Steps S101 to S107 of this embodiment involve acquiring sample audio data of the speaker; inputting the sample audio data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; reconstructing the sample audio data through the encoding network to obtain initial audio data, which eliminates pitch information in the sample audio data without changing the audio content and timbre information, avoiding the impact of differences in pitch features of different speaker samples on model training, thereby improving the training effect of the model. Speech alignment is performed on the initial audio data to obtain a sample audio embedding vector, ensuring that the audio length of the sample audio embedding vector is consistent with the initial audio data, achieving feature constraints on model training, thereby strengthening the feature decoupling of the neural network model for speech features. The decoding network decouples the sample audio embedding vector, the pre-acquired sample pitch parameters, and the sample timbre feature vector to obtain synthesized audio data. This method ensures that the synthesized audio data contains speech content, timbre information, and pitch information that are close to the sample audio data, resulting in synthesized audio data with better audio quality. Finally, the loss of the synthesized audio data and sample speech data is calculated using a preset loss function to obtain the model loss value. The parameters of the neural network model are updated based on the model loss value to train the neural network model and obtain the speech conversion model, thereby effectively improving the training effect of the model.

[0088] In step S101 of some embodiments, a web crawler can be written, and after setting the data source, data can be crawled in a targeted manner to obtain sample audio data of the sample speaking object. The data source can be various types of network platforms, social media, or certain specific audio databases, etc. The sample audio data can be the music material, speech report, chat dialogue, etc. of the sample speaking object. The sample audio data includes sample audio content and sample acoustic features. The sample acoustic features include sample timbre information and sample pitch information.

[0089] In step S102 of some embodiments, sample audio data is input into a preset neural network model. The neural network model can be constructed based on the AuTo VC model. The neural network model includes an encoding network and a decoding network. The encoding network is mainly used to perform speech reconstruction and speech alignment on the input audio data to eliminate pitch information in the audio data. It also performs fine-tuning processing on the sample audio data according to the phoneme features of the phoneme data to obtain an audio embedding vector. The decoding network is mainly used to decouple the audio embedding vector from the pitch and timbre features of the target speaker, so as to convert the original speaker into the target speaker without changing the speech content of the input audio data. That is, the audio embedding vector and the pitch and timbre features of the target speaker are fused to form new audio data.

[0090] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S203:

[0091] Step S201: Extract parameters from the sample audio data through the coding network to obtain the initial fundamental frequency parameters, aperiodic parameters, and spectral envelope parameters of the sample audio data;

[0092] Step S202: Calculate the mean of the initial fundamental frequency parameters to obtain the target fundamental frequency parameters;

[0093] Step S203: The target fundamental frequency parameters, aperiodic parameters, and spectral envelope parameters are reconstructed into speech using an encoding network to obtain initial audio data.

[0094] In step S201 of some embodiments, when extracting parameters from sample audio data through the encoding network, commonly used audio parameter analysis tools (such as World Analyzer) can be used to analyze the parameters of the sample audio data to determine various audio parameters of the sample audio data. Then, these audio parameters are filtered through preset keywords in the encoding network to extract the more important parameter information in the sample audio data, such as the aperiodic parameter A, the spectral envelope parameter E, and the initial fundamental frequency parameter F corresponding to each frame of audio. The preset keywords may include the name, label value, etc. of the important parameters.

[0095] In step S202 of some embodiments, all initial fundamental frequency parameters in the sample audio data are summed, and then the sum of the fundamental frequencies is divided by the number of frames of the sample audio data (i.e. the number of initial fundamental frequency parameters) to achieve the mean value processing of the initial fundamental frequency parameters and obtain the target fundamental frequency parameter F0. The target fundamental frequency parameter can reflect the overall frequency situation of the sample audio data more clearly.

[0096] In step S203 of some embodiments, the target fundamental frequency parameter, aperiodic parameter and spectral envelope parameter are fused by an encoding network to obtain fused audio features, thereby reconstructing a new speech waveform based on the fused audio features to obtain initial audio data. The initial audio data includes sample audio content and sample timbre information, but does not include sample pitch information.

[0097] Through the above steps S201 to S203, the pitch information in the sample audio data can be eliminated without changing the audio content and timbre information of the sample audio data. This avoids the impact of differences in the pitch characteristics of different speaking objects on the training of the model, thereby improving the training effect of the model.

[0098] Please see Figure 3 In some embodiments, step S104 may include, but is not limited to, steps S301 to S302:

[0099] Step S301: Perform phoneme feature recognition on the initial audio data to obtain phoneme feature data, and obtain the duration sequence of the initial audio data based on the phoneme feature data;

[0100] Step S302: Perform speech alignment on the initial audio data according to the duration sequence to obtain the sample audio embedding vector.

[0101] In step S301 of some embodiments, when performing phoneme feature recognition on the initial audio data, the initial audio data is first segmented into frames to determine how many frames it includes. Then, the phonemes in each frame are identified to determine the phoneme category corresponding to each frame, thereby statistically determining the types of phonemes (i.e., phoneme categories) and the number of times each phoneme appears (i.e., phoneme count) in the entire initial audio data. The phoneme types and the number of times each phoneme appears are used as phoneme feature data. When constructing the duration sequence of the initial audio data, the phoneme category is used as the number of elements, i.e., the number of elements equals the number of phonemes that appear. The number of times each phoneme appears is used as the element value, i.e., the value of the element is the number of times each phoneme appears. For example, if the entire initial audio data contains 2 phoneme types, i.e., phoneme a and phoneme b, where phoneme a appears 3 times and phoneme b appears 4 times, then the duration sequence of the initial audio data is [3, 4].

[0102] In step S302 of some embodiments, text features are first extracted from the initial audio data to obtain the audio text embedding vector corresponding to the initial audio data. Then, the audio text embedding vector is segmented according to the duration sequence. Then, the segmented audio text embedding vector is merged according to the elements of the duration sequence to obtain the sample audio embedding vector corresponding to the speech length of the initial audio data.

[0103] Through the above steps S301 to S302, the number of elements and element values ​​of the duration sequence can be determined based on the phoneme information of the initial audio data. The text content information of the initial audio data and the audio length of the initial audio data are aligned according to the element information of the duration sequence to obtain a sample audio embedding vector that can represent the text content features of the initial audio data and whose audio length is consistent with the initial audio data.

[0104] Please see Figure 4 In some embodiments, the phoneme feature data includes phoneme category and phoneme number, and step S301 may include, but is not limited to, steps S401 to S403:

[0105] Step S401: Perform frame segmentation on the initial audio data to obtain multiple audio segments;

[0106] Step S402: The audio segment is identified according to the preset phoneme reference table to obtain the phoneme category of the initial audio data and the number of phonemes in each phoneme category.

[0107] Step S403: Obtain the duration sequence based on the phoneme category and the number of phonemes.

[0108] In step S401 of some embodiments, the initial audio data is divided into frames according to the audio length of the initial audio data to obtain multiple audio segments, each audio segment corresponding to a certain frame of audio in the initial audio data.

[0109] For example, if the duration of an initial audio data is 7 seconds, then the number of frames in the initial audio data is 7. Based on the number of frames in the initial audio data, the initial audio data is divided into 7 audio segments, and each audio segment corresponds to a Mel cepstral frame.

[0110] In step S402 of some embodiments, the preset phoneme lookup table includes a mapping relationship between phonemes and Mel cepstral frames. This mapping relationship can be one-to-one or one-to-many. For example, one Mel cepstral frame corresponds to a unique phoneme, or one phoneme corresponds to multiple different Mel cepstral frames. To improve the training effect of the model, in this embodiment, a phoneme lookup table with a one-to-one mapping relationship can be used. After each audio segment is processed by a short-time Fourier transform and filtered by a Mel cepstral filter, the Mel cepstral frame corresponding to each audio segment is obtained. By consulting the preset phoneme lookup table, the phoneme corresponding to the Mel cepstral frame is obtained, and this phoneme is used as the phoneme of the audio segment. In this way, all phonemes corresponding to the initial audio data can be obtained relatively easily, and these phonemes can be classified and counted to determine the phoneme categories contained in the initial audio data and the number of phonemes in each phoneme category.

[0111] For example, if the duration of an initial audio data point is 9 seconds, then the number of frames in the initial audio data is 9. Based on the number of frames, the initial audio data is divided into 9 audio segments, each corresponding to a Mel-Cepstral frame. After each audio segment undergoes a short-time Fourier transform and is filtered by a Mel-Cepstral filter, the Mel-Cepstral frame corresponding to each audio segment is obtained. By consulting a pre-defined phoneme lookup table, the phoneme corresponding to the Mel-Cepstral frame is obtained. These phonemes are then classified and statistically analyzed, revealing that the initial audio data contains two types of phonemes: phoneme a and phoneme b. Phoneme a has 4 phonemes, and phoneme b has 5 phonemes.

[0112] In step S403 of some embodiments, the number of elements in the duration sequence is set according to the number of phoneme categories, and the element values ​​of the duration sequence are set according to the number of phonemes. Specifically, when there are two phoneme categories, namely phoneme a and phoneme b, the number of elements in the duration sequence is two. If the number of phoneme a is 4 and the number of phoneme b is 5, the duration sequence can be represented as [4, 5].

[0113] Through the above steps S401 to S403, the number of elements and element values ​​of the duration sequence can be determined based on the phoneme information of the initial audio data, and the duration of the phonemes in the initial audio data can be converted into a sequence representation. This allows the speech length to be controlled based on the duration of each phoneme in the subsequent speech conversion process, thereby improving the speech conversion effect.

[0114] Please see Figure 5 In some embodiments, step S302 may include, but is not limited to, steps S501 to S505:

[0115] Step S501: Embed the initial audio data to obtain the audio text embedding vector;

[0116] Step S502: The audio text embedding vector is segmented according to the duration sequence to obtain the intermediate embedding vector corresponding to each phoneme category, wherein the number of intermediate embedding vectors is the same as the number of phonemes in the audio category.

[0117] Step S503: Calculate the mean of the intermediate embedding vectors for each phoneme category to obtain the candidate embedding vectors corresponding to each phoneme category.

[0118] Step S504: The candidate embedding vectors are copied according to the number of phonemes to obtain the target embedding vector corresponding to each phoneme category, wherein the number of target embedding vectors is the same as the number of phonemes in the audio category.

[0119] Step S505: Concatenate all target embedding vectors to obtain sample audio embedding vectors.

[0120] In step S501 of some embodiments, the initial audio data is embedded through the embedding layer of the encoding network to achieve vectorization of the initial audio data and obtain an audio text embedding vector, which contains the text content information corresponding to the initial audio data.

[0121] In step S502 of some embodiments, the audio text embedding vector is segmented based on the sum of the element values ​​of the duration sequence to obtain an intermediate embedding vector corresponding to each phoneme category. Specifically, the elements appearing in the duration sequence are summed to obtain the sum of element values. For example, if a certain duration sequence is [4,5], then the sum of element values ​​is 4+5=9. The audio text embedding vector is then segmented into 9 intermediate embedding vectors. Each intermediate embedding vector corresponds to a frame segment of the initial audio data. Since one frame segment corresponds to one phoneme, each intermediate embedding vector also corresponds to one phoneme. That is, the number of intermediate embedding vectors is the same as the number of phonemes in the audio category.

[0122] In step S503 of some embodiments, the average value of the intermediate embedding vectors belonging to the same phoneme category is calculated to obtain the candidate embedding vector corresponding to each phoneme category. Specifically, the vector summation of all intermediate embedding vectors of a certain phoneme category is first performed, and the number of phonemes in that phoneme category is determined. Then, the result of the vector summation is divided by the number of phonemes to obtain the average vector of that phoneme category, and this average vector is used as the candidate embedding vector.

[0123] In step S504 of some embodiments, the candidate embedding vector is copied according to the number of phonemes to obtain the target embedding vector corresponding to each phoneme category. That is, if a certain phoneme category includes k phonemes, then the number of phonemes is k. The candidate embedding vector is copied k times to obtain k target embedding vectors for that phoneme category. Therefore, the number of target embedding vectors is the same as the number of phonemes in the audio category, and the target embedding vector is the same vector as the candidate embedding vector.

[0124] In step S505 of some embodiments, all target embedding vectors belonging to the same initial audio data are concatenated according to a preset concatenation order to obtain the sample audio embedding vector corresponding to the initial audio data.

[0125] For example, if the duration of an initial audio data is 9 seconds, then the number of frames in the initial audio data is 9. Based on the number of frames, the initial audio data is divided into 9 audio segments, each corresponding to a Mel-Cepstral frame. A short-time Fourier transform and Mel-Cepstral filter are performed on each audio segment to obtain the corresponding Mel-Cepstral frame. By consulting a pre-defined phoneme lookup table, the phoneme corresponding to the Mel-Cepstral frame is obtained. These phonemes are classified and statistically analyzed, revealing that the initial audio data contains two phonemes: phoneme a and phoneme b. Phoneme a has 4 phonemes, and phoneme b has 5 phonemes. The duration sequence can then be represented as [4, 5]. Further, the audio text embedding vector is divided into 9 intermediate embedding vectors based on the sum of the element values ​​of the duration sequence. Four intermediate embedding vectors (A1, A2, A3, A4) correspond to phoneme a, and five intermediate embedding vectors (B1, B2, B3, B4, B5) correspond to phoneme b. The average of the four intermediate embedding vectors of phoneme 'a' is calculated to obtain the candidate embedding vector 'An' for phoneme 'a', i.e., (A1+A2+A3+A4) / 4 = An. An is then copied four times to obtain the target embedding vector (An, An, An, An) for phoneme 'a'. Similarly, the average of the five intermediate embedding vectors of phoneme 'b' is calculated to obtain the candidate embedding vector 'An' for phoneme 'b', i.e., (B1+B2+B3+B4+B5) / 5 = Bn. Bn is then copied five times to obtain the target embedding vector (Bn, Bn, Bn, Bn, Bn) for phoneme 'b'. Finally, the target embedding vector (An, An, An, An) for phoneme 'a' and the target embedding vector (Bn, Bn, Bn, Bn, Bn) for phoneme 'b' are concatenated to obtain the sample audio embedding vector corresponding to the initial audio data.

[0126] Through the above steps S501 to S505, the text content information of the initial audio data and the audio length of the initial audio data can be aligned according to the element information of the duration sequence, so as to obtain a sample audio embedding vector that can represent the text content features of the initial audio data and whose audio length is consistent with the initial audio data. This can better constrain the model training, strengthen the feature decoupling of the neural network model on speech features and speaker features, and improve the training effect of the model.

[0127] Please see Figure 6 In some embodiments, prior to step S105, the model training method may include, but is not limited to, steps S601 to S602:

[0128] Step S601: Input the sample audio data into the preset voiceprint recognition model, wherein the voiceprint recognition model includes an LSTM layer and a linear layer;

[0129] Step S602: Extract features from the sample audio data using an LSTM layer to obtain the latent feature vector of the sample audio.

[0130] Step S603: The latent vector of the sample audio features is predicted by a linear layer to obtain the sample timbre feature vector.

[0131] In step S601 of some embodiments, sample audio data is input into a preset voiceprint recognition model through a pre-written computer program or script program. The voiceprint recognition model can be constructed based on a deep convolutional network structure or a long short-term memory network structure. The voiceprint recognition model includes LSTM layers and linear layers. The LSTM layers and linear layers are used to extract the feature expression of a certain speaking object. The feature expression can be used to characterize the speaking style characteristics of the speaking object. The speaking style characteristics include speech rate, pitch, timbre, etc.

[0132] In step S602 of some embodiments, the sample audio data is encoded by an LSTM layer in a left-to-right order to obtain a first audio feature vector. Then, the sample audio data is encoded by an LSTM layer in a right-to-left order to obtain a second audio feature vector. Finally, the first audio feature vector and the second audio feature vector are concatenated to obtain the latent vector of the sample audio features.

[0133] Furthermore, the voiceprint recognition model can include multiple LSTM layers. In each LSTM layer, the sample audio data is encoded from left to right and from right to left. The results of the two encoding processes are then concatenated and input into the next LSTM layer for the same encoding and concatenation operations. The output of the last LSTM layer is used as the final latent vector of the sample audio features.

[0134] In step S603 of some embodiments, the linear layer includes prediction functions such as the softmax function. The probability distribution of the sample audio feature latent vector is calculated by the prediction function of the linear layer to obtain the probability distribution of the sample audio feature latent vector on each preset reference timbre feature label. This probability distribution can clearly reflect the possibility that the sample audio feature latent vector belongs to each reference timbre feature label. Therefore, the reference timbre feature label with the largest probability distribution is selected as the target timbre feature label of the sample audio feature latent vector, and the target timbre feature label is converted into a vector form to obtain the sample timbre feature vector of the sample audio data.

[0135] Through the above steps S601 to S602, the timbre information of the sample audio data can be extracted relatively easily. The timbre feature labels that can be used to represent the timbre information of the sample audio data are selected from multiple reference timbre feature labels, and the timbre feature labels are converted into vector form so that the timbre feature labels can be used for subsequent speech synthesis, thereby improving the effect of speech synthesis.

[0136] In step S105 of some embodiments, firstly, the sample audio embedding vector, sample pitch parameters, and sample timbre feature vector are concatenated by a decoding network to obtain a synthesized audio vector. Then, the synthesized audio vector is decoupled by the decoding network and converted into waveform form to obtain synthesized audio data. The sample timbre feature vector is used to characterize the speaking style of the sample speaker, and the sample pitch parameters include the pitch characteristics of the sample speaker. This method enables the synthesized audio data to contain speech content, timbre information, and pitch information that are relatively close to the sample audio data, so that the obtained synthesized audio data has better audio quality.

[0137] In step S106 of some embodiments, the process of calculating the loss of the synthesized audio data and sample speech data using a preset loss function can be represented as shown in formula (1):

[0138] L recon =||xx′||1 Formula (1)

[0139] Among them, L recon Let L be the model loss value, x be the sample speech data, and x′ be the synthesized audio data. recon The magnitude of the value can clearly reflect the similarity between the sample speech data and the synthesized audio data. Meanwhile, the model loss value L... recon The size of the value can also clearly reflect the training level of the model.

[0140] In step S107 of some embodiments, since the sample speech data originates from the sample speaking object, and the sample pitch parameters and sample timbre feature vectors used to synthesize the audio data also originate from the sample speaking object, the synthesized audio data obtained through the neural network model needs to be as close as possible to the sample speech data, that is, the model loss value needs to be as small as possible. Therefore, the parameters of the neural network model are updated according to the model loss value. By updating the model parameters of the neural network model, the synthesized audio data obtained through the neural network model is made closer to the sample speech data. When, after multiple parameter updates, the model loss value is less than or equal to a preset loss threshold, it indicates that the similarity between the synthesized audio data and the sample speech data is good, and the speech conversion effect of the neural network model can meet the current requirements. Then, training the neural network model is stopped, and the speech conversion model is obtained.

[0141] The training method of the model in this application embodiment acquires sample audio data of sample speaking objects. The sample audio data includes sample audio content and sample acoustic features, with the acoustic features including sample timbre information and sample pitch information. The sample audio data is input into a preset neural network model, which includes an encoding network and a decoding network. The encoding network reconstructs the sample audio data to obtain initial audio data, which includes sample audio content and sample timbre information, but excludes sample pitch information. This eliminates pitch information in the sample audio data without altering its audio content and timbre information, preventing differences in pitch features between different speaking objects from affecting model training and thus improving training effectiveness. Furthermore, speech alignment is performed on the initial audio data to obtain sample audio embedding vectors. This ensures the audio length of the sample audio embedding vectors matches that of the initial audio data, achieving feature constraints for model training and strengthening feature decoupling of the neural network model from speech features. Furthermore, the sample audio embedding vector, pre-acquired sample pitch parameters, and sample timbre feature vector are decoupled through a decoding network to obtain synthetic audio data. The sample timbre feature vector characterizes the speaking style of the sample speaker. This approach ensures that the synthesized audio data contains speech content, timbre information, and pitch information closely similar to the sample audio data, resulting in better audio quality. Finally, a loss function is used to calculate the loss between the synthesized audio data and the sample speech data to obtain the model loss value. Based on this loss value, the parameters of the neural network model are updated, thus training the neural network model and obtaining a speech conversion model. This effectively improves the model's training performance and enhances its speech conversion accuracy on the input audio data.

[0142] Please see Figure 7 This application also provides a speech conversion method, which may include, but is not limited to, steps S701 to S702:

[0143] Step S701: Obtain the raw audio data to be processed;

[0144] Step S702: Input the original audio data, the target pitch features and target timbre features of the target speaker into the speech conversion model to perform speech conversion and obtain the target audio data. The speech conversion model is trained according to the training method in the first aspect.

[0145] In step S701 of some embodiments, a web crawler can be written, and after setting up a data source, data can be crawled in a targeted manner to obtain the raw audio data to be processed. The data source can be various types of online platforms, social media, or certain specific audio databases, etc. The raw audio data can be music material of a speaker, a speech, a chat conversation, etc. Raw audio data can also be obtained through other means, and is not limited to these.

[0146] Furthermore, audio data of the target speaker is obtained from online platforms, social media, or audio databases. The target pitch features and target timbre features of the target speaker are obtained through voiceprint recognition models or other d-vector techniques. The target pitch features can characterize the pitch of the target speaker, and the target timbre features can be used to characterize the speaking style of the target speaker.

[0147] In step S702 of some embodiments, the original audio data, the target pitch features and target timbre features of the target speaker are input into the speech conversion model for speech conversion. The speech conversion model obtains the speech content of the original audio data and removes the pitch features and timbre features of the original audio data. Then, the target pitch features and target timbre features of the target speaker are fused with the speech content of the original audio data to realize the conversion of the pitch information and timbre information of the original audio data and obtain the target audio data.

[0148] The speech conversion method of this application reconstructs the original audio data through the coding network of the speech conversion model to obtain candidate audio data. The candidate audio data does not contain the timbre and pitch information of the original audio data, but only retains the speech content of the original audio data. Then, the candidate audio data is speech aligned, and the speech-aligned candidate audio data is decoupled from the target timbre and target pitch features of the target speaker to form new audio data, namely target audio data. The speech content of the target audio data is the same as that of the original audio data. At the same time, the target audio data contains the timbre and pitch features of the target speaker. This method can transform the original speaker corresponding to the original audio data into the target speaker without changing the speech content information of the original audio data. This method can better represent the speech content information and the timbre and pitch features of the target speaker, and can effectively improve the speech conversion effect.

[0149] Please see Figure 8 This application also provides a model training apparatus that can implement the above-described model training method. The apparatus includes:

[0150] The audio data acquisition module 801 is used to acquire sample audio data of the sample speaking object; wherein, the sample audio data includes sample audio content and sample acoustic features, and the sample acoustic features include sample timbre information and sample pitch information;

[0151] The data input module 802 is used to input sample audio data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0152] The reconstruction module 803 is used to reconstruct the sample audio data through the encoding network to obtain the initial audio data, wherein the initial audio data includes the sample audio content and sample timbre information, and the initial audio data does not include the sample pitch information.

[0153] The speech alignment module 804 is used to perform speech alignment on the initial audio data to obtain the sample audio embedding vector;

[0154] The decoupling module 805 is used to decouple the sample audio embedding vector, the pre-acquired sample pitch parameters and the sample timbre feature vector through the decoding network to obtain synthesized audio data. The sample timbre feature vector is used to characterize the speaking style of the sample speaker.

[0155] The loss calculation module 806 is used to calculate the loss of the synthesized audio data and sample speech data using a preset loss function to obtain the model loss value.

[0156] The parameter update module 807 is used to update the parameters of the neural network model based on the model loss value in order to train the neural network model and obtain the speech conversion model.

[0157] In some embodiments, the reconstruction module 803 includes:

[0158] The parameter extraction unit is used to extract parameters from the sample audio data through the coding network to obtain the initial fundamental frequency parameters, aperiodic parameters, and spectral envelope parameters of the sample audio data.

[0159] The mean calculation unit is used to calculate the mean of the initial fundamental frequency parameters to obtain the target fundamental frequency parameters;

[0160] The speech reconstruction unit is used to reconstruct the speech from the target fundamental frequency parameters, aperiodic parameters, and spectral envelope parameters through a coding network to obtain the initial audio data.

[0161] In some embodiments, the voice alignment module 804 includes:

[0162] The feature recognition unit is used to perform phoneme feature recognition on the initial audio data to obtain phoneme feature data, and to obtain the duration sequence of the initial audio data based on the phoneme feature data.

[0163] Alignment units are used to perform speech alignment on the initial audio data based on the duration sequence to obtain sample audio embedding vectors.

[0164] In some embodiments, the phoneme feature data includes phoneme category and phoneme number, and the feature recognition unit includes:

[0165] The framing subunit is used to process the initial audio data into frames to obtain multiple audio segments;

[0166] The recognition subunit is used to identify audio segments according to a preset phoneme lookup table to obtain the phoneme category of the initial audio data and the number of phonemes in each phoneme category.

[0167] The sequence determination subunit is used to obtain the duration sequence based on the phoneme category and the number of phonemes.

[0168] In some embodiments, the alignment unit includes:

[0169] The embedding subunit is used to embed the initial audio data to obtain the audio text embedding vector;

[0170] The segmentation subunit is used to segment the audio text embedding vector according to the duration sequence to obtain the intermediate embedding vector corresponding to each phoneme category. The number of intermediate embedding vectors is the same as the number of phonemes in the audio category.

[0171] The computation subunit is used to calculate the mean of the intermediate embedding vector for each phoneme category to obtain the candidate embedding vector corresponding to each phoneme category.

[0172] The copying subunit is used to copy the candidate embedding vectors according to the number of phonemes to obtain the target embedding vector corresponding to each phoneme category. The number of target embedding vectors is the same as the number of phonemes in the audio category.

[0173] The splicing subunit is used to splice all target embedding vectors to obtain the sample audio embedding vector.

[0174] In some embodiments, the training apparatus for the model further includes a sample timbre feature acquisition module, specifically including:

[0175] The data input unit is used to input sample audio data into a preset voiceprint recognition model, wherein the voiceprint recognition model includes an LSTM layer and a linear layer;

[0176] The extraction unit is used to extract features from the sample audio data through the LSTM layer to obtain the latent feature vector of the sample audio.

[0177] The prediction unit is used to predict the latent vector of sample audio features through a linear layer to obtain the sample timbre feature vector.

[0178] The specific implementation of the training device for this model is basically the same as the specific implementation of the training method for the model described above, and will not be repeated here.

[0179] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the training method or speech conversion method of the aforementioned model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0180] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0181] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0182] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the training method or speech conversion method of the model in the embodiments of this application.

[0183] The input / output interface 903 is used to implement information input and output;

[0184] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0185] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0186] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0187] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the training method or speech conversion method of the above-described model.

[0188] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0189] The training method, speech conversion method, training device, electronic device, and computer-readable storage medium provided in this application embodiment acquire sample audio data of sample speaking objects. The sample audio data includes sample audio content and sample acoustic features, with the acoustic features including sample timbre information and sample pitch information. The sample audio data is input into a preset neural network model, which includes an encoding network and a decoding network. The encoding network reconstructs the sample audio data to obtain initial audio data, which includes sample audio content and sample timbre information but excludes sample pitch information. This eliminates pitch information in the sample audio data without altering its audio content and timbre, preventing differences in pitch features between different speaking objects from affecting model training and thus improving training effectiveness. Furthermore, speech alignment is performed on the initial audio data to obtain sample audio embedding vectors. This ensures the audio length of the sample audio embedding vectors matches that of the initial audio data, constraining model training and strengthening feature decoupling of the neural network model from speech features. Furthermore, the sample audio embedding vector, pre-acquired sample pitch parameters, and sample timbre feature vector are decoupled through a decoding network to obtain synthetic audio data. The sample timbre feature vector characterizes the speaking style of the sample speaker. This method ensures that the synthesized audio data contains speech content, timbre information, and pitch information that are closely similar to the sample audio data, resulting in better audio quality. Finally, a loss function is used to calculate the loss between the synthesized audio data and the sample speech data to obtain the model loss value. The parameters of the neural network model are updated based on the model loss value to train the neural network model, thus obtaining the speech conversion model and effectively improving the training effect. In the application stage of the speech conversion model, the timbre and pitch features of the original audio data and the target speaker are merged and converted. This transforms the original speaker corresponding to the original audio data into the target speaker without changing the speech content information of the original audio data. This method can better represent the speech content information and the timbre and pitch characteristics of the target speaker, effectively improving the speech conversion effect.

[0190] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0191] It will be understood by those skilled in the art that Figure 1-7The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0194] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0195] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0197] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training a model, characterized in that, The training method includes: Obtain sample audio data of the speaking object; wherein, the sample audio data includes sample audio content and sample acoustic features, and the sample acoustic features include sample timbre information and sample pitch information; The sample audio data is input into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; The sample audio data is reconstructed using the encoding network to obtain initial audio data, wherein the initial audio data includes the sample audio content and the sample timbre information, and the initial audio data does not include the sample pitch information; The initial audio data is aligned to obtain sample audio embedding vectors; The decoding network decouples the sample audio embedding vector, the pre-acquired sample pitch parameters, and the sample timbre feature vector to obtain synthesized audio data, wherein the sample timbre feature vector is used to characterize the speaking style of the sample speaker. The model loss value is obtained by calculating the loss of the synthesized audio data and the sample speech data using a preset loss function. The parameters of the neural network model are updated based on the model loss value to train the neural network model and obtain a speech conversion model.

2. The training method according to claim 1, characterized in that, The process of reconstructing the sample audio data through the coding network to obtain initial audio data includes: The initial fundamental frequency parameters, aperiodic parameters, and spectral envelope parameters of the sample audio data are obtained by extracting parameters from the encoded network. The target fundamental frequency parameters are obtained by averaging the initial fundamental frequency parameters. The initial audio data is obtained by reconstructing the speech using the target fundamental frequency parameters, the aperiodic parameters, and the spectral envelope parameters through the coding network.

3. The training method according to claim 1, characterized in that, The step of performing speech alignment on the initial audio data to obtain a sample audio embedding vector includes: Phoneme feature recognition is performed on the initial audio data to obtain phoneme feature data, and the duration sequence of the initial audio data is obtained based on the phoneme feature data; The initial audio data is speech aligned according to the duration sequence to obtain the sample audio embedding vector.

4. The training method according to claim 3, characterized in that, The phoneme feature data includes phoneme category and phoneme quantity. The process of performing phoneme feature recognition on the initial audio data to obtain phoneme feature data, and obtaining the duration sequence of the initial audio data based on the phoneme feature data, includes: The initial audio data is segmented into frames to obtain multiple audio segments; The audio segment is identified and processed according to a preset phoneme lookup table to obtain the phoneme category of the initial audio data and the number of phonemes in each phoneme category; The duration sequence is obtained based on the phoneme category and the number of phonemes.

5. The training method according to claim 3, characterized in that, The step of performing speech alignment on the initial audio data based on the duration sequence to obtain the sample audio embedding vector includes: The initial audio data is embedded to obtain an audio text embedding vector; The audio text embedding vector is segmented according to the duration sequence to obtain an intermediate embedding vector corresponding to each phoneme category, wherein the number of intermediate embedding vectors is the same as the number of phonemes in the audio category; The mean of the intermediate embedding vectors for each phoneme category is calculated to obtain the candidate embedding vectors corresponding to each phoneme category; The candidate embedding vectors are copied according to the number of phonemes to obtain the target embedding vectors corresponding to each phoneme category, wherein the number of target embedding vectors is the same as the number of phonemes in the audio category; All the target embedding vectors are concatenated to obtain the sample audio embedding vector.

6. The training method according to any one of claims 1 to 5, characterized in that, Before obtaining the synthesized audio data by decoupling the sample audio embedding vector, the pre-acquired sample pitch parameters, and the sample timbre feature vector through the decoding network, the training method further obtains the sample timbre feature vector, specifically including: The sample audio data is input into a preset voiceprint recognition model, wherein the voiceprint recognition model includes an LSTM layer and a linear layer; The LSTM layer is used to extract features from the sample audio data to obtain the sample audio feature latent vector; The sample timbre feature vector is obtained by predicting the latent vector of the sample audio features through the linear layer.

7. A speech conversion method, characterized in that, The method includes: Obtain the raw audio data to be processed The original audio data, the pre-acquired target pitch features and target timbre features of the target speaker are input into the speech conversion model to perform speech conversion, thereby obtaining the target audio data. The speech conversion model is trained according to the training method described in any one of claims 1 to 6.

8. A training device for a model, characterized in that, The training device includes: An audio data acquisition module is used to acquire sample audio data of a sample speaking object; wherein, the sample audio data includes sample audio content and sample acoustic features, and the sample acoustic features include sample timbre information and sample pitch information; A data input module is used to input the sample audio data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; The reconstruction module is used to reconstruct the sample audio data through the encoding network to obtain initial audio data, wherein the initial audio data includes the sample audio content and the sample timbre information, and the initial audio data does not include the sample pitch information; The speech alignment module is used to perform speech alignment on the initial audio data to obtain sample audio embedding vectors; The decoupling module is used to decouple the sample audio embedding vector, the pre-acquired sample pitch parameters and the sample timbre feature vector through the decoding network to obtain synthesized audio data, wherein the sample timbre feature vector is used to characterize the speaking style characteristics of the sample speaking object; The loss calculation module is used to calculate the loss of the synthesized audio data and the sample speech data using a preset loss function to obtain the model loss value; The parameter update module is used to update the parameters of the neural network model based on the model loss value in order to train the neural network model and obtain a speech conversion model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the training method as described in any one of claims 1 to 6 or the speech conversion method as described in claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of any one of claims 1 to 6 or the speech conversion method of claim 7.

Images