Data processing method, audio generation method and speech synthesis model training method

Abstract

The invention provides a data processing method, an audio generation method, a speech synthesis model training method, devices, equipment, a medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models, audio processing and the like. The specific implementation scheme of the data processing method comprises the following steps: in response to a received reference audio, extracting features of the reference audio to obtain reference audio features; retrieving an audio feature most similar to the reference audio feature from a predetermined sample feature library as a target audio feature; audio feature-text feature pairs included in the training data of the speech synthesis model are stored in the predetermined sample feature library; in response to the condition that the similarity between the target audio feature and the reference audio feature is greater than or equal to a similarity threshold value, obtaining a target text feature associated with the target audio feature; and taking the preset identification information as an index, and storing the target audio features and the target text features in a specified feature library in an associated manner.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, large models and audio processing, and can be applied to film and television, games, intelligent voice assistants, virtual digital humans and various scenarios that require personalized audio. Background Technology

[0002] With the development of computer and network technologies, deep learning models are being applied more and more widely, and have made groundbreaking progress in various fields. Among them, text-to-speech (TTS) technology is an important direction of deep learning.

[0003] Speech synthesis technology aims to replicate the voice features of a specified object based on a deep learning model, generating a virtual voice that is highly similar to the voice of that specified object. Summary of the Invention

[0004] This disclosure aims to provide a data processing method, audio generation method, and training method, apparatus, device, medium, and program product that can improve timbre similarity and reduce timbre loss.

[0005] According to a first aspect of this disclosure, a data processing method is provided, comprising: in response to receiving reference audio, extracting features of the reference audio to obtain reference audio features; retrieving from a predetermined sample feature library the audio feature most similar to the reference audio features as a target audio feature; the predetermined sample feature library stores audio feature-text feature pairs included in the training data of a speech synthesis model, wherein the text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature; in response to the similarity between the target audio feature and the reference audio feature being greater than or equal to a similarity threshold, obtaining a target text feature associated with the target audio feature; and storing the target audio feature and the target text feature in association in a specified feature library using predetermined identification information as an index; the predetermined identification information uniquely indicates the recipient of the reference audio.

[0006] According to a second aspect of this disclosure, an audio generation method is provided, comprising: in response to receiving reference audio and target text, querying a specified feature library based on identification information associated with the reference audio; the specified feature library stores audio features and text features indexed by the identification information, wherein the text corresponding to the text features records the text content expressed by the audio corresponding to the audio features; the identification information uniquely indicates the object providing the reference audio; in response to finding a feature associated with the reference audio, using the found feature as a reference feature; extracting text features from the target text to obtain target text features; and generating target audio corresponding to the target text based on the reference features and the target text features, wherein the data in the specified feature library is stored using the data processing method provided in the first aspect of this disclosure.

[0007] According to a third aspect of this disclosure, a method for training a speech synthesis model is provided, comprising: periodically updating a predetermined sample feature library based on data stored in an updated sample feature library; the updated sample feature library storing audio feature-text feature pairs obtained based on reference audio received online; the predetermined sample feature library storing audio feature-text feature pairs included in the training data of the speech synthesis model; the text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature; and training the speech synthesis model based on the updated feature library to obtain a trained speech synthesis model, wherein the data stored in the updated sample feature library is obtained based on the reference audio received in the data processing method provided in the first aspect of this disclosure.

[0008] According to a fourth aspect of this disclosure, a data processing apparatus is provided, comprising: an audio feature extraction module, configured to extract features of the reference audio in response to receiving reference audio, thereby obtaining reference audio features; a feature retrieval module, configured to retrieve, from a predetermined sample feature library, an audio feature most similar to the reference audio features, as a target audio feature; the predetermined sample feature library stores audio feature-text feature pairs included in the training data of a speech synthesis model, wherein the text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature; a text feature acquisition module, configured to acquire a target text feature associated with the target audio feature in response to a similarity between the target audio feature and the reference audio feature being greater than or equal to a similarity threshold; and a first feature storage module, configured to store the target audio feature and the target text feature in association to a specified feature library using predetermined identification information as an index; the predetermined identification information uniquely indicates the recipient of the reference audio.

[0009] According to a fifth aspect of this disclosure, an audio generation apparatus is provided, comprising: a feature query module, configured to query a specified feature library based on identification information associated with the reference audio in response to receiving reference audio and target text; the specified feature library stores audio features and text features indexed by the identification information, wherein the text corresponding to the text features records the text content expressed by the audio corresponding to the audio features; the identification information uniquely indicates the recipient of the reference audio; a feature determination module, configured to use the queried features as reference features in response to finding features associated with the reference audio; a target feature extraction module, configured to extract text features of the target text to obtain target text features; and an audio generation module, configured to generate target audio corresponding to the target text based on the reference features and the target text features, wherein the data in the specified feature library is stored using a data processing apparatus provided in a fourth aspect of this disclosure.

[0010] According to a sixth aspect of this disclosure, a training apparatus for a speech synthesis model is provided, comprising: a feature library update module for periodically updating a predetermined sample feature library based on data stored in an updated sample feature library; the updated sample feature library stores audio feature-text feature pairs obtained based on reference audio received online; the predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model; the text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature; and a model training module for training the speech synthesis model based on the updated feature library to obtain a trained speech synthesis model, wherein the data stored in the updated sample feature library is obtained based on reference audio received by the data processing apparatus provided in the fourth aspect of this disclosure.

[0011] According to a seventh aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform at least one of the following methods provided in this disclosure: a data processing method, an audio generation method, and a training method for a speech synthesis model.

[0012] According to an eighth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform at least one of the following methods provided in this disclosure: a data processing method, an audio generation method, and a training method for a speech synthesis model.

[0013] According to a ninth aspect of this disclosure, a computer program product is provided, comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement at least one of the following methods provided in this disclosure: a data processing method, an audio generation method, and a training method for a speech synthesis model.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0016] Figure 1 This is a schematic diagram illustrating application scenarios of the data processing method, audio generation method, and speech synthesis model training method and apparatus according to embodiments of this disclosure.

[0017] Figure 2 This is a schematic flowchart of a data processing method according to an embodiment of the present disclosure;

[0018] Figure 3 This is a schematic diagram illustrating the implementation principle of a data processing method according to an embodiment of the present disclosure;

[0019] Figure 4 This is a schematic flowchart of an audio generation method according to an embodiment of the present disclosure;

[0020] Figure 5 This is a schematic diagram illustrating the principle of generating target audio according to an embodiment of this disclosure;

[0021] Figure 6 This is a flowchart illustrating a training method for a speech synthesis model according to an embodiment of the present disclosure;

[0022] Figure 7 This is a structural block diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0023] Figure 8 This is a structural block diagram of an audio generation apparatus according to an embodiment of the present disclosure;

[0024] Figure 9 This is a structural block diagram of a training apparatus for a speech synthesis model according to embodiments of the present disclosure; and

[0025] Figure 10 This is a block diagram of an electronic device used to implement at least one of the data processing method, audio generation method, and speech synthesis model training method of the embodiments of this disclosure. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] Voice replication, also known as voice cloning, is a personalized application of speech synthesis technology. It aims to replicate the vocal characteristics of a specified object using a deep learning model, generating a virtual voice highly similar to that object. This technology can be widely applied in various scenarios requiring personalized voices, such as film and games, generating voices with the vocal characteristics of specific characters to enhance audience immersion. Furthermore, this technology can also be applied to intelligent voice assistants and virtual digital humans to provide more personalized human-computer interaction and improve user experience.

[0028] Deep learning models used for timbre replication typically require users to record long audio clips, then use Automatic Speech Recognition (ASR) technology to annotate the recorded audio with corresponding text. The audio and text are then uploaded to the cloud for manual or semi-automatic model training and deployment. This technique often fails to mimic the target speaker's speech habits, such as pauses and style, and struggles to fully simulate the subtle differences between different timbres. Furthermore, training high-quality models usually requires a large amount of high-quality audio, and the existence of rare or hard-to-obtain timbres significantly limits the quality of the trained model.

[0029] To address the problems existing in related technologies, this disclosure provides a data processing method, an audio generation method, and a training method, apparatus, device, medium, and program product for a speech synthesis model. The following will first combine... Figure 1 The application scenarios of the methods and apparatus provided in this disclosure are described.

[0030] Figure 1 This is a schematic diagram illustrating the application scenarios of the data processing method, audio generation method, and speech synthesis model training method and apparatus according to embodiments of this disclosure.

[0031] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110. The electronic device 110 may be a variety of electronic devices with processing and interaction functions, including but not limited to smartphones, tablets, portable computers, desktop computers and smart wearable devices.

[0032] For example, electronic device 110 may have various client applications or applets installed or loaded to provide AIGC functionality. For instance, electronic device 110 may process target text 101 and audio segment 102 based on the installed client applications or loaded applets to generate target audio segment 103 corresponding to target text 101. The text content expressed by target audio segment 103 is consistent with the text content in target text 101, and the timbre and other audio features of target audio segment 103 are highly similar to the timbre and other audio features of audio segment 102.

[0033] In one embodiment, the electronic device 110 may use a speech synthesis model 104 to process the target text 101 and the audio segment 102 to replicate the sound features in the audio segment and generate the target audio segment 103. For example, before processing with the speech synthesis model 104, the text content expressed by the audio segment 102 may be determined first, and the text including that text content may be used as reference text. The speech synthesis model 104 can generate the target audio segment 102 by processing the target text 101, the audio segment 102, and the reference text. Alternatively, the data provider may provide the reference text along with the audio segment 102 and the target text 101; this disclosure does not limit this.

[0034] like Figure 1 As shown, application scenario 100 may also include server 120. Electronic device 110 can communicate with server 120 via a network. For example, server 120 can be a backend management server supporting the operation of client applications or applets installed on electronic devices. Server 120 can also be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS") in terms of high management difficulty and weak business scalability. Server 120 can also be a server for a distributed system or a server combined with blockchain.

[0035] For example, server 120 can train the base model using samples stored in a predetermined sample library, and send the trained model that meets the accuracy requirements as speech synthesis model 104 to electronic device 110. Alternatively, the speech synthesis model 104 can be uploaded to the cloud, and electronic device 110 can call the speech synthesis model 104 through interaction with the cloud.

[0036] In one embodiment, a predetermined sample library can be periodically updated based on audio segments 102 acquired online, and the speech synthesis model 104 can be trained (or fine-tuned) based on samples in the updated predetermined sample library to continuously improve the robustness and accuracy of the model.

[0037] In one embodiment, the electronic device 110 may also send the target text 101 and audio segment 102 to the server 120 via a network, and the server 120 may process the target text 101 and audio segment 102 to obtain the target audio segment 103.

[0038] It should be noted that the data processing method and audio generation method provided in this disclosure can be executed by the electronic device 110 or by the server 120. Correspondingly, the data processing device and audio generation device provided in this disclosure can be located in the electronic device 110 or in the server 120. The training method for the speech synthesis model provided in this disclosure can be executed by the server 120. Correspondingly, the training device for the speech synthesis model provided in this disclosure can be located in the server 120.

[0039] It should be understood that Figure 1 The number and type of electronic devices 110 and servers 120 shown are merely illustrative. Depending on implementation needs, there may be any number and type of electronic devices 110 and servers 120.

[0040] The following will combine Figures 2-3 The data processing methods provided in this disclosure are described in detail.

[0041] Figure 2 This is a schematic flowchart of a data processing method according to an embodiment of the present disclosure.

[0042] like Figure 2 As shown, the data processing method 200 of this embodiment may include operations S210 to S240.

[0043] In operation S210, in response to receiving the reference audio, the features of the reference audio are extracted to obtain the reference audio features.

[0044] According to embodiments of this disclosure, the reference audio may be, for example, an audio segment of 5 seconds in length or any shorter duration (e.g., less than 10 seconds), and this disclosure does not limit it.

[0045] For example, Mel Frequency Cepstral Coefficients (MFCCs) can be extracted from the reference audio, and then a neural network model can be used to further extract features from the extracted MFCCs to obtain the features of the reference audio.

[0046] For example, a pre-trained speech recognition model can be used to extract features from a reference audio. This pre-trained model could be a HUBERT (Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units) model, etc., to extract sub-coded audio features. The input to this pre-trained speech recognition model is the reference audio, and the model forms latent variables through an encoder similar to the BERT principle. This embodiment can use these latent variables as features of the reference audio. Alternatively, a convolutional network can be used to process these latent variables to obtain the semantic encoding of the reference audio, and the obtained semantic encoding can be used as features of the reference audio.

[0047] According to embodiments of this disclosure, reference audio features can express speech features that uniquely correspond to the object providing the reference audio. For example, the reference audio is obtained by recording the audio of the object providing the reference audio.

[0048] For example, the reference audio feature can express the timbre of the reference audio provider, or the reference audio feature can express the tone, speaking habits, etc. of the reference audio provider.

[0049] In operation S220, the audio feature most similar to the reference audio feature is retrieved from the predetermined sample feature library and used as the target audio feature.

[0050] According to embodiments of this disclosure, a predetermined sample feature library may store audio features used as samples. For example, the predetermined sample feature library may store audio features used when training a speech synthesis model used online. The audio features in this predetermined sample feature library can be obtained by extracting features from publicly available audio, which is obtained from a set of publicly available audio. The principle of extracting features from publicly available audio is similar to the above-described operation S210, and will not be repeated here.

[0051] In this embodiment, the similarity between audio features in the predetermined sample feature library and reference audio features extracted in operation S210 can be calculated iteratively, and the audio feature with the highest similarity to the reference audio feature in the predetermined sample feature library is taken as the target audio feature. The similarity between audio features can be represented by cosine similarity, Jaccard similarity coefficient, etc., and this disclosure does not limit this.

[0052] In operation S230, in response to the similarity between the target audio feature and the reference audio feature being greater than or equal to a similarity threshold, the target text feature associated with the target audio feature is obtained.

[0053] According to embodiments of this disclosure, the predetermined sample feature library may also store text features associated with the audio features. For example, the associated audio features and text features are stored in the predetermined sample feature library in the form of text pairs. For example, text features associated with the audio features of the publicly available audio can be obtained by extracting features from the text content expressed by the acquired publicly available audio. For example, an audio-text pair can be obtained from a publicly available database, the audio features can be extracted to obtain audio features, and the text features can be extracted to obtain text features. The obtained text features and audio features can constitute a feature pair. The text in the audio-text pair records the text content expressed by the audio.

[0054] For example, given the text "How do I get to address A?", by recording the audio of the speaker reading the text, we can obtain the corresponding audio, thus creating an audio-text pair.

[0055] When the similarity between the target audio feature and the reference audio feature is greater than or equal to a similarity threshold, this embodiment can use text features stored in a predetermined sample feature library that are associated with the target audio feature as the target text feature. The similarity threshold can be any value close to 1, such as 0.8, 0.9, or 0.95, set according to actual needs; this disclosure does not limit it.

[0056] In operation S240, the target audio features and target text features are stored in a specified feature library in association, using the predetermined identification information as an index.

[0057] In this embodiment, information uniquely indicating the provider of the reference audio can be used as predetermined identification information. This associates the speech features uniquely corresponding to the provider of the reference audio with the identification information uniquely indicating the provider.

[0058] Considering that when a user repeatedly replicates their voice, the provided reference audio is often a fixed audio file, this embodiment of the disclosure, by maintaining a specified feature library, can quickly obtain reference features reflecting speech characteristics when processing audio in an online system. Furthermore, since the obtained features are samples used to train the online speech synthesis model, it not only improves the efficiency of replicating voice features but also increases the similarity between the voice features of the generated speech segment and the voice features of the reference audio. This is because, compared to unprocessed audio, the model often has a higher accuracy in reasoning ability for the audio samples used during training.

[0059] Furthermore, by storing retrieved audio features with a similarity greater than or equal to a similarity threshold in association with text features, there is no need to extract audio features or text features of the reference text during audio generation. This improves the efficiency of audio generation while avoiding the impact of unstable accuracy of the extracted features on the audio generation results, effectively ensuring the accuracy and stability of audio generation.

[0060] According to embodiments of this disclosure, when the similarity between the target audio features obtained through operation S220 and the reference audio features is less than a similarity threshold, the audio features extracted in real time from the reference audio and the associated text features can be used as reference features for online inference, thereby improving inference efficiency.

[0061] Specifically, when the similarity between the target audio features and the reference audio features is less than a similarity threshold, a reference text corresponding to the reference audio can be determined. This reference text can be provided by the provider of the reference audio, or it can be obtained through speech recognition of the reference audio; this disclosure does not limit the method of obtaining the reference text. As long as the reference text records the text content expressed by the reference audio, it is acceptable. After determining the reference text, its features can be extracted to obtain reference text features. Subsequently, using predetermined identification information as an index, the extracted reference audio features and reference text features are associated and stored in a designated feature library. In this way, the range of providers corresponding to the features stored in the designated feature library can be continuously expanded, providing a foundation for improving the reasoning efficiency of subsequent reasoning processes. For example, the reference text features can express the phoneme information of the reference text.

[0062] One approach is to embed the reference text and use the result as the reference text's features. Alternatively, after obtaining the embedding result, a recurrent neural network or a network based on an attention mechanism can be used to encode the embedding result to extract the semantic features of the reference text, which can then be used as the reference text's features.

[0063] For example, the result of the embedding process is an embedding vector of the reference text. In one embodiment, the embedding vector can be concatenated with the extracted semantic features to form the reference text features.

[0064] Figure 3 This is a schematic diagram illustrating the implementation principle of the data processing method according to an embodiment of the present disclosure.

[0065] According to embodiments of this disclosure, a predetermined sample feature library can be expanded based on audio features extracted in real time from reference audio and associated text features, so as to continuously train the speech synthesis model based on online data, thereby solving the technical problem that the accuracy of the speech synthesis model is limited due to the large amount of data required.

[0066] Specifically, the reference text corresponding to the reference audio can be determined using the method described in the above embodiments, and then the features of the reference text can be extracted to obtain reference text features. This embodiment can store the obtained reference audio features and reference text features as feature pairs in an updated sample feature library. The data stored in the updated sample feature library can be used to expand the aforementioned predetermined sample feature library. For example, the data in the updated sample feature library can be periodically merged into the predetermined sample feature library, and the updated sample feature library can be cleared.

[0067] like Figure 3 As shown, in one embodiment 300, the system can guide the user to provide reference audio and reference text 320 by providing a recording interface 310. The reference audio can be recorded in real-time or be an audio file directly provided by the user. Alternatively, the user can provide only the reference audio without providing the reference text; in this case, the system obtains the reference text through speech recognition of the reference audio.

[0068] After obtaining the reference audio and reference text, cloud model 330 can be invoked to extract reference audio features and reference text features 340. For example, cloud model 330 may include HUBERT and BERT models, where HUBERT is used to process the reference audio to obtain reference audio features, and BERT is used to process the reference text to obtain reference text features. It is understood that the type of cloud model 330 described above is merely an example to facilitate understanding of this disclosure; any cloud model can be set according to actual needs, and this disclosure does not limit its use.

[0069] After obtaining the reference audio features and reference text features, the reference audio features and reference text features can be stored in the updated sample feature library 302 in the form of feature pairs, so as to provide a data basis for the expansion of the predetermined sample feature library 301.

[0070] In one embodiment, after obtaining the reference audio features, a predetermined sample feature library 301 can be retrieved based on the reference audio features to find the audio features most similar to the reference audio features, which will be used as the target audio feature 350. After obtaining the target audio feature 350, it can be determined whether the similarity between the target audio feature 350 and the reference audio feature is greater than or equal to a similarity threshold. If the similarity between the two audio features is greater than or equal to the similarity threshold, then the text features associated with the target audio feature are obtained from the predetermined sample feature library 301 and stored as the target text feature 360. Subsequently, the target audio feature 350 and the target text feature 360 ​​can be associatedly stored in a designated feature library 303 using the unique identifier information indicating the user as an index, as reference features used during online inference to ensure the accuracy of the inference. If the similarity between the two audio features is less than the similarity threshold, then the obtained reference audio feature and reference text feature can be associatedly stored in the designated feature library 303 using the unique identifier information indicating the user as an index, as reference features used during online inference. Thus, during online inference, specifically when replicating the timbre of a user's voice, there is no need to extract relevant features based on the reference audio provided by the user, thereby improving the efficiency of online inference.

[0071] In one embodiment, if the similarity between the target audio feature 350 and the extracted reference audio feature is less than a similarity threshold, the reference audio feature and the reference text feature can be associatedly stored in the updated sample feature library 302. Thus, after updating the predetermined sample feature library 301 based on the data in the updated sample feature library 302, the samples in the predetermined sample feature library 301 have significant differences from each other and low redundancy. This improves the efficiency of fine-tuning the speech synthesis model based on the samples in the updated predetermined sample feature library 301 and avoids the situation where the robustness of the speech synthesis model is poor due to sample imbalance.

[0072] Based on the data processing method provided in this disclosure, this disclosure also provides an audio generation method. The following will combine... Figures 4-6 The audio generation method is described in detail.

[0073] Figure 4 This is a schematic flowchart of an audio generation method according to an embodiment of the present disclosure.

[0074] like Figure 4 As shown, the audio generation method 400 of this embodiment may include operations S410 to S440.

[0075] In operation S410, in response to receiving reference audio and target text, a specified feature library is queried based on the identification information associated with the reference audio.

[0076] In operation S420, in response to the query finding a feature associated with the reference audio, the queried feature is used as the reference feature.

[0077] According to embodiments of this disclosure, the identification information associated with the reference audio is identification information that uniquely corresponds to the object providing the reference audio; that is, the identification information uniquely indicates the object providing the reference audio. This identification information may, for example, be the account of the object providing the audio on the audio generation platform, and this disclosure does not limit it in this regard.

[0078] When a user uses an audio generation platform to replicate a voice, they must provide at least a reference audio and a target text. The reference audio provides speech characteristics, and the target text provides the textual content expressed by the target audio obtained through voice replication. After receiving the reference audio and target text, the terminal device running the audio generation platform can query the aforementioned specified feature library based on the identification information of the provided object. If the specified feature library stores features indexed by the identification information of the provided object, then the feature indexed by the identification information of the provided object will be used as the queried feature, thus obtaining the reference feature.

[0079] In operation S430, the text features of the target text are extracted to obtain the target text features.

[0080] According to embodiments of this disclosure, target text can be embedded, and the result of the embedding process can be used as the target text feature. Alternatively, after obtaining the embedding result, a recurrent neural network or a network based on an attention mechanism can be used to encode the embedding result to extract semantic features of the target text, and the extracted semantic features can be used as the target text feature.

[0081] For example, the result of the embedding process is an embedding vector of the target text. In one embodiment, the embedding vector can be concatenated with the extracted semantic features of the target text to form the target text features.

[0082] In operation S440, target audio corresponding to the target text is generated based on reference features and target text features.

[0083] According to embodiments of this disclosure, reference features and target text features can be input into a speech synthesis model, which then outputs audio data with the user's voice. The speech synthesis model can employ a zero-shot voice cloning model, or a model architecture combining a Generative Pre-trained Transformer (GPT) model and a Speech-to-Video Voice Transformation System (SoVITS), etc., and this disclosure does not limit the specific approach.

[0084] In one embodiment, the reference features include reference audio features and reference text features. This embodiment can input the reference audio features, reference text features, and target text features into a generative network (e.g., a generative pre-trained converter model), which outputs predicted target audio features. Subsequently, the target audio features can be decoded to obtain the Mel spectrum of the target audio. This Mel spectrum is then input into a vocoder, which outputs the waveform signal of the target audio.

[0085] In one embodiment, reference audio features and reference text features can be fused first to obtain fused features. Then, the fused features are concatenated with the target text features and input into a variational adaptive in a zero-shot audio cloning model. Subsequently, the features generated by the variational adaptive are input into a decoder in the zero-shot audio cloning model, which outputs the Mel spectrum of the target audio.

[0086] In one embodiment, an autoregressive approach can be used to process the reference features and target text features to predict the target audio features corresponding to the target audio.

[0087] This embodiment of the disclosure generates target audio by querying reference features from a specified feature library based on the identification information of the provided object, and combining the reference features with the text features of the target text. Compared with the technical solution of first extracting audio features in real time and then generating audio, this method can improve audio generation efficiency and avoid the effect of unstable accuracy of the extracted features on the audio generation effect, effectively ensuring the accuracy and stability of audio generation.

[0088] Figure 5 This is a schematic diagram illustrating the principle of generating target audio according to an embodiment of the present disclosure.

[0089] like Figure 5 As shown, in one embodiment 500, the overall architecture of the speech synthesis model may include a generative pre-trained model 510 and a decoder 520.

[0090] In this embodiment 500, the target text feature 501, the reference text feature 502, and the reference audio feature 503 can be input into the generative pre-trained model 510, which then outputs the target audio feature 504 in an autoregressive manner. The target audio feature 504 can express the speaking habits of the person providing the reference audio, such as intonation and pauses.

[0091] In one embodiment, the generative pre-trained model 510 can first concatenate the target text feature 501 and the reference text feature 502, and then concatenate the reference audio feature 503 and the feature obtained by concatenating the two text features to obtain the fused feature. Simultaneously, a mask is generated and merged by combining the reference text feature 502 and the reference audio feature. A Transformer encoder is then used to encode the feature obtained by generating and merging the mask and the fused feature to obtain the target audio feature 504.

[0092] In this embodiment 500, the audio signal of the reference audio 505 can be preprocessed sequentially, including pre-emphasis, framing, and windowing. Then, the preprocessed audio signal is sequentially subjected to short-time Fourier transform and Mel spectrum transform to obtain the Mel spectrum 506 of the reference audio 505.

[0093] After obtaining the Mel spectrum 506 and the target audio feature 504, this embodiment can input the Mel spectrum 506 of the reference audio 505, the target audio feature 504, and the target text feature 501 into the decoder 520, and the decoder 520 outputs the target audio 507. When decoding the target audio feature 504, by considering the target text feature 501, phoneme information can be provided for the generation of the target audio 507; by considering the Mel spectrum 506 of the reference audio 505, timbre features can be provided for the generation of the target audio 507; the target audio feature 504 can provide information such as the speaking habits and style of the subject of the reference audio for the generation of the target audio 507.

[0094] In one embodiment, the decoder 520 can employ the discrimination model in the SoVITS module. The decoder 520 first obtains a prior distribution through a prior encoder, then inputs it into an inverse flow network to transform the data into a posterior distribution. This posterior distribution can be decoded to obtain the target audio 507. That is, when decoding the features of the target audio, the decoder 520 uses the Mel spectrum of the reference audio as a reference. Therefore, the decoded target audio can not only possess the pauses and intonation of the reference audio provider, but also the timbre characteristics of that provider. Thus, the generated target audio can be natural, clear, and able to mimic the subtle differences in a real human voice.

[0095] According to embodiments of this disclosure, if no features associated with the reference audio are found in the aforementioned operation S410, this embodiment can extract reference features based on the reference audio provided by the user. For example, audio features of the reference audio can be extracted to obtain reference audio features. Simultaneously, reference text corresponding to the reference audio can be determined. If the user provides reference text, the user's reference text can be used as the reference text corresponding to the reference audio. If the user does not provide reference text, the reference text can be obtained through speech recognition of the reference audio. After obtaining the reference text, text features of the reference text can be extracted to obtain reference text features. The reference audio features and reference text features can then constitute reference features. The extraction principle of the reference audio features is similar to that described in the aforementioned operation S210, and the extraction principle of the reference text features is similar to the principle of obtaining reference text features described above, and will not be repeated here.

[0096] According to embodiments of this disclosure, after obtaining the target audio, it can be output using a streaming transmission method. Furthermore, during the transmission of the target audio, a queue buffering mechanism can be set up to achieve continuous, stable, and efficient transmission of the target audio.

[0097] Based on the data processing method provided in this disclosure, this disclosure also provides a training method for a speech synthesis model. The following will combine... Figure 6 The training method is described in detail.

[0098] Figure 6 This is a flowchart illustrating a training method for a speech synthesis model according to an embodiment of the present disclosure.

[0099] like Figure 6 As shown, the training method 600 for the speech synthesis model in this embodiment may include operations S610 to S620.

[0100] In operation S610, the predetermined sample feature library is periodically updated based on the data stored in the updated sample feature library.

[0101] According to embodiments of this disclosure, in addition to storing speech feature-text feature pairs, the aforementioned predetermined sample feature library may also store target text associated with the speech feature-text feature pairs, or text features of the target text. In this embodiment, speech feature-text feature pairs can be periodically retrieved from the updated sample feature library, and target text can be randomly assigned or randomly generated for the retrieved speech feature-text feature pairs. The target text is then added to the predetermined sample feature library in association with the speech feature-text feature pairs retrieved from the updated sample feature library. Alternatively, the target text features of the randomly assigned or randomly generated target text can be extracted first, and the target text features are added to the predetermined sample feature library in association with the corresponding speech feature-text feature pairs. In this way, the features in the predetermined sample feature library are updated. It is understood that speech feature-text feature pairs can constitute reference features.

[0102] The update cycle can be determined, for example, based on the amount of reference audio data generated online. If the amount of reference audio data generated online is large, the update cycle can be set to a smaller cycle. Alternatively, the update cycle can be determined based on the average time it takes for the number of features stored in the updated sample feature library to reach a predetermined amount. Alternatively, an update operation on a predetermined sample feature library can be triggered in response to the number of features stored in the updated sample feature library reaching a predetermined amount; this disclosure does not limit this.

[0103] When operating the S620, the speech synthesis model is trained based on the updated feature library to obtain the trained speech synthesis model.

[0104] According to embodiments of this disclosure, a principle similar to that described in operation S440 can be employed to generate target audio that expresses the target text corresponding to the target text features, based on reference features in the updated feature library and target text features. Subsequently, the Mel spectrum of the target audio and the Mel spectrum of the audio corresponding to the reference audio features in the reference features are extracted. The loss of the speech synthesis model in generating the target audio is determined based on the difference between the two Mel spectra. The speech synthesis model is fine-tuned with the goal of minimizing this loss, thereby achieving training of the speech synthesis model. For example, the mean square error of the two Mel spectra can be used to represent the loss of the speech synthesis model in generating the target audio, or other alternative loss functions can be used to determine the loss of the speech synthesis model in generating the target audio; this disclosure does not limit this approach.

[0105] In one embodiment, the speech synthesis model may employ the architecture described in Embodiment 500 above. When training the speech synthesis model, the generation loss of the generative pre-trained model may be determined, for example, based on the difference between the target audio features and the reference audio features. The generative pre-trained model is trained with the objective of minimizing this generation loss. It is understood that both the target audio features and the reference audio features can be understood as token sequences. This embodiment may use the cross-entropy loss function to determine the generation loss of the generative pre-trained model, or other alternative loss functions may be used; this disclosure does not limit this approach.

[0106] This disclosure embodiment periodically updates the predetermined sample feature library based on the data stored in the updated sample feature library, and trains the speech synthesis model based on the updated feature library. By continuously expanding the dataset and fine-tuning, the generalization ability of the speech synthesis model can be significantly improved, so that the speech synthesis model can exhibit good timbre reproduction effect in different scenarios and for different users.

[0107] After obtaining the trained speech synthesis model, its model file can be uploaded to the cloud. This allows for the updating of the model running in the cloud. This ensures that when generating audio online by calling the cloud-based speech synthesis model, the latest and most optimized model can be used, resulting in higher quality and more personalized audio output.

[0108] Based on the data processing method provided in this disclosure, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 7 The device is described in detail.

[0109] Figure 7 This is a structural block diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0110] like Figure 7 As shown, the data processing device 700 of this embodiment may include an audio feature extraction module 710, a feature retrieval module 720, a text feature acquisition module 730, and a first feature storage module 740.

[0111] The audio feature extraction module 710 is used to extract features of the reference audio in response to receiving the reference audio, thereby obtaining reference audio features. In one embodiment, the audio feature extraction module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0112] The feature retrieval module 720 is used to retrieve the audio feature most similar to the reference audio feature from a predetermined sample feature library, and use it as the target audio feature. The predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model. The text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature. In one embodiment, the feature retrieval module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0113] The text feature acquisition module 730 is used to acquire target text features associated with the target audio features in response to a similarity threshold between the target audio features and the reference audio features. In one embodiment, the text feature acquisition module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0114] The first feature storage module 740 is used to store target audio features and target text features in association to a designated feature library using predetermined identification information as an index. The predetermined identification information uniquely indicates the object providing the reference audio. In one embodiment, the first feature storage module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0115] According to embodiments of this disclosure, the data processing apparatus 700 may further include a reference text determination module and a text feature extraction module. The reference text determination module is used to determine a reference text corresponding to the reference audio in response to a similarity threshold between the target audio features and the reference audio features; the reference text contains the text content expressed by the reference audio. The text feature extraction module is used to extract features from the reference text to obtain reference text features. The first feature storage module 740 is further used to store the reference audio features and reference text features in association to a designated feature library using predetermined identification information as an index.

[0116] According to embodiments of this disclosure, the data processing apparatus 700 may further include a reference text determination module, a text feature extraction module, and a second feature storage module. The reference text determination module determines a reference text corresponding to a reference audio; the reference text contains the text content expressed by the reference audio. The text feature extraction module extracts features from the reference text to obtain reference text features. The second feature storage module stores the reference audio features and reference text features in the form of feature pairs in an updated sample feature library. The data stored in the updated sample feature library is used to expand a predetermined sample feature library.

[0117] According to embodiments of this disclosure, the second feature storage module described above can be specifically used to: in response to the similarity between the target audio feature and the reference audio feature being less than a similarity threshold, store the reference audio feature and the reference text feature in the form of feature pairs to the updated sample feature library.

[0118] Based on the audio generation method provided in this disclosure, this disclosure also provides an audio generation apparatus. The following will be combined with... Figure 8 The device is described in detail.

[0119] Figure 8 This is a structural block diagram of an audio generation apparatus according to an embodiment of the present disclosure.

[0120] like Figure 8 As shown, the audio generation device 800 of this embodiment may include a feature query module 810, a feature determination module 820, a target feature extraction module 830, and an audio generation module 840.

[0121] The feature query module 810 is used to query a specified feature library based on the identification information associated with the reference audio in response to receiving reference audio and target text. The specified feature library stores audio features and text features indexed by the identification information. The text features correspond to text that contains the text content expressed by the audio features. The identification information uniquely indicates the recipient of the reference audio. The data in the specified feature library is stored using the data processing apparatus provided in this disclosure. In one embodiment, the feature query module 810 can be used to perform the operation S410 described above, which will not be repeated here.

[0122] The feature determination module 820 is used to use the queried features as reference features in response to the discovery of features associated with the reference audio. In one embodiment, the feature determination module 820 may be used to perform the operation S420 described above, which will not be repeated here.

[0123] The target feature extraction module 830 is used to extract text features from the target text to obtain target text features. In one embodiment, the target feature extraction module 830 can be used to perform the operation S430 described above, which will not be repeated here.

[0124] The audio generation module 840 is used to generate target audio corresponding to the target text based on reference features and target text features. In one embodiment, the audio generation module 840 can be used to perform the operation S440 described above, which will not be repeated here.

[0125] According to embodiments of this disclosure, the audio generation apparatus 800 may further include a reference feature extraction module, used to extract reference features based on the reference audio in response to the absence of features associated with the reference audio.

[0126] According to embodiments of this disclosure, the reference feature extraction module may include a text determination submodule, an audio feature extraction submodule, and a text feature extraction submodule. The text determination submodule is used to determine the reference text corresponding to the reference audio. The reference text contains the text content expressed by the reference audio. The audio feature extraction submodule is used to extract features from the reference audio to obtain reference audio features. The text feature extraction submodule is used to extract text features from the reference text to obtain reference text features. The reference features include reference audio features and reference text features.

[0127] According to embodiments of this disclosure, the audio generation module 840 may include a feature generation submodule and a decoding submodule. The feature generation submodule is used to process reference features and target text features using a generative network to generate target audio features corresponding to the target audio. The decoding submodule is used to decode the target audio features with reference to the Mel spectrum of the reference audio to obtain the target audio.

[0128] Based on the training method for the speech synthesis model provided in this disclosure, this disclosure also provides a training apparatus for the speech synthesis model. The following will combine... Figure 9 The device is described in detail.

[0129] Figure 9 This is a structural block diagram of a training apparatus for a speech synthesis model according to an embodiment of the present disclosure.

[0130] like Figure 9 As shown, the training device 900 for the speech synthesis model in this embodiment may include a feature library update module 910 and a model training module 920.

[0131] The feature library update module 910 is used to periodically update a predetermined sample feature library based on the data stored in the updated sample feature library. The updated sample feature library stores audio feature-text feature pairs obtained based on reference audio received online; the predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model; the text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature. The data stored in the updated sample feature library is obtained based on reference audio received by the data processing apparatus provided in this disclosure. In one embodiment, the feature library update module 910 can be used to perform the operation S610 described above, which will not be repeated here.

[0132] The model training module 920 is used to train the speech synthesis model based on the updated feature library to obtain the trained speech synthesis model. In one embodiment, the model training module 920 can be used to perform the operation S620 described above, which will not be repeated here.

[0133] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information in this disclosed technical solution comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. In this disclosed technical solution, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0134] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0135] Figure 10 A schematic block diagram of an example electronic device 1000 is shown, which can be used to implement at least one of the data processing method, audio generation method, and speech synthesis model training method of embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded into random access memory (RAM) 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0137] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs at least one of the various methods and processes described above, such as data processing methods, audio generation methods, and speech synthesis model training methods. For example, in some embodiments, at least one of the data processing methods, audio generation methods, and speech synthesis model training methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, it can perform one or more steps of at least one of the data processing method, audio generation method, and speech synthesis model training method described above. Alternatively, in other embodiments, computing unit 1001 can be configured by any other suitable means (e.g., by means of firmware) to perform at least one of the data processing method, audio generation method, and speech synthesis model training method.

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0144] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0145] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: In response to receiving reference audio, features of the reference audio are extracted to obtain reference audio features; The audio feature most similar to the reference audio feature is retrieved from the predetermined sample feature library and used as the target audio feature; The predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model. The text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature. In response to a similarity threshold between the target audio feature and the reference audio feature being greater than or equal to the target audio feature, the target text feature associated with the target audio feature is obtained; as well as Using predetermined identification information as an index, the target audio features and the target text features are stored in a designated feature library in association; the predetermined identification information uniquely indicates the object that provides the reference audio.

2. The method according to claim 1, further comprising: In response to the similarity between the target audio feature and the reference audio feature being less than the similarity threshold, a reference text corresponding to the reference audio is determined; The reference text contains the text content expressed by the reference audio. Extract the features of the reference text to obtain the reference text features; and Using the predetermined identification information as an index, the reference audio features and the reference text features are associated and stored in the designated feature library.

3. The method according to claim 1, further comprising: Determine the reference text corresponding to the reference audio; The reference text contains the text content expressed by the reference audio. Extract the features of the reference text to obtain the reference text features; and The reference audio features and the reference text features are stored as feature pairs in the updated sample feature library. The data stored in the updated sample feature library is used to expand the predetermined sample feature library.

4. The method according to claim 3, wherein, The step of storing the reference audio features and the reference text features as feature pairs in the updated sample feature library includes: In response to the similarity between the target audio feature and the reference audio feature being less than the similarity threshold, the reference audio feature and the reference text feature are stored as feature pairs in the updated sample feature library.

5. An audio generation method, comprising: In response to receiving reference audio and target text, a specified feature library is queried based on the identification information associated with the reference audio; the specified feature library stores audio features and text features indexed by the identification information, and the text corresponding to the text feature records the text content expressed by the audio corresponding to the audio feature; the identification information uniquely indicates the recipient of the reference audio; In response to finding a feature associated with the reference audio, the found feature is used as a reference feature; Extract the text features of the target text to obtain the target text features; as well as Based on the reference features and the target text features, a target audio corresponding to the target text is generated. The data in the specified feature library is stored using the method described in any one of claims 1 to 4.

6. The method according to claim 5, further comprising: In response to the absence of features associated with the reference audio, the reference features are extracted based on the reference audio.

7. The method according to claim 6, wherein, Extracting the reference features based on the reference audio includes: Determine the reference text corresponding to the reference audio; the reference text records the text content expressed by the reference audio; Extract the features from the reference audio to obtain the reference audio features; Extract the text features of the reference text to obtain the reference text features. The reference features include the reference audio features and the reference text features.

8. The method according to claim 6, wherein, The step of generating the target audio corresponding to the target text based on the reference features and the target text features includes: A generative network is used to process the reference features and the target text features to generate target audio features corresponding to the target audio; and Using the Mel spectrum of the reference audio as a reference, the target audio features are decoded to obtain the target audio.

9. A training method for a speech synthesis model, comprising: The predetermined sample feature library is periodically updated based on the data stored in the updated sample feature library; The updated sample feature library stores audio feature-text feature pairs obtained based on reference audio received online; the predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model. In an audio feature-text feature pair, the text corresponding to the audio feature records the text content expressed by the audio corresponding to the audio feature; and Based on the updated feature library, the speech synthesis model is trained to obtain the trained speech synthesis model. The data stored in the updated sample feature library is obtained based on the reference audio received in the method of any one of claims 1 to 4.

10. A data processing apparatus, comprising: The audio feature extraction module is used to extract features of the reference audio in response to receiving the reference audio, and obtain the reference audio features; The feature retrieval module is used to retrieve the audio feature most similar to the reference audio feature from a predetermined sample feature library, and use it as the target audio feature; The predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model. The text corresponding to the text feature in the audio feature-text feature pair records the text content expressed by the audio corresponding to the audio feature. The text feature acquisition module is used to acquire target text features associated with the target audio features in response to the similarity between the target audio features and the reference audio features being greater than or equal to a similarity threshold; as well as The first feature storage module is used to store the target audio features and the target text features in association to a designated feature library using predetermined identification information as an index; the predetermined identification information uniquely indicates the object that provides the reference audio.

11. The apparatus of claim 10, further comprising: A reference text determination module is used to determine reference text corresponding to the reference audio in response to a similarity threshold between the target audio feature and the reference audio feature; the reference text contains the text content expressed by the reference audio; and The text feature extraction module is used to extract features from the reference text to obtain reference text features. The first feature storage module is further configured to store the reference audio features and the reference text features in association to the designated feature library, using the predetermined identification information as an index.

12. The apparatus of claim 10, further comprising: A reference text determination module is used to determine the reference text corresponding to the reference audio. The reference text contains the text content expressed by the reference audio. The text feature extraction module is used to extract features from the reference text to obtain reference text features; and The second feature storage module is used to store the reference audio features and the reference text features as feature pairs in the updated sample feature library. The data stored in the updated sample feature library is used to expand the predetermined sample feature library.

13. The apparatus according to claim 12, wherein, The second feature storage module is used for: In response to the similarity between the target audio feature and the reference audio feature being less than the similarity threshold, the reference audio feature and the reference text feature are stored as feature pairs in the updated sample feature library.

14. An audio generation apparatus, comprising: The feature query module is used to query a specified feature library based on the identification information associated with the reference audio in response to receiving reference audio and target text. The specified feature library stores audio features and text features indexed by the identification information. The text corresponding to the text feature records the text content expressed by the audio corresponding to the audio feature. The identification information uniquely indicates the recipient of the reference audio. The feature determination module is used to use the queried features as reference features in response to the query of features associated with the reference audio. The target feature extraction module is used to extract the text features of the target text to obtain the target text features; and The audio generation module is used to generate target audio corresponding to the target text based on the reference features and the target text features. The data in the specified feature library is stored using the apparatus described in any one of claims 10 to 13.

15. The apparatus of claim 14, further comprising: A reference feature extraction module is used to extract the reference features based on the reference audio in response to the absence of features associated with the reference audio.

16. The apparatus according to claim 15, wherein, The reference feature extraction module includes: The text determination submodule is used to determine the reference text corresponding to the reference audio; the reference text contains the text content expressed by the reference audio. An audio feature extraction submodule is used to extract features from the reference audio to obtain reference audio features; and The text feature extraction submodule is used to extract the text features of the reference text to obtain the reference text features. The reference features include the reference audio features and the reference text features.

17. The apparatus according to claim 15, wherein, The audio generation module includes: The feature generation submodule is used to process the reference features and the target text features using a generative network to generate target audio features corresponding to the target audio; and The decoding submodule is used to decode the target audio features with reference to the Mel spectrum of the reference audio to obtain the target audio.

18. A training device for a speech synthesis model, comprising: The feature library update module is used to periodically update the predetermined sample feature library based on the data stored in the updated sample feature library; The updated sample feature library stores audio feature-text feature pairs obtained based on reference audio received online; the predetermined sample feature library stores audio feature-text feature pairs included in the training data of the speech synthesis model. In an audio feature-text feature pair, the text corresponding to the audio feature records the text content expressed by the audio corresponding to the audio feature; and The model training module is used to train the speech synthesis model based on the updated feature library, resulting in a trained speech synthesis model. The data stored in the updated sample feature library is obtained based on the reference audio received by the device according to any one of claims 10 to 13.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images