Audio synthesis method, computer device, storage medium, and program product

Abstract

The application provides an audio synthesis method, a computer device, a storage medium, and a program product. The method comprises the following steps: performing feature fusion processing on phoneme feature information of a target text and label information corresponding to the target text to obtain target phoneme feature information; performing splicing processing on the target phoneme feature information and target pitch feature information to obtain spliced feature information; generating a predicted mel spectrum according to the spliced feature information; and performing conversion processing on the predicted mel spectrum by using a vocoder to obtain audio data matched with a score corresponding to the target text. By using the application, the cost of audio synthesis can be reduced, and the timbre stability of the synthesized audio data can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to audio synthesis methods, computer devices and computer-readable storage media, and computer programs. Background Technology

[0002] Audio synthesis technology converts lyrics and musical scores into audio, and it has wide applications in voice assistants, chatbots, and virtual assistants. Currently, training audio synthesis models typically requires subjects to record large amounts of audio data (several hours or more). However, collecting several hours of audio from a single subject is difficult and costly. In this case, it's more efficient to collect audio data from dozens to hundreds of subjects, each with tens of minutes of audio data. However, training an audio synthesis model by collecting audio data from multiple subjects can lead to confusion in the timbre characteristics of different subjects, resulting in poor timbre stability in the synthesized audio data. Summary of the Invention

[0003] This application provides an audio synthesis method, computer equipment and storage medium, and program product, which can reduce the cost of audio synthesis and improve the timbre stability of the synthesized audio data.

[0004] On one hand, embodiments of this application provide an audio synthesis method, which includes: performing feature fusion processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain target phoneme feature information; performing splicing processing on the target phoneme feature information and target pitch feature information to obtain spliced ​​feature information; generating a predicted Mel spectrum based on the spliced ​​feature information; and performing conversion processing on the predicted Mel spectrum through a vocoder to obtain audio data that matches the score corresponding to the target text.

[0005] It is evident that by fusing the phoneme features of the target text with the corresponding tag information, and fusing the pitch features of the musical score with the tag information, and then performing audio synthesis, it is beneficial to better preserve the features of different tag information in the synthesized audio data, thereby improving the timbre stability of the synthesized audio data.

[0006] In some embodiments, the above-mentioned feature fusion processing of the phoneme feature information of the target text and the tag information corresponding to the target text to obtain target phoneme feature information includes: fusing and encoding the phoneme feature information of the target text and the tag information corresponding to the target text to obtain phoneme encoded feature information; decoding the phoneme encoded feature information to obtain phoneme decoded feature information; and supervising the phoneme decoded feature information based on the tag information to obtain target phoneme feature information. Similarly, the above-mentioned feature fusion processing of the pitch feature information of the musical score corresponding to the target text and the tag information to obtain target phoneme feature information includes: fusing and encoding the pitch feature information of the musical score corresponding to the target text and the tag information to obtain pitch encoded feature information; decoding the pitch encoded feature information to obtain pitch decoded feature information; and supervising the pitch decoded feature information based on the musical score corresponding to the target text to obtain target pitch feature information. It is evident that supervised training of phoneme decoding feature information using label information is beneficial for further consolidating the label information retained in the phoneme decoding feature information; supervised training of pitch decoding feature information using the musical score corresponding to the target text is beneficial for further improving the pitch accuracy of the pitch decoding feature information.

[0007] In some embodiments, the above-mentioned fusion encoding process of the phoneme feature information of the target text and the tag information corresponding to the target text to obtain phoneme encoded feature information includes: encoding the phoneme feature information of the target text to obtain initial phoneme encoded feature information; encoding the tag information corresponding to the target text to obtain tag encoded feature information; concatenating the initial phoneme encoded feature information and the tag encoded feature information to obtain phoneme encoded feature information; or, superimposing the initial phoneme encoded feature information and the tag encoded feature information to obtain phoneme encoded feature information. It is evident that using different methods to fuse the phoneme feature information of the target text and the tag information corresponding to the target text allows for more flexible fusion of these two types of information.

[0008] In some embodiments, the above-mentioned fusion encoding process of the pitch feature information and tag information of the musical score corresponding to the target text to obtain pitch encoded feature information includes: encoding the pitch feature information of the musical score corresponding to the target text to obtain initial pitch encoded feature information; encoding the tag information to obtain tag encoded feature information; concatenating the initial pitch encoded feature information and the tag encoded feature information to obtain pitch encoded feature information; or, superimposing the initial pitch encoded feature information and the tag encoded feature information to obtain pitch encoded feature information. It is evident that using different methods to fuse the pitch feature information and tag information of the musical score corresponding to the target text allows for more flexible fusion of these two sets of information.

[0009] In some embodiments, the above-mentioned supervised training of phoneme decoding feature information based on label information to obtain target phoneme feature information includes: supervising the phoneme decoding feature information based on label information using a first loss function until the first loss function satisfies a first constraint condition, and determining the phoneme decoding feature information obtained through supervised training as the target phoneme feature information; the above-mentioned supervised training of pitch decoding feature information based on the musical score corresponding to the target text to obtain target pitch feature information includes: supervising the pitch decoding feature information based on the musical score corresponding to the target text using a second loss function until the second loss function satisfies a second constraint condition, and determining the pitch decoding feature information obtained through supervised training as the target pitch feature information. It can be seen that using different loss functions to supervise the training of phoneme decoding feature information and pitch decoding feature information respectively allows the target phoneme feature information obtained through supervised training to better retain label information, and the target pitch feature information obtained through supervised training to have a higher similarity to the musical score, thereby improving the accuracy of the synthesized audio data.

[0010] In some embodiments, the above-described decoding process of phoneme-encoded feature information to obtain phoneme-decoded feature information includes: performing duration prediction processing on the phoneme-encoded feature information to obtain predicted duration information corresponding to the phoneme-encoded feature information; performing expansion processing on the phoneme-encoded feature information based on the predicted duration information to obtain extended phoneme feature information; and performing decoding processing on the extended phoneme feature information to obtain phoneme-decoded feature information. It is evident that expanding the phoneme-encoded feature information allows the expanded phoneme-encoded feature information to match the spectral length required for synthesizing audio data, thereby facilitating the synthesis of audio data with more uniform rhythm.

[0011] In some embodiments, the method further includes: acquiring initial audio data generated by the target object for the target text; extracting phoneme feature information of the target text from the initial audio data; and determining the identification information of the target object as the tag information corresponding to the target text.

[0012] On one hand, embodiments of this application provide an audio synthesis apparatus, which includes: an acquisition unit for acquiring phoneme feature information of a target text and tag information corresponding to the target text; a processing unit for performing feature fusion processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain target phoneme feature information; performing feature fusion processing on the pitch feature information of the musical score corresponding to the target text and the tag information to obtain target pitch feature information; and synthesizing audio data matching the musical score corresponding to the target text based on the target phoneme feature information and the target pitch feature information.

[0013] Accordingly, this application also provides a computer device, including: a processor, a communication interface, and a memory, wherein the processor, the communication interface, and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to implement the method in this application embodiment.

[0014] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in the embodiments of this application.

[0015] Accordingly, this application also provides a computer program product, which includes a computer program or computer instructions, which, when executed by a processor, implement the methods in this application. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the system architecture applied to the embodiments of this application;

[0018] Figure 2 This is a flowchart illustrating an audio synthesis method provided in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the architecture of an audio synthesis model provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram illustrating an application scenario of audio synthesis provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of an audio synthesis device provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] To better understand the solutions of the embodiments of this application, some nouns and terms that may be involved in the embodiments of this application will be introduced first.

[0025] (1) Phoneme: A phoneme is the smallest unit of speech. It can be analyzed based on the articulation actions within a syllable; one action constitutes one phoneme. In Chinese, pronunciation is determined by Pinyin, and phonemes are the initials and finals in Pinyin. In English, pronunciation is determined by phonetic symbols, and phonemes are the vowels and consonants in English phonetic symbols.

[0026] (2) Score (note) and pitch (pitch): Score can be used to represent the melody of a song. Score can include different notes, and each note can have its own corresponding pitch (that is, tone).

[0027] (3) Mel spectrum: The Mel spectrum is obtained by performing a Fourier transform on the acoustic signal and then transforming it using the Mel scale.

[0028] Secondly, the system architecture involved in this application will be described.

[0029] The audio synthesis method provided in this application embodiment can be applied to... Figure 1 In the system architecture shown. For example... Figure 1 As shown, the system architecture includes terminal device 110 and server 120, and terminal device 110 and server 120 can communicate via a network. It is understood that... Figure 1 The system architecture shown is merely an example and does not constitute a limitation on the embodiments of this application. In practical applications, it may include different architectures. Figure 1 The number of terminal devices and servers shown.

[0030] The server 120 can be a server with audio synthesis capabilities. The terminal device 110 can send an audio synthesis request to the server. The audio synthesis request can include phoneme feature information of the target text, tag information corresponding to the target text, and pitch feature information of the musical score corresponding to the target text. After receiving the audio synthesis request from the terminal device 110, the server 120 can synthesize audio data that matches the musical score corresponding to the target text based on the phoneme feature information of the target text, the tag information corresponding to the target text, and the pitch feature information of the musical score corresponding to the target text.

[0031] Specifically, server 120 can perform feature fusion processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain target phoneme feature information; then, it can perform feature fusion processing on the pitch feature information of the musical score corresponding to the target text and the tag information to obtain target pitch feature information; next, it can perform concatenation processing on the target phoneme feature information and the target pitch feature information to obtain concatenated feature information; then, it can generate a predicted Mel spectrum based on the concatenated feature information; finally, it can perform conversion processing on the predicted Mel spectrum through a vocoder to obtain audio data that matches the musical score corresponding to the target text.

[0032] In some embodiments, the audio synthesis method provided in this application can also be executed by the terminal device 110, which can send the synthesized audio data to the server 120.

[0033] It should be noted that the terminal device 110 can be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and other cloud computing-based services. The terminal device 110 and the server 120 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the application does not impose any limitations.

[0034] The audio synthesis method provided in the embodiments of this application will be described in detail below.

[0035] Please see Figure 2 This is a flowchart illustrating an audio synthesis method provided in an embodiment of this application. This audio synthesis method can be executed by an audio synthesis device, which can be a terminal device or a server. Figure 2 As shown, the method includes, but is not limited to, the following steps:

[0036] S201. Perform feature fusion processing on the phoneme feature information of the target text and the label information corresponding to the target text to obtain the target phoneme feature information.

[0037] In the embodiments of this application, the audio synthesis device can perform fusion processing on the phoneme feature information of the target text and the label information corresponding to the target text to obtain the target phoneme feature information. Among them, the target text can be the text content corresponding to any lyrics, and the target text can be a Chinese text or an English text. The phoneme feature information includes the phoneme features extracted from the initial audio data generated by the target object for the target text. The initial audio data can be generated by the target object reading the target text or singing the target text. The phoneme feature information can be, for example, a phoneme feature sequence, and the phoneme features included in the phoneme feature sequence are arranged in the order of the characters in the target text. For example, the target text "今天" includes the phonemes "j", "in", "t", and "ian", and the phoneme feature sequence corresponding to the target text "今天" is "jintian". The label information corresponding to the target text refers to the identification information of the target object that generates the initial audio data for the target text, which can be used to distinguish different objects. For example, if object 1 sings the target text "你今天真好看", then the label information corresponding to the target text "你今天真好看" can be the identification information of object 1 (such as the digital identification 1).

[0038] In some embodiments, the specific way for the audio synthesis device to perform feature fusion processing on the phoneme feature information of the target text and the label information corresponding to the target text to obtain the target phoneme feature information can be as follows: First, perform fusion encoding processing on the phoneme feature information of the target text and the label information corresponding to the target text to obtain phoneme encoding feature information; then perform decoding processing on the phoneme encoding feature information to obtain phoneme decoding feature information; finally, perform supervised training on the phoneme decoding feature information according to the label information to obtain the target phoneme feature information. Among them, performing fusion encoding on the phoneme feature information of the target text and the label information corresponding to the target text can fuse the phoneme features in the phoneme feature information of the target text and the label information corresponding to the target text. In this way, the phoneme encoding feature information obtained by fusion encoding can retain the label information to facilitate distinguishing the phoneme features of different objects. After decoding the phoneme encoding feature information that fuses the label information, then performing supervised training on the phoneme decoding feature information according to the label information can consolidate the label information fused in the phoneme decoding feature information, so that the target phoneme feature information obtained by supervised training retains sufficient label information. In this way, it is beneficial to improve the timbre stability of the subsequently synthesized audio data.

[0039] In some embodiments, the audio synthesis device can encode the phoneme feature information of the target text to obtain initial phoneme encoded feature information, and encode the label information corresponding to the target text to obtain label encoded feature information. The initial phoneme encoded feature information and the label encoded feature information are then fused to obtain the final phoneme encoded feature information. Specifically, the audio synthesis device can use an encoder to encode both the phoneme feature information and the label information corresponding to the target text. The encoder can be a neural network, for example, it can be a stack of N feed-forward transformer blocks (FFT blocks), each FFT block consisting of a self-attention module and a one-dimensional convolutional (conv-1D) module. The initial phoneme encoded feature information obtained by encoding the phoneme feature information of the target text can be a phoneme embedding feature. The label information corresponding to the target text is encoded by the encoder to obtain a label embedding feature. After obtaining the initial phoneme encoding feature information and the tag encoding feature information, the audio synthesis device can perform the fusion processing on the initial phoneme encoding feature information and the tag encoding feature information by concatenating the initial phoneme encoding feature information and the tag encoding feature information, or by concatenating the initial phoneme encoding feature information and the tag encoding feature information and superimposing them. For example, assuming that the initial phoneme encoding feature information is a vector [a,b,c,d] and the tag encoding feature information is a vector [3,2,1,1], the phoneme encoding feature information obtained by fusing the initial phoneme encoding feature information and the tag encoding feature information can be the result of concatenating the vector [a,b,c,d] and the vector [3,2,1,1] as [a,b,c,d,3,2,1,1], or the result of superimposing the vector [a,b,c,d] and the vector [3,2,1,1] as [a+3,b+4,c+5,d+6]. In other embodiments, the audio synthesis device may perform fusion processing on the initial phoneme coding feature information and the tag coding feature information by multiplying the initial phoneme coding feature information and the tag coding feature information, or other possible methods, which are not limited in this application.

[0040] In some embodiments, after obtaining phoneme-encoded feature information, the audio synthesis device can decode the phoneme-encoded feature information to obtain phoneme-decoded feature information. Specifically, the audio synthesis device can decode the phoneme-encoded feature information by first performing duration prediction processing on the phoneme-encoded feature information to obtain predicted duration information corresponding to the phoneme-encoded feature information; then performing expansion processing on the phoneme-encoded feature information to obtain extended phoneme feature information; and finally decoding the extended phoneme feature information to obtain phoneme-decoded feature information. This is because the length of the phoneme-encoded feature information is usually less than the spectral length required for subsequent audio data synthesis. Decoding the phoneme-encoded feature information in this way allows the decoded phoneme-decoded feature information to match the spectral length required for synthesizing audio data, thereby facilitating the synthesis of audio data with more uniform tempo and rhythm. The audio synthesis device can perform duration prediction processing on the phoneme-encoded feature information using a duration predictor. The duration predictor can predict the duration of each phoneme-encoded feature in the phoneme-encoded feature information to obtain the predicted duration information corresponding to the phoneme-encoded feature information. The duration of each phoneme coding feature can be understood as the number of times each phoneme coding feature needs to be repeated. The audio synthesis device repeats each phoneme coding feature in the phoneme coding feature information according to the duration predicted by the duration predictor, thus achieving the effect of expanding the phoneme coding feature information and obtaining extended phoneme feature information. For example, assuming the phoneme coding feature information is a vector [a,b,c,d], and the duration predicted by the duration predictor is a vector [1,2,1,3], this means repeating phoneme coding feature a by 1 time, phoneme coding feature b by 2 times, phoneme coding feature c by 1 time, and phoneme coding feature d by 3 times. The repetition of phoneme coding features a and b by one can also be understood as keeping the lengths of phoneme coding features a and b unchanged, and repeating each phoneme coding feature in the phoneme coding feature information [a,b,c,d] according to the duration [1,2,1,3] to obtain the extended phoneme coding feature information [a,b,b,c,d,d,d].

[0041] In some embodiments, the audio synthesis device can decode the phoneme encoded feature information using a decoder to obtain phoneme decoded feature information. The decoder can be a neural network, and the structure of the neural network can be, for example, a stack of N FFT blocks, each FFT block consisting of a self-attention module and a conv-1D module.

[0042] In some embodiments, after obtaining phoneme decoding feature information, the audio synthesis device can perform supervised training on the phoneme decoding feature information based on label information to obtain target phoneme feature information. Specifically, the audio synthesis device can perform supervised training on the phoneme decoding feature information based on label information using a first loss function until the first loss function satisfies a first constraint condition, and then determine the phoneme decoding feature information obtained through supervised training as the target phoneme feature information. A loss function is used to estimate the degree of inconsistency between the model's predicted values ​​and the true values. It is a non-negative real-valued function; the smaller the loss function, the better the robustness of the model. The audio synthesis device can perform supervised training on the phoneme decoding feature information based on label information using the first loss function by performing classification training on the phoneme decoding feature information based on label information using the first loss function. Specifically, it can input the label information and phoneme decoding feature information into a classifier (also called a classification model) to optimize the classification ability of the classifier to the best extent, with the goal of satisfying the first constraint condition for the first loss function. The classifier structure can be a fully-connected layer (FC-layer) or other more complex network structures, such as an emphasized channel attention propagation and aggregation in TDNN (ECAPA-TDNN). The first loss function can be, for example, the SoftMax cross-entropy function. The first constraint can be determined based on the training requirements of the classifier, such as whether the number of training iterations of the classifier has reached a preset number, or whether the value of the first loss function has converged, etc., and can be determined according to actual needs. This application does not specifically limit the first loss function, the structure of the classifier, or the content of the first constraint. The target phoneme feature information obtained in this way is beneficial to improving the stability of the subsequently synthesized audio data.

[0043] S202, the pitch feature information of the musical score corresponding to the target text is fused with the tag information to obtain the target pitch feature information.

[0044] In this embodiment, the audio synthesis device can fuse the pitch feature information of the musical score corresponding to the target text with the tag information to obtain the target pitch feature information. The pitch feature information of the musical score corresponding to the target text refers to the pitch feature information of each note in the musical score corresponding to the target text, which can also be understood as the tone feature of each note.

[0045] In some embodiments, the audio synthesis device performs feature fusion processing on the pitch feature information of the score corresponding to the target text and the tag information to obtain the target pitch feature information. Specifically, this can be achieved by first fusing and encoding the pitch feature information of the score corresponding to the target text and the tag information to obtain pitch-encoded feature information; then decoding the pitch-encoded feature information to obtain pitch-decoded feature information; and finally, supervised training of the pitch-decoded feature information based on the score corresponding to the target text to obtain the target pitch feature information. The feature fusion processing of the pitch feature information of the score corresponding to the target text and the tag information allows for the fusion of the pitch feature information and the tag information. This way, the pitch-encoded feature information obtained through fusion encoding retains the tag information, facilitating the differentiation of pitch features between different objects. After decoding the pitch-encoded feature information fused with the tag information, supervised training of the pitch-decoded feature information based on the score corresponding to the target text makes the target pitch feature information obtained through supervised training closer to the pitch features of each note in the score. This improves the pitch accuracy of the subsequently synthesized audio data.

[0046] In some embodiments, the audio synthesis device can encode the pitch feature information of the score corresponding to the target text to obtain initial pitch encoded feature information, and encode the tag information to obtain tag encoded feature information. The initial pitch encoded feature information and the tag encoded feature information are then fused to obtain the final pitch encoded feature information. Specifically, the audio synthesis device can use an encoder to encode both the pitch feature information and the tag information of the score corresponding to the target text. The encoder can be a neural network, and its structure can be the same as or different from the neural network structure used to encode the phoneme feature information of the target text; for example, it can be a stack of N FFT blocks. The initial pitch encoded feature information obtained by encoding the pitch feature information of the target text can be a pitch embedding feature. Similarly, the tag information obtained by encoding the tag information can be a tag embedding feature. After obtaining the initial pitch coding feature information and the tag coding feature information, the audio synthesis device can perform fusion processing on the initial pitch coding feature information and the tag coding feature information in ways including but not limited to concatenating the initial pitch coding feature information and the tag coding feature information, superimposing the initial pitch coding feature information and the tag coding feature information, or multiplying the initial pitch coding feature information and the tag coding feature information.

[0047] In some embodiments, after the audio synthesis device obtains the pitch-encoded feature information, it can decode the pitch-encoded feature information using a decoder to obtain the pitch-decoded feature information. The decoder can be a neural network, and the structure of the neural network can be, for example, a stack of N FFT blocks, each FFT block consisting of a self-attention module and a conv-1D module.

[0048] In some embodiments, after obtaining pitch decoding feature information, the audio synthesis device can perform supervised training on the pitch decoding feature information based on the score corresponding to the target text to obtain target pitch feature information. Specifically, the audio synthesis device can perform supervised training on the pitch decoding feature information based on the score corresponding to the target text using a second loss function until the second loss function satisfies the second constraint condition, and then determine the pitch decoding feature information obtained from the supervised training as the target pitch feature information. Here, the second loss function used for supervised training of the pitch decoding feature information can be the same as or different from the first loss function used for supervised training of the phoneme decoding feature information. The way the audio synthesis device supervises training the pitch decoding feature information based on the score corresponding to the target text using the second loss function can be by inputting the score corresponding to the target text and the pitch decoding feature information into a pitch predictor (also called a pitch prediction model), with the goal of optimizing the pitch predictor so that the second loss function satisfies the second constraint condition. The second loss function can be a distance function, such as the L1-norm (also called Manhattan distance) function. It is understood that in this embodiment, the second loss function can also be other functions, as long as the function can be used to measure the degree of difference between the predicted pitch feature information and the standard pitch feature information corresponding to the musical score. For example, the second loss function can also be an L2-norm (also known as Euclidean distance) function. This application does not specifically limit the second loss function, the structure of the pitch predictor, or the content of the second constraint. The target pitch feature information obtained in this way can better control the pitch of the subsequently synthesized audio data, which is beneficial to improving the quality of the synthesized audio data.

[0049] S203, the target phoneme feature information and the target pitch feature information are spliced ​​together to obtain spliced ​​feature information.

[0050] In this embodiment, after obtaining the target phoneme feature information and the target pitch feature information, the audio synthesis device can concatenate the target phoneme feature information and the target pitch feature information to obtain concatenated feature information. The target phoneme feature information can be a phoneme embedding feature, the target pitch feature information can be a pitch embedding feature, and the concatenated feature information can be obtained by concatenating the phoneme embedding feature and the pitch embedding feature.

[0051] In some embodiments, the target phoneme feature information and target pitch feature information can be concatenated vertically. For example, assuming the target phoneme feature information is a 1*200 dimensional vector and the target pitch feature information is also a 1*200 dimensional vector, the concatenated feature information obtained after vertically concatenating the target phoneme feature information and target pitch feature information is a 2*200 dimensional matrix.

[0052] In some embodiments, the method for concatenating the target phoneme feature information and the target pitch feature information can also be to concatenate them horizontally. For example, assuming that the target phoneme feature information is a 1*100-dimensional vector and the target pitch feature information is also a 1*100-dimensional vector, the concatenated feature information obtained after horizontally concatenating the target phoneme feature information and the target pitch feature information is a 1*200-dimensional vector.

[0053] S204, Generate the predicted Mel spectrum based on the splicing feature information.

[0054] In this embodiment, the audio synthesis apparatus can generate a predicted Mel spectrum based on splicing feature information. The Mel spectrum is a spectrum obtained by performing a Fourier transform on the acoustic signal and then a Mel-scale transform. After obtaining the Mel spectrum, a vocoder is used to perform a time-domain transformation on the Mel spectrum to obtain the synthesized audio data.

[0055] In some embodiments, the audio synthesis device inputs the spliced ​​feature information obtained by splicing the target phoneme feature information and the target pitch feature information into a Mel spectrum predictor (also known as a Mel spectrum prediction model) to generate a predicted Mel spectrum. The Mel spectrum predictor can be a multi-layer FC-layer, or it can be a recurrent neural network such as a recurrent neural network (RNN).

[0056] S205, the predicted Mel spectrum is converted by a vocoder to obtain audio data that matches the musical score corresponding to the target text.

[0057] In this embodiment, the audio synthesis apparatus converts the predicted Mel spectrum using a vocoder to obtain audio data that matches the musical score corresponding to the target text. Specifically, the audio data matching the musical score corresponding to the target text is the audio data synthesized using the text content of the target text as lyrics and the musical score matching the lyrics as accompaniment melody.

[0058] In some embodiments, the vocoder can be a conventional vocoder, such as a STRAIGHT vocoder or a WORLD vocoder, or a neural vocoder, such as a WaveRnn (waveform recurrent neural network) vocoder, a WaveNet (waveform neural network) vocoder, a WaveGlow (a network that synthesizes high-quality speech from a Mel spectrogram based on a stream) vocoder, or other types of vocoders. This application embodiment does not limit this.

[0059] For ease of understanding, the following uses audio synthesis devices as terminal devices and servers respectively to illustrate the implementation of the audio synthesis method provided in the embodiments of this application.

[0060] Example 1: The audio synthesis device can be a terminal device, which can have an audio synthesis application installed and running. The terminal device, through the audio synthesis application, can obtain initial audio data generated by an object in relation to a target text, as well as the object's identification information. From the obtained initial audio data, the terminal device can extract the phoneme feature information of the object in relation to the target text and determine the object's identification information as the tag information of the target text. Based on the text content of the target text, the terminal device can obtain the corresponding musical score, then determine the pitch feature information of the score, and subsequently, the terminal device can use the aforementioned audio synthesis method to synthesize audio data matching the musical score corresponding to the target text.

[0061] Example 2: The audio synthesis device can be a server. The server can obtain the phoneme feature information of the target text, the tag information of the target text, and the pitch feature information of the musical score corresponding to the target text from a database or other devices. Then, the server can use the above-mentioned audio synthesis method to synthesize audio data that matches the musical score corresponding to the target text.

[0062] In this embodiment, the audio synthesis device obtains target phoneme feature information by fusing the phoneme feature information of the target text with the tag information corresponding to the target text, and obtains target pitch feature information by fusing the pitch feature information of the musical score corresponding to the target text with the tag information. This ensures that the target phoneme feature information and the target pitch feature information used to synthesize audio data retain sufficient tag information, which is beneficial for better distinguishing the timbre of different objects and improving the timbre stability of the synthesized audio data.

[0063] In some embodiments, the audio synthesis method provided in this application can be executed by an audio synthesis model, the overall architecture of which is as follows: Figure 3 As shown.

[0064] Specifically, in the audio synthesis model, the phoneme feature information of the target text is fused and encoded with the corresponding label information by the encoder, resulting in phoneme-encoded feature information. The decoder then decodes this phoneme-encoded feature information to obtain phoneme-decoded feature information. Adding a classifier to the audio synthesis model allows for supervised training of the phoneme-decoded feature information based on the label information, thus obtaining the target phoneme feature information. Similarly, the pitch feature information of the musical score corresponding to the target text is fused and encoded with the label information by the encoder, resulting in pitch-encoded feature information. The decoder then decodes this pitch-encoded feature information to obtain pitch-decoded feature information. Adding a pitch predictor to the audio synthesis model allows for supervised training of the pitch-decoded feature information based on the musical score corresponding to the target text, thus obtaining the target pitch feature information. After obtaining the target phoneme feature information and the target pitch feature information, they can be concatenated to obtain concatenated feature information. This concatenated feature information is then used as input by a Mel spectrum predictor to output a predicted Mel spectrum. Finally, the synthesized audio data can be obtained by performing time-domain transformation on the Mel spectrum using a vocoder.

[0065] In some embodiments, the encoder in the audio synthesis model can be composed of N stacked FFT blocks, and the decoder can also be composed of N stacked FFT blocks. A duration predictor can be added between the encoder and decoder. This duration predictor can length-extend the phoneme-coded feature information, thereby matching the length-extended phoneme-coded feature information with the spectral length required for synthesizing the audio data. This facilitates the synthesis of audio data with more uniform tempo and rhythm.

[0066] In some embodiments, the audio synthesis method provided in this application can be applied to Figure 4 The application scenarios shown are as follows. Figure 4As shown in the figure, an audio synthesis application program can be installed on the terminal device 400. After an object logs in to the audio synthesis application program, the object can record initial audio data for a target text (i.e., lyrics). For example, if the target text is "Spring slumber goes unawares", the initial audio data can be generated by the object reading the target text or singing the target text. When detecting a touch operation on the audio synthesis control 41 by the object, the terminal device 400 can send an audio chorus request to the server. The audio synthesis request includes the initial audio data of the object collected by the terminal device 400 and the identification information of the object. The identification information of the object can be, for example, the account information of the object logging in to the audio synthesis application. After receiving the audio chorus request, the server can extract the phoneme feature information of the target text from the initial audio data of the object and determine the identification information of the object as the label information corresponding to the target text. It can be understood that for the same musical score, the server can obtain the phoneme feature information of different objects for the target text. The server performs feature fusion processing on the phoneme feature information of different objects for the target text and the label information corresponding to the target text to obtain target phoneme feature information, and performs feature fusion processing on the pitch feature reputation label information of the musical score corresponding to the target text to obtain target pitch feature information. Finally, according to the target phoneme feature information and the target pitch feature information, complete audio data matching the musical score corresponding to the target text is synthesized. Since the server尽可能保留了目标文本对应的标签信息,也即是不同对象的标识信息,使得最终合成的音频数据可以保留到不同对象的音色特征,从而有利于提升合成的音频数据的音色稳定性。

[0067] It should be noted that the execution carrier for executing each step in the above method embodiments can be composed of hardware, can also be composed of software, or can be composed of a combination of software and hardware. In the specific implementation of this application, regarding relevant data such as phoneme feature information, label information, and initial audio data, the data involved and used are all authorized by the object. When the above embodiments of this application are applied to specific products or technologies, the data involved and used need to obtain the permission or consent of the object, and the collection, use, and processing of relevant data need to comply with the relevant regulations and standards of relevant countries and regions.

[0068] Furthermore, please refer to Figure 5 , which is a schematic structural diagram of an audio synthesis device provided by an embodiment of this application. The audio synthesis device 50 can include:

[0069] An acquisition unit 501, configured to acquire phoneme feature information of a target text and label information corresponding to the target text;

[0070] It should be noted that the content in the double brackets "尽可能保留了目标文本对应的标签信息,也即是不同对象的标识信息,使得最终合成的音频数据可以保留到不同对象的音色特征,从而有利于提升合成的音频数据的音色稳定性。" in the translation of seems to be incorrect or incomplete in the original Chinese. It is recommended to check and correct the original text for a more accurate translation.The processing unit 502 is used to perform feature fusion processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain target phoneme feature information; perform feature fusion processing on the pitch feature information of the musical score corresponding to the target text and the tag information to obtain target pitch feature information; perform concatenation processing on the target phoneme feature information and the target pitch feature information to obtain concatenated feature information; generate a predicted Mel spectrum based on the concatenated feature information; and perform conversion processing on the predicted Mel spectrum through a vocoder to obtain audio data matching the musical score corresponding to the target text.

[0071] In some embodiments, the processing unit 502 is further configured to perform fusion encoding processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain phoneme encoded feature information; perform decoding processing on the phoneme encoded feature information to obtain phoneme decoded feature information; and perform supervised training on the phoneme decoded feature information based on the tag information to obtain target phoneme feature information.

[0072] In some embodiments, the processing unit 502 is further configured to perform fusion encoding processing on the pitch feature information of the musical score corresponding to the target text and the tag information to obtain pitch encoded feature information; perform decoding processing on the pitch encoded feature information to obtain pitch decoded feature information; and perform supervised training on the pitch decoded feature information based on the musical score corresponding to the target text to obtain target pitch feature information.

[0073] In some embodiments, the processing unit 502 is further configured to encode the phoneme feature information of the target text to obtain initial phoneme encoded feature information; encode the tag information corresponding to the target text to obtain tag encoded feature information; concatenate the initial phoneme encoded feature information with the tag encoded feature information to obtain phoneme encoded feature information; or superimpose the initial phoneme encoded feature information with the tag encoded feature information to obtain phoneme encoded feature information.

[0074] In some embodiments, the processing unit 502 is further configured to encode the pitch feature information of the musical score corresponding to the target text to obtain initial pitch encoding feature information; encode the tag information to obtain tag encoding feature information; concatenate the initial pitch encoding feature information with the tag encoding feature information to obtain pitch encoding feature information; or superimpose the initial pitch encoding feature information with the tag encoding feature information to obtain pitch encoding feature information.

[0075] In some embodiments, the processing unit 502 is further configured to supervise the training of phoneme decoding feature information based on the label information using a first loss function until the first loss function satisfies the first constraint condition, and determine the phoneme decoding feature information obtained by supervised training as the target phoneme feature information.

[0076] In some embodiments, the processing unit 502 is further configured to supervise the training of pitch decoding feature information based on the musical score corresponding to the target text using a second loss function until the second loss function satisfies the second constraint condition, and then determine the pitch decoding feature information obtained by supervised training as the target pitch feature information.

[0077] In some embodiments, the processing unit 502 is further configured to perform duration prediction processing on the phoneme coding feature information to obtain predicted duration information corresponding to the phoneme coding feature information; perform extension processing on the phoneme coding feature information according to the predicted duration information to obtain extended phoneme feature information; and perform decoding processing on the extended phoneme feature information to obtain decoded phoneme feature information.

[0078] In some embodiments, the acquisition unit 501 is further configured to acquire the initial audio data generated by the target object for the target text; the processing unit 502 is further configured to extract the phoneme feature information of the target text from the initial audio data, and determine the identification information of the target object as the tag information corresponding to the target text.

[0079] It should be noted that, Figure 5 For details not mentioned in the corresponding embodiments and the specific implementation methods of each step, please refer to [link to relevant documentation]. Figures 2-4 The embodiments shown and the foregoing content will not be repeated here.

[0080] For further details, please see Figure 6 , Figure 6This is a schematic diagram of the structure of a computer device 60 provided in an embodiment of this application. The computer device may include a network interface 601, a memory 602, and a processor 603. The network interface 601, memory 602, and processor 603 are connected via one or more communication buses, which are used to enable communication between these components. The network interface 601 may include a standard wired interface or a wireless interface (such as a WIFI interface). The memory 602 may include volatile memory, such as random-access memory (RAM); the memory 602 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 602 may also include a combination of the above types of memory. The processor 603 may be a central processing unit (CPU). The processor 603 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The aforementioned PLD can be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.

[0081] Optionally, the memory 602 is also used to store program instructions, which the processor 603 can also call to implement the relevant methods and steps in this application.

[0082] In some embodiments, the processor 603 calls program instructions stored in the memory 602 to: perform feature fusion processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain target phoneme feature information; perform feature fusion processing on the pitch feature information of the musical score corresponding to the target text and the tag information to obtain target pitch feature information; perform concatenation processing on the target phoneme feature information and the target pitch feature information to obtain concatenated feature information; generate a predicted Mel spectrum based on the concatenated feature information; and perform conversion processing on the predicted Mel spectrum through a vocoder to obtain audio data matching the musical score corresponding to the target text.

[0083] In some embodiments, the processor 603 may also invoke the program instructions to: concatenate the target phoneme feature information and the target pitch feature information to obtain concatenated feature information; generate a predicted Mel spectrum based on the concatenated feature information; and convert the predicted Mel spectrum using a vocoder to obtain audio data that matches the score corresponding to the target text.

[0084] In some embodiments, the processor 603 may also call the program instructions to: perform fusion encoding processing on the phoneme feature information of the target text and the tag information corresponding to the target text to obtain phoneme encoded feature information; perform decoding processing on the phoneme encoded feature information to obtain phoneme decoded feature information; and perform supervised training on the phoneme decoded feature information based on the tag information to obtain target phoneme feature information.

[0085] In some embodiments, the processor 603 may also call the program instructions to: perform fusion encoding processing on the pitch feature information of the score corresponding to the target text and the tag information to obtain pitch encoded feature information; perform decoding processing on the pitch encoded feature information to obtain pitch decoded feature information; and perform supervised training on the pitch decoded feature information based on the score corresponding to the target text to obtain target pitch feature information.

[0086] In some embodiments, the processor 603 may also invoke the program instructions to: encode the phoneme feature information of the target text to obtain initial phoneme encoded feature information; encode the tag information corresponding to the target text to obtain tag encoded feature information; concatenate the initial phoneme encoded feature information with the tag encoded feature information to obtain phoneme encoded feature information; or superimpose the initial phoneme encoded feature information with the tag encoded feature information to obtain phoneme encoded feature information.

[0087] In some embodiments, the processor 603 may also invoke the program instructions to: encode the pitch feature information of the musical score corresponding to the target text to obtain initial pitch encoding feature information; encode the tag information to obtain tag encoding feature information; concatenate the initial pitch encoding feature information with the tag encoding feature information to obtain pitch encoding feature information; or superimpose the initial pitch encoding feature information with the tag encoding feature information to obtain pitch encoding feature information.

[0088] In some embodiments, the processor 603 may also call the program instructions to: perform supervised training on the phoneme decoding feature information based on the tag information using a first loss function until the first loss function satisfies the first constraint condition, and determine the phoneme decoding feature information obtained from the supervised training as the target phoneme feature information.

[0089] In some embodiments, the processor 603 may also call the program instructions to: supervise the training of pitch decoding feature information based on the musical score corresponding to the target text using a second loss function until the second loss function satisfies the second constraint condition, and determine the pitch decoding feature information obtained by supervised training as the target pitch feature information.

[0090] In some embodiments, the processor 603 may also call the program instructions to: perform duration prediction processing on the phoneme coding feature information to obtain the predicted duration information corresponding to the phoneme coding feature information; perform extension processing on the phoneme coding feature information according to the predicted duration information to obtain extended phoneme feature information; and perform decoding processing on the extended phoneme feature information to obtain decoded phoneme feature information.

[0091] In some embodiments, the processor 603 may also invoke the program instructions to: acquire the initial audio data generated by the target object for the target text; extract the phoneme feature information of the target text from the initial audio data; and determine the identification information of the target object as the tag information corresponding to the target text.

[0092] It should be understood that the principles and beneficial effects of the computer device 60 in solving the problem described in the embodiments of this application are the same as those in this application. Figures 2-4 The embodiments shown and the foregoing content have similar principles and beneficial effects in solving the problem, and will not be repeated here for the sake of brevity.

[0093] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods described in the embodiments of this application.

[0094] Accordingly, this application also provides a computer program product, which includes a computer program or computer instructions, which, when executed by a processor, implement the methods in this application.

[0095] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0096] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An audio synthesis method, characterized by, The method comprises: Fusing and encoding phoneme feature information of a target text and label information corresponding to the target text to obtain phoneme encoded feature information; the label information corresponding to the target text refers to identification information of a target object generating initial audio data of the target text, and the initial audio data is used to extract the phoneme feature information of the target text; Decoding the phoneme encoded feature information to obtain phoneme decoded feature information; Supervising and training the phoneme decoded feature information according to the label information to obtain target phoneme feature information; Fusing and encoding pitch feature information of a score corresponding to the target text and the label information to obtain pitch encoded feature information; Decoding the pitch encoded feature information to obtain pitch decoded feature information; Supervising and training the pitch decoded feature information according to the score corresponding to the target text to obtain target pitch feature information; Splicing the target phoneme feature information and the target pitch feature information to obtain spliced feature information; Generating predicted mel spectrum according to the spliced feature information; Converting the predicted mel spectrum through a vocoder to obtain audio data matching the score corresponding to the target text.

2. The method of claim 1, wherein, The method comprises: Encoding the phoneme feature information of the target text to obtain initial phoneme encoded feature information; Encoding the label information corresponding to the target text to obtain label encoded feature information; Splicing or superimposing the initial phoneme encoded feature information and the label encoded feature information to obtain phoneme encoded feature information. The method comprises:

3. The method of claim 1, wherein, Encoding the pitch feature information of the score corresponding to the target text to obtain initial pitch encoded feature information; Encoding the label information to obtain label encoded feature information; Splicing or superimposing the initial pitch encoded feature information and the label encoded feature information to obtain pitch encoded feature information. The method comprises: Supervising and training the phoneme decoded feature information according to the label information through a first loss function until the first loss function meets a first constraint condition, and determining the phoneme decoded feature information obtained through the supervision and training as target phoneme feature information.

4. The method of claim 1, wherein, The method comprises: ​ ​ The pitch decoding feature information is supervised trained according to the score corresponding to the target text by a second loss function until the second loss function meets a second constraint condition, and the pitch decoding feature information obtained by the supervised training is determined as target pitch feature information.

5. The method of claim 1, wherein, The decoding processing on the phoneme coding feature information is performed to obtain phoneme decoding feature information, including: The duration prediction processing is performed on the phoneme coding feature information to obtain predicted duration information corresponding to the phoneme coding feature information; The extension processing is performed on the phoneme coding feature information according to the predicted duration information to obtain phoneme extension feature information; The decoding processing is performed on the phoneme extension feature information to obtain phoneme decoding feature information.

6. The method according to any one of claims 1 to 5, wherein, The method further includes: obtaining initial audio data generated by a target object for a target text; extracting phoneme feature information of the target text from the initial audio data, and determining identification information of the target object as label information corresponding to the target text.

7. A computer device, comprising: including: a processor, a communication interface and a memory, the processor, the communication interface and the memory are connected with each other, wherein the memory stores executable program code, the processor is used to call the executable program code to realize the method in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method in any one of claims 1-6.

9. A computer program product, characterised in that, The computer program product includes computer program or computer instruction, and the computer program or computer instruction is executed by the processor to realize the method in any one of claims 1-6.

Images