Audio processing method, device, equipment and product

Abstract

The invention discloses an audio processing method, device, equipment and product, and belongs to the technical field of audio processing. The audio processing method comprises the following steps: acquiring a first audio and a text corresponding to the first audio; determining the phoneme duration of each phoneme in a phoneme sequence corresponding to the text according to the first audio; according to the text, the phoneme duration of each phoneme and the language information corresponding to the text, prosodic features of the first audio are predicted; performing timbre extraction on the first audio to obtain timbre features of the first audio; and performing synthesis processing on the text, the timbre feature and the rhythm feature to obtain a second audio.

Description

Technical Field

[0001] The present application belongs to the field of audio processing technology, and specifically relates to an audio processing method, device, equipment and product. Background Art

[0002] Audio synthesis refers to the use of artificial intelligence models to learn and train on large-scale data to imitate the audio of a specified object so that it can be reproduced on electronic devices or media.

[0003] Current audio synthesis technology only relies on the audio of specified objects, so it requires a large amount of training corpus, resulting in a training scale of several hours or even dozens of hours to obtain relatively stable results, thereby realizing audio synthesis through artificial intelligence models.

[0004] It can be seen that the current audio synthesis technology takes a long time and has low efficiency. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an audio processing method, device, equipment and product, which can solve the technical problem of low efficiency in synthesizing audio in related technologies.

[0006] In a first aspect, an embodiment of the present application provides an audio processing method, comprising:

[0007] Acquire a first audio and a text corresponding to the first audio;

[0008] Determine the phoneme duration of each phoneme in the phoneme sequence corresponding to the text based on the first audio; predict the prosodic features of the first audio based on the text, the phoneme duration of each phoneme and the language information corresponding to the text;

[0009] Performing timbre extraction on the first audio to obtain timbre characteristics of the first audio;

[0010] The text, the timbre features and the rhythm features are synthesized to obtain a second audio.

[0011] In a second aspect, an embodiment of the present application provides an audio processing device, including:

[0012] An acquisition module, used to acquire a first audio and a text corresponding to the first audio;

[0013] A determination module, configured to determine the phoneme duration of each phoneme in a phoneme sequence corresponding to the text according to the first audio;

[0014] A prediction module, configured to predict a prosodic feature of the first audio according to the text, the phoneme duration of each phoneme, and language information corresponding to the text;

[0015] An extraction module, used to extract the timbre of the first audio to obtain the timbre characteristics of the first audio;

[0016] The synthesis module is used to synthesize the text, timbre features and rhythm features to obtain the second audio.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0021] In an embodiment of the present application, a first audio and a text corresponding to the first audio are obtained; the phoneme duration of each phoneme in the phoneme sequence corresponding to the text is determined based on the first audio; the rhythmic features of the first audio are predicted based on the text, the phoneme duration of each phoneme, and the language information corresponding to the text; the timbre of the first audio is extracted to obtain the timbre features of the first audio; the text, timbre features, and rhythmic features are synthesized to obtain a second audio. Based on the audio, an embodiment of the present application introduces the text corresponding to the audio, and based on the audio, the phoneme duration of each phoneme corresponding to the text is obtained, and the user's speaking habits and rhythmic style can be learned by combining the sound-character relationship, so that the re-synthesized audio is as close to the user's real audio as possible, without the need for a large amount of audio training data, thereby improving the synthesis efficiency of the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the structure of a global timbre feature extraction model provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0025] Figure 4 A flowchart of a training process of a duration prediction model provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of alignment of a sample audio, a sample audio text and a sample phoneme sequence provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of another audio processing method provided in an embodiment of the present application;

[0028] Figure 7 A flowchart of another audio processing method provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a prosodic feature prediction process provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] The audio processing method provided by the embodiments of this application can be applied to scenarios where audio needs to be synthesized, such as voice navigation, novel reading, simultaneous interpretation, consultation broadcast, short video production, film and television dubbing, virtual human voice generation, etc. Specifically, for a piece of audio, through the method of the embodiments of this application, an audio that is as similar as possible to the timbre, rhythm, etc. of the audio can be synthesized.

[0036] For example, user A enters the audio "I'm so happy today. Hurry up and go on a trip!" in an electronic device. During the audio recording process, "zhen shi" (true) is read as "zhen si" and "chu" (out) is read as "cu". Using the method of the embodiments of this application, the synthesized audio obtained will also read "zhen shi" as "zhen si" and "chu" as "cu", that is, the enunciation style is the same as the audio entered by the user.

[0037] Another example is that user B said "The weather is so bad today. I'm a little unhappy", with a rather sad tone. Then, for this audio, using the method of the embodiments of this application, the tone of the synthesized audio obtained is also sad.

[0038] The execution subject of the audio processing method provided by the embodiments of this application can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, etc. In some embodiments of this application, taking an electronic device as the execution subject to execute the information display method as an example, the audio processing method provided by the embodiments of this application is described.

[0039] Next, in conjunction with the accompanying drawings, the audio processing method provided by the embodiments of this application will be described in detail through specific embodiments and their application scenarios.

[0040] Figure 1 It is a flowchart of an audio processing method provided by the embodiments of this application. As Figure 1 shown, the audio processing method may include the following steps:

[0041] S110. Obtain a first audio and the text corresponding to the first audio.

[0042] S120. Determine the phoneme duration of each phoneme in the phoneme sequence corresponding to the text according to the first audio.

[0043] S130. Predict the prosodic features of the first audio according to the text, the phoneme duration of each phoneme, and the language information corresponding to the text.

[0044] S140. Extract the timbre of the first audio to obtain the timbre feature of the first audio.

[0045] S150. Perform synthesis processing on the text, the timbre feature, and the prosodic features to obtain a second audio.

[0046] Based on the audio, the embodiment of the present application introduces the text corresponding to the audio, and obtains the phoneme duration of each phoneme according to the audio and the text, and learns the speaking habits and prosody styles of the user in combination with the relationship between Chinese characters and phonemes, so that the re-synthesized audio is as close as possible to the user's real audio, thereby improving the synthesis efficiency of the audio.

[0047] The above steps will be described in detail as follows:

[0048] In S110, the first audio here can be an audio segment pre-recorded by the user, or an audio segment intercepted from a video, or an audio of the user collected in real time. The duration of the first audio can be, for example, 5s, 10s, etc. The embodiment of the present application does not limit the language of the first audio. For example, it can be a Chinese audio, an English audio, a Chinese-English audio, or an audio in other languages.

[0049] The text here is the text corresponding to the first audio. Exemplarily, the first audio can be speech-recognized by a speech recognition model to obtain the text corresponding to the first audio. The speech recognition model can adopt, for example, the whisper model, or other models, as long as it can perform speech recognition on the first audio to obtain the text corresponding to the first audio.

[0050] Considering that the first audio may contain white noise or background noise, which causes great interference to the first audio and further affects the subsequent audio synthesis effect. In some embodiments, before recognizing the first audio, the first audio is first denoised, and then the denoised first audio is speech-recognized to obtain the corresponding text.

[0051] In S120, exemplarily, each Chinese character in the text can be converted into pinyin. For example, the Chinese character "中" can be converted into pinyin "zhong1". If the text contains English, each English word can be kept unchanged, that is, the English words are not converted.

[0052] After obtaining the converted text, the phoneme corresponding to each pinyin or word can be determined in combination with a pronunciation dictionary. The pronunciation dictionary is used to store the mapping relationship between pinyin or words and phonemes. Table 1 exemplarily lists the mapping relationship between the English word "human" and the Chinese character "中" and phonemes:

[0053] Table 1 Mapping relationship between some pinyin or words and phonemes

[0054] Words\Pinyin phoneme human HH Y UW1 M AH0 N zhong1 zh ong1

[0055] Arrange the phonemes in the order of each word or pinyin in the audio text, and the phoneme sequence can be obtained.

[0056] In some embodiments, before converting the text into a phoneme sequence, the text may be processed. For example, text normalization may be used to normalize expressions, abbreviations, punctuation marks, dates, etc. in the text to obtain normalized text. For example, the date 08 / 23 may be processed into August 23. The normalized text is then converted into a phoneme sequence in the manner of the above embodiment.

[0057] After the phoneme sequence is obtained, the phoneme duration of each phoneme can be determined in combination with the first audio.

[0058] For example, the duration prediction model may be used to determine the phoneme duration of each phoneme. Specifically, the first audio and the phoneme sequence may be input into the duration prediction model, and the phoneme duration of each phoneme may be output.

[0059] For another example, the first audio and the phoneme sequence may be aligned using an alignment method to obtain the position of each phoneme in the first audio, and then obtain the phoneme duration of each phoneme according to the position.

[0060] In S130 , the prosodic features are used to characterize changes in the strength, duration, tone, and speech speed of the voice.

[0061] According to the text, the phoneme duration of each phoneme, and the language information corresponding to the text, the prosodic features of the first audio can be predicted, providing a basis for the subsequent synthesis of new audio. The embodiment of the present application does not limit the specific prediction process. For example, the text, the phoneme duration of each phoneme, and the language information corresponding to the text can be input into the prosodic prediction model to predict the prosodic features of the first audio.

[0062] For another example, the text, the phoneme duration of each phoneme, and the language information corresponding to the text may be matched with a prosody template to obtain the prosody feature of the first audio. The prosody template is used to store the association between the text, the phoneme duration, the language information, and the prosody feature.

[0063] In S140, the timbre feature is used to characterize the quality of the sound, the pronunciation style or the speaking style, etc. The embodiment of the present application does not limit the timbre extraction process.

[0064] Exemplarily, a global timbre feature extraction model may be used to extract a global timbre feature of the first audio, where the global timbre feature is used to characterize the overall timbre feature of the first audio.

[0065] It should be understood that, generally speaking, the user's timbre is globally stable and unchanged, and the sound frequency of each user only changes within a certain range, and this frequency range is narrow and fixed. Although the user's timbre is stable as a whole, it is considered that in some cases, the timbre may vary due to the different oral structures and positions when speaking. For example, speaking different languages, having large emotional fluctuations, speaking fast or slow speeds, etc., will cause the timbre to change to a certain extent due to the different positions of the mouth, tongue, lips, etc. Based on this, illustratively, the local timbre features of the first audio can also be extracted using a local timbre feature extraction model, and the local timbre features are used to characterize the local timbre features of the first audio.

[0066] Exemplarily, the local timbre feature extraction model may be used to extract the timbre feature of a certain audio segment in the first audio to obtain the local timbre feature.

[0067] Exemplarily, the global timbre features and local timbre features of the first audio may also be extracted simultaneously, so as to capture both the overall characteristics of the timbre and the local timbre changes, thereby making the resynthesized audio more similar to the original audio.

[0068] The structures of the global timbre feature extraction model and the local timbre feature extraction model can be similar. Taking the global timbre feature extraction model as an example, Figure 2 As shown, the global timbre feature extraction model 200 may include a linear layer 201, a normalization layer 202 and a convolutional layer 203. The convolutional layer 203 may be formed by stacking a plurality of convolutional neural network (CNN) blocks. In some embodiments, the convolutional layer 203 may be formed by stacking 4 CNN blocks.

[0069] Before the first audio is input into the global timbre feature extraction model 200, the first audio may be dimensionally transformed, for example, the first audio may be converted into a vector of dimension T*D, where T represents the time length of the first audio and D represents the feature dimension of the vector. Through the global timbre feature extraction model 200, the first audio of dimension T*D may be compressed into a feature vector of 1*D for representing the global timbre feature.

[0070] In S150, the text, the timbre feature and the rhythm feature are synthesized to obtain a second audio, which is a synthesized audio of the first audio.

[0071] Exemplarily, text, timbre features, and prosody features may be input into an audio synthesis model, and synthesized audio may be output as the second audio.

[0072] Exemplarily, feature vectors corresponding to text, timbre features, and rhythmic features may be concatenated to obtain a concatenated feature vector, which may then be processed by a decoder to convert the concatenated feature vector into a Mel spectrum, which may then be processed by a vocoder to convert the Mel spectrum into audio as synthesized audio.

[0073] The decoder can use Figure 3 The structure shown may also adopt other structures in actual application, and the embodiments of the present application are not limited thereto.

[0074] like Figure 3 As shown, the decoder 300 may include multiple stacked decoding modules 301. In some embodiments, the decoder 300 may include 4 stacked decoding modules 301. For example, the decoding module 301 may include a linear layer 3011, a multi-head attention layer 3012, a convolutional layer 3013, and a normalization layer 3014. The concatenated feature vector is input into the decoder 300 to output a Mel spectrum. The role of each layer can be referred to in the relevant technology, which will not be repeated here.

[0075] Through the above solution, the timbre and rhythm of the second audio can be made as similar as possible to the timbre and rhythm of the first audio, thereby improving the audio synthesis effect.

[0076] In some embodiments, the above S120 may include the following steps:

[0077] The first audio and the phoneme sequence corresponding to the text are input into a second duration prediction model, and the phoneme duration of each phoneme in the phoneme sequence is output.

[0078] The second duration prediction model is a trained model for predicting the phoneme duration of each phoneme in a phoneme sequence. Different phoneme durations can represent different speaking styles of users. Exemplarily, the second duration prediction model can adopt a model combining a Gaussian Mixture Model (GMM) and Deep Learning (DL), or other models that can predict phoneme duration. The second duration prediction model can automatically obtain the phoneme duration of each phoneme, simplifying the user's operation and improving the accuracy of the result.

[0079] The training process of the second duration prediction model is described below. Figure 4 A flowchart of a training process of a duration prediction model provided in an embodiment of the present application, the training process may be performed before S110, such as Figure 4 As shown, the training process may include the following S410-S440:

[0080] S410: Obtain multiple duration training samples.

[0081] Each duration training sample includes sample audio, a sample phoneme sequence corresponding to the sample text, and a sample duration corresponding to the sample phoneme sequence, and the sample text is the text corresponding to the sample audio.

[0082] Exemplarily, the duration training sample can be obtained in the following manner:

[0083] Obtaining sample audio and sample text corresponding to the sample audio;

[0084] Segmenting the sample text according to the language information of the sample text to obtain at least one segmented text;

[0085] Searching for each segmented text to obtain at least one phoneme corresponding to each segmented text; concatenating each phoneme corresponding to each segmentation according to the order of each segmented text in the sample text to obtain a sample phoneme sequence corresponding to the sample text;

[0086] Determine the starting position and ending position of each phoneme in the sample phoneme sequence from the sample audio; obtain the sample duration of each phoneme according to the time difference between the starting position and the ending position of each phoneme;

[0087] The sample audio, the sample phoneme sequence and the sample duration corresponding to the sample phoneme sequence are determined as duration training samples.

[0088] There are many ways to obtain sample audios, and the duration of each sample audio can be the same or different.

[0089] In the model training stage, in order to more conveniently determine the relationship between phonetic characters, the sample text may be segmented according to the language information of the sample text to obtain at least one segmented text.

[0090] For example, when the sample text includes Chinese text, each character in the Chinese text can be determined as a segmented text, and when the sample text includes English text, each word in the English text can be determined as a segmented text. Exemplarily, when the segmented text is Chinese characters, the Chinese characters can be further converted into splicing, and when the segmented text is English, the segmented text can be kept unchanged.

[0091] For each segmented text, the pronunciation dictionary can be searched according to its pinyin or word to obtain the phonemes corresponding to the segmented text, thereby obtaining a sample phoneme sequence corresponding to the sample text.

[0092] The following is an explanation of the process of determining the sample duration of each phoneme, as shown below:

[0093] Exemplarily, the converted sample text and sample audio can be aligned using the Montreal Forced Aligner (MFA), where the converted sample audio text is text containing pinyin or words. After alignment, the start position and end position of each phoneme in the sample audio can be determined by combining the sample phoneme sequence and the sample audio. Based on the time difference between the end position and the start position, the sample duration of each phoneme can be obtained.

[0094] Taking the sample text "哇塞" as an example, where the pinyin of the Chinese character "哇" is "wa5", and the corresponding phoneme is "ua5", and the pinyin of the Chinese character "塞" is "sail", and the corresponding phoneme is "s ail". Figure 5 Exemplarily, an alignment schematic diagram of a sample audio, a sample text, and a sample phoneme sequence is listed. Among them, 500 is the waveform of the audio "哇", 501 is the waveform of the audio "塞", 502 is the spectrum of the audio "哇", 503 is the spectrum of the audio "塞", 504 is the pinyin of the audio "哇", 505 is the pinyin of the audio "塞", 506 is the phoneme of the pinyin "wa5", and 507 is the phoneme of the pinyin "sail". Through alignment, the boundary of each phoneme in the sample audio can be determined, and then the sample duration of each phoneme can be obtained.

[0095] In the embodiment of the present application, the alignment relationship between the sample audio and the sample text is obtained from the perspective of pinyin or words, and then the sample duration of each phoneme is obtained by combining the pronunciation dictionary, so that subsequent model training can be carried out based on the sample duration of each phoneme, improving the training effect of the model.

[0096] S420: Input the sample phoneme sequence and the sample audio into the first duration prediction model, and output the predicted duration of each phoneme in the sample phoneme sequence.

[0097] In the specific training process, the sample phoneme sequence and the sample audio can be input into the first duration prediction model, and the predicted duration of each sample phoneme in the sample phoneme sequence is output.

[0098] S430: Determine the duration loss function value according to the sample duration and the predicted duration.

[0099] Exemplarily, according to the predicted duration and the sample duration, that is, the actual duration of each phoneme, combined with the loss function, the duration loss function value can be obtained. The embodiment of the present application does not limit the specific loss function. For example, the mean square error loss function, the mean absolute value loss function, etc. can be used.

[0100] S440: Train the first duration prediction model according to the duration loss function value to obtain the second duration prediction model.

[0101] The model parameters of the model can be updated according to the value of the duration loss function until the value of the duration loss function converges or the number of iterations reaches a threshold, and the training is terminated. At this time, a trained duration prediction model can be obtained.

[0102] Before the actual application of the embodiment of the present application, a first duration prediction model can be trained in combination with the sound-character relationship to obtain a second duration prediction model, so that the second duration prediction model can be used to predict the phoneme duration of each phoneme corresponding to each text to obtain the user's speaking habits, thereby improving the accuracy of the synthesized audio.

[0103] Figure 6 A flowchart of another audio processing method provided in an embodiment of the present application, Figure 6 and Figure 1 The difference is that Figure 1 The S130 in can be refined into Figure 6 S610-S640 in.

[0104] S610: Perform text encoding on the text to obtain a text feature vector corresponding to the text.

[0105] Exemplarily, the text corresponding to the first audio may be encoded using a text encoder to obtain a text feature vector corresponding to the text. Exemplarily, the dimension of the text feature vector is T*D. The text feature vector contains the semantic information of the text.

[0106] S620, vectorizing the phoneme duration of each phoneme to obtain a duration feature vector of each phoneme; and vectorizing the language information corresponding to the text to obtain a language feature vector corresponding to the language information.

[0107] The vectorization processing here is to convert the one-dimensional phoneme duration and language information into vectors of specific dimensions respectively, for example, they can be converted into vectors of dimension T*D, that is, the dimensions of the text feature vector, duration feature vector and language feature vector in the embodiment of the present application are the same.

[0108] For example, the phoneme duration may be input into the embedding layer embedding to obtain a duration feature vector corresponding to each phoneme. Similarly, the language information may be input into the embedding layer embedding to obtain a language feature vector corresponding to the language information.

[0109] In actual application, the embodiment of the present application does not limit the execution order of S610 and S620. For example, S610 may be executed first and then S620, or S620 may be executed first and then S610, or S610 and S620 may be executed simultaneously.

[0110] S630 , concatenating the text feature vector, the duration feature vector of each phoneme, and the language feature vector according to the time dimension to obtain a concatenated feature vector.

[0111] Assuming that the dimensions of the text feature vector, duration feature vector, and language feature vector are all T*D, the dimension of the concatenated feature vector is T*3D.

[0112] S640: Input the concatenated feature vector into a prosody prediction model, and output prosody features of the first audio.

[0113] The prosody prediction model is obtained by training a large language model (LLM) by taking a sample concatenated feature vector as input and a sample prosody feature of a sample audio as output.

[0114] The embodiment of the present application continues training on the basis of the large language model to obtain a prosody prediction model. When subsequently predicting the prosody features of a speaker other than the prosody training sample, there is no need to retrain the model or fine-tune it. Moreover, since the large language model has been pre-trained using a large amount of corpus, when the embodiment of the present application continues training on this basis, it is only necessary to obtain a short period of audio as a training sample.

[0115] The embodiment of the present application combines the semantic information of the text, the duration information of the phonemes and the language information on the basis of the large language model to predict the prosodic features, so that the prediction result is closer to the prosodic style of the user.

[0116] The following is an explanation of the training process of the rhythm prediction model, as shown below:

[0117] In some embodiments, before S110, the audio processing method may further include the following steps:

[0118] Acquire multiple prosody training samples, each prosody training sample includes a sample concatenation feature vector and a sample prosody feature corresponding to the sample audio, the sample concatenation feature vector includes a sample duration feature vector of each phoneme, a sample language feature vector corresponding to the sample text, and a sample text feature vector corresponding to the sample text, the sample text being the text corresponding to the sample audio;

[0119] Inputting a first prosodic feature of the sample audio at a first sampling moment and a sample concatenated vector corresponding to a second sampling moment into a large language model, and outputting a second prosodic feature of the sample audio at a second sampling moment, wherein the first sampling moment and the second sampling moment are sampling moments within a time period corresponding to the sample audio, and the first sampling moment is a moment before the second sampling moment;

[0120] Determine a prosody loss function value according to the second prosody feature and the sample prosody feature of the sample audio at the second sampling moment;

[0121] According to the value of the prosody loss function, a large language model is trained to obtain a prosody prediction model.

[0122] The generation process of the sample concatenation feature vector can refer to the above embodiment, and for the sake of brevity, it will not be repeated here. The sample prosodic feature is the actual prosodic feature contained in the sample audio. In the embodiment of the present application, the prosodic feature of the sample audio at a discrete moment can be obtained, and different prosodic features can be represented by different tokens.

[0123] The first sampling moment and the second sampling moment are respectively discrete sampling moments of the time period corresponding to the sample audio, and the first sampling moment and the second sampling moment are adjacent sampling moments. For example, the first sampling moment may be the moment before the second sampling moment. Of course, in actual application, the first sampling moment may also be the moment after the second sampling moment.

[0124] The first prosodic feature is the predicted prosodic feature of the sample audio at the first sampling moment, and the second prosodic feature is the predicted prosodic feature of the sample audio at the second sampling moment. That is, the embodiment of the present application can predict the prosodic feature of the sample audio at time t based on the prosodic feature predicted before time t combined with the sample concatenation vector at time t, that is, predict the prosodic feature by autoregression, thereby improving the accuracy of the prediction result.

[0125] Exemplarily, the first prosodic feature and the sample concatenation vector corresponding to the second sampling moment are input into the large language model, and the second prosodic feature output by the large language model can be obtained:

[0126] Exemplarily, the prosodic features at time t predicted by the large language model can be determined by the following formula:

[0127]

[0128] where c t represents the sample concatenation vector at time t, u t is the prosodic token predicted at time t, i.e., the second prosodic feature, and θ is the training parameter of the large language model. The main body of the large language model can adopt the Decoder-only GPT structure. The prediction process of prosodic features can be found in Figure 7 , that is, the prosody prediction is performed by autoregression, which improves the accuracy of the prediction results. Here s is empty, ci represents the sample concatenation vector at the i-th sampling moment, 1≤i≤n, n is the number of sampling moments, ui represents the prosody token at the i-th sampling moment, and different prosody tokens represent different prosody features.

[0129] According to the second prosodic feature output by the large language model and the actual prosodic feature corresponding to the same time, combined with the prosodic loss function, the prosodic loss function value can be determined. According to the prosodic loss function value, the model parameter θ of the large language model can be updated until the training termination condition is met to obtain the prosodic prediction model. Among them, the prosodic loss function can adopt the mean square error loss function, the mean absolute value loss function, etc. The training termination condition can be, for example, that the prosodic loss function value tends to be stable, or the number of iterations reaches a preset number threshold.

[0130] The embodiment of the present application performs model training by autoregression on the basis of a large language model, so that in subsequent prosody prediction, the contextual information of the audio can be fully considered to obtain a prosodic style close to the user.

[0131] Figure 8 A flowchart of another audio processing method provided in an embodiment of the present application, Figure 8 and Figure 1 The difference is that Figure 1 The S140 in can be refined into Figure 8 S810-S830 in.

[0132] S810: Perform timbre extraction on the first audio to obtain a global timbre feature of the first audio.

[0133] Exemplarily, the global timbre feature extraction model can be used to extract the global timbre of the first audio to obtain the global timbre feature of the first audio. The structure of the global timbre feature extraction model can be found in the above embodiment, and will not be repeated here for the sake of brevity.

[0134] S820: Extract at least one audio segment from the first audio.

[0135] The embodiment of the present application does not limit the duration, number, and position of the audio segments in the first audio. For example, multiple audio segments can be respectively extracted from the first audio according to different duration ratios.

[0136] In some embodiments, the above S820 may include the following steps:

[0137] Determining a segment duration of at least one audio segment captured;

[0138] Determine the starting position of each audio segment extracted from the first audio according to the segment duration of each audio segment;

[0139] Starting from the start position, audio segments having the same segment duration as each audio segment are extracted from the first audio along a reference direction.

[0140] For example, the segment duration may include but is not limited to 1 / 3, 1 / 4, 1 / 5, and 1 / 6 of the first audio duration. There may be one or more audio segments of the same segment duration.

[0141] The starting position of each audio segment can be determined according to the segment duration of each audio segment. Taking the segment duration of 1 / 3 of the first audio duration as an example, illustratively, if intercepted in the forward direction, the starting position of the audio segment can be arbitrarily selected from the starting position of the first audio to the 2 / 3 duration position as the starting position of the audio segment. If intercepted in the reverse direction, the starting position of the audio segment can be arbitrarily selected from the 2 / 3 duration position to the end position of the first audio as the starting position of the audio segment. Among them, forward interception means intercepting in the order of the first audio from front to back, and reverse interception means intercepting in the order of the first audio from back to front. The process of determining the starting position corresponding to other segment durations is similar.

[0142] The reference direction here may include a forward direction or a reverse direction. Taking forward interception as an example, the audio segment of the corresponding duration may be intercepted in the forward direction starting from a determined starting position.

[0143] When there are multiple audio segments corresponding to each segment duration, the starting position corresponding to each audio segment may be different.

[0144] The embodiment of the present application can predetermine the segmentation duration of the audio segment to be intercepted, and then intercept audio segments of different lengths from the first audio according to the segmentation duration. This can not only improve the flexibility of audio segmentation, but also help capture the timbre changes of the audio at different times, thereby improving the synthesis effect of subsequent audio.

[0145] S830: Perform timbre extraction on each audio segment to obtain segment timbre features of each audio segment.

[0146] For each audio segment, a local timbre feature extraction model can be used to extract timbre, so that the segmented timbre feature of each audio segment can be obtained, that is, the local timbre feature of the first audio. The structure of the local timbre feature extraction model can be referred to in the above embodiment, and for the sake of brevity, it will not be repeated here.

[0147] Taking the first audio "The weather is so bad today, I am a little unhappy" as an example, the extracted global timbre features include the user's fundamental frequency and overall timbre characteristics such as hoarse, warm, bright, etc.; when extracting local timbre features, the above-mentioned audio segmentation interception strategy can capture the timbre changes in the local part of the first audio, such as the timbre changes when the user speaks different languages.

[0148] The embodiment of the present application adopts global and local timbre extraction schemes, which can not only capture the overall characteristics of the first timbre, but also capture the local timbre changes of the first audio, so as to more comprehensively learn the user's timbre characteristics and improve the accuracy of the synthesized audio.

[0149] When performing audio synthesis, the embodiment of the present application not only considers the audio itself, but also considers the text corresponding to the audio, so that the user's pronunciation habits can be learned from the relationship between sound and characters, and the timbre can be extracted from the overall and local multi-scales respectively, which can not only capture the overall characteristics of the user's timbre, but also capture the user's timbre changes, so that the timbre of the synthesized audio is closer to the user's actual timbre. In addition, the embodiment of the present application also uses a large language model, combined with an autoregressive method to predict rhythm, and fully considers the contextual information of the audio, so that a rhythmic style close to the user can be generated. Therefore, through the solution of the embodiment of the present application, a realistic anthropomorphic synthesis effect can be obtained using only short-duration audio such as 5s, and the synthesis efficiency is also greatly improved.

[0150] It should be noted that the audio processing method provided in the embodiment of the present application can be executed by an audio processing device or a processing module in the audio processing device for executing the audio processing method. In the embodiment of the present application, the audio processing device provided in the embodiment of the present application is described by taking the audio processing device executing the audio processing method as an example.

[0151] Fig. 9 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application.

[0152] like Fig. 9 As shown, the audio processing device 900 may include:

[0153] An acquisition module 901 is used to acquire a first audio and a text corresponding to the first audio;

[0154] A determination module 902, configured to determine the phoneme duration of each phoneme in a phoneme sequence corresponding to the text according to the first audio;

[0155] A prediction module 903, configured to predict the prosodic features of the first audio according to the text, the phoneme duration of each phoneme, and the language information corresponding to the text;

[0156] An extraction module 904 is used to extract the timbre of the first audio to obtain the timbre characteristics of the first audio;

[0157] The synthesis module 905 is used to synthesize the text, timbre features and rhythm features to obtain the second audio.

[0158] In an embodiment of the present application, a first audio and a text corresponding to the first audio are obtained; the phoneme duration of each phoneme in the phoneme sequence corresponding to the text is determined based on the first audio; the rhythmic features of the first audio are predicted based on the text, the phoneme duration of each phoneme, and the language information corresponding to the text; the timbre of the first audio is extracted to obtain the timbre features of the first audio; the text, timbre features, and rhythmic features are synthesized to obtain a second audio. Based on the audio, an embodiment of the present application introduces the text corresponding to the audio, and based on the audio, the phoneme duration of each phoneme corresponding to the text is obtained, and the user's speaking habits and rhythmic style are learned in combination with the sound-character relationship, so that the re-synthesized audio is as close to the user's real audio as possible, without the need for a large amount of audio training data, thereby improving the synthesis efficiency of the audio.

[0159] In some possible implementations of the embodiments of the present application, the determination module 902 is specifically configured to:

[0160] Inputting the first audio and the phoneme sequence corresponding to the text into a second duration prediction model, and outputting the phoneme duration of each phoneme in the phoneme sequence;

[0161] The acquisition module 901 is further used to acquire a plurality of duration training samples before acquiring the first audio and the text corresponding to the first audio, each duration training sample including the sample audio, a sample phoneme sequence corresponding to the sample text, and a sample duration corresponding to the sample phoneme sequence, and the sample text is the text corresponding to the sample audio;

[0162] The determination module 902 is further used to input the sample phoneme sequence and the sample audio into the first duration prediction model, output the predicted duration of each phoneme in the sample phoneme sequence; and determine the duration loss function value according to the sample duration and the predicted duration;

[0163] The audio processing device 900 may further include:

[0164] The training module is used to train the first duration prediction model according to the duration loss function value to obtain the second duration prediction model.

[0165] In some possible implementations of the embodiments of the present application, the acquisition module 901 is specifically used to:

[0166] Obtaining sample audio and sample text corresponding to the sample audio;

[0167] The audio processing device 900 may further include:

[0168] A segmentation module, used to segment the sample text according to the language information of the sample text to obtain at least one segmented text;

[0169] A search module is used to search according to each segmented text to obtain at least one phoneme corresponding to each segmented text; according to the order of each segmented text in the sample text, each phoneme corresponding to each segmentation is concatenated to obtain a sample phoneme sequence corresponding to the sample text;

[0170] Determination module 902 is also used to determine the starting position and ending position of each phoneme in the sample phoneme sequence from the sample audio; obtain the sample duration of each phoneme based on the time difference between the starting position and the ending position of each phoneme; and determine the sample duration corresponding to the sample audio, the sample phoneme sequence and the sample phoneme sequence as the duration training sample.

[0171] In some possible implementations of the embodiments of the present application, the prediction module 903 is specifically used to:

[0172] Encode the text to obtain the text feature vector corresponding to the text;

[0173] Vectorizing the phoneme duration of each phoneme to obtain a duration feature vector of each phoneme; and vectorizing the language information corresponding to the text to obtain a language feature vector corresponding to the language information;

[0174] Concatenate the text feature vector, the duration feature vector of each phoneme, and the language feature vector according to the time dimension to obtain a concatenated feature vector;

[0175] Inputting the concatenated feature vector into a prosody prediction model, and outputting prosody features of the first audio;

[0176] Among them, the rhythm prediction model is obtained by training a large language model with a sample concatenated feature vector as input and a sample rhythm feature of the sample audio as output.

[0177] In some possible implementations of the embodiments of the present application, the acquisition module 901 is further used to acquire a plurality of prosody training samples before acquiring the first audio and the text corresponding to the first audio, each prosody training sample including a sample concatenation feature vector and a sample prosody feature corresponding to the sample audio, the sample concatenation feature vector including a sample duration feature vector of each phoneme, a sample language feature vector corresponding to the sample text, and a sample text feature vector corresponding to the sample text, the sample text being the text corresponding to the sample audio;

[0178] The training module is also used to input the first prosodic feature of the sample audio at the first sampling moment and the sample concatenation vector corresponding to the second sampling moment into the large language model, and output the second prosodic feature of the sample audio at the second sampling moment, where the first sampling moment and the second sampling moment are sampling moments within the time period corresponding to the sample audio, and the first sampling is the moment before the second sampling moment; determine the prosodic loss function value according to the second prosodic feature and the sample prosodic feature of the sample audio at the second sampling moment; and train the large language model according to the prosodic loss function value to obtain a prosodic prediction model.

[0179] In some possible implementations of the embodiment of the present application, the extraction module 904 is specifically used to:

[0180] Performing timbre extraction on the first audio to obtain a global timbre feature of the first audio;

[0181] Extracting at least one audio segment from the first audio;

[0182] The timbre of each audio segment is extracted to obtain the segmented timbre features of each audio segment.

[0183] In some possible implementations of the embodiment of the present application, the extraction module 904 is specifically used to:

[0184] Determining a segment duration of at least one audio segment captured;

[0185] Determine the starting position of each audio segment extracted from the first audio according to the segment duration of each audio segment;

[0186] Starting from the start position, audio segments having the same segment duration as each audio segment are extracted from the first audio along a reference direction.

[0187] When performing audio synthesis, the embodiment of the present application not only considers the audio itself, but also considers the text corresponding to the audio, so that the user's pronunciation habits can be learned from the relationship between sound and characters, and the timbre can be extracted from the overall and local multi-scales respectively, which can not only capture the overall characteristics of the user's timbre, but also capture the user's timbre changes, so that the timbre of the synthesized audio is closer to the user's actual timbre. In addition, the embodiment of the present application also uses a large language model, combined with an autoregressive method to predict rhythm, and fully considers the contextual information of the audio, so that a rhythmic style close to the user can be generated. Therefore, through the solution of the embodiment of the present application, a realistic anthropomorphic synthesis effect can be obtained using only short-duration audio such as 5s, and the synthesis efficiency is also greatly improved.

[0188] The audio processing device in the embodiment of the present application can be a device, or a component in an electronic device, such as an integrated circuit or a chip. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a user computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0189] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0190] The audio processing device provided in the embodiment of the present application can achieve Figures 1 to 8 The various processes in the audio processing method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0191] like Fig.10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 801 and a memory 1002, wherein the memory 1002 stores programs or instructions that can be executed on the processor 1001, and when the program or instructions are executed by the processor 1001, the various steps of the above-mentioned audio processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0192] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0193] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0194] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and a processor 1110 and other components.

[0195] Those skilled in the art will appreciate that the electronic device 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The structure of the electronic device 1100 shown in the figure does not constitute a limitation of the electronic device 1100. The electronic device 1100 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0196] The processor 1110 is configured to obtain a first audio and a text corresponding to the first audio;

[0197] Determine the phoneme duration of each phoneme in the phoneme sequence corresponding to the text according to the first audio;

[0198] predicting a prosodic feature of the first audio according to the text, the phoneme duration of each phoneme, and the language information of the audio text;

[0199] Performing timbre extraction on the first audio to obtain timbre characteristics of the first audio;

[0200] The text, the timbre features and the rhythm features are synthesized to obtain a synthesized audio of the first audio.

[0201] In an embodiment of the present application, a first audio and a text corresponding to the first audio are obtained; the phoneme duration of each phoneme in the phoneme sequence corresponding to the text is determined based on the first audio; the rhythmic features of the first audio are predicted based on the text, the phoneme duration of each phoneme, and the language information corresponding to the text; the timbre of the first audio is extracted to obtain the timbre features of the first audio; the text, timbre features, and rhythmic features are synthesized to obtain a second audio. Based on the audio, an embodiment of the present application introduces the text corresponding to the audio, and based on the audio, the phoneme duration of each phoneme corresponding to the text is obtained, and the user's speaking habits and rhythmic style are learned in combination with the sound-character relationship, so that the re-synthesized audio is as close to the user's real audio as possible, without the need for a large amount of audio training data, thereby improving the synthesis efficiency of the audio.

[0202] In some possible implementations of the embodiments of the present application, the processor 1110 is specifically configured to:

[0203] Inputting the first audio and the phoneme sequence corresponding to the text into a second duration prediction model, and outputting the phoneme duration of each phoneme in the phoneme sequence;

[0204] Before obtaining the first audio and the text corresponding to the first audio, obtain a plurality of duration training samples, each duration training sample including the sample audio, a sample phoneme sequence corresponding to the sample text, and a sample duration corresponding to the sample phoneme sequence, and the sample text is the text corresponding to the sample audio;

[0205] Input the sample phoneme sequence and the sample audio into a first duration prediction model, and output the predicted duration of each phoneme in the sample phoneme sequence;

[0206] Determine the duration loss function value based on the sample duration and the predicted duration;

[0207] According to the duration loss function value, the first duration prediction model is trained to obtain the second duration prediction model.

[0208] In some possible implementations of the embodiments of the present application, the processor 1110 is specifically configured to:

[0209] Obtaining sample audio and sample text corresponding to the sample audio;

[0210] Segmenting the sample text according to the language information of the sample text to obtain at least one segmented text;

[0211] Searching for each segmented text to obtain at least one phoneme corresponding to each segmented text; concatenating each phoneme corresponding to each segmentation according to the order of each segmented text in the sample text to obtain a sample phoneme sequence corresponding to the sample text;

[0212] Determine the starting position and ending position of each phoneme in the sample phoneme sequence from the sample audio; obtain the sample duration of each phoneme according to the time difference between the starting position and the ending position of each phoneme;

[0213] The sample audio, the sample phoneme sequence and the sample duration corresponding to the sample phoneme sequence are determined as duration training samples.

[0214] In some possible implementations of the embodiments of the present application, the processor 1110 is specifically configured to:

[0215] Encode the text to obtain the text feature vector corresponding to the text;

[0216] Vectorizing the phoneme duration of each phoneme to obtain a duration feature vector of each phoneme; and vectorizing the language information corresponding to the text to obtain a language feature vector corresponding to the language information;

[0217] Concatenate the text feature vector, the duration feature vector of each phoneme, and the language feature vector according to the time dimension to obtain a concatenated feature vector;

[0218] Inputting the concatenated feature vector into a prosody prediction model, and outputting prosody features of the first audio;

[0219] Among them, the rhythm prediction model is obtained by training a large language model with a sample concatenated feature vector as input and a sample rhythm feature of the sample audio as output.

[0220] In some possible implementations of the embodiments of the present application, the processor 1110 is specifically configured to:

[0221] Before obtaining the first audio and the text corresponding to the first audio, obtain a plurality of prosody training samples, each prosody training sample including a sample concatenation feature vector and a sample prosody feature corresponding to the sample audio, the sample concatenation feature vector including a sample duration feature vector of each phoneme in the sample text, a sample language feature vector corresponding to the sample text, and a sample text feature vector corresponding to the sample text, the sample text being the text corresponding to the sample audio;

[0222] Inputting a first prosodic feature of the sample audio at a first sampling moment and a sample concatenated vector corresponding to a second sampling moment into a large language model, and outputting a second prosodic feature of the sample audio at a second sampling moment, wherein the first sampling moment and the second sampling moment are sampling moments within a time period corresponding to the sample audio, and the first sampling moment is a moment before the second sampling moment;

[0223] Determine a prosody loss function value according to the second prosody feature and the sample prosody feature of the sample audio at the second sampling moment;

[0224] According to the value of the prosody loss function, a large language model is trained to obtain a prosody prediction model.

[0225] In some possible implementations of the embodiments of the present application, the processor 1110 is specifically configured to:

[0226] Performing timbre extraction on the first audio to obtain a global timbre feature of the first audio;

[0227] Extracting at least one audio segment from the first audio;

[0228] The timbre of each audio segment is extracted to obtain the segmented timbre features of each audio segment.

[0229] In some possible implementations of the embodiments of the present application, the processor 1110 is specifically configured to:

[0230] Determining a segment duration of at least one audio segment captured;

[0231] Determine the starting position of each audio segment extracted from the first audio according to the segment duration of each audio segment;

[0232] Starting from the start position, audio segments having the same segment duration as each audio segment are extracted from the first audio along a reference direction.

[0233] When performing audio synthesis, the embodiment of the present application not only considers the audio itself, but also considers the text corresponding to the audio, so that the user's pronunciation habits can be learned from the relationship between sound and characters, and the timbre can be extracted from the overall and local multi-scales respectively, which can not only capture the overall characteristics of the user's timbre, but also capture the user's timbre changes, so that the timbre of the synthesized audio is closer to the user's actual timbre. In addition, the embodiment of the present application also uses a large language model, combined with an autoregressive method to predict rhythm, and fully considers the contextual information of the audio, so that a rhythmic style close to the user can be generated. Therefore, through the solution of the embodiment of the present application, a realistic anthropomorphic synthesis effect can be obtained using only short-duration audio such as 5s, and the synthesis efficiency is also greatly improved.

[0234] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processor (Graphics Processing Unit, GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0235] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0236] The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.

[0237] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned audio processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0238] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0239] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0240] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0241] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned audio processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0242] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0243] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0244] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio processing method, characterized in that: include: Acquire a first audio and a text corresponding to the first audio; Determine the phoneme duration of each phoneme in the phoneme sequence corresponding to the text according to the first audio; predicting a prosodic feature of the first audio according to the text, the phoneme duration of each phoneme, and language information corresponding to the text; Performing timbre extraction on the first audio to obtain timbre features of the first audio; The text, the timbre feature and the rhythm feature are synthesized to obtain a second audio.

2. The method according to claim 1, characterized in that: The determining, according to the first audio, the phoneme duration of each phoneme in the phoneme sequence corresponding to the text comprises: Inputting the first audio and the phoneme sequence corresponding to the text into a second duration prediction model, and outputting the phoneme duration of each phoneme in the phoneme sequence; Before obtaining the first audio and the text corresponding to the first audio, the method further includes: Acquire multiple duration training samples, each duration training sample includes a sample audio, a sample phoneme sequence corresponding to a sample text, and a sample duration corresponding to the sample phoneme sequence, wherein the sample text is a text corresponding to the sample audio; Inputting the sample phoneme sequence and the sample audio into a first duration prediction model, and outputting a predicted duration of each phoneme in the sample phoneme sequence; Determine a duration loss function value according to the sample duration and the predicted duration; The first duration prediction model is trained according to the duration loss function value to obtain a second duration prediction model.

3. The method according to claim 2, characterized in that Get duration training samples, including: Acquire sample audio and sample text corresponding to the sample audio; Segmenting the sample text according to the language information of the sample text to obtain at least one segmented text; Searching each segmented text to obtain at least one phoneme corresponding to each segmented text; concatenating each phoneme corresponding to each segmentation according to the order of each segmented text in the sample text to obtain a sample phoneme sequence corresponding to the sample text; Determine the starting position and the ending position of each phoneme in the sample phoneme sequence from the sample audio; obtain the sample duration of each phoneme according to the time difference between the starting position and the ending position of each phoneme; The sample audio, the sample phoneme sequence and the sample duration corresponding to the sample phoneme sequence are determined as duration training samples.

4. The method according to claim 1, characterized in that The predicting the prosodic feature of the first audio according to the text, the phoneme duration of each phoneme, and the language information corresponding to the text includes: Encoding the text to obtain a text feature vector corresponding to the text; Vectorizing the phoneme duration of each phoneme to obtain a duration feature vector of each phoneme; and vectorizing the language information corresponding to the text to obtain a language feature vector corresponding to the language information; splicing the text feature vector, the duration feature vector of each phoneme, and the language feature vector according to the time dimension to obtain a spliced ​​feature vector; Inputting the concatenated feature vector into a prosody prediction model, and outputting prosody features of the first audio; The prosody prediction model is obtained by training a large language model with a sample concatenated feature vector as input and a sample prosody feature of a sample audio as output.

5. The method according to claim 4, characterized in that Before obtaining the first audio and the text corresponding to the first audio, the method further includes: Acquire multiple prosody training samples, each prosody training sample includes a sample concatenation feature vector and a sample prosody feature corresponding to the sample audio, the sample concatenation feature vector includes a sample duration feature vector of each phoneme, a sample language feature vector corresponding to a sample text, and a sample text feature vector corresponding to the sample text, the sample text being the text corresponding to the sample audio; Inputting a first prosodic feature of the sample audio at a first sampling moment and a sample concatenation vector corresponding to a second sampling moment into a large language model, and outputting a second prosodic feature of the sample audio at the second sampling moment, wherein the first sampling moment and the second sampling moment are sampling moments within a time period corresponding to the sample audio, and the first sampling moment is a moment before the second sampling moment; Determine a prosody loss function value according to the second prosody feature and the sample prosody feature of the sample audio at the second sampling time; The large language model is trained according to the prosody loss function value to obtain a prosody prediction model.

6. The method according to claim 1, characterized in that The step of extracting the timbre of the first audio to obtain the timbre characteristics of the first audio includes: Performing timbre extraction on the first audio to obtain a global timbre feature of the first audio; Extracting at least one audio segment from the first audio; The timbre of each audio segment is extracted to obtain the segmented timbre features of each audio segment.

7. The method according to claim 6, characterized in that The extracting at least one audio segment from the first audio comprises: Determining a segment duration of at least one audio segment captured; Determining a starting position of each audio segment extracted from the first audio according to the segment duration of each audio segment; Starting from the starting position, audio segments having the same segment duration as each audio segment are extracted from the first audio along a reference direction.

8. An audio processing device, characterized in that: include: An acquisition module, used for acquiring a first audio and a text corresponding to the first audio; A determination module, configured to determine the phoneme duration of each phoneme in a phoneme sequence corresponding to the text according to the first audio; A prediction module, configured to predict the prosodic features of the first audio according to the text, the phoneme duration of each phoneme, and the language information corresponding to the text; An extraction module, used for performing timbre extraction on the first audio to obtain timbre features of the first audio; A synthesis module is used to synthesize the text, the timbre feature and the rhythm feature to obtain a second audio.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 7.

Images