Method, device and storage medium for generating audio

Abstract

The embodiments of the present application disclose a method, device and storage medium for generating audio, which belongs to the field of audio technology. Phoneme sequences and phoneme duration sequences are used. The phoneme sequence is generally obtained from the text corresponding to the audio and is completely unaffected by the noise in the audio. The phoneme duration sequence is generally obtained based on the audio and the text corresponding to the audio and is very little affected by the noise in the audio. In addition, the phoneme sequence can provide the pronunciation information of the audio, and the phoneme duration sequence can provide the rhythm information of the audio. Therefore, in addition to inputting the pronunciation rhythm features and timbre features into the acoustic model, the method also uses the phoneme sequence and the phoneme duration sequence as the input of the model at the same time to generate audio, which can reduce the impact of noise on the generated audio and further improve the quality of the generated audio.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a method, device, and storage medium for generating audio. Background Art

[0002] In the field of audio technology, VC (voice conversion) technology refers to converting a source audio into a target audio with the timbre of a specified speaker, and the target audio also retains the pronunciation rhythm characteristics of the source audio, wherein the pronunciation rhythm characteristics are used to represent each phoneme in the pronunciation process and the corresponding pronunciation duration. For example, in a children's reading application, in order to retain the father's or mother's timbre and the pronunciation rhythm characteristics of the reading audio at the same time, the father or mother can select the reading audio of a fairy tale in the application. In addition, the father or mother also needs to record an audio of his or her own speech. After clicking the conversion control, the audio with his or her own timbre and the pronunciation rhythm characteristics of the reading audio can be obtained.

[0003] Related applications can include a voice conversion model, which includes a PPG (phoneme posteriorgram) extractor, an acoustic model, and a vocoder. The specific usage process is as follows: input the spoken audio into the PPG extractor to obtain a PPG matrix. The PPG matrix represents the probability of each phoneme corresponding to each frame of the spoken audio and is a characteristic of the pronunciation rhythm. Inputting the PPG matrix and the timbre characteristics of the target person into the voice conversion model yields audio that has the target person's timbre and the pronunciation rhythm characteristics of the spoken audio.

[0004] In the related art, if the reading audio has a lot of noise, it will affect the accuracy of the pronunciation rhythm features (PPG matrix) extracted by the PPG extractor, and further affect the quality of the converted audio. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, and storage medium for generating audio, which can solve the problems of related technologies. The technical solution is as follows:

[0006] In a first aspect, a method for generating audio is provided, the method comprising:

[0007] Acquire the timbre characteristics of the first person, and acquire the pronunciation rhythm characteristics, phoneme sequence, and phoneme duration sequence corresponding to the reading audio of the second person;

[0008] Inputting the timbre characteristics of the first character, the pronunciation rhythm characteristics corresponding to the second character, the phoneme sequence, and the phoneme duration sequence into a pre-trained target acoustic model to obtain spectrum data output by the target acoustic model;

[0009] Inputting the phoneme sequence and the phoneme duration sequence into a pre-trained first feature adjustment model, wherein the first feature adjustment model adjusts the length of the phoneme sequence using the phoneme duration sequence and outputs a first adjustment feature, wherein the length of the first adjustment feature is the same as the length of the spectrum data;

[0010] adjusting the spectrum data using the first adjustment feature to obtain adjusted spectrum data;

[0011] The adjusted spectrum data is input into a pre-trained target vocoder to obtain a target audio output by the target vocoder.

[0012] In one possible implementation, the target acoustic model includes a first encoder, a decoder, a second feature adjustment model, and a first superimposer;

[0013] The step of inputting the timbre characteristics of the first character, the pronunciation rhythm characteristics corresponding to the second character, the phoneme sequence, and the phoneme duration sequence into a pre-trained target acoustic model to obtain spectrum data output by the target acoustic model includes:

[0014] Inputting the pronunciation rhythm feature corresponding to the second character into the pre-trained first encoder to obtain a first encoding feature output by the first encoder;

[0015] Inputting the phoneme sequence corresponding to the second character and the phoneme duration sequence corresponding to the second character into the pre-trained second feature adjustment model, the second feature adjustment model adjusting the length of the phoneme sequence corresponding to the second character using the phoneme duration sequence corresponding to the second character and outputting a second adjustment feature, wherein the second adjustment feature has the same length as the first encoding feature;

[0016] Inputting the second adjustment feature and the first coding feature into the first superimposer, and having the first superimposer perform positional addition on the second adjustment feature and the first coding feature to obtain and output a second coding feature;

[0017] The second coding feature and the timbre feature of the first character are input into the pre-trained decoder to obtain the spectrum data output by the decoder.

[0018] In one possible implementation, the second feature adjustment model includes a second encoder and a first adjuster;

[0019] The step of inputting the phoneme sequence corresponding to the second character and the phoneme duration sequence corresponding to the second character into a pre-trained second feature adjustment model, and the second feature adjustment model adjusting the length of the phoneme sequence corresponding to the second character using the phoneme duration sequence corresponding to the second character and then outputting a second adjustment feature, includes:

[0020] Inputting the phoneme sequence corresponding to the second character into the pre-trained second encoder to obtain a first phoneme feature output by the second encoder;

[0021] The first phoneme feature and the phoneme duration sequence are input into the first regulator, and the first regulator uses the phoneme duration sequence corresponding to the second character to adjust the length of the first phoneme feature and then outputs the second adjusted feature.

[0022] In one possible implementation, the first feature adjustment model includes a third encoder and a second adjuster;

[0023] The step of inputting the phoneme sequence and the phoneme duration sequence into a pre-trained first feature adjustment model, and having the first feature adjustment model adjust the length of the phoneme sequence using the phoneme duration sequence and then outputting a first adjustment feature, includes:

[0024] Inputting the phoneme sequence into the pre-trained third encoder to obtain a second phoneme feature output by the third encoder;

[0025] The second phoneme feature and the phoneme duration sequence are input into the second regulator, and the second regulator uses the phoneme duration sequence to adjust the length of the second phoneme feature and then outputs the first adjusted feature.

[0026] In a possible implementation, using the first adjustment feature to adjust the spectrum data to obtain adjusted spectrum data includes:

[0027] The first adjustment feature and the spectrum data are input into a second adder, and the second adder performs a position-wise addition on the first adjustment feature and the spectrum data to obtain and output the adjusted spectrum data.

[0028] In one possible implementation, obtaining a phoneme duration sequence corresponding to the second person's reading audio includes:

[0029] The reading audio of the second character and the text corresponding to the reading audio are input into a phoneme duration determination model to obtain a phoneme duration sequence corresponding to the second character.

[0030] In a possible implementation, the method further includes:

[0031] Obtaining timbre features, pronunciation rhythm features, phoneme sequence, phoneme duration sequence, and spectrum data corresponding to the first sample audio;

[0032] Inputting the timbre feature, the pronunciation rhythm feature, the phoneme sequence, and the phoneme duration sequence corresponding to the first sample audio into the acoustic model to be trained, to obtain spectrum data output by the acoustic model to be trained;

[0033] Training the acoustic model to be trained with the goal of minimizing the difference between the spectrum data corresponding to the first sample audio and the spectrum data output by the acoustic model to be trained;

[0034] If the trained acoustic model meets the training end condition, the trained acoustic model is determined as the target acoustic model.

[0035] In a possible implementation, the method further includes:

[0036] Obtaining a phoneme sequence, a phoneme duration sequence, and spectrum data corresponding to the second sample audio;

[0037] Inputting the phoneme sequence and the phoneme duration sequence corresponding to the second sample audio into the trained first feature adjustment model to obtain a third adjustment feature output by the first feature adjustment model;

[0038] adjusting the frequency spectrum data corresponding to the second audio sample using the third adjustment feature to obtain adjusted frequency spectrum data corresponding to the second audio sample;

[0039] Inputting the adjusted spectrum data corresponding to the second sample audio into the vocoder to be trained to obtain predicted audio output by the vocoder to be trained;

[0040] The vocoder to be trained is trained to minimize the difference between the predicted audio and the sample audio; if the trained vocoder meets a training end condition, the trained vocoder is determined as the target vocoder.

[0041] In a second aspect, a device for generating audio is provided, the device comprising:

[0042] An acquisition module, configured to acquire the timbre characteristics of the first character, and acquire the pronunciation rhythm characteristics, phoneme sequence, and phoneme duration sequence corresponding to the reading audio of the second character;

[0043] Generate modules for:

[0044] Inputting the timbre characteristics of the first character, the pronunciation rhythm characteristics corresponding to the second character, the phoneme sequence, and the phoneme duration sequence into a pre-trained target acoustic model to obtain spectrum data output by the target acoustic model;

[0045] Inputting the phoneme sequence and the phoneme duration sequence into a pre-trained first feature adjustment model, wherein the first feature adjustment model adjusts the length of the phoneme sequence using the phoneme duration sequence and outputs a first adjustment feature, wherein the length of the first adjustment feature is the same as the length of the spectrum data;

[0046] adjusting the spectrum data using the first adjustment feature to obtain adjusted spectrum data;

[0047] The adjusted spectrum data is input into a pre-trained target vocoder to obtain a target audio output by the target vocoder.

[0048] In one possible implementation, the target acoustic model includes a first encoder, a decoder, a second feature adjustment model, and a first superimposer;

[0049] The generating module is used to:

[0050] Inputting the pronunciation rhythm feature corresponding to the second character into the pre-trained first encoder to obtain a first encoding feature output by the first encoder;

[0051] Inputting the phoneme sequence corresponding to the second character and the phoneme duration sequence corresponding to the second character into the second feature adjustment model, and the second feature adjustment model adjusting the length of the phoneme sequence corresponding to the second character using the phoneme duration sequence corresponding to the second character, and then outputting a second adjustment feature, wherein the second adjustment feature has the same length as the first coding feature;

[0052] Inputting the second adjustment feature and the first coding feature into the first superimposer to obtain a second coding feature output by the first superimposer;

[0053] The second coding feature and the timbre feature of the first character are input into the pre-trained decoder to obtain the spectrum data output by the decoder.

[0054] In one possible implementation, the second feature adjustment model includes a second encoder and a first adjuster;

[0055] The generating module is used to:

[0056] Inputting the phoneme sequence corresponding to the second character into the pre-trained second encoder to obtain a first phoneme feature output by the second encoder;

[0057] The first phoneme feature and the phoneme duration sequence are input into the first regulator, and the first regulator uses the phoneme duration sequence corresponding to the second character to adjust the length of the first phoneme feature and then outputs the second adjusted feature.

[0058] In one possible implementation, the first feature adjustment model includes a third encoder and a second adjuster;

[0059] The generating module is used to:

[0060] Inputting the phoneme sequence into the pre-trained third encoder to obtain a second phoneme feature output by the third encoder;

[0061] The second phoneme feature and the phoneme duration sequence are input into the second regulator, and the second regulator uses the phoneme duration sequence to adjust the length of the second phoneme feature and then outputs the first adjusted feature.

[0062] In a possible implementation, the generating module is configured to:

[0063] The first adjustment feature and the spectrum data are input into a second adder, and the second adder performs a position-wise addition on the first adjustment feature and the spectrum data to obtain and output the adjusted spectrum data.

[0064] In a possible implementation, the acquisition module is configured to:

[0065] The reading audio of the second character and the text corresponding to the reading audio are input into a phoneme duration determination model to obtain a phoneme duration sequence corresponding to the second character.

[0066] In a possible implementation, the apparatus further includes a training module:

[0067] The acquisition module is used to obtain timbre characteristics, pronunciation rhythm characteristics, phoneme sequence, phoneme duration sequence and spectrum data corresponding to the first sample audio;

[0068] The generating module is configured to input the timbre feature, the pronunciation rhythm feature, the phoneme sequence, and the phoneme duration sequence corresponding to the first sample audio into the acoustic model to be trained, to obtain spectrum data output by the acoustic model to be trained;

[0069] The training module is used to:

[0070] Training the acoustic model to be trained with the goal of minimizing the difference between the spectrum data corresponding to the first sample audio and the spectrum data output by the acoustic model to be trained;

[0071] If the trained acoustic model meets the training end condition, the trained acoustic model is determined as the target acoustic model.

[0072] In a possible implementation, the acquisition module is further configured to:

[0073] Obtaining a phoneme sequence, a phoneme duration sequence, and spectrum data corresponding to the second sample audio;

[0074] The generating module is used to:

[0075] Inputting the phoneme sequence and the phoneme duration sequence corresponding to the second sample audio into the trained first feature adjustment model to obtain a third adjustment feature output by the first feature adjustment model;

[0076] adjusting the frequency spectrum data corresponding to the second audio sample using the third adjustment feature to obtain adjusted frequency spectrum data corresponding to the second audio sample;

[0077] Inputting the adjusted spectrum data corresponding to the second sample audio into the vocoder to be trained to obtain predicted audio output by the vocoder to be trained;

[0078] The training module is used to:

[0079] The vocoder to be trained is trained to minimize the difference between the predicted audio and the sample audio; if the trained vocoder meets a training end condition, the trained vocoder is determined as the target vocoder.

[0080] In a third aspect, a computer device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions; the processor executes the computer instructions stored in the memory so that the computer device executes the method of the first aspect and its possible implementation methods.

[0081] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer program code. In response to the computer program code being executed by a computer device, the computer device executes the method of the first aspect and possible implementations thereof.

[0082] In a fifth aspect, a computer program product is provided, the computer program product comprising computer program code. In response to the computer program code being executed by a computer device, the computer device executes the method of the first aspect and possible implementations thereof.

[0083] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0084] By the method provided in the embodiment of the present application, a phoneme sequence and a phoneme duration sequence are adopted. The phoneme sequence is generally obtained from the text corresponding to the audio and is completely unaffected by the noise in the audio. The phoneme duration sequence is generally obtained based on the audio and the text corresponding to the audio and is very little affected by the noise in the audio. In addition, the phoneme sequence can provide the pronunciation information of the audio, and the phoneme duration sequence can provide the rhythm information of the audio. Therefore, in addition to using the pronunciation rhythm features and timbre features as the input of the model, using the phoneme sequence and the phoneme duration sequence as the input of the model at the same time to generate audio can reduce the impact of noise on the generated audio. Further, the quality of the generated audio can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0086] Figure 1 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0087] Figure 2 This is a flowchart of a method for generating audio provided by an embodiment of the present application;

[0088] Figure 3 This is a schematic diagram of a model operation for generating target audio provided by an embodiment of the present application;

[0089] Figure 4 This is a flowchart of a method for generating audio provided by an embodiment of the present application;

[0090] Figure 5 1 is a schematic diagram of a model operation for training an acoustic model provided in an embodiment of the present application;

[0091] Figure 6 This is a flowchart of a method for generating audio provided by an embodiment of the present application;

[0092] Figure 7 1 is a schematic diagram of a model operation for training a vocoder provided in an embodiment of the present application;

[0093] Figure 8 1 is a schematic structural diagram of a device for generating audio provided in an embodiment of the present application;

[0094] Figure 9This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] Embodiments of the present application provide a method for generating audio, which is used to convert the timbre of audio. The method can be implemented by a computer device, which can be a terminal or a server. The terminal can be a mobile phone, tablet computer, laptop computer, desktop computer, etc. An application with a sound conversion function can be installed in the terminal. The server can be the background server of the application with the sound conversion function, and the server can be a single server or a device group consisting of multiple devices.

[0096] From the perspective of hardware composition, the structure of computer equipment may include processor and memory, such as Figure 1 As shown. The processor 110 can be a CPU (central processing unit) or an SoC (system on chip), etc., and the processor can be used to execute various instructions involved in the method. The memory 120 can include various volatile memories or non-volatile memories, such as SSD (solid state disk), DRAM (dynamic random access memory), etc. The memory can be used to generate pre-stored data, intermediate data, and result data in the audio generation process, such as sample audio.

[0097] In addition to processors and memories, computer devices may also include display components, communication components, audio acquisition components, audio output components, etc.

[0098] The display component can be an independent screen, or a screen integrated with the terminal body, a projector, etc. The screen can be a touch screen or a non-touch screen. The display component is used to display the system interface, application interface, etc., for example, the interface for recording audio. The communication component can be a wired network connector, a WiFi (wireless fidelity) module, a Bluetooth module, a cellular network communication module, etc. The communication component can be used to transmit data with other devices, such as servers or other terminals. The audio acquisition component can be a microphone for collecting the user's voice. The audio output component can be a speaker, headphones, etc. for playing audio.

[0099] The following are some terms used in this embodiment:

[0100] MFC (mel-frequency spectrum): hereinafter referred to as the Mel spectrum. The Mel spectrum can be used to represent the audio features of an audio segment, including information related to timbre, text content, pitch, etc.

[0101] Phoneme: The smallest speech unit divided according to the natural attributes of speech. Analyzed based on the pronunciation actions in a syllable, one action constitutes one phoneme. For example, "ā" has only one phoneme, "ài" has two phonemes, "dài" has three phonemes, etc.

[0102] Phoneme sequence: Refers to the set of all phonemes in an audio segment. For example, if the text corresponding to an audio segment is "There is a bird in the tree", the phoneme sequence is "shùshàng yǒu zhīn iǎo".

[0103] Phoneme duration: Refers to the pronunciation duration of each phoneme. For an audio segment, the pronunciation duration of a phoneme can be represented by "frames". For different people or in different contexts, the pronunciation duration of the same phoneme is different. For different phoneme strings, the pronunciation duration of the same phoneme may also be different. For example, the pronunciation duration of "à" in "ài" and "bàn" is different.

[0104] Phoneme duration sequence: Refers to the set of the pronunciation durations of each phoneme in an audio segment. For example, if the phoneme sequence is "shùshàng yǒu zhīn iǎo", the phoneme duration sequence can be "2 2 2 2 2 2 2 2 2 2 22", indicating that the pronunciation duration of each phoneme is 2 frames.

[0105] PPG (phoneme posteriorgram) matrix: Refers to the posterior probabilities of each phoneme in the set of phonemes corresponding to each frame of an audio. For example, the dimension of the PPG matrix corresponding to a speech audio segment is M*N, where M represents the number of frames of the speech audio, N represents the number of phonemes corresponding to one frame of the speech audio, and the point (m, n) represents the probability of the nth phoneme corresponding to the mth frame of the speech audio.

[0106] The audio generation solutions provided in the embodiments of this application can be applied to application programs related to audio to convert an audio segment into an audio with a user-specified timbre. For example, in a children's reading application program, parents can select a fairy tale, record a short audio of their own speech, and perform timbre conversion on the audio of the fairy tale.

[0107] In view of the above application scenarios, the embodiments of this application provide a method for generating audio. The processing flow of this method can be as Figure 2 shown, and the corresponding data flow diagram can be as Figure 3 shown, including the following processing steps:

[0108] 201 , obtaining the timbre characteristics of the first person, the pronunciation rhythm characteristics corresponding to the reading audio of the second person, the phoneme sequence, and the phoneme duration sequence.

[0109] In practice, when a user needs to convert a reading audio into a reading audio with their own timbre characteristics, they can first select a reading audio (the second person's reading audio) in the relevant application. The reading audio can be of any length. Then, they can record an audio of their own speech (the first person's reading audio). For example, they can record an audio of about three minutes.

[0110] like Figure 3 As shown, after obtaining the audio of the user's speech, the computer device can input the audio into the timbre feature extraction model to obtain the user's timbre features (i.e., the timbre features of the first person). Alternatively, the audio can be input into the speaker recognition model to obtain the user's character code (i.e., the timbre features of the first person), and the character code is used to represent the user's timbre. Both the timbre feature extraction model and the speaker recognition model can be trained machine learning models, such as neural network models. Among them, the timbre features can be specifically represented as timbre vectors, and the pronunciation rhythm features can be specifically represented as PPG matrices.

[0111] After the computer device obtains the user's selection operation for the second person's reading audio, it can find the second person's reading audio and its corresponding text in the reading audio library, input the reading audio into the PPG extractor, and obtain the pronunciation rhythm characteristics corresponding to the second person. In addition, the text corresponding to the reading audio is input into the phoneme predictor to obtain the phoneme sequence corresponding to the second person. Furthermore, the reading audio and its corresponding text are input into the phoneme duration determination model to obtain the phoneme duration sequence corresponding to the second person. Alternatively, the text of the reading audio can also be obtained by inputting the reading audio into the text recognition model after finding the second person's reading audio. It should be noted that the PPG extractor, phoneme predictor, phoneme duration determination model and text recognition model can be trained machine learning models.

[0112] 202 , input the timbre characteristics of the first person, the pronunciation rhythm characteristics, phoneme sequence, and phoneme duration sequence corresponding to the second person into a pre-trained target acoustic model to obtain spectrum data output by the target acoustic model.

[0113] The target acoustic model includes a first encoder, a first superimposer, a decoder, and a second feature adjustment model. In a specific implementation, the second feature adjustment model may specifically include a second encoder and a first regulator. The first regulator may specifically include an LR (Length Regulator), etc.

[0114] like Figure 3 As shown, the pronunciation rhythm feature corresponding to the second character is input into the first encoder to obtain a first encoding feature. The first encoding feature can be specifically represented as a PPG vector. The phoneme sequence corresponding to the second character is input into the second encoder to obtain a first phoneme feature. The first phoneme feature can be specifically represented as a phoneme vector.

[0115] The phoneme duration sequence and the first phoneme feature corresponding to the second character are input into the first regulator. The first regulator can copy each phoneme vector in the first phoneme feature a corresponding number of times based on the phoneme duration corresponding to each phoneme in the phoneme duration sequence to obtain a second adjustment feature. The second adjustment feature can be specifically represented as a corrected phoneme vector that has been corrected for phoneme duration, and has the same length as the first coding feature. For example, the phoneme sequence is "aoe", and its corresponding phoneme feature is "a1 a2 a3", where a1 is the vector corresponding to the phoneme "a", a2 is the vector corresponding to the phoneme "o", and a3 is the vector corresponding to the phoneme "e". The phoneme duration sequence is "2 3 4", and the phoneme features and the phoneme duration sequence are input into the second regulator, and the obtained adjustment feature is "a1 a1 a2 a2 a2 a3 a3 a3 a3".

[0116] The first coding feature and the second adjustment feature are input into a first adder, and the first coding feature and the second adjustment feature are added to obtain a second coding feature. The second coding feature can be specifically represented as an addition vector of the PPG vector and the correction phoneme vector.

[0117] The second encoding feature and the first character's timbre feature are input into a decoder to obtain spectral data. In one implementation, the spectral data can be specifically mel-spectrogram data. The first encoder, the second encoder, and the decoder can all be trained machine learning models.

[0118] Phoneme sequences contain text and pronunciation information, while phoneme duration sequences contain rhythm information. Duration-corrected phoneme vectors can reflect stable text, pronunciation, and rhythm information. Even if the pronunciation and rhythm features extracted by the PPG extractor are inaccurate due to audio noise, the acoustic model can be used to input the phoneme sequence and phoneme duration sequence to adjust the first encoding feature, minimizing the impact of noise on the final output spectrum data.

[0119] The above-mentioned target acoustic model is composed of four modules: a first encoder, a first superimposer, a decoder and a second feature adjustment model. In other implementations, the target acoustic model can be composed of other modules, or can also include only a single component module.

[0120] 203 , inputting the phoneme sequence and the phoneme duration sequence into a pre-trained first feature adjustment model, and the first feature adjustment model uses the phoneme duration sequence to adjust the length of the phoneme sequence and then outputs a first adjustment feature.

[0121] The first feature adjustment model includes a third encoder and a second regulator, wherein the third encoder may be a trained machine learning model, and the second regulator may specifically include an LR (Length Regulator).

[0122] like Figure 3 As shown, the phoneme sequence corresponding to the second character is input into the third encoder to obtain the second phoneme feature. The second phoneme feature can be specifically represented as a phoneme vector. The phoneme duration sequence and the second phoneme feature corresponding to the second character are input into the second regulator. The second regulator can copy each phoneme vector in the second phoneme feature a corresponding number of times based on the phoneme duration corresponding to each phoneme in the phoneme duration sequence to obtain the first adjustment feature. The first adjustment feature can be a phoneme feature corrected by the phoneme duration. In this way, a phoneme feature with the same length as the spectral data output by the above-mentioned target acoustic model can be obtained, so that the spectral data can be adjusted using the first adjustment feature.

[0123] The above-mentioned first feature adjustment model is composed of two modules: the third encoder and the second regulator. In other implementations, the first feature adjustment model can be composed of a combination of other modules, or can also include only a single component module.

[0124] 204 : Use the first adjustment feature to adjust the spectrum data to obtain adjusted spectrum data.

[0125] In practice, the pronunciation of phonemes can be divided into unvoiced and voiced sounds. The principles of voice production of unvoiced and voiced sounds are different. Voiced sounds have their own fundamental frequencies, while unvoiced sounds are similar to noise (for example, "b", "p", etc.) and do not have their own fundamental frequencies. However, the vocoder will synthesize unvoiced sounds into pronunciations with fixed fundamental frequencies, resulting in the synthesized pronunciations having an electronic sound feel. Therefore, in an embodiment of the present application, the phoneme sequence and phoneme duration sequence corresponding to the second character are input into the second regulator to obtain a first adjustment feature. The first adjustment feature and the spectrum data are input into the second superimposer, and the first adjustment feature and the spectrum data are added to obtain the adjusted spectrum data. In this way, after the spectrum data is adjusted by the first adjustment feature, the adjusted spectrum data can have phoneme-related information. The spectrum data with phoneme-related information is further used as the input of the target vocoder, so that the vocoder can clearly distinguish between unvoiced and voiced sounds, thereby making the unvoiced part of the target audio generated finally more natural and eliminating the electronic sound feel.

[0126] The above method uses the second adder to adjust the spectrum data. In other possible implementations, a model composed of other modules can be used.

[0127] 205 , input the adjusted spectrum data into a pre-trained target vocoder to obtain a target audio output by the target vocoder.

[0128] The target vocoder is a trained machine learning model, specifically HifiGan, which includes a generator and a discriminator. The target audio output by the target vocoder contains both the timbre information of the first person and the text information and pronunciation rhythm information of the second person's reading audio.

[0129] Since the phoneme sequence is generally obtained from the text corresponding to the audio and is completely unaffected by the noise in the audio, the phoneme duration sequence is generally obtained based on the audio and the text corresponding to the audio and is very little affected by the noise in the audio. Thus, when converting audio, in addition to using the pronunciation rhythm characteristics corresponding to the second person's reading audio and the timbre characteristics of the first person as inputs to the target acoustic model, the phoneme sequence and phoneme duration sequence corresponding to the second person's reading audio are also used as inputs to the target acoustic model. This can reduce the impact of noise on the generated target audio, thereby making the spectral data output by the model closer to the actual spectral data of the second person's reading audio (the actual spectral data is the spectral data obtained by directly inputting the audio into the spectral acquisition model), thereby improving the quality of the target audio output by the vocoder.

[0130] The embodiment of the present application further provides a method for training an acoustic model for the above application scenario. The processing flow of the method can be as follows: Figure 4 As shown, the corresponding model operation diagram can be Figure 5 As shown in the figure, the models with bold lines in the figure represent models that need to be trained, and the models without bold lines represent models that do not need to be trained, such as trained models or non-machine learning models. The processing steps include the following:

[0131] 401 , obtaining timbre features, pronunciation rhythm features, phoneme sequences, phoneme duration sequences, and spectrum data corresponding to a first sample audio.

[0132] In implementation, the first audio sample is input into a timbre feature extraction model to obtain timbre features corresponding to the first audio sample. The first audio sample is input into a PPG extractor to obtain pronunciation rhythm features corresponding to the first audio sample. The text corresponding to the first audio sample is input into a phoneme predictor to obtain a phoneme sequence corresponding to the first audio sample. The first audio sample and its corresponding text are input into a phoneme duration determination model to obtain a phoneme duration sequence. The first audio sample is input into a spectrum acquisition model to obtain spectrum data.

[0133] 402 : Input the timbre features, pronunciation rhythm features, phoneme sequence, and phoneme duration sequence corresponding to the first sample audio into the acoustic model to be trained to obtain spectrum data output by the acoustic model to be trained.

[0134] The acoustic model to be trained includes a first superimposer, a first encoder to be trained, a decoder to be trained, and a second feature adjustment model to be trained. The second feature adjustment model to be trained includes a first adjuster and a second encoder to be trained.

[0135] In implementation, the pronunciation rhythm feature corresponding to the first audio sample is input into the first encoder to be trained to obtain a third encoding feature. The third encoding feature can specifically be a PPG vector. The phoneme sequence corresponding to the first audio sample is input into the second encoder to be trained to obtain a third phoneme feature. The third phoneme feature can specifically be a phoneme vector. The third phoneme feature and the phoneme duration sequence corresponding to the first audio sample are input into a first adjuster. The first adjuster can adjust the third phoneme feature by replicating each phoneme vector in the third phoneme feature a corresponding number of times based on the phoneme duration corresponding to each phoneme in the phoneme duration sequence, thereby obtaining a fourth adjusted feature. The fourth adjusted feature can be a phoneme vector corrected for phoneme duration. The third encoding feature and the fourth adjusted feature are input into a first adder, and the third and fourth adjusted features are added together to obtain a fourth encoding feature. The fourth encoding feature can specifically be represented as the sum of the PPG vector and the phoneme vector corrected for phoneme duration. The fourth encoding feature and the timbre feature corresponding to the first audio sample are input into a decoder to be trained to obtain spectral data.

[0136] 403 : The acoustic model to be trained is trained with the training objective of minimizing the difference between the spectrum data corresponding to the first sample audio and the spectrum data output by the acoustic model to be trained.

[0137] In practice, because the spectral data corresponding to the first sample audio is directly obtained by inputting the first sample audio into the spectral acquisition model, and the spectral data output by the acoustic model to be trained is obtained by inputting the second pronunciation feature into the acoustic model, and the pronunciation rhythm feature corresponding to the first sample audio is obtained by inputting the first sample audio into the PPG extractor, if the first sample audio has large noise, it will cause the extracted pronunciation rhythm feature to be inaccurate, and further cause the spectral data output by the acoustic model to be inaccurate. Therefore, the training purpose is to minimize the difference between the spectral data corresponding to the first sample audio and the spectral data output by the acoustic model to be trained, so that the spectral data predicted by the acoustic model through the input pronunciation rhythm feature can be more accurate.

[0138] During training, the above differences can be obtained by calculating the loss value. The loss value can be calculated using the MSE (mean square error) loss function, which is expressed as follows:

[0139]

[0140] Among them, y i Represents the spectrum data output by the acoustic model to be trained, represents the spectrum data corresponding to the first sample audio, n represents the number of sample audios, and i represents the sequence number of the sample audios.

[0141] 404. If the trained acoustic model meets the training end condition, the trained acoustic model is determined as the target acoustic model.

[0142] If the training end condition is not met, another set of sample audios can be obtained and the above process can be repeated.

[0143] There are many options for training end conditions, and the following are some of them:

[0144] Condition 1: The specified number of training runs has been achieved. Condition 2: The loss value is less than the specified value. Condition 3: The loss value no longer shows a decreasing trend. Condition 4: The acoustic model is used to generate spectral data, which is then converted into audio using a vocoder. The generated audio is then compared with the audio used to extract the pronunciation rhythm features, and the matching degree reaches the specified value, or the matching degree reaches the specified value for multiple consecutive experiments.

[0145] The above training process can be a basic training process for the acoustic model. After obtaining the audio of the user speaking, the computer device can also use the audio of the user speaking as sample audio to fine-tune the acoustic model. Since the decoder is the model in the acoustic model that processes timbre features, in this training, only the decoder in the acoustic model can be trained. The specific processing process is similar to the steps 401 to 404 above and will not be repeated here. The number of training sessions for fine-tuning training can be much smaller than the number of training sessions for basic training.

[0146] In addition, during the above training process, the sample audio can be noised to obtain a noisy sample audio. The noisy sample audio is input into the PPG extractor to obtain a PPG matrix. The phoneme sequence, phoneme duration sequence and timbre features corresponding to the PPG matrix and the sample audio (either noisy sample audio or unnoisy sample audio) are then input into the acoustic model to be trained to obtain spectrum data. The unnoised sample audio is input into the spectrum acquisition model to obtain spectrum data. The acoustic model to be trained is trained with the purpose of minimizing the difference between the spectrum data output by the acoustic model to be trained and the spectrum data output by the spectrum acquisition model. If the trained acoustic model meets the training end condition, the trained acoustic model is determined as the target acoustic model. By using the noisy sample audio to train the acoustic model to be trained, with the purpose of making the spectrum data output by the acoustic model to be trained close to the spectrum data output by the spectrum acquisition model, the noise resistance of the acoustic model can be better trained.

[0147] The embodiment of the present application further provides a method for training a vocoder for the above application scenario. The processing flow of the method can be as follows: Figure 6 As shown, the corresponding model operation diagram can be Figure 7 The models with bold outlines in the figure represent models that need to be trained, and the models without bold outlines represent models that do not need to be trained, such as trained models or non-machine learning models. The processing steps include the following:

[0148] 601. Obtain a phoneme sequence, a phoneme duration sequence, and spectrum data corresponding to a second audio sample.

[0149] In implementations, the text corresponding to the second audio sample is input into a phoneme predictor to obtain a phoneme sequence. The second audio sample and its corresponding text are input into a phoneme duration determination model to obtain a phoneme duration sequence. The second audio sample can be input into a spectrum acquisition model to obtain spectrum data. Alternatively, the second audio sample can be input into a PPG extractor to obtain a PPG matrix, which is then input into a target acoustic model to obtain spectrum data.

[0150] 602. Input the phoneme sequence and phoneme duration sequence corresponding to the second sample audio into the trained first feature adjustment model to obtain a third adjustment feature output by the first feature adjustment model.

[0151] In an implementation, the phoneme sequence corresponding to the second sample audio is input into a third encoder to obtain a fourth phoneme feature. The fourth phoneme feature can specifically be a phoneme vector. The fourth phoneme feature and the phoneme duration sequence corresponding to the second sample audio are input into a second adjuster. The second adjuster can replicate each phoneme vector in the fourth phoneme feature a corresponding number of times based on the phoneme duration corresponding to each phoneme in the phoneme duration sequence to obtain a third adjusted feature. The third adjusted feature can specifically be a phoneme vector that has been corrected for phoneme duration.

[0152] 603 : Use the third adjustment feature to adjust the spectrum data corresponding to the second audio sample to obtain adjusted spectrum data corresponding to the second audio sample.

[0153] In implementation, the spectrum data corresponding to the third adjustment feature and the second sample audio are input into the second adder, and the spectrum data corresponding to the third adjustment feature and the second sample audio are added to obtain adjusted spectrum data.

[0154] 604 : Input the adjusted spectrum data corresponding to the second sample audio into the vocoder to be trained to obtain the predicted audio output by the vocoder to be trained.

[0155] At 605 , the vocoder to be trained is trained to minimize the difference between the predicted audio and the sample audio.

[0156] During training, the above differences can be specifically obtained by calculating the loss value of the vocoder. The loss value of the vocoder can be the weighted sum of the loss value of the generator and the loss value of the discriminator. The mathematical expressions of the loss function of the generator and the loss function of the discriminator can be as follows:

[0157] L G =L Adv (G; D)+C1L FM (G; D)+C2L Mel (G)

[0158] L D =L Adv (D; G)

[0159] Among them, L Adv (G; D) and L Adv (D; G) is the adversarial loss value, L FM (G; D) Feature matching loss value, L Mel (G) is the Mel spectrum loss value, C1 and C2 are constants that can be set by technicians according to their needs.

[0160] 606. If the trained vocoder meets the training end condition, the trained vocoder is determined as the target vocoder.

[0161] If the training end condition is not met, another set of sample audios can be obtained and the above process can be repeated.

[0162] There are many options for training end conditions, and the following are some of them:

[0163] Condition 1: The specified number of training cycles has been reached. Condition 2: The loss value is less than the specified value. Condition 3: The loss value no longer shows a decreasing trend. Condition 4: A vocoder is used to generate audio based on the spectral data, and the generated audio is compared with the sample audio. The matching degree must reach the specified value, or the matching degree must reach the specified value after multiple consecutive experiments.

[0164] In addition, during the above training process, the sample audio can be noised to obtain a noisy sample audio. The noisy sample audio is input into the target acoustic model or the spectrum acquisition model to obtain spectrum data. The spectrum data and the phoneme sequence and phoneme duration sequence corresponding to the sample audio (either the noisy sample audio or the unnoisy sample audio) are then input into the vocoder to be trained to obtain the predicted audio. The vocoder to be trained is trained with the purpose of minimizing the difference between the predicted audio and the unnoisy sample audio. If the trained vocoder meets the training end condition, the trained vocoder is determined as the target vocoder.

[0165] By the method provided in the embodiment of the present application, a phoneme sequence and a phoneme duration sequence are adopted. The phoneme sequence is generally obtained from the text corresponding to the audio and is completely unaffected by the noise in the audio. The phoneme duration sequence is generally obtained based on the audio and the text corresponding to the audio and is very little affected by the noise in the audio. In addition, the phoneme sequence can provide the pronunciation information of the audio, and the phoneme duration sequence can provide the rhythm information of the audio. Therefore, in addition to using the pronunciation rhythm features and timbre features as the input of the model, using the phoneme sequence and the phoneme duration sequence as the input of the model at the same time to generate audio can reduce the impact of noise on the generated audio. Further, the quality of the generated audio can be improved.

[0166] Based on the same technical concept, the embodiment of the present application also provides a device for generating audio, such as Figure 8 As shown, the device includes:

[0167] An acquisition module 810 is configured to acquire the timbre characteristics of the first character, and acquire the pronunciation rhythm characteristics, phoneme sequence, and phoneme duration sequence corresponding to the reading audio of the second character;

[0168] The generation module 820 is used to input the timbre characteristics of the first character and the pronunciation rhythm characteristics, phoneme sequence and phoneme duration sequence corresponding to the second character into a pre-trained target acoustic model to obtain spectral data output by the target acoustic model; input the phoneme sequence and phoneme duration sequence into a pre-trained first feature adjustment model, and the first feature adjustment model uses the phoneme duration sequence to adjust the length of the phoneme sequence and output a first adjustment feature, wherein the length of the first adjustment feature is the same as the length of the spectral data; use the first adjustment feature to adjust the spectral data to obtain adjusted spectral data; input the adjusted spectral data into a pre-trained target vocoder to obtain the target audio output by the target vocoder.

[0169] In one possible implementation, the target acoustic model includes a first encoder, a decoder, a second feature adjustment model, and a first superimposer;

[0170] Generation module 820 is used to: input the pronunciation rhythm feature corresponding to the second character into a pre-trained first encoder to obtain a first coding feature output by the first encoder; input the phoneme sequence corresponding to the second character and the phoneme duration sequence corresponding to the second character into a pre-trained second feature adjustment model, and the second feature adjustment model uses the phoneme duration sequence corresponding to the second character to adjust the length of the phoneme sequence corresponding to the second character and then output a second adjustment feature, wherein the second adjustment feature has the same length as the first coding feature; input the second adjustment feature and the first coding feature into a first superimposer to obtain a second coding feature output by the first superimposer; input the second coding feature and the timbre feature of the first character into a pre-trained decoder to obtain spectral data output by the decoder.

[0171] In one possible implementation, the second feature adjustment model includes a second encoder and a first adjuster;

[0172] Generation module 820 is used to: input the phoneme sequence corresponding to the second character into a pre-trained second encoder to obtain the first phoneme feature output by the second encoder; input the first phoneme feature and the phoneme duration sequence into the first regulator, and the first regulator uses the phoneme duration sequence corresponding to the second character to adjust the length of the first phoneme feature and then output the second adjusted feature.

[0173] In one possible implementation, the first feature conditioning model includes a third encoder and a second conditioner;

[0174] Generation module 820 is used to: input the phoneme sequence into a pre-trained third encoder to obtain a second phoneme feature output by the third encoder; input the second phoneme feature and the phoneme duration sequence into a second regulator, and the second regulator uses the phoneme duration sequence to adjust the length of the second phoneme feature and then output a first adjusted feature.

[0175] In one possible implementation, the generating module 820 is configured to:

[0176] The first adjustment feature and the spectrum data are input into a second adder, and the second adder performs positional addition on the first adjustment feature and the spectrum data to obtain and output adjusted spectrum data.

[0177] In a possible implementation, the acquisition module 810 is configured to:

[0178] The reading audio of the second person and the text corresponding to the reading audio are input into the phoneme duration determination model to obtain a phoneme duration sequence corresponding to the second person.

[0179] In a possible implementation, the apparatus further includes a training module 830, specifically:

[0180] An acquisition module 810 is configured to acquire timbre features, pronunciation rhythm features, phoneme sequences, phoneme duration sequences, and spectrum data corresponding to the first audio sample;

[0181] A generation module 820 is configured to input the timbre characteristics, pronunciation rhythm characteristics, phoneme sequence, and phoneme duration sequence corresponding to the first sample audio into the acoustic model to be trained, to obtain spectrum data output by the acoustic model to be trained;

[0182] The training module 830 is used to train the acoustic model to be trained with the training purpose of minimizing the difference between the spectral data corresponding to the first sample audio and the spectral data output by the acoustic model to be trained; if the trained acoustic model meets the training end condition, the trained acoustic model is determined as the target acoustic model.

[0183] In a possible implementation, the acquisition module 810 is further configured to: acquire a phoneme sequence, a phoneme duration sequence, and spectrum data corresponding to the second sample audio;

[0184] The generation module 820 is configured to: input the phoneme sequence and phoneme duration sequence corresponding to the second audio sample into the first feature adjustment model to obtain a third adjustment feature output by the trained first feature adjustment model; adjust the spectrum data corresponding to the second audio sample using the third adjustment feature to obtain adjusted spectrum data corresponding to the second audio sample; and input the adjusted spectrum data corresponding to the second audio sample into the vocoder to be trained to obtain predicted audio output by the vocoder to be trained;

[0185] The training module 830 is configured to train the vocoder to be trained so as to minimize the difference between the predicted audio and the sample audio; and determine the trained vocoder as the target vocoder if the trained vocoder meets the training end condition.

[0186] The device provided by the embodiment of the present application adopts a phoneme sequence and a phoneme duration sequence. The phoneme sequence is generally obtained from the text corresponding to the audio and is completely unaffected by the noise in the audio. The phoneme duration sequence is generally obtained based on the audio and the text corresponding to the audio and is very little affected by the noise in the audio. In addition, the phoneme sequence can provide the pronunciation information of the audio, and the phoneme duration sequence can provide the rhythm information of the audio. Therefore, in addition to using the pronunciation rhythm features and timbre features as the input of the model, using the phoneme sequence and the phoneme duration sequence as the input of the model at the same time to generate audio can reduce the impact of noise on the generated audio. Further, the quality of the generated audio can be improved.

[0187] It should be noted that the audio generation device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate audio generation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio generation device provided in the above embodiment and the audio generation method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0188] Figure 9 The following is a block diagram of the structure of an electronic device 900 provided in an embodiment of the present application. The electronic device may be a terminal in the above-mentioned embodiment. The electronic device 900 may be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player (moving picture experts group audio layer III), an MP4 player (moving picture experts group audio layer IV), a laptop computer, or a desktop computer. The electronic device 900 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0189] Typically, the electronic device 900 includes a processor 901 and a memory 902 .

[0190] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (digital signal processing), FPGA (field-programmable gate array), or PLA (programmable logic array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (central processing unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (graphics processing unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (artificial intelligence) processor, which is used to process computing operations related to machine learning.

[0191] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is used to be executed by the processor 901 to implement the method provided in the embodiment of the present application.

[0192] In some embodiments, electronic device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0193] The peripheral device interface 903 can be used to connect at least one I / O (input / output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0194] The radio frequency circuit 904 is used to receive and transmit RF (radio frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (wireless fidelity) network. In some embodiments, the radio frequency circuit 904 may also include circuits related to near field communication (NFC), which is not limited in this application.

[0195] The display screen 905 is used to display a user interface (UI). This UI can include graphics, text, icons, videos, or any combination thereof. If the display screen 905 is a touchscreen display, it can also capture touch signals on or above the surface of the display screen 905. These touch signals can be input as control signals to the processor 901 for processing. In this case, the display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 905, located on the front panel of the electronic device 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of the electronic device 900 or in a foldable design. In still other embodiments, the display screen 905 can be a flexible display, located on a curved or foldable surface of the electronic device 900. Furthermore, the display screen 905 can be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. The display screen 905 can be made of materials such as LCD (liquid crystal display) and OLED (organic light-emitting diode).

[0196] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (virtual reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0197] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the electronic device 900. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0198] The positioning component 908 is used to locate the current geographic location of the electronic device 900 to implement navigation or LBS (location-based service). The positioning component 908 can be a positioning component based on the GPS (global positioning system), Beidou system or Galileo system.

[0199] The power supply 909 is used to power the various components in the electronic device 900. The power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0200] In some embodiments, the electronic device 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0201] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the electronic device 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0202] The gyroscope sensor 912 can detect the orientation and rotation angle of the electronic device 900. It can work in conjunction with the accelerometer 911 to collect 3D motions of the user on the electronic device 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0203] The pressure sensor 913 can be set on the side frame of the electronic device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the electronic device 900, it can detect the user's grip signal of the electronic device 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0204] The fingerprint sensor 914 is used to collect the user's fingerprint, and the processor 901 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 901 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be set on the front, back, or side of the electronic device 900. When a physical button or manufacturer logo is set on the electronic device 900, the fingerprint sensor 914 can be integrated with the physical button or manufacturer logo.

[0205] The optical sensor 915 is used to detect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity detected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity detected by the optical sensor 915.

[0206] Proximity sensor 916, also known as a distance sensor, is typically located on the front panel of electronic device 900. 916 is used to detect the distance between the user and the front of electronic device 900. In one embodiment, when 916 detects that the distance between the user and the front of electronic device 900 is gradually decreasing, processor 901 controls display screen 905 to switch from the screen-on state to the screen-off state. When 916 detects that the distance between the user and the front of electronic device 900 is gradually increasing, processor 901 controls display screen 905 to switch from the screen-off state to the screen-on state.

[0207] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the electronic device 900, and the electronic device 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0208] In an embodiment of the present application, a computer-readable storage medium is further provided, such as a memory including instructions, wherein the instructions can be executed by a processor in a terminal to perform the method for performing interactive operations in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (read-only memory), a RAM (random access memory), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0209] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0210] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0211] The above descriptions are only some possible embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating audio, characterized in that The method comprises: Acquire the timbre characteristics of the first person, and acquire the pronunciation rhythm characteristics, phoneme sequence, and phoneme duration sequence corresponding to the reading audio of the second person; Inputting the timbre characteristics of the first character, the pronunciation rhythm characteristics corresponding to the second character, the phoneme sequence, and the phoneme duration sequence into a pre-trained target acoustic model to obtain spectrum data output by the target acoustic model; Inputting the phoneme sequence and the phoneme duration sequence into a pre-trained first feature adjustment model, wherein the first feature adjustment model adjusts the length of the phoneme sequence using the phoneme duration sequence and outputs a first adjustment feature, wherein the length of the first adjustment feature is the same as the length of the spectrum data; adjusting the spectrum data using the first adjustment feature to obtain adjusted spectrum data; The adjusted spectrum data is input into a pre-trained target vocoder to obtain a target audio output by the target vocoder.

2. The method according to claim 1, characterized in that The target acoustic model includes a first encoder, a decoder, a second feature adjustment model and a first superimposer; The step of inputting the timbre characteristics of the first character, the pronunciation rhythm characteristics corresponding to the second character, the phoneme sequence, and the phoneme duration sequence into a pre-trained target acoustic model to obtain spectrum data output by the target acoustic model includes: Inputting the pronunciation rhythm feature corresponding to the second character into the pre-trained first encoder to obtain a first encoding feature output by the first encoder; Inputting the phoneme sequence corresponding to the second character and the phoneme duration sequence corresponding to the second character into the pre-trained second feature adjustment model, the second feature adjustment model adjusting the length of the phoneme sequence corresponding to the second character using the phoneme duration sequence corresponding to the second character and outputting a second adjustment feature, wherein the second adjustment feature has the same length as the first encoding feature; Inputting the second adjustment feature and the first coding feature into the first superimposer, and having the first superimposer perform positional addition on the second adjustment feature and the first coding feature to obtain and output a second coding feature; The second coding feature and the timbre feature of the first character are input into the pre-trained decoder to obtain the spectrum data output by the decoder.

3. The method according to claim 2, characterized in that The second feature conditioning model includes a second encoder and a first conditioner; The step of inputting the phoneme sequence corresponding to the second character and the phoneme duration sequence corresponding to the second character into a pre-trained second feature adjustment model, and the second feature adjustment model adjusting the length of the phoneme sequence corresponding to the second character using the phoneme duration sequence corresponding to the second character and then outputting a second adjustment feature, includes: Inputting the phoneme sequence corresponding to the second character into the pre-trained second encoder to obtain a first phoneme feature output by the second encoder; The first phoneme feature and the phoneme duration sequence are input into the first regulator, and the first regulator uses the phoneme duration sequence corresponding to the second character to adjust the length of the first phoneme feature and then outputs the second adjusted feature.

4. The method according to claim 1, wherein The first feature conditioning model includes a third encoder and a second conditioner; The step of inputting the phoneme sequence and the phoneme duration sequence into a pre-trained first feature adjustment model, and having the first feature adjustment model adjust the length of the phoneme sequence using the phoneme duration sequence and then outputting a first adjustment feature, includes: Inputting the phoneme sequence into the pre-trained third encoder to obtain a second phoneme feature output by the third encoder; The second phoneme feature and the phoneme duration sequence are input into the second regulator, and the second regulator uses the phoneme duration sequence to adjust the length of the second phoneme feature and then outputs the first adjusted feature.

5. The method according to claim 1, characterized in that The adjusting the spectrum data using the first adjustment feature to obtain adjusted spectrum data includes: The first adjustment feature and the spectrum data are input into a second adder, and the second adder performs a position-wise addition on the first adjustment feature and the spectrum data to obtain and output the adjusted spectrum data.

6. The method according to claim 1, characterized in that The step of obtaining a phoneme duration sequence corresponding to the second character's reading audio includes: The reading audio of the second character and the text corresponding to the reading audio are input into a phoneme duration determination model to obtain a phoneme duration sequence corresponding to the second character.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtaining timbre features, pronunciation rhythm features, phoneme sequence, phoneme duration sequence, and spectrum data corresponding to the first sample audio; Inputting the timbre feature, the pronunciation rhythm feature, the phoneme sequence, and the phoneme duration sequence corresponding to the first sample audio into the acoustic model to be trained, to obtain spectrum data output by the acoustic model to be trained; The acoustic model to be trained is trained with the training purpose of minimizing the difference between the spectral data corresponding to the first sample audio and the spectral data output by the acoustic model to be trained; if the trained acoustic model meets the training end condition, the trained acoustic model is determined as the target acoustic model.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtaining a phoneme sequence, a phoneme duration sequence, and spectrum data corresponding to the second sample audio; Inputting the phoneme sequence and the phoneme duration sequence corresponding to the second sample audio into the trained first feature adjustment model to obtain a third adjustment feature output by the first feature adjustment model; adjusting the frequency spectrum data corresponding to the second audio sample using the third adjustment feature to obtain adjusted frequency spectrum data corresponding to the second audio sample; Inputting the adjusted spectrum data corresponding to the second sample audio into a vocoder to be trained to obtain a predicted audio output by the vocoder to be trained; The vocoder to be trained is trained to minimize the difference between the predicted audio and the sample audio; if the trained vocoder meets a training end condition, the trained vocoder is determined as the target vocoder.

9. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory is used to store computer instructions; The processor executes the computer instructions stored in the memory to enable the computer device to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program code. In response to the computer program code being executed by a computer device, the computer device executes the method according to any one of claims 1 to 8.

Images