Audio processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide an audio processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining first audio data having first text content, a first tone and a first rhythm; processing the first audio data through an audio processing model to obtain content features of second audio data, rhythm features of the second audio data and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second tone of the second audio data is different from the first tone; a second rhythm of the second audio data is the same as the first rhythm or is the same as a third rhythm of the second tone; and through the audio processing model, according to the content feature of the second audio data, the rhythm feature of the second audio data and the tone feature of the second tone, generating the second audio data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] In order to meet the audio processing needs of users in audio and video entertainment, virtual social networking and other scenarios, the existing technology provides a variety of audio processing methods, such as cutting the length of audio data, changing the timbre of audio data, etc. Among them, changing the timbre of audio data is one of the key directions of audio processing. Based on this, it is necessary to provide a technical solution to change the timbre of audio data and achieve high-quality audio processing. Summary of the invention

[0003] The embodiments of the present disclosure provide an audio processing method, an apparatus, an electronic device, and a storage medium, which can change the timbre of audio data and achieve high-quality audio processing.

[0004] In a first aspect, an embodiment of the present disclosure provides an audio processing method, including:

[0005] Acquire first audio data having first text content, first timbre and first rhythm;

[0006] The first audio data is processed by an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0007] The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre.

[0008] In a second aspect, an embodiment of the present disclosure provides an audio processing device, including:

[0009] A data acquisition unit, configured to acquire first audio data having first text content, a first timbre, and a first rhythm;

[0010] a feature acquisition unit, configured to process the first audio data through an audio processing model to acquire content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0011] The audio generation unit is used to generate the second audio data according to the content characteristics of the second audio data, the rhythm characteristics of the second audio data and the timbre characteristics of the second timbre through the audio processing model.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0014] In one or more embodiments of the present disclosure, first, a first audio data having a first text content, a first timbre and a first rhythm is obtained, then, the first audio data is processed through an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data and timbre features of the second timbre, the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre, and finally, the second audio data is generated according to the content features of the second audio data, the rhythm features of the second audio data and the timbre features of the second timbre through the audio processing model. It can be seen that through this embodiment, the first audio data can be processed into the second audio data with the same content and different timbre through the audio processing model, so as to achieve the effect of changing the timbre of the audio data, and the second rhythm of the second audio data can be made the same as the first rhythm of the first audio data or the same as the third rhythm of the second timbre, so as to achieve the effect of changing the rhythm while changing the timbre or keeping the rhythm unchanged, so as to achieve high-quality audio processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0016] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of the reasoning process of an audio processing model provided in one embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of the training process of an audio processing model provided in one embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0022] The embodiment of the present disclosure provides an audio processing method, which can change the timbre of audio data and achieve high-quality audio processing. The audio processing method can be applied to a server and executed by the server.

[0023] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the process includes the following steps:

[0024] Step S102, obtaining first audio data having first text content, first timbre and first rhythm;

[0025] Step S104, processing the first audio data through the audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0026] Step S106: Generate second audio data through an audio processing model according to content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre.

[0027] In this embodiment, first, the first audio data having the first text content, the first timbre and the first rhythm is obtained, then, the first audio data is processed by the audio processing model to obtain the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre, the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, the second rhythm of the second audio data is the same as the first rhythm or the third rhythm of the second timbre, and finally, the second audio data is generated according to the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre by the audio processing model. It can be seen that through this embodiment, the first audio data can be processed into the second audio data with the same content and different timbre by the audio processing model, so as to achieve the effect of changing the timbre of the audio data, and the second rhythm of the second audio data can be made the same as the first rhythm of the first audio data or the third rhythm of the second timbre, so as to achieve the effect of changing the rhythm while changing the timbre or keeping the rhythm unchanged, so as to achieve high-quality audio processing.

[0028] Below Figure 1 The method flow is described in detail.

[0029] In the above step S102, the first audio data having the first text content, the first timbre and the first rhythm is obtained. In this embodiment, the first audio data includes a speech narration, a cappella songs, a nursery rhyme read aloud, etc. performed by a human voice or a robot imitating a human voice. The first text content of the first audio data is the text content corresponding to the first audio data, such as "Hello, today is September 10th, Teacher's Day". The first timbre of the first audio data is the timbre of the human voice or the robot's voice in the first audio data. The first rhythm of the first audio data is the tone, pitch, speed of the sound, strength of the sound, etc. when the human voice or the robot's voice in the first audio data reads or sings the first text content aloud.

[0030] The text mentioned in each embodiment of this specification refers to a written language in the form of characters. For example, a piece of novel text on the Internet is a piece of text. The simplest text can be exemplified as "Hello". The rhythm mentioned in each embodiment of this specification refers to information such as tone, pitch, speed of sound, strength of sound, etc. Audio data includes three elements: timbre, content, and rhythm. Except for content and timbre, the rest of the information in the audio data heard by human ears can be classified as rhythm.

[0031] In the above step S104, the first audio data is processed by the audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre. The second text content of the second audio data is the same as the first text content. The second timbre of the second audio data is different from the first timbre. The second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre.

[0032] Similar to the first audio data, in this embodiment, the second audio data includes a human voice or a robot imitating a human voice to speak, a cappella song, a nursery rhyme, etc. The second text content of the second audio data is the text content corresponding to the second audio data, such as "Hello, today is September 10th, Teacher's Day". The second timbre of the second audio data is the timbre of the human voice or the robot's voice in the second audio data. The second rhythm of the second audio data is the tone, pitch, speed of the voice, strength of the voice, etc. when the human voice or the robot's voice in the second audio data reads or sings the second text content.

[0033] In one example, the first audio data is a text narrated by user A, and the text content is: "Hello, today is September 10th, Teacher's Day", then the second text content of the second audio data is the same as the first text content of the first audio data, both of which are "Hello, today is September 10th, Teacher's Day". The first timbre of the first audio data is a male timbre, and the second timbre of the second audio data is a female timbre. The second rhythm of the second audio data is the same as the first rhythm of the first audio data, or the second rhythm of the second audio data is the same as the third rhythm of the second timbre.

[0034] In this embodiment, the first audio data has a first rhythm, the second audio data has a second rhythm, and the second timbre has a third rhythm. The first rhythm of the first audio data is the same as the second rhythm of the second audio data, which means that the tone, pitch, speed, strength, etc. of the human voice or robot voice in the first audio data when reading or singing the first text content at each moment are the same as the tone, pitch, speed, strength, etc. of the human voice or robot voice in the second audio data when reading or singing the second text content at each moment. In one embodiment, the first rhythm of the first audio data is the same as the second rhythm of the second audio data, which can be expressed as the rhythmic feature of the first audio data is the same as the rhythmic feature of the second audio data. The rhythmic feature of the audio data can be represented by a vector, and whether the two rhythmic features are the same can be determined by determining whether the values ​​of the two vectors are the same.

[0035] The third rhythm of the second timbre represents the average tone, average pitch, average sound speed, average strength of the sound and other information when the second timbre reads or sings various contents. When the second timbre reads or sings the second text content, the second rhythm of the second audio data is the same as the third rhythm of the second timbre, which means that the average tone, average pitch, average sound speed, average strength of the sound and other information at each moment when the human voice or robot voice in the second audio data reads or sings the second text content are the same as the rhythm of the second timbre.

[0036] It can be seen that through this embodiment, the first audio data can be processed into the second audio data with the same content but different timbre, so as to achieve the effect of changing the timbre of the audio data. Moreover, compared with the technical solution in the prior art that can only change the timbre of the audio data but cannot control the change effect of the rhythm of the audio data, the rhythm of the audio data can be controllably changed, and the second rhythm of the second audio data can be made the same as the first rhythm of the first audio data, so as to achieve the effect of changing the timbre while keeping the rhythm unchanged, or the second rhythm of the second audio data can be made the same as the third rhythm of the second timbre, so as to achieve the effect of changing the timbre and changing the rhythm accordingly, so that the timbre and rhythm of the second audio data are harmonious and consistent.

[0037] Furthermore, in this embodiment, the language of the first audio data is not limited, and the first audio data in any language can be processed to obtain the second audio data, thereby achieving the voice changing effect of audio data in different languages.

[0038] In the above step S104, the first audio data is processed by the audio processing model to obtain the content features of the second audio data, the rhythm features of the second audio data, and the timbre features of the second timbre. In this embodiment, the audio processing model is pre-trained, and the audio processing model is used to process the first audio data into the second audio data. Through the audio processing model, the content features of the second audio data, the rhythm features of the second audio data, and the timbre features of the second timbre can be obtained to generate the second audio data.

[0039] In one embodiment, the second rhythm of the second audio data is the same as the third rhythm of the second timbre; and the first audio data is processed by the audio processing model to obtain content features of the second audio data and rhythm features of the second audio data, including:

[0040] Extracting, by an audio encoding module in the audio processing model, content features of the first audio data as content features of the second audio data;

[0041] The prosody feature of the second audio data is predicted based on the content feature of the first audio data and the prosody information of the third prosody through the prosody prediction module in the audio processing model.

[0042] The content features of the second audio data include but are not limited to the text features of the text content of the second audio data at each moment. The rhythm features of the second audio data include but are not limited to the tone, pitch, sound speed, sound intensity, etc. of the second audio data at each moment.

[0043] Figure 2 A schematic diagram of the inference process of an audio processing model provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the trained audio processing model includes an audio encoding module and a rhythm prediction module. In the process of processing the first audio data through the audio processing model to generate the second audio data, if the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the spectral features of the first audio data are obtained, and the spectral features of the first audio data are input into the audio encoding module. The audio encoding module processes the spectral features of the first audio data and extracts the content features of the first audio data as the content features of the second audio data.

[0044] In one example, the first audio data is subjected to Fourier transformation and Mel filtering to obtain a Mel spectrogram of the first audio data as the spectral feature of the first audio data. The spectral feature of the first audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the first audio data at the corresponding frequency at the corresponding moment. The spectral feature of the first audio data is input into an audio encoding module, and the audio encoding module processes the spectral feature of the first audio data to obtain the content feature of the first audio data as the content feature of the second audio data. The content feature of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension, 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature of the second audio data at the corresponding moment, which is represented by a vector of size [1, 512].

[0045] In one example, the audio coding module can be an encoder in an ASR (Automatic Speech Recognition) model, which is composed of 18 Conformers (Convolution-augmented Transformer for Speech Recognition) connected in sequence, and the output of the last Conformer is the content feature of the second audio data. When training the audio coding module, audio data in multiple languages ​​can be used for training, so that the audio coding module has the ability to process audio data in multiple languages, so that the audio processing model can process the first audio data in any language to obtain the second audio data, and achieve the voice changing effect of audio data in different languages. In one example, the audio coding module is trained with audio data in English and Chinese, so that the audio coding module can have the ability to process audio data in multiple languages ​​(including English and Chinese).

[0046] like Figure 2 As shown, the trained audio processing model also includes a rhythm prediction module. In the process of processing the first audio data through the audio processing model to generate the second audio data, if the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the content features of the second audio data output by the audio encoding module are input into the rhythm prediction module. The rhythm prediction module predicts the rhythm features of the second audio data based on the content features of the input second audio data and the rhythm information of the third rhythm stored in the rhythm prediction module.

[0047] According to the previous introduction, the rhythmic information of the third rhythm of the second timbre includes the average tone, average pitch, average speed of sound, average strength of sound and other information when the second timbre reads or sings various contents, and the rhythmic features of the second audio data include the tone at each moment, the pitch at each moment, the speed of sound at each moment, the strength of sound at each moment and other information when the human voice or robot voice in the second audio data reads or sings the second text content. Therefore, predicting the rhythmic features of the second audio data based on the content features of the second audio data and the rhythmic information of the third rhythm is a process of predicting the tone at each moment, the pitch at each moment, the speed of sound at each moment, the strength of sound at each moment and other information according to the specific content of the second audio data and the average tone, average pitch, average speed of sound, average strength of sound and other information represented by the third rhythm.

[0048] In order to achieve the effect of predicting the tone at each moment, the pitch at each moment, the speed of the sound at each moment, the strength of the sound at each moment and other information, the input of the rhythm prediction module includes the content features of the second audio data, and the rhythm information of the third rhythm is learned by the rhythm prediction module during the model training process and stored in the rhythm prediction module as a model parameter. The rhythm prediction module can take the content features of the second audio data as input, and predict the tone at each moment, the pitch at each moment, the speed of the sound at each moment, the strength of the sound at each moment and other information when the human voice or robot voice in the second audio data reads or sings the second text content according to the content features of the second audio data and the rhythm information of the third rhythm of the second timbre, as the rhythm features of the second audio data.

[0049] In one embodiment, the audio processing model learns the ability to transform the timbre of audio data into a variety of preset timbres during training, and the rhythm prediction module learns the rhythm information of various preset timbres during training and stores the rhythm information of each preset timbre as a model parameter in the rhythm prediction module. Based on this, when a user uses the audio processing model in this embodiment to achieve a voice change effect, the user can input the first audio data and specify the second timbre of the second audio data to be transformed and the second rhythm of the second audio data, the second timbre is in the above-mentioned preset timbre, and when the audio processing model determines that the second rhythm is the same as the third rhythm of the second timbre, the rhythm prediction module in the audio processing model predicts the rhythm characteristics of the second audio data based on the content characteristics of the second audio data and the rhythm information of the stored third rhythm.

[0050] In one example, the prosodic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding prosodic feature of the second audio data at the corresponding moment, represented by a vector of size [1, 512].

[0051] It can be seen that, through this embodiment, the audio processing model implements the rhythm prediction function through the rhythm prediction module, and the rhythm prediction function can achieve the effect of changing the rhythm of the first audio data.

[0052] In one embodiment, the second rhythm of the second audio data is the same as the first rhythm of the first audio data; and the first audio data is processed by the audio processing model to obtain content features of the second audio data and rhythm features of the second audio data, including:

[0053] Extracting, by an audio encoding module in the audio processing model, content features of the first audio data as content features of the second audio data;

[0054] The prosody features of the first audio data are extracted as prosody features of the second prosody through the audio encoding module in the audio processing model.

[0055] The content features of the second audio data include but are not limited to the text features of the text content of the second audio data at each moment. The rhythm features of the second audio data include but are not limited to the tone, pitch, sound speed, sound intensity, etc. of the second audio data at each moment.

[0056] refer to Figure 2 As described above, the process of extracting the content features of the first audio data as the content features of the second audio data through the audio encoding module in the audio processing model is the same as the previous description and will not be repeated here.

[0057] In this embodiment, the spectral features of the first audio data are obtained, and the spectral features of the first audio data are input into the audio encoding module. The audio encoding module processes the spectral features of the first audio data and extracts the rhythmic features of the first audio data as the rhythmic features of the second audio data.

[0058] In one example, the first audio data is subjected to Fourier transformation and Mel filtering to obtain a Mel spectrogram of the first audio data as the spectral feature of the first audio data. The spectral feature of the first audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the first audio data at the corresponding frequency at the corresponding moment. The spectral feature of the first audio data is input into an audio encoding module, and the audio encoding module processes the spectral feature of the first audio data to obtain the rhythmic feature of the first audio data as the rhythmic feature of the second audio data. The rhythmic feature of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension, 512 represents the feature dimension, and each value in the matrix represents the corresponding rhythmic feature of the second audio data at the corresponding moment, which is represented by a vector of size [1, 512].

[0059] In one example, the audio encoding module may be an encoder in an ASR model, which is composed of 18 Conformers connected in sequence, the output of the last layer is content features, and the output of the middle layer is rhythm features. For example, the output of the 12th Conformer is the rhythm features of the second audio data.

[0060] It can be seen that through this embodiment, prosodic feature extraction and content feature extraction can be implemented through the audio encoding module, so that multiple functions can be realized through the same module. There is no need to distribute the modules for extracting content features and the modules for extracting prosodic features, thereby improving the efficiency of audio processing.

[0061] In one embodiment, obtaining the timbre characteristics of the second timbre includes:

[0062] Acquire the timbre feature of the second timbre from the timbre feature library of the audio processing model.

[0063] In this embodiment, the audio processing model learns the ability to transform the timbre of audio data into a variety of preset timbres during training, the rhythm prediction module learns the rhythm information of various preset timbres during training and stores the rhythm information of each preset timbre as a model parameter in the rhythm prediction module, and after the audio processing model is trained, the timbre features of various preset timbres are stored in the timbre feature library. Based on this, when the user uses the audio processing model in this embodiment to achieve the voice change effect, the user can input the first audio data and specify the second timbre of the second audio data to be transformed and the second rhythm of the second audio data. The second timbre is located in the above-mentioned preset timbre, so that the audio processing model extracts the timbre features of the second timbre from the timbre feature library. In this embodiment, the audio processing model can also store the timbre features of various preset timbres in the timbre feature library during training.

[0064] It can be seen that through this embodiment, the timbre features of various timbre can be stored in the timbre feature library of the audio processing model. During the audio processing process, the timbre features of the second timbre can be directly obtained from the timbre feature library, eliminating the process of extracting timbre features and improving the audio processing efficiency.

[0065] Since audio data includes three elements: timbre, content and rhythm, the above step S106 also generates second audio data according to the content features of the second audio data, the rhythm features of the second audio data and the timbre features of the second timbre through the audio processing model.

[0066] In one embodiment, when the second rhythm of the second audio data is the same as the third rhythm of the second timbre, generating the second audio data according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre by using the audio processing model includes:

[0067] Generate, by the first timbre conversion module in the audio processing model, a frequency spectrum feature corresponding to the second audio data according to the content feature of the second audio data output by the audio encoding module, the rhythm feature of the second audio data output by the rhythm prediction module, and the timbre feature of the second timbre;

[0068] The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

[0069] In this embodiment, reference Figure 2 As shown, the audio processing model also includes a first timbre conversion module and a vocoder. When the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the first timbre conversion module has the ability to synthesize audio data according to content features, rhythm features and timbre features. Therefore, the first timbre conversion module generates a spectral feature corresponding to the second audio data according to the content features of the second audio data output by the audio encoding module, the rhythm features of the second audio data output by the rhythm prediction module, and the timbre features of the second timbre. Among them, the first timbre conversion module can obtain the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0070] In one example, the content features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the rhythmic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding rhythmic features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the timbre features of the second timbre can be represented by a matrix of size [100, 256], where 100 represents the time dimension and 256 represents the feature dimension, and each value in the matrix represents the timbre features of the second timbre at the corresponding moment, represented by a vector of size [1, 256].

[0071] In this embodiment, the content features of the second audio data output by the audio encoding module, the rhythm features of the second audio data output by the rhythm prediction module, and the timbre features of the second timbre are feature spliced ​​to obtain a splicing result [100, 512+512+256], and the splicing result is input to the first timbre conversion module, and the first timbre conversion module generates a spectral feature corresponding to the second audio data based on the splicing result. The spectral feature corresponding to the second audio data is represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the second audio data at the corresponding frequency at the corresponding time. The spectral feature corresponding to the second audio data can be represented by a Mel-spectrogram.

[0072] The first timbre conversion module is connected to the vocoder, the input of the vocoder includes the frequency spectrum characteristics corresponding to the second audio data output by the first timbre conversion module, and the vocoder generates the second audio data according to the frequency spectrum characteristics corresponding to the second audio data.

[0073] It can be seen that through this embodiment, when the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the frequency spectrum characteristics corresponding to the second audio data can be generated by the first timbre conversion module, and the second audio data can be generated by the voice coder according to the frequency spectrum characteristics corresponding to the second audio data, thereby efficiently and quickly processing the first audio data into the second audio data, thereby improving the audio processing efficiency.

[0074] In one embodiment, when the second rhythm of the second audio data is the same as the first rhythm of the first audio data, generating the second audio data according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre by using the audio processing model includes:

[0075] Generate, by a second timbre conversion module in the audio processing model, a spectral feature corresponding to the second audio data according to content features of the second audio data output by the audio encoding module, rhythm features of the second audio data output by the audio encoding module, and timbre features of the second timbre;

[0076] The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

[0077] In this embodiment, reference Figure 2 As shown, the audio processing model also includes a second timbre conversion module and a vocoder. When the second rhythm of the second audio data is the same as the first rhythm of the first audio data, the second timbre conversion module has the ability to synthesize audio data according to content features, rhythm features and timbre features. Therefore, the second timbre conversion module generates a spectral feature corresponding to the second audio data according to the content features of the second audio data output by the audio encoding module, the rhythm features of the second audio data output by the audio encoding module, and the timbre features of the second timbre. Among them, the second timbre conversion module can obtain the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0078] In one example, the content features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the rhythmic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding rhythmic features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the timbre features of the second timbre can be represented by a matrix of size [100, 256], where 100 represents the time dimension and 256 represents the feature dimension, and each value in the matrix represents the timbre features of the second timbre at the corresponding moment, represented by a vector of size [1, 256].

[0079] In this embodiment, the content features of the second audio data output by the audio encoding module, the rhythmic features of the second audio data output by the audio encoding module, and the timbre features of the second timbre are feature spliced ​​to obtain a splicing result [100, 512+512+256], and the splicing result is input to the second timbre conversion module, and the second timbre conversion module generates a spectral feature corresponding to the second audio data based on the splicing result. The spectral feature corresponding to the second audio data is represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the second audio data at the corresponding frequency at the corresponding time. The spectral feature corresponding to the second audio data can be represented by a Mel-spectrogram.

[0080] The second timbre conversion module is connected to the vocoder, the input of the vocoder includes the frequency spectrum characteristics corresponding to the second audio data output by the second timbre conversion module, and the vocoder generates the second audio data according to the frequency spectrum characteristics corresponding to the second audio data.

[0081] It can be seen that through this embodiment, when the second rhythm of the second audio data is the same as the first rhythm of the first audio data, the frequency spectrum characteristics corresponding to the second audio data can be generated by the second timbre conversion module, and the second audio data can be generated by the voice coder according to the frequency spectrum characteristics corresponding to the second audio data, so that the first audio data can be efficiently and quickly processed into the second audio data, thereby improving the audio processing efficiency.

[0082] like Figure 2 As shown, the audio processing model also includes a switching module. In one example, when the user uses the audio processing model in this embodiment to achieve a voice change effect, the user can input the first audio data and specify the second timbre of the second audio data to be transformed and the second rhythm of the second audio data, and the second timbre is located in the above-mentioned preset timbre. After the spectral characteristics of the first audio data, the second timbre specified by the user, and the second rhythm specified by the user are input into the audio processing model, the switching module can receive the second timbre specified by the user and the second rhythm specified by the user. After determining that the second rhythm is the same as the third rhythm of the second timbre, the switching module starts the audio encoding module, the rhythm prediction module, the first timbre conversion module and the vocoder, and outputs the content features output by the audio encoding module to the rhythm prediction module and the first timbre conversion module respectively, and outputs the second audio data to the user through the process described above. At this time, the switching module controls the audio encoding module not to output the rhythm features. After determining that the second rhythm is the same as the first rhythm, the switching module starts the audio encoding module, the second timbre conversion module and the voice coder, outputs the content features output by the audio encoding module to the second timbre conversion module, and outputs the second audio data to the user through the process described above. At this time, the switching module controls the audio encoding module to output the rhythm features and content features.

[0083] The above describes in detail the process of audio processing by the audio processing model. The following describes the training process of the audio processing model. In one embodiment, the above method flow also includes:

[0084] Obtaining sample audio data having a sample timbre;

[0085] Extracting content features of sample audio data, timbre features of sample timbre, and spectrum features of sample audio data;

[0086] A neural network structure is trained according to content features of sample audio data, timbre features of sample timbre, and spectrum features of sample audio data; the trained neural network structure is used to form an audio processing model.

[0087] First, sample audio data with sample timbre is obtained. The sample timbre is the preset timbre mentioned above, that is, the target timbre that the audio processing model can transform. For example, if the trained audio processing model is set to be able to transform the audio data into the timbre of an old man, a boy, a girl, etc., then the timbre of an old man, a boy, a girl, etc. is the sample timbre. In one embodiment, sample audio data of multiple languages ​​for each sample timbre is obtained, for example, sample audio data of Chinese and sample audio data of English for each sample timbre are obtained.

[0088] Next, the content features of the sample audio data, the timbre features of the sample timbre, and the spectrum features of the sample audio data are extracted. The content features of the sample audio data include the text features of the text content of the sample audio data at each moment. The spectrum features of the sample audio data can be represented by a mel-spectrogram.

[0089] Finally, a neural network structure is trained according to the content characteristics of the sample audio data, the timbre characteristics of the sample timbre, and the spectrum characteristics of the sample audio data; the trained neural network structure is used to form an audio processing model.

[0090] It can be seen that through this embodiment, sample audio data with sample timbre can be obtained, and an audio processing model can be trained according to the content characteristics of the sample audio data, the timbre characteristics of the sample timbre, and the spectral characteristics of the sample audio data, so that the trained audio processing model can process the timbre of the audio data into the sample timbre, thereby achieving a voice changing effect.

[0091] In one embodiment, extracting content features of sample audio data includes:

[0092] The content features of the sample audio data are extracted through a pre-trained audio encoding module; wherein the audio encoding module is used to form an audio processing model.

[0093] In this embodiment, the audio encoding module is Figure 2The audio coding module in the audio processing model. Compared with other modules in the audio processing model, the audio coding module is obtained by pre-training. After the audio coding module is trained, for each sample audio data, the spectrum features of the sample audio data are obtained, and the spectrum features of the sample audio data are input into the audio coding module. The audio coding module processes the spectrum features of the sample audio data and extracts the content features of the sample audio data.

[0094] In one example, the sample audio data is subjected to Fourier transform and Mel filtering to obtain a Mel spectrogram of the sample audio data as the spectral feature of the sample audio data. The spectral feature of the sample audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the sample audio data at the corresponding frequency at the corresponding moment. The spectral feature of the sample audio data is input into a trained audio encoding module, and the audio encoding module processes the spectral feature of the sample audio data to obtain the content feature of the sample audio data. The content feature of the sample audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension, 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature of the sample audio data at the corresponding moment, which is represented by a vector of size [1, 512].

[0095] As mentioned above, the audio encoding module can be an encoder in the ASR model, which is composed of 18 Conformers connected in sequence, and the output of the last Conformer is the content feature of the sample audio data. When training the audio encoding module, audio data in multiple languages ​​can be used for training, so that the audio encoding module has the ability to process audio data in multiple languages, so that the audio processing model can process the first audio data in any language to obtain the second audio data, and achieve the voice changing effect of audio data in different languages. In one example, the audio encoding module is trained with audio data in English and Chinese, so that the audio encoding module can have the ability to process audio data in multiple languages ​​(including English and Chinese).

[0096] It can be seen that through this embodiment, the content features of the sample audio data can be extracted efficiently and quickly through the pre-trained audio encoding module.

[0097] In one embodiment, the audio data for training the audio encoding module may be the sample audio data for training the audio processing model, or may not be the sample audio data for training the audio processing model. The audio data for training the audio encoding module may have multiple timbres and multiple languages, and the sample audio data may also have multiple timbres and multiple languages.

[0098] In one embodiment, audio data for training an audio coding module is obtained, and the audio data has multiple timbres and multiple languages. The audio coding module is formed by connecting 18 Conformers in sequence. During training, a fully connected layer is added to the end of the last Conformer to extract the mel-spectrogram of the audio data used for training, and the mel-spectrogram is input into the audio coding module for processing. The training goal is that the fully connected layer can accurately predict the category of each phoneme in the audio data. After the training is completed, the fully connected layer is removed, and the first audio data is processed using an audio coding module formed by connecting 18 Conformers in sequence, wherein the output of the last Conformer is the content feature of the first audio data, which is also the content feature of the second audio data, and the output of the middle layer is the rhythm feature, for example, the output of the 12th Conformer is the rhythm feature of the first audio data, which is also the rhythm feature of the second audio data.

[0099] Different from the content encoding module and rhythm encoding module in the prior art, the present embodiment adopts the same audio encoding module to realize the dual functions of content encoding and rhythm encoding, and uses multi-language data to train the audio encoding module, so that the trained audio encoding module can extract content and rhythm from multi-language audio data, thereby enabling the audio processing model to process multi-language audio data.

[0100] In this embodiment, after the audio coding module training is completed, the training Figure 2 In one embodiment, the neural network structure for constituting the audio processing model includes a rhythm prediction module and a first timbre conversion module; training the neural network structure according to the content characteristics of the sample audio data, the timbre characteristics of the sample timbre, and the spectrum characteristics of the sample audio data includes:

[0101] Predicting the prosodic features of the sample audio data through a prosodic prediction module;

[0102] Generate, by the first timbre conversion module, frequency spectrum features of the third audio data according to the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre;

[0103] Based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data, the prosody prediction module and the first timbre conversion module are trained.

[0104] Figure 3 A schematic diagram of the training process of an audio processing model provided in an embodiment of the present disclosure is shown in FIG. Figure 3As shown, the neural network structure for constituting the audio processing model includes a rhythm prediction module and a first timbre conversion module in the audio processing model. Based on this, after the audio encoding module extracts the content features of the sample audio data, the content features of the sample audio data are input into the rhythm prediction module, and the rhythm features of the sample audio data are predicted by the rhythm prediction module. Then, the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the first timbre conversion module for processing to obtain the spectrum features of the third audio data. Finally, based on the difference between the spectrum features of the sample audio data and the spectrum features of the third audio data, the rhythm prediction module and the first timbre conversion module are trained.

[0105] In this embodiment, the timbre features of various sample timbres can be pre-stored in the neural network structure. If the difference between the spectral features of the sample audio data and the spectral features of the third audio data is less than the preset difference, it means that the rhythm prediction module can accurately predict the rhythm features of the sample audio data, and it means that the first timbre conversion module can accurately generate the spectral features of the sample audio data according to the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre, thereby determining that the training of the rhythm prediction module and the first timbre conversion module is completed.

[0106] The expression methods and specific meanings of the rhythmic features of the sample audio data, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the third audio data are similar to the expression methods and specific meanings of the rhythmic features, content features, timbre features, and spectral features involved in processing the first audio data. For example, the rhythmic features include the tone, pitch, sound speed, and sound weight of the audio data at each moment, the content features include the text features of the text content of the audio data at each moment, and the spectral features can be represented by a Mel-spectrogram. For details, please refer to the previous introduction to the processing process of the first audio data, which will not be repeated here.

[0107] In this embodiment, since the trained prosody prediction module can accurately predict the prosody features of the sample audio data, it can be determined that the trained prosody prediction module can accurately learn the prosody information of various sample timbres. Since the trained first timbre conversion module can accurately generate the spectrum features of the sample audio data according to the content features of the sample audio data output by the audio encoding module, the prosody features of the sample audio data output by the prosody prediction module, and the timbre features of the sample timbre, it can be determined that the trained first timbre conversion module has the ability to generate audio data according to the prosody features, content features, and timbre features, thereby achieving the effect of converting the prosody of the first audio data into the prosody of the sample timbre through the joint use of the prosody prediction module and the first timbre conversion module.

[0108] It can be seen that through this embodiment, the rhythm prediction module and the first timbre conversion module can be accurately trained, and the trained rhythm prediction module can accurately learn the rhythm information of various sample timbres, so that the trained first timbre conversion module has the ability to generate audio data according to rhythm features, content features and timbre features, thereby achieving the effect of converting the rhythm of the first audio data into the rhythm of the sample timbre through the joint use of the rhythm prediction module and the first timbre conversion module.

[0109] In one embodiment, the neural network structure includes a second timbre conversion module; the audio encoding module is further used to output the rhythmic features of the sample audio data; and the neural network structure is trained according to the content features of the sample audio data, the timbre features of the sample timbre, and the spectrum features of the sample audio data, including:

[0110] Generate frequency spectrum features of fourth audio data by the second timbre conversion module according to content features of the sample audio data output by the audio encoding module, rhythm features of the sample audio data output by the audio encoding module, and timbre features of the sample timbre;

[0111] Based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data, the second timbre conversion module is trained.

[0112] like Figure 3As shown, the neural network structure for constituting the audio processing model also includes a second timbre conversion module. Based on this, after the content features of the sample audio data are extracted at the last layer of the audio encoding module and the rhythm features of the sample audio data are extracted at the middle layer of the audio encoder, the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the audio encoding module and the timbre features of the sample timbre are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the second timbre conversion module for processing to obtain the spectrum features of the fourth audio data. Finally, based on the difference between the spectrum features of the sample audio data and the spectrum features of the fourth audio data, the second timbre conversion module is trained.

[0113] In this embodiment, the timbre features of various sample timbres can be pre-stored in the neural network structure. If the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data is less than the preset difference, it means that the second timbre conversion module can accurately generate the spectral features of the sample audio data according to the content features of the sample audio data output by the audio encoding module, the rhythmic features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre, thereby determining that the training of the second timbre conversion module is completed.

[0114] The expression methods and specific meanings of the rhythmic features of the sample audio data, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the fourth audio data are similar to the expression methods and specific meanings of the rhythmic features, content features, timbre features, and spectral features involved in processing the first audio data. For example, the rhythmic features include the tone, pitch, sound speed, and sound weight of the audio data at each moment, the content features include the text features of the text content of the audio data at each moment, and the spectral features can be represented by a Mel-spectrogram. For details, please refer to the previous introduction to the processing process of the first audio data, which will not be repeated here.

[0115] In this embodiment, since the trained second timbre conversion module can accurately generate the spectral characteristics of the sample audio data based on the content characteristics of the sample audio data output by the audio encoding module, the rhythmic characteristics of the sample audio data output by the audio encoding module, and the timbre characteristics of the sample timbre, it can be determined that the trained second timbre conversion module has the ability to generate audio data based on the rhythmic characteristics, content characteristics and timbre characteristics, thereby achieving the effect of keeping the rhythm of the first audio data unchanged through the use of the second timbre conversion module.

[0116] It can be seen that through this embodiment, the second timbre conversion module can be accurately trained, so that the trained second timbre conversion module has the ability to generate audio data according to rhythmic features, content features and timbre features, thereby achieving the effect of keeping the rhythm of the first audio data unchanged through the use of the second timbre conversion module.

[0117] In one embodiment, the first timbre conversion module and the second timbre conversion module in the audio processing model can be connected by multiple conformers, and the parameters of the first timbre conversion module and the second timbre conversion module are independently updated during training. The rhythm prediction module is stacked by multiple convolutional networks. The vocoder in the audio processing model can be pre-trained like the audio processing module.

[0118] In one embodiment, reference Figure 3 , the complete training process of the audio processing model is:

[0119] First, a vocoder and an audio processing module are pre-trained, then sample audio data with sample timbre is obtained, content features and rhythm features of the sample audio data are extracted through the pre-trained audio coding module, rhythm features of the sample audio data are predicted through the rhythm prediction module, spectrum features of the third audio data are generated through the first timbre conversion module according to the content features of the sample audio data output by the audio coding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre, spectrum features of the fourth audio data are generated through the second timbre conversion module according to the content features of the sample audio data output by the audio coding module, the rhythm features of the sample audio data output by the audio coding module, and the timbre features of the sample timbre, based on the principle of minimizing the difference between the spectrum features of the sample audio data, the spectrum features of the third audio data, and the spectrum features of the fourth audio data, the rhythm prediction module, the first timbre conversion module, the second timbre conversion module, the pre-trained audio coding module, the trained rhythm prediction module, the first timbre conversion module, the second timbre conversion module, the pre-trained vocoder, and Figure 2 The switching modules in the above example are combined to obtain the audio processing model. The switching modules can also be pre-trained. Since the vocoder is pre-trained, Figure 3 The vocoder is not shown.

[0120] In summary, the following technical effects can be achieved through the above audio processing model and audio processing method:

[0121] 1. Processing the first audio data into second audio data with the same content but different timbre, so as to achieve the effect of changing the timbre of the audio data;

[0122] 2. The second rhythm of the second audio data can be made the same as the first rhythm of the first audio data or the same as the third rhythm of the second timbre, so as to achieve the effect of changing the rhythm while changing the timbre or keeping the rhythm unchanged, realize the effect of controllable rhythm, and realize high-quality audio processing;

[0123] 3. The pre-trained audio encoding module can be trained based on data in multiple languages, so that the audio encoding module has the ability to process audio data in multiple languages, and the audio processing model can process the first audio data in any language to obtain the second audio data, thereby achieving the voice changing effect of audio data in different languages.

[0124] An embodiment of the present disclosure further provides an audio processing device for implementing the above-mentioned audio processing method. Figure 4 A structural diagram of an audio processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the device comprises:

[0125] The data acquisition unit 401 is used to acquire first audio data having first text content, first timbre and first rhythm;

[0126] The feature acquisition unit 402 is used to process the first audio data through an audio processing model to acquire content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0127] The audio generation unit 403 is used to generate the second audio data according to the content features of the second audio data, the rhythm features of the second audio data and the timbre features of the second timbre through the audio processing model.

[0128] Optionally, the second rhythm is the same as the third rhythm; the feature acquisition unit 402 is specifically used to: extract content features of the first audio data as content features of the second audio data through the audio encoding module in the audio processing model; and predict the rhythm features of the second audio data based on the content features of the second audio data and the rhythm information of the third rhythm through the rhythm prediction module in the audio processing model.

[0129] Optionally, the second rhythm is the same as the first rhythm; the feature acquisition unit 402 is specifically used to: extract content features of the first audio data as content features of the second audio data through the audio encoding module in the audio processing model; extract rhythm features of the first audio data as rhythm features of the second rhythm through the audio encoding module in the audio processing model.

[0130] Optionally, the feature acquisition unit 402 is specifically configured to acquire the timbre feature of the second timbre from a timbre feature library of the audio processing model.

[0131] Optionally, the audio generation unit 403 is specifically used to: generate frequency spectrum characteristics corresponding to the second audio data according to content characteristics of the second audio data output by the audio encoding module, rhythm characteristics of the second audio data output by the rhythm prediction module, and timbre characteristics of the second timbre through the first timbre conversion module in the audio processing model; and generate the second audio data according to the frequency spectrum characteristics corresponding to the second audio data through the vocoder in the audio processing model.

[0132] Optionally, the audio generation unit 403 is specifically used to: generate, through the second timbre conversion module in the audio processing model, spectral features corresponding to the second audio data according to content features of the second audio data output by the audio encoding module, rhythmic features of the second audio data output by the audio encoding module, and timbre features of the second timbre; and generate, through the vocoder in the audio processing model, the second audio data according to the spectral features corresponding to the second audio data.

[0133] Optionally, it also includes a training unit, which is used to: obtain sample audio data with sample timbre; extract content features of the sample audio data, timbre features of the sample timbre and spectral features of the sample audio data; train a neural network structure according to the content features of the sample audio data, the timbre features of the sample timbre and the spectral features of the sample audio data; the trained neural network structure is used to form the audio processing model.

[0134] Optionally, the training unit is specifically used to: extract content features of the sample audio data through a pre-trained audio encoding module; wherein the audio encoding module is used to form the audio processing model.

[0135] Optionally, the neural network structure includes a rhythm prediction module and a first timbre conversion module; the training unit is specifically used to: predict the rhythm features of the sample audio data through the rhythm prediction module; generate spectral features of the third audio data through the first timbre conversion module according to the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre; and train the rhythm prediction module and the first timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

[0136] Optionally, the neural network structure includes a second timbre conversion module; the audio encoding module is also used to output the rhythmic features of the sample audio data; the training unit is specifically used to: generate spectral features of fourth audio data through the second timbre conversion module according to the content features of the sample audio data output by the audio encoding module, the rhythmic features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre; and train the second timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

[0137] The audio processing device in the embodiment of the present disclosure can implement the various processes of the above-mentioned audio processing method embodiment and achieve the same effects and functions, which will not be repeated here.

[0138] An embodiment of the present disclosure further provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 5 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 501 and memory 502, and one or more applications or data may be stored in the memory 502. Among them, the memory 502 may be a temporary storage or a permanent storage. The application stored in the memory 502 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 501 can be configured to communicate with the memory 502 to execute a series of computer executable instructions in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input or output interfaces 505, one or more keyboards 506, etc.

[0139] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0140] Acquire first audio data having first text content, first timbre and first rhythm;

[0141] The first audio data is processed by an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0142] The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre.

[0143] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0144] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0145] Acquire first audio data having first text content, first timbre and first rhythm;

[0146] The first audio data is processed by an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0147] The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre.

[0148] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0149] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0150] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0151] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0152] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0153] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0154] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0159] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0160] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0161] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. An audio processing method, characterized in that: include: Acquire first audio data having first text content, first timbre and first rhythm; The first audio data is processed by an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre; The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre.

2. The method according to claim 1, characterized in that The second rhythm is the same as the third rhythm; and the processing of the first audio data by the audio processing model to obtain content features of the second audio data and rhythm features of the second audio data includes: extracting, by means of an audio encoding module in the audio processing model, content features of the first audio data as content features of the second audio data; The prosody feature of the second audio data is predicted based on the content feature of the second audio data and the prosody information of the third prosody through the prosody prediction module in the audio processing model.

3. The method according to claim 1, characterized in that The second rhythm is the same as the first rhythm; and the processing of the first audio data by the audio processing model to obtain content features of the second audio data and rhythm features of the second audio data includes: extracting, by means of an audio encoding module in the audio processing model, content features of the first audio data as content features of the second audio data; The prosody features of the first audio data are extracted through the audio encoding module in the audio processing model as the prosody features of the second prosody.

4. The method according to claim 1, characterized in that Acquiring the timbre characteristics of the second timbre includes: The timbre feature of the second timbre is obtained from the timbre feature library of the audio processing model.

5. The method according to claim 2, characterized in that: The step of generating the second audio data by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre comprises: Generate, by the first timbre conversion module in the audio processing model, a frequency spectrum feature corresponding to the second audio data according to the content feature of the second audio data output by the audio encoding module, the prosody feature of the second audio data output by the prosody prediction module, and the timbre feature of the second timbre; The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

6. The method according to claim 3, characterized in that The step of generating the second audio data by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre comprises: Generate, by a second timbre conversion module in the audio processing model, a frequency spectrum feature corresponding to the second audio data according to content features of the second audio data output by the audio encoding module, rhythm features of the second audio data output by the audio encoding module, and timbre features of the second timbre; The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining sample audio data having a sample timbre; Extracting content features of the sample audio data, timbre features of the sample timbre, and spectrum features of the sample audio data; A neural network structure is trained according to the content features of the sample audio data, the timbre features of the sample timbre and the spectrum features of the sample audio data; the trained neural network structure is used to form the audio processing model.

8. The method according to claim 7, characterized in that The extracting content features of the sample audio data comprises: The content features of the sample audio data are extracted through a pre-trained audio encoding module; wherein the audio encoding module is used to form the audio processing model.

9. The method according to claim 8, characterized in that The neural network structure includes a rhythm prediction module and a first timbre conversion module; the training of the neural network structure according to the content characteristics of the sample audio data, the timbre characteristics of the sample timbre and the spectrum characteristics of the sample audio data includes: Predicting the prosody features of the sample audio data by the prosody prediction module; Generate, by the first timbre conversion module, frequency spectrum features of third audio data according to content features of the sample audio data output by the audio encoding module, rhythm features of the sample audio data output by the rhythm prediction module, and timbre features of the sample timbre; The prosody prediction module and the first timbre conversion module are trained based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

10. The method according to claim 8, characterized in that The neural network structure includes a second timbre conversion module; the audio encoding module is further used to output the rhythmic features of the sample audio data; the training of the neural network structure according to the content features of the sample audio data, the timbre features of the sample timbre and the spectrum features of the sample audio data includes: generating, by the second timbre conversion module, frequency spectrum features of fourth audio data according to content features of the sample audio data output by the audio encoding module, rhythm features of the sample audio data output by the audio encoding module, and timbre features of the sample timbre; The second timbre conversion module is trained based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

11. An audio processing device, characterized in that: include: A data acquisition unit, configured to acquire first audio data having first text content, a first timbre, and a first rhythm; a feature acquisition unit, configured to process the first audio data through an audio processing model to acquire content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre; The audio generation unit is used to generate the second audio data according to the content characteristics of the second audio data, the rhythm characteristics of the second audio data and the timbre characteristics of the second timbre through the audio processing model.

12. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10.

Images