Audio processing method and apparatus, and electronic device and storage medium

Abstract

An audio processing method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring first audio data, which has first text content, a first tone and a first rhythm (S102); processing the first audio data by means of an audio processing model, so as to acquire a content feature of second audio data, a rhythm feature of the second audio data and a tone feature of a second tone, wherein second text content of the second audio data is the same as the first text content, the second tone of the second audio data is different from the first tone, and a second rhythm of the second audio data is the same as the first rhythm or is the same as a third rhythm of the second tone (S104); and by means of the audio processing model, generating the second audio data on the basis of the content feature of the second audio data, the rhythm feature of the second audio data and the tone feature of the second tone (S106).

Description

Audio processing method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 28, 2023, with application number 202311278074.7 and invention name “Audio processing method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of audio processing technology, and in particular to an audio processing method, device, electronic device, and storage medium. Background Art

[0003] To meet users' audio processing needs in scenarios such as audio and video entertainment and virtual social interaction, existing technologies offer a variety of audio processing methods, such as trimming the length of audio data and changing the timbre of audio data. Changing the timbre of audio data is a key area of ​​audio processing. Therefore, it is necessary to provide a technical solution that can modify the timbre of audio data to achieve high-quality audio processing.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide an audio processing method, apparatus, electronic device, and storage medium, which can change the timbre of audio data and achieve high-quality audio processing.

[0006] In a first aspect, an embodiment of the present disclosure provides an audio processing method, including:

[0007] Acquire first audio data having first text content, a first timbre, and a first rhythm;

[0008] Processing the first audio data using an audio processing model to obtain content features of second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0009] The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre.

[0010] In a second aspect, an embodiment of the present disclosure provides an audio processing device, including:

[0011] A data acquisition unit, configured to acquire first audio data having first text content, a first timbre, and a first rhythm;

[0012] a feature acquisition unit, configured to process the first audio data using an audio processing model to acquire content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; wherein the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; and the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0013] The audio generation unit is configured to generate the second audio data according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre through the audio processing model.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In one or more embodiments of the present disclosure, first, first audio data having first text content, a first timbre and a first rhythm is obtained, then, the first audio data is processed through an audio processing model to obtain content features of second audio data, rhythm features of the second audio data and timbre features of the second timbre, the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre, finally, through the audio processing model, second audio data is generated according to the content features of the second audio data, the rhythm features of the second audio data and the timbre features of the second timbre. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of the present disclosure or technical solutions in the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments described in the present disclosure. Those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0018] FIG1 is a schematic flow chart of an audio processing method provided by an embodiment of the present disclosure;

[0019] FIG2 is a schematic diagram of the inference process of the audio processing model provided by an embodiment of the present disclosure;

[0020] FIG3 is a schematic diagram of the training process of the audio processing model provided by an embodiment of the present disclosure;

[0021] FIG4 is a schematic structural diagram of an audio processing device provided by an embodiment of the present disclosure;

[0022] FIG5 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0024] The present disclosure provides an audio processing method that can change the timbre of audio data to achieve high-quality audio processing. The audio processing method can be applied to and executed by a server.

[0025] FIG1 is a flow chart of an audio processing method provided by an embodiment of the present disclosure. As shown in FIG1 , the flow chart includes the following steps:

[0026] Step S102, obtaining first audio data having first text content, a first timbre, and a first rhythm;

[0027] Step S104: Processing the first audio data using an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0028] Step S106 : Generate second audio data through the audio processing model according to the content features of the second audio data, the rhythm features of the second audio data, and the timbre features of the second timbre.

[0029] In this embodiment, first, first audio data having first text content, a first timbre, and a first rhythm is obtained. Then, the first audio data is processed through an audio processing model to obtain content features of second audio data, rhythm features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre. Finally, the audio processing model is used to generate second audio data based on the content features of the second audio data, the rhythm features of the second audio data, and the timbre features of the second timbre. It can be seen that through this embodiment, the audio processing model can be used to process the first audio data into second audio data with the same content but different timbre, thereby achieving the effect of changing the timbre of the audio data. In addition, the second rhythm of the second audio data can be made the same as the first rhythm of the first audio data or the same as the third rhythm of the second timbre, thereby achieving the effect of changing the timbre while changing the rhythm or keeping the rhythm unchanged, thereby achieving high-quality audio processing.

[0030] The method flow in FIG1 is described in detail below.

[0031] In the above step S102, first audio data having first text content, first timbre, and first rhythm is obtained. In this embodiment, the first audio data includes a human voice or a robot imitating a human voice speaking, a cappella song, a nursery rhyme, etc. The first text content of the first audio data is the text content corresponding to the first audio data, such as "Hello, today is September 10th, Teacher's Day." The first timbre of the first audio data is the timbre of the human voice or robot voice in the first audio data. The first rhythm of the first audio data is the tone, pitch, speed, strength, etc. of the human voice or robot voice in the first audio data when reading or singing the first text content.

[0032] The text mentioned in various embodiments of this specification refers to written language in the form of characters. For example, a paragraph of online fiction is considered text. The simplest example of text is "Hello." The prosody mentioned in various embodiments of this specification refers to information such as tone, pitch, speed, and intensity. Audio data consists of three elements: timbre, content, and prosody. Beyond content and timbre, all other information in audio data perceived by the human ear can be categorized as prosody.

[0033] In step S104, the first audio data is processed using the audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre. The second text content of the second audio data is the same as the first text content. The second timbre of the second audio data is different from the first timbre. The second cadence of the second audio data is the same as the first cadence or the same as the third cadence of the second timbre.

[0034] Similar to the first audio data, in this embodiment, the second audio data includes a human voice or a robot imitating a human voice speaking, a cappella song, a nursery rhyme, etc. The second text content of the second audio data is the text content corresponding to the second audio data, such as "Hello, today is September 10th, Teacher's Day." The second timbre of the second audio data is the timbre of the human voice or robot voice in the second audio data. The second rhythm of the second audio data is information such as the tone, pitch, speed, and strength of the human voice or robot voice in the second audio data when reciting or singing the second text content.

[0035] In one example, the first audio data is a text narrated by user A, with the text content being: "Hello, today is September 10th, Teacher's Day." The second text content of the second audio data is the same as the first text content of the first audio data, both being "Hello, today is September 10th, Teacher's Day." The first timbre of the first audio data is a male timbre, while the second timbre of the second audio data is a female timbre. The second rhythm of the second audio data is the same as the first rhythm of the first audio data, or the second rhythm of the second audio data is the same as the third rhythm of the second timbre.

[0036] In this embodiment, the first audio data has a first rhythm, the second audio data has a second rhythm, and the second timbre has a third rhythm. The first rhythm of the first audio data is the same as the second rhythm of the second audio data, which means that the tone, pitch, speed, and strength of the sound at each moment when the human voice or robot voice in the first audio data reads or sings the first text content are the same as the tone, pitch, speed, and strength of the sound at each moment when the human voice or robot voice in the second audio data reads or sings the second text content. In one embodiment, the first rhythm of the first audio data is the same as the second rhythm of the second audio data, which can be expressed as the rhythmic features of the first audio data are the same as the rhythmic features of the second audio data. The rhythmic features of the audio data can be represented by vectors, and whether the two rhythmic features are the same can be determined by determining whether the values ​​of the two vectors are the same.

[0037] The third rhythm of the second timbre represents the average tone, average pitch, average speed of sound, average strength of sound and other information when the second timbre reads or sings various contents. When the second timbre reads or sings the second text content, the second rhythm of the second audio data is the same as the third rhythm of the second timbre, which means that the average tone, average pitch, average speed of sound, average strength of sound and other information at each moment when the human voice or robot voice in the second audio data reads or sings the second text content is the same as the rhythm of the second timbre.

[0038] It can be seen that through this embodiment, the first audio data can be processed into second audio data with the same content but different timbre, thereby achieving the effect of changing the timbre of the audio data. Moreover, compared with the technical solutions in the prior art that can only change the timbre of audio data but cannot control the change effect of the rhythm of audio data, the rhythm of audio data can be controllably changed, and the second rhythm of the second audio data can be made the same as the first rhythm of the first audio data, thereby achieving the effect of changing the timbre while keeping the rhythm unchanged, or the second rhythm of the second audio data can be made the same as the third rhythm of the second timbre, thereby achieving the effect of changing the timbre while changing the rhythm accordingly, so that the timbre and rhythm of the second audio data are harmonious and consistent.

[0039] Furthermore, in this embodiment, the language of the first audio data is not limited, and the first audio data in any language can be processed to obtain the second audio data, thereby achieving the voice changing effect of audio data in different languages.

[0040] In step S104, the first audio data is processed by the audio processing model to obtain content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre. In this embodiment, a pre-trained audio processing model is used to process the first audio data into the second audio data. The audio processing model can be used to obtain content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, thereby generating the second audio data.

[0041] In one embodiment, the second rhythm of the second audio data is the same as the third rhythm of the second timbre; and processing the first audio data by the audio processing model to obtain content features and rhythm features of the second audio data includes:

[0042] Extracting, by an audio encoding module in an audio processing model, content features of the first audio data as content features of the second audio data;

[0043] The prosody prediction module in the audio processing model predicts the prosody features of the second audio data based on the content features of the first audio data and the prosody information of the third prosody.

[0044] The content features of the second audio data include but are not limited to the text features of the text content of the second audio data at each moment. The rhythm features of the second audio data include but are not limited to the tone, pitch, sound speed, sound intensity, etc. of the second audio data at each moment.

[0045] Figure 2 is a schematic diagram of the inference process of the audio processing model provided by an embodiment of the present disclosure. As shown in Figure 2, the trained audio processing model includes an audio encoding module and a rhythm prediction module. In the process of processing the first audio data to generate the second audio data through the audio processing model, if the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the spectral features of the first audio data are obtained, and the spectral features of the first audio data are input into the audio encoding module. The audio encoding module processes the spectral features of the first audio data and extracts the content features of the first audio data as the content features of the second audio data.

[0046] In one example, the first audio data is subjected to Fourier transform and Mel filtering to obtain a Mel spectrogram of the first audio data as the spectral features of the first audio data. The spectral features of the first audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the first audio data at a corresponding frequency at a corresponding moment. The spectral features of the first audio data are input into an audio encoding module, which processes the spectral features of the first audio data to obtain content features of the first audio data as the content features of the second audio data. The content features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension, 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature of the second audio data at a corresponding moment, represented by a vector of size [1, 512].

[0047] In one example, the audio encoding module can be an encoder in an ASR (Automatic Speech Recognition) model, which is composed of 18 Conformers (Convolution-augmented Transformer for Speech Recognition, convolution-augmented transformer for speech recognition) connected in sequence, and the output of the last Conformer is the content feature of the second audio data. When training the audio encoding module, audio data in multiple languages ​​can be used for training, so that the audio encoding module has the ability to process audio data in multiple languages, so that the audio processing model can process the first audio data in any language to obtain the second audio data, thereby achieving the voice changing effect of audio data in different languages. In one example, using audio data in English and Chinese to train the audio encoding module can enable the audio encoding module to have the ability to process audio data in multiple languages ​​(including English and Chinese).

[0048] As shown in Figure 2, the trained audio processing model also includes a rhythm prediction module. In the process of processing the first audio data through the audio processing model to generate the second audio data, if the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the content features of the second audio data output by the audio encoding module are input into the rhythm prediction module. The rhythm prediction module predicts the rhythm features of the second audio data based on the content features of the input second audio data and the rhythm information of the third rhythm stored in the rhythm prediction module.

[0049] As can be seen from the previous introduction, the rhythmic information of the third rhythm of the second timbre includes the average tone, average pitch, average speed of sound, average strength of sound and other information when the second timbre reads or sings various contents. The rhythmic features of the second audio data include the tone, pitch, speed of sound and strength of sound at each moment when the human voice or robot voice in the second audio data reads or sings the second text content. Therefore, predicting the rhythmic features of the second audio data based on the content features of the second audio data and the rhythmic information of the third rhythm is a process of predicting the tone at each moment, pitch, speed of sound and strength of sound at each moment based on the specific content of the second audio data and the average tone, average pitch, average speed of sound and average strength of sound represented by the third rhythm.

[0050] In order to achieve the effect of predicting the tone at each moment, the pitch at each moment, the speed of the sound at each moment, the strength of the sound at each moment and other information, the input of the rhythm prediction module includes the content features of the second audio data, and the rhythmic information of the third rhythm is learned by the rhythm prediction module during the model training process and stored as a model parameter in the rhythm prediction module. The rhythm prediction module can take the content features of the second audio data as input, and predict the tone at each moment, the pitch at each moment, the speed of the sound at each moment, the strength of the sound at each moment and other information when the human voice or robot voice in the second audio data reads or sings the second text content according to the content features of the second audio data and the rhythmic information of the third rhythm of the second timbre, as the rhythmic features of the second audio data.

[0051] In one embodiment, during training, the audio processing model learns to transform the timbre of audio data into a variety of preset timbres. The prosody prediction module learns the prosody information of each preset timbre during training and stores the prosody information of each preset timbre as a model parameter in the prosody prediction module. Therefore, when a user uses the audio processing model of this embodiment to achieve a voice-changing effect, they can input first audio data and specify the second timbre of the second audio data to be transformed and the second prosody of the second audio data, where the second timbre is within the aforementioned preset timbres. Upon determining that the second prosody is the same as the third prosody of the second timbre, the prosody prediction module in the audio processing model predicts the prosody characteristics of the second audio data based on the content characteristics of the second audio data and the stored prosody information of the third prosody.

[0052] In an example, the prosodic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension. Each value in the matrix represents the corresponding prosodic feature of the second audio data at the corresponding moment and is represented by a vector of size [1, 512].

[0053] It can be seen that, through this embodiment, the audio processing model implements the rhythm prediction function through the rhythm prediction module, and the rhythm prediction function can achieve the effect of changing the rhythm of the first audio data.

[0054] In one embodiment, the second rhythm of the second audio data is the same as the first rhythm of the first audio data; and processing the first audio data by the audio processing model to obtain content features and rhythm features of the second audio data includes:

[0055] Extracting, by an audio encoding module in an audio processing model, content features of the first audio data as content features of the second audio data;

[0056] The audio encoding module in the audio processing model extracts the prosody features of the first audio data as the prosody features of the second prosody.

[0057] The content features of the second audio data include but are not limited to the text features of the text content of the second audio data at each moment. The rhythm features of the second audio data include but are not limited to the tone, pitch, sound speed, sound intensity, etc. of the second audio data at each moment.

[0058] Referring to Figure 2 and the previous description, the process of extracting the content features of the first audio data as the content features of the second audio data through the audio encoding module in the audio processing model is the same as the previous description and will not be repeated here.

[0059] In this embodiment, the spectral features of the first audio data are obtained and input into the audio encoding module. The audio encoding module processes the spectral features of the first audio data and extracts the rhythmic features of the first audio data as the rhythmic features of the second audio data.

[0060] In one example, a Fourier transform and Mel filter are performed on the first audio data to obtain a Mel spectrogram of the first audio data as the spectral features of the first audio data. The spectral features of the first audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension, 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the first audio data at a corresponding frequency at a corresponding moment. The spectral features of the first audio data are input into an audio encoding module, which processes the spectral features of the first audio data to obtain the prosodic features of the first audio data as the prosodic features of the second audio data. The prosodic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension, 512 represents the feature dimension, and each value in the matrix represents the corresponding prosodic feature of the second audio data at a corresponding moment, represented by a vector of size [1, 512].

[0061] In one instance, the audio encoding module can be an encoder in an ASR model, which is composed of 18 Conformers connected in sequence. The output of the last layer is the content feature, and the output of the middle layer is the rhythm feature. For example, the output of the 12th Conformer is the rhythm feature of the second audio data.

[0062] It can be seen that through this embodiment, rhythm feature extraction and content feature extraction can be realized through the audio encoding module, so that multiple functions can be realized through the same module. There is no need to distribute the modules for extracting content features and the modules for extracting rhythm features, thereby improving the efficiency of audio processing.

[0063] In one embodiment, obtaining the timbre characteristics of the second timbre includes:

[0064] Acquire the timbre feature of the second timbre from the timbre feature library of the audio processing model.

[0065] In this embodiment, the audio processing model learns the ability to transform the timbre of audio data into a variety of preset timbres during training, and the rhythm prediction module learns the rhythm information of various preset timbres during training and stores the rhythm information of each preset timbre as a model parameter in the rhythm prediction module. After the audio processing model is trained, the timbre features of various preset timbres are stored in the timbre feature library. Based on this, when the user uses the audio processing model in this embodiment to achieve a voice change effect, the user can input the first audio data and specify the second timbre of the second audio data to be transformed and the second rhythm of the second audio data. The second timbre is located in the above-mentioned preset timbre, so that the audio processing model extracts the timbre features of the second timbre from the timbre feature library. In this embodiment, the audio processing model can also store the timbre features of various preset timbres in the timbre feature library during training.

[0066] It can be seen that through this embodiment, the timbre characteristics of various timbres can be stored in the timbre feature library of the audio processing model. During the audio processing process, the timbre characteristics of the second timbre can be directly obtained from the timbre feature library, eliminating the process of extracting timbre features and improving the audio processing efficiency.

[0067] Since audio data includes three elements: timbre, content, and rhythm, the above step S106 also generates second audio data according to the content characteristics of the second audio data, the rhythm characteristics of the second audio data, and the timbre characteristics of the second timbre through the audio processing model.

[0068] In one embodiment, when the second rhythm of the second audio data is the same as the third rhythm of the second timbre, generating the second audio data according to the content features of the second audio data, the rhythm features of the second audio data, and the timbre features of the second timbre by the audio processing model includes:

[0069] generating, by a first timbre conversion module in the audio processing model, spectral features corresponding to the second audio data based on content features of the second audio data output by the audio encoding module, prosodic features of the second audio data output by the prosody prediction module, and timbre features of the second timbre;

[0070] The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

[0071] In this embodiment, as shown in FIG2 , the audio processing model further includes a first timbre conversion module and a vocoder. When the second prosody of the second audio data is identical to the third prosody of the second timbre, the first timbre conversion module is capable of synthesizing audio data based on content features, prosody features, and timbre features. Therefore, the first timbre conversion module generates spectral features corresponding to the second audio data based on the content features of the second audio data output by the audio encoding module, the prosody features of the second audio data output by the prosody prediction module, and the timbre features of the second timbre. The first timbre conversion module can obtain the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0072] In an example, the content features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the rhythmic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding rhythmic features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the timbre features of the second timbre can be represented by a matrix of size [100, 256], where 100 represents the time dimension and 256 represents the feature dimension, and each value in the matrix represents the timbre features of the second timbre at the corresponding moment, represented by a vector of size [1, 256].

[0073] In this embodiment, the content features of the second audio data output by the audio encoding module, the rhythm features of the second audio data output by the rhythm prediction module, and the timbre features of the second timbre are feature spliced ​​to obtain a splicing result [100, 512+512+256]. The splicing result is input into the first timbre conversion module, which generates spectral features corresponding to the second audio data based on the splicing result. The spectral features corresponding to the second audio data are represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension. Each value in the matrix represents the amplitude of the second audio data at the corresponding frequency at the corresponding moment. The spectral features corresponding to the second audio data can be represented by a mel-spectrogram.

[0074] The first timbre conversion module is connected to the vocoder. The input of the vocoder includes the spectrum characteristics corresponding to the second audio data output by the first timbre conversion module. The vocoder generates the second audio data according to the spectrum characteristics corresponding to the second audio data.

[0075] It can be seen that through this embodiment, when the second rhythm of the second audio data is the same as the third rhythm of the second timbre, the spectral characteristics corresponding to the second audio data can be generated by the first timbre conversion module, and the second audio data can be generated according to the spectral characteristics corresponding to the second audio data by the voice coder, so that the first audio data can be processed into the second audio data efficiently and quickly, thereby improving the audio processing efficiency.

[0076] In one embodiment, when the second prosody of the second audio data is the same as the first prosody of the first audio data, generating the second audio data according to content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre by using an audio processing model includes:

[0077] generating, by a second timbre conversion module in the audio processing model, spectral features corresponding to the second audio data based on content features of the second audio data output by the audio encoding module, rhythmic features of the second audio data output by the audio encoding module, and timbre features of the second timbre;

[0078] The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

[0079] In this embodiment, as shown in FIG2 , the audio processing model further includes a second timbre conversion module and a vocoder. When the second cadence of the second audio data is the same as the first cadence of the first audio data, the second timbre conversion module is capable of synthesizing audio data based on content features, cadence features, and timbre features. Therefore, the second timbre conversion module generates spectral features corresponding to the second audio data based on the content features of the second audio data output by the audio encoding module, the cadence features of the second audio data output by the audio encoding module, and the timbre features of the second timbre. The second timbre conversion module can obtain the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0080] In an example, the content features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the rhythmic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding rhythmic features of the second audio data at the corresponding moment, represented by a vector of size [1, 512]; the timbre features of the second timbre can be represented by a matrix of size [100, 256], where 100 represents the time dimension and 256 represents the feature dimension, and each value in the matrix represents the timbre features of the second timbre at the corresponding moment, represented by a vector of size [1, 256].

[0081] In this embodiment, the content features of the second audio data output by the audio encoding module, the rhythmic features of the second audio data output by the audio encoding module, and the timbre features of the second timbre are feature spliced ​​to obtain a splicing result [100, 512+512+256]. The splicing result is input into the second timbre conversion module, and the second timbre conversion module generates a spectral feature corresponding to the second audio data based on the splicing result. The spectral feature corresponding to the second audio data is represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension. Each value in the matrix represents the amplitude of the second audio data at the corresponding frequency at the corresponding moment. The spectral feature corresponding to the second audio data can be represented by a mel-spectrogram.

[0082] The second timbre conversion module is connected to the vocoder. The input of the vocoder includes the spectrum characteristics corresponding to the second audio data output by the second timbre conversion module. The vocoder generates the second audio data according to the spectrum characteristics corresponding to the second audio data.

[0083] It can be seen that through this embodiment, when the second rhythm of the second audio data is the same as the first rhythm of the first audio data, the second timbre conversion module can generate the spectral characteristics corresponding to the second audio data, and the voice coder can generate the second audio data according to the spectral characteristics corresponding to the second audio data, so as to efficiently and quickly process the first audio data into the second audio data, thereby improving the audio processing efficiency.

[0084] As shown in Figure 2, the audio processing model also includes a switching module. In one example, when the user uses the audio processing model in this embodiment to achieve a voice change effect, the user can input the first audio data and specify the second timbre of the second audio data to be converted and the second rhythm of the second audio data, where the second timbre is located in the above-mentioned preset timbre. After the spectral characteristics of the first audio data, the second timbre specified by the user, and the second rhythm specified by the user are input into the audio processing model, the switching module can receive the second timbre specified by the user and the second rhythm specified by the user. After determining that the second rhythm is the same as the third rhythm of the second timbre, the switching module starts the audio encoding module, the rhythm prediction module, the first timbre conversion module, and the voice coder, and outputs the content features output by the audio encoding module to the rhythm prediction module and the first timbre conversion module respectively, and outputs the second audio data to the user through the process described above. At this time, the switching module controls the audio encoding module not to output the rhythm features. After determining that the second rhythm is the same as the first rhythm, the switching module starts the audio encoding module, the second timbre conversion module and the voice coder, outputs the content features output by the audio encoding module to the second timbre conversion module, and outputs the second audio data to the user through the process described above. At this time, the switching module controls the audio encoding module to output rhythm features and content features.

[0085] The above describes in detail the process of audio processing by the audio processing model. The following describes the training process of the audio processing model. In one embodiment, the above method flow also includes:

[0086] Obtaining sample audio data having a sample timbre;

[0087] extracting content features of the sample audio data, timbre features of the sample timbre, and spectrum features of the sample audio data;

[0088] A neural network structure is trained based on content features of sample audio data, timbre features of sample timbre, and spectrum features of sample audio data; the trained neural network structure is used to form an audio processing model.

[0089] First, sample audio data with sample timbres is obtained. The sample timbre is the preset timbre described above, i.e., the target timbre that the audio processing model can transform. For example, if the trained audio processing model is set to be able to transform the audio data into the timbre of an elderly person, a male, or a female, then the timbre of an elderly person, a male, or a female is the sample timbre. In one embodiment, sample audio data in multiple languages ​​is obtained for each sample timbre, for example, sample audio data in Chinese and sample audio data in English are obtained for each sample timbre.

[0090] Next, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data are extracted. The content features of the sample audio data include the text features of the text content of the sample audio data at each moment. The spectral features of the sample audio data can be represented by a mel-spectrogram.

[0091] Finally, a neural network structure is trained based on the content characteristics of the sample audio data, the timbre characteristics of the sample timbre, and the spectrum characteristics of the sample audio data; the trained neural network structure is used to form an audio processing model.

[0092] It can be seen that through this embodiment, sample audio data with sample timbre can be obtained, and the audio processing model can be trained according to the content characteristics of the sample audio data, the timbre characteristics of the sample timbre and the spectral characteristics of the sample audio data, so that the trained audio processing model can process the timbre of the audio data into the sample timbre to achieve the voice changing effect.

[0093] In one embodiment, extracting content features of sample audio data includes:

[0094] The content features of the sample audio data are extracted through a pre-trained audio encoding module; wherein the audio encoding module is used to form an audio processing model.

[0095] In this embodiment, the audio encoding module is the audio encoding module in Figure 2. Compared to other modules in the audio processing model, this audio encoding module is pre-trained. After the audio encoding module is trained, the spectral features of each audio sample are obtained and input into the audio encoding module. The audio encoding module processes the spectral features of the audio sample and extracts the content features of the audio sample.

[0096] In one example, Fourier transform and Mel filtering are performed on sample audio data to obtain a Mel-spectrogram of the sample audio data as the spectral features of the sample audio data. The spectral features of the sample audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension. Each value in the matrix represents the amplitude of the sample audio data at a corresponding frequency at a corresponding moment. The spectral features of the sample audio data are input into a trained audio encoding module, which processes the spectral features of the sample audio data to obtain content features of the sample audio data. The content features of the sample audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension. Each value in the matrix represents the corresponding content feature of the sample audio data at a corresponding moment, represented by a vector of size [1, 512].

[0097] As mentioned above, the audio encoding module can be an encoder in the ASR model, which is composed of 18 Conformers connected in sequence, and the output of the last Conformer is the content feature of the sample audio data. When training the audio encoding module, audio data in multiple languages ​​can be used for training, so that the audio encoding module has the ability to process audio data in multiple languages, so that the audio processing model can process the first audio data in any language to obtain the second audio data, thereby achieving the voice changing effect of audio data in different languages. In one example, the audio encoding module is trained with audio data in English and Chinese, which can enable the audio encoding module to have the ability to process audio data in multiple languages ​​(including English and Chinese).

[0098] It can be seen that through this embodiment, the content features of the sample audio data can be extracted efficiently and quickly through the pre-trained audio encoding module.

[0099] In one embodiment, the audio data used to train the audio encoding module may be the sample audio data used to train the audio processing model, or may not be the sample audio data used to train the audio processing model. The audio data used to train the audio encoding module may have multiple timbres and multiple languages, and the sample audio data may also have multiple timbres and multiple languages.

[0100] In one embodiment, audio data for training an audio coding module is obtained. The audio data has multiple timbres and multiple languages. The audio coding module is composed of 18 Conformers connected in sequence. During training, a fully connected layer is added to the end of the last Conformer to extract the mel-spectrogram of the audio data used for training. The mel-spectrogram is input into the audio coding module for processing. The training goal is that the fully connected layer can accurately predict the category of each phoneme in the audio data. After the training is completed, the fully connected layer is removed, and the first audio data is processed using an audio coding module composed of 18 Conformers connected in sequence, wherein the output of the last Conformer is the content feature of the first audio data, which is also the content feature of the second audio data, and the output of the middle layer is the rhythmic feature. For example, the output of the 12th Conformer is the rhythmic feature of the first audio data, which is also the rhythmic feature of the second audio data.

[0101] Different from the content encoding module and rhythm encoding module in the prior art, this embodiment uses the same audio encoding module to achieve the dual functions of content encoding and rhythm encoding, and uses multilingual data to train the audio encoding module, so that the trained audio encoding module can extract content and rhythm from multilingual audio data, thereby enabling the audio processing model to process multilingual audio data.

[0102] In this embodiment, after the audio encoding module is trained, the prosody prediction module, the first timbre conversion module, and the second timbre conversion module in FIG2 are trained. In one embodiment, the neural network structure used to form the audio processing model includes the prosody prediction module and the first timbre conversion module; training the neural network structure based on the content characteristics of the sample audio data, the timbre characteristics of the sample timbre, and the spectral characteristics of the sample audio data includes:

[0103] Predicting the rhythmic features of sample audio data through a rhythm prediction module;

[0104] generating, by the first timbre conversion module, spectral features of the third audio data based on the content features of the sample audio data output by the audio encoding module, the prosody features of the sample audio data output by the prosody prediction module, and the timbre features of the sample timbre;

[0105] Based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data, the prosody prediction module and the first timbre conversion module are trained.

[0106] FIG3 is a schematic diagram of the training process of the audio processing model provided by an embodiment of the present disclosure. As shown in FIG3 , the neural network structure for constituting the audio processing model includes a rhythm prediction module and a first timbre conversion module in the audio processing model. Based on this, after the audio encoding module extracts the content features of the sample audio data, the content features of the sample audio data are input into the rhythm prediction module. The rhythm features of the sample audio data are predicted by the rhythm prediction module. Then, the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre are spliced ​​to obtain spliced ​​features. The spliced ​​features are input into the first timbre conversion module for processing to obtain the spectrum features of the third audio data. Finally, based on the difference between the spectrum features of the sample audio data and the spectrum features of the third audio data, the rhythm prediction module and the first timbre conversion module are trained.

[0107] In this embodiment, the timbre characteristics of various sample timbres can be pre-stored in the neural network structure. If the difference between the spectral characteristics of the sample audio data and the spectral characteristics of the third audio data is less than a preset difference, it indicates that the prosody prediction module can accurately predict the prosody characteristics of the sample audio data, and that the first timbre conversion module can accurately generate the spectral characteristics of the sample audio data based on the content characteristics of the sample audio data output by the audio encoding module, the prosody characteristics of the sample audio data output by the prosody prediction module, and the timbre characteristics of the sample timbre, thereby confirming that the training of the prosody prediction module and the first timbre conversion module is complete.

[0108] The expression methods and specific meanings of the rhythmic features of the sample audio data, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the third audio data are similar to the expression methods and specific meanings of the rhythmic features, content features, timbre features, and spectral features involved in processing the first audio data. For example, the rhythmic features include the tone, pitch, sound speed, and sound intensity of the audio data at each moment, the content features include the text features of the text content of the audio data at each moment, and the spectral features can be represented by a Mel-spectrogram. For details, please refer to the previous introduction to the processing process of the first audio data, which will not be repeated here.

[0109] In this embodiment, since the trained prosody prediction module can accurately predict the prosody features of the sample audio data, it can be determined that the trained prosody prediction module can accurately learn the prosody information of various sample timbres. Since the trained first timbre conversion module can accurately generate the spectral features of the sample audio data based on the content features of the sample audio data output by the audio encoding module, the prosody features of the sample audio data output by the prosody prediction module, and the timbre features of the sample timbre, it can be determined that the trained first timbre conversion module has the ability to generate audio data based on the prosody features, content features, and timbre features. Thus, through the combined use of the prosody prediction module and the first timbre conversion module, the rhythm of the first audio data can be converted into the rhythm of the sample timbre.

[0110] It can be seen that through this embodiment, the rhythm prediction module and the first timbre conversion module can be accurately trained, and the trained rhythm prediction module can accurately learn the rhythm information of various sample timbres, so that the trained first timbre conversion module has the ability to generate audio data according to rhythm features, content features and timbre features, thereby achieving the effect of converting the rhythm of the first audio data into the rhythm of the sample timbre through the joint use of the rhythm prediction module and the first timbre conversion module.

[0111] In one embodiment, the neural network structure includes a second timbre conversion module; the audio encoding module is further configured to output prosodic features of the sample audio data; and training the neural network structure based on content features of the sample audio data, timbre features of the sample timbre, and spectral features of the sample audio data includes:

[0112] generating, by the second timbre conversion module, spectral features of the fourth audio data based on the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre;

[0113] The second timbre conversion module is trained based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

[0114] As shown in Figure 3, the neural network structure used to form the audio processing model also includes a second timbre conversion module. Based on this, after extracting the content features of the sample audio data at the last layer of the audio encoding module and the prosodic features of the sample audio data at the middle layer of the audio encoder, the content features of the sample audio data output by the audio encoding module, the prosodic features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre are concatenated to obtain concatenated features. The concatenated features are then input into the second timbre conversion module for processing to obtain spectral features of the fourth audio data. Finally, the second timbre conversion module is trained based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

[0115] In this embodiment, timbre features of various sample timbres can be pre-stored in the neural network structure. If the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data is less than a preset difference, it indicates that the second timbre conversion module can accurately generate the spectral features of the sample audio data based on the content features of the sample audio data output by the audio encoding module, the rhythmic features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre, thereby determining that the training of the second timbre conversion module is complete.

[0116] The expression methods and specific meanings of the rhythmic features of the sample audio data, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the fourth audio data are similar to the expression methods and specific meanings of the rhythmic features, content features, timbre features, and spectral features involved in processing the first audio data. For example, the rhythmic features include the tone, pitch, sound speed, and sound weight of the audio data at each moment, the content features include the text features of the text content of the audio data at each moment, and the spectral features can be represented by a Mel-spectrogram. For details, please refer to the previous introduction to the processing process of the first audio data, which will not be repeated here.

[0117] In this embodiment, since the trained second timbre conversion module can accurately generate the spectral characteristics of the sample audio data based on the content characteristics of the sample audio data output by the audio encoding module, the rhythmic characteristics of the sample audio data output by the audio encoding module, and the timbre characteristics of the sample timbre, it can be determined that the trained second timbre conversion module has the ability to generate audio data based on the rhythmic characteristics, content characteristics and timbre characteristics, thereby achieving the effect of keeping the rhythm of the first audio data unchanged through the use of the second timbre conversion module.

[0118] It can be seen that through this embodiment, the second timbre conversion module can be accurately trained, so that the trained second timbre conversion module has the ability to generate audio data based on rhythmic features, content features and timbre features, thereby achieving the effect of keeping the rhythm of the first audio data unchanged through the use of the second timbre conversion module.

[0119] In one embodiment, both the first and second timbre conversion modules in the audio processing model can be composed of multiple connected conformers, with parameters of the first and second timbre conversion modules independently updated during training. The prosody prediction module is composed of multiple stacked convolutional networks. The vocoder in the audio processing model can be pre-trained, similar to the audio processing modules.

[0120] In one embodiment, referring to FIG3 , the complete training process of the audio processing model is as follows:

[0121] First, a vocoder and an audio processing module are pre-trained. Then, sample audio data with a sample timbre is obtained. The content features and rhythm features of the sample audio data are extracted by the pre-trained audio encoding module. The rhythm features of the sample audio data are predicted by the rhythm prediction module. The spectrum features of the third audio data are generated by the first timbre conversion module based on the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the rhythm prediction module, and the timbre features of the sample timbre. The spectrum features of the fourth audio data are generated by the second timbre conversion module based on the content features of the sample audio data output by the audio encoding module, the rhythm features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre. Based on the principle of minimizing the difference between the spectral features of the sample audio data, the spectral features of the third audio data, and the spectral features of the fourth audio data, the rhythm prediction module, the first timbre conversion module, and the second timbre conversion module are trained. The pre-trained audio encoding module, the trained rhythm prediction module, the first timbre conversion module, the second timbre conversion module, the pre-trained vocoder, and the switching module in Figure 2 are combined to obtain an audio processing model. The switching module can also be pre-trained. Since the vocoder is pre-trained, the vocoder is not shown in FIG3 .

[0122] In summary, the above audio processing model and audio processing method can achieve the following technical effects:

[0123] 1. Processing the first audio data into second audio data with the same content but different timbre, thereby changing the timbre of the audio data;

[0124] 2. The second rhythm of the second audio data can be made the same as the first rhythm of the first audio data or the same as the third rhythm of the second timbre, thereby achieving the effect of changing the rhythm while changing the timbre or keeping the rhythm unchanged, realizing the effect of rhythm control and achieving high-quality audio processing;

[0125] 3. The pre-trained audio encoding module can be trained based on data in multiple languages, so that the audio encoding module has the ability to process audio data in multiple languages, and the audio processing model can process the first audio data in any language to obtain the second audio data, achieving the voice changing effect of audio data in different languages.

[0126] An embodiment of the present disclosure further provides an audio processing device for implementing the above-mentioned audio processing method. FIG4 is a structural diagram of the audio processing device provided by an embodiment of the present disclosure. As shown in FIG4 , the device includes:

[0127] The data acquisition unit 401 is configured to acquire first audio data having first text content, a first timbre, and a first rhythm;

[0128] A feature acquisition unit 402 is configured to process the first audio data using an audio processing model to acquire content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; and the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0129] The audio generation unit 403 is configured to generate the second audio data according to the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre through the audio processing model.

[0130] Optionally, the second rhythm is the same as the third rhythm; the feature acquisition unit 402 is specifically used to: extract the content features of the first audio data as the content features of the second audio data through the audio encoding module in the audio processing model; and predict the rhythm features of the second audio data based on the content features of the second audio data and the rhythm information of the third rhythm through the rhythm prediction module in the audio processing model.

[0131] Optionally, the second rhythm is the same as the first rhythm; the feature acquisition unit 402 is specifically used to: extract the content features of the first audio data as the content features of the second audio data through the audio encoding module in the audio processing model; extract the rhythm features of the first audio data as the rhythm features of the second rhythm through the audio encoding module in the audio processing model.

[0132] Optionally, the feature acquisition unit 402 is specifically configured to acquire the timbre feature of the second timbre from a timbre feature library of the audio processing model.

[0133] Optionally, the audio generation unit 403 is specifically used to: generate, through the first timbre conversion module in the audio processing model, spectral features corresponding to the second audio data according to the content features of the second audio data output by the audio encoding module, the rhythm features of the second audio data output by the rhythm prediction module, and the timbre features of the second timbre; and generate, through the vocoder in the audio processing model, the second audio data according to the spectral features corresponding to the second audio data.

[0134] Optionally, the audio generation unit 403 is specifically used to: generate, through the second timbre conversion module in the audio processing model, spectral features corresponding to the second audio data according to the content features of the second audio data output by the audio encoding module, the rhythmic features of the second audio data output by the audio encoding module, and the timbre features of the second timbre; and generate, through the vocoder in the audio processing model, the second audio data according to the spectral features corresponding to the second audio data.

[0135] Optionally, it also includes a training unit, which is used to: obtain sample audio data with sample timbre; extract content features of the sample audio data, timbre features of the sample timbre and spectral features of the sample audio data; train a neural network structure based on the content features of the sample audio data, the timbre features of the sample timbre and the spectral features of the sample audio data; the trained neural network structure is used to form the audio processing model.

[0136] Optionally, the training unit is specifically used to: extract content features of the sample audio data through a pre-trained audio encoding module; wherein the audio encoding module is used to form the audio processing model.

[0137] Optionally, the neural network structure includes a prosody prediction module and a first timbre conversion module; the training unit is specifically used to: predict the prosody features of the sample audio data through the prosody prediction module; generate spectral features of the third audio data through the first timbre conversion module according to the content features of the sample audio data output by the audio encoding module, the prosody features of the sample audio data output by the prosody prediction module, and the timbre features of the sample timbre; and train the prosody prediction module and the first timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

[0138] Optionally, the neural network structure includes a second timbre conversion module; the audio encoding module is also used to output the rhythmic features of the sample audio data; the training unit is specifically used to: generate spectral features of fourth audio data through the second timbre conversion module according to the content features of the sample audio data output by the audio encoding module, the rhythmic features of the sample audio data output by the audio encoding module, and the timbre features of the sample timbre; and train the second timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

[0139] The audio processing device in the embodiment of the present disclosure can implement the various processes of the above-mentioned audio processing method embodiment and achieve the same effects and functions, which will not be repeated here.

[0140] An embodiment of the present disclosure also provides an electronic device. FIG5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG5 , the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 501 and a memory 502. One or more applications or data may be stored in the memory 502. Among them, the memory 502 may be a temporary storage or a persistent storage. The application stored in the memory 502 may include one or more modules (not shown in the figure), each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 501 may be configured to communicate with the memory 502 and execute a series of computer executable instructions in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input or output interfaces 505, one or more keyboards 506, etc.

[0141] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the following process:

[0142] Acquire first audio data having first text content, a first timbre, and a first rhythm;

[0143] Processing the first audio data using an audio processing model to obtain content features of second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0144] The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre.

[0145] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effects and functions, which will not be repeated here.

[0146] Another embodiment of the present disclosure further provides a computer-readable storage medium for storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:

[0147] Acquire first audio data having first text content, a first timbre, and a first rhythm;

[0148] Processing the first audio data using an audio processing model to obtain content features of second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre;

[0149] The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre.

[0150] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effects and functions, which will not be repeated here.

[0151] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0152] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0153] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0154] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0155] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0156] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0158] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0160] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0161] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0162] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0163] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. An audio processing method, comprising: Acquire first audio data having first text content, first timbre and first rhythm; The first audio data is processed by an audio processing model to obtain content features of the second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre; The second audio data is generated by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data and the timbre feature of the second timbre.

2. The method according to claim 1, wherein: The second rhythm is the same as the third rhythm; and the processing of the first audio data by the audio processing model to obtain content features of the second audio data and rhythm features of the second audio data includes: extracting, by means of an audio encoding module in the audio processing model, content features of the first audio data as content features of the second audio data; The prosody feature of the second audio data is predicted based on the content feature of the second audio data and the prosody information of the third prosody through the prosody prediction module in the audio processing model.

3. The method according to claim 1, wherein: The second rhythm is the same as the first rhythm; and the processing of the first audio data by the audio processing model to obtain content features of the second audio data and rhythm features of the second audio data includes: extracting, by means of an audio encoding module in the audio processing model, content features of the first audio data as content features of the second audio data; The prosody features of the first audio data are extracted through the audio encoding module in the audio processing model as the prosody features of the second prosody.

4. The method according to claim 1, wherein: Acquiring the timbre characteristics of the second timbre includes: The timbre feature of the second timbre is obtained from the timbre feature library of the audio processing model.

5. The method according to claim 2, wherein: The step of generating the second audio data by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre comprises: Generate, by the first timbre conversion module in the audio processing model, a frequency spectrum feature corresponding to the second audio data according to the content feature of the second audio data output by the audio encoding module, the prosody feature of the second audio data output by the prosody prediction module, and the timbre feature of the second timbre; The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

6. The method according to claim 3, wherein: The step of generating the second audio data by the audio processing model according to the content feature of the second audio data, the rhythm feature of the second audio data, and the timbre feature of the second timbre comprises: Generate, by a second timbre conversion module in the audio processing model, a frequency spectrum feature corresponding to the second audio data according to content features of the second audio data output by the audio encoding module, rhythm features of the second audio data output by the audio encoding module, and timbre features of the second timbre; The second audio data is generated according to the frequency spectrum features corresponding to the second audio data through the vocoder in the audio processing model.

7. The method according to claim 1, further comprising: Obtaining sample audio data having a sample timbre; Extracting content features of the sample audio data, timbre features of the sample timbre, and spectrum features of the sample audio data; A neural network structure is trained according to the content features of the sample audio data, the timbre features of the sample timbre and the spectrum features of the sample audio data; the trained neural network structure is used to form the audio processing model.

8. The method according to claim 7, wherein: The extracting content features of the sample audio data comprises: The content features of the sample audio data are extracted through a pre-trained audio encoding module; wherein the audio encoding module is used to form the audio processing model.

9. The method according to claim 8, wherein: The neural network structure includes a rhythm prediction module and a first timbre conversion module; the training of the neural network structure according to the content characteristics of the sample audio data, the timbre characteristics of the sample timbre and the spectrum characteristics of the sample audio data includes: Predicting the prosody features of the sample audio data by the prosody prediction module; Generate, by the first timbre conversion module, frequency spectrum features of third audio data according to content features of the sample audio data output by the audio encoding module, rhythm features of the sample audio data output by the rhythm prediction module, and timbre features of the sample timbre; The prosody prediction module and the first timbre conversion module are trained based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

10. The method according to claim 8, wherein: The neural network structure includes a second timbre conversion module; the audio encoding module is further used to output the rhythmic features of the sample audio data; the training of the neural network structure according to the content features of the sample audio data, the timbre features of the sample timbre and the spectrum features of the sample audio data includes: generating, by the second timbre conversion module, frequency spectrum features of fourth audio data according to content features of the sample audio data output by the audio encoding module, rhythm features of the sample audio data output by the audio encoding module, and timbre features of the sample timbre; The second timbre conversion module is trained based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

11. An audio processing device, comprising: A data acquisition unit, configured to acquire first audio data having first text content, a first timbre, and a first rhythm; a feature acquisition unit, configured to process the first audio data using an audio processing model, Acquire content features of second audio data, rhythm features of the second audio data, and timbre features of the second timbre; the second text content of the second audio data is the same as the first text content; the second timbre of the second audio data is different from the first timbre; the second rhythm of the second audio data is the same as the first rhythm or the same as the third rhythm of the second timbre; The audio generation unit is used to generate the second audio data according to the content characteristics of the second audio data, the rhythm characteristics of the second audio data and the timbre characteristics of the second timbre through the audio processing model.

12. An electronic device comprising: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10.

Images