Audio processing method, electronic device, and program

Abstract

Disclosed are an audio processing method, apparatus, electronic device, and storage medium, the method comprising: a step (S102) of acquiring first audio data having first text content, a first timbre, and a first prosody; a step (S104) of processing the first audio data by an audio processing model to acquire content features of second audio data having second text content which is the same as the first text content, a second timbre which is different from the first timbre, and a second prosody which is the same as the first prosody or the same as the third prosody of the second timbre, as well as timbre features of the second audio data and timbre features of the second timbre; and a step (S106) of generating second audio data by an audio processing model based on the content features of the second audio data, the timbre features of the second audio data, and the timbre features of the second timbre.

Description

Technical Field

[0001] This application claims the priority of a Chinese patent application filed with the China National Intellectual Property Administration on September 28, 2023, with an application number of 202311278074.7 and an invention title of "Audio Processing Method, Apparatus, Electronic Device and Storage Medium", the content of which is hereby incorporated by reference in its entirety into this application.

[0002] The present disclosure relates to the technical field of audio processing, and in particular, to an audio processing method, apparatus, electronic device and storage medium.

Background Art

[0003] In order to meet the user's audio processing needs in scenarios such as audio-visual entertainment and virtual social, in the prior art, various audio processing methods are provided, for example, trimming the length of audio data, changing the timbre of audio data, etc. Changing the timbre of audio data is one of the important directions of audio processing. Based on this, it is necessary to provide a technical solution for changing the timbre of audio data and realizing high-quality audio processing.

Summary of the Invention

[0004] Embodiments of the present disclosure provide an audio processing method, apparatus, electronic device and storage medium that can change the timbre of audio data and realize high-quality audio processing.

[0005] According to a first aspect, embodiments of the present disclosure provide an audio processing method, the method comprising: acquiring first audio data having first text content, a first timbre and a first rhythm; A step of processing the first audio data using an audio processing model to obtain content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre. The audio processing model includes the step of generating the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre.

[0006] According to a second aspect, an embodiment of the present disclosure provides an audio processing device. The device is A data acquisition unit that acquires first audio data having first text content, first timbre, and first prosody, A feature acquisition unit that processes the first audio data using an audio processing model and acquires content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre, The audio generation unit includes an audio generation unit that generates the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre, according to the audio processing model.

[0007] According to a third aspect, an embodiment of the present disclosure provides an electronic device comprising a processor and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor causes the processor to perform the steps of the method according to the first aspect.

[0008] According to a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium. The computer-readable storage medium is used to store computer executable instructions, and when the computer executable instructions are executed by a processor, the steps of the method described in the first aspect are realized.

[0009] In one or more embodiments of the present disclosure, first, first audio data having first text content, first timbre, and first prosody is obtained; next, the first audio data is processed by an audio processing model to obtain content features of second audio data, prosody features of second audio data, and timbre features of second timbre; the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of second timbre; and finally, second audio data is generated by the audio processing model based on the content features of second audio data, prosody features of second audio data, and timbre features of second timbre. [Brief explanation of the drawing]

[0010] To better illustrate one or more embodiments of this disclosure or the technical concepts in the prior art, the drawings that may be used to describe the embodiments or the prior art are briefly described below. Obviously, the drawings in the following description are only a few embodiments of the disclosure, and those skilled in the art can obtain other drawings based on these without any creative effort. [Figure 1] This is a flowchart of an audio processing method according to one embodiment of the present disclosure. [Figure 2]This is a schematic diagram of the inference process of an audio processing model according to one embodiment of the present disclosure. [Figure 3] This is a schematic diagram of the training process of an audio processing model according to one embodiment of the present disclosure. [Figure 4] This is a schematic diagram of the structure of an audio processing device according to one embodiment of the present disclosure. [Figure 5] This is a schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0011] To enable a person skilled in the art to better understand the technical concepts in one or more embodiments of this disclosure, the technical concepts in one or more embodiments of this disclosure will be described clearly and completely below, with reference to the drawings of one or more embodiments of this disclosure. Clearly, the embodiments described are only a subset of the embodiments of this disclosure, not all embodiments. All other embodiments derived from one or more embodiments of this disclosure without the creative effort of a person skilled in the art should fall within the scope of this disclosure.

[0012] Embodiments of this disclosure provide an audio processing method that can modify the timbre of audio data and achieve high-quality audio processing. This audio processing method may be applied to a server and executed by the server.

[0013] Figure 1 is a flowchart of an audio processing method according to one embodiment of the present disclosure, and as shown in Figure 1, the flowchart is Step S102 involves acquiring first audio data having first text content, first timbre, and first prosody. Step S104 involves processing the first audio data using an audio processing model to obtain content features, prosodic features, and timbre features of the second audio data, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre. The process includes step S106, which generates second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre, using an audio processing model.

[0014] In the embodiments of this disclosure, first, first audio data having first text content, first timbre, and first prosody is obtained; next, the first audio data is processed by an audio processing model to obtain content features of second audio data, prosody features of second audio data, and timbre features of second timbre; the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of second timbre; and finally, second audio data is generated by the audio processing model based on the content features of second audio data, prosody features of second audio data, and timbre features of second timbre. As can be seen from the above, according to this embodiment, the audio processing model can process the first audio data into second audio data with the same content but a different timbre, thereby achieving the effect of changing the timbre of the audio data. Furthermore, it is possible to make the second prosody of the second audio data the same as the first prosody of the first audio data, or the same as the third prosody of the second timbre, thereby achieving the effect of changing the timbre and the prosody, or keeping the prosody as is, and realizing high-quality audio processing.

[0015] The method flow shown in Figure 1 will be explained in detail below.

[0016] In step S102 above, first audio data having first text content, first timbre, and first prosody is acquired. In this embodiment, the first audio data includes spoken language in a human voice or a robot voice that imitates a human voice, a song sung a cappella, a nursery rhyme recited, etc. The first text content of the first audio data is text content corresponding to the first audio data, for example, "Hello, today is September 10th, Teacher's Day." The first timbre of the first audio data is the timbre of the human voice or robot voice in the first audio data. The first prosody of the first audio data is information such as pitch, tone, speed, and intensity of the voice when the first text content is recited or sung a cappella in the human voice or robot voice in the first audio data.

[0017] In each embodiment of this specification, "text" refers to written language in written form. For example, a portion of a novel on a network is a portion of text, and the simplest example of text is, for example, "hello." In each embodiment of this specification, "prosody" refers to information such as intonation, pitch, speed of speech, and volume of speech. Audio data consists of three elements: timbre, content, and prosody. In audio data audible to the human ear, all information other than content and timbre can be classified as prosody.

[0018] In step S104 above, the first audio data is processed by the audio processing model to obtain the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre. The second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre.

[0019] Similar to the first audio data, in this embodiment, the second audio data includes human speech or spoken words by a robot imitating human speech, an acapella song, a recited nursery rhyme, etc. The second text content of the second audio data is the text content corresponding to the second audio data. For example, it is "Hello, today is September 10th, Teacher's Day". The second timbre of the second audio data is the timbre of human speech or robot speech in the second audio data. The second rhythm of the second audio data is information such as pitch, tone, speech speed, and speech intensity when reading the second text content or singing in acapella with human speech or robot speech in the second audio data.

[0020] In one example, when the first audio data is the text spoken by User A and the text content is "Hello, today is September 10th, Teacher's Day", the second text content of the second audio data is the same as the first text content of the first audio data, both being "Hello, today is September 10th, Teacher's Day". The first timbre of the first audio data is the timbre of a male, and the second timbre of the second audio data is the timbre of a female. The second rhythm of the second audio data is the same as the first rhythm of the first audio data or the same as the third rhythm of the second timbre.

[0021] In this embodiment, the first audio data has a first rhythm, the second audio data has a second rhythm, and the second timbre has a third rhythm. The fact that the first rhythm of the first audio data is the same as the second rhythm of the second audio data means that information such as the pitch at each moment, the pitch at each moment, the speed of the voice at each moment, and the intensity of the voice at each moment when reading the first text content with a human voice or a robot voice in the first audio data or singing a cappella is the same as the information such as the pitch at each moment, the pitch at each moment, the speed of the voice at each moment, and the intensity of the voice at each moment when reading the second text content with a human voice or a robot voice in the second audio data or singing a cappella. In one embodiment, the fact that the first rhythm of the first audio data is the same as the second rhythm of the second audio data can be represented by the fact that the rhythm characteristics of the first audio data are the same as the rhythm characteristics of the second audio data. The rhythm characteristics of audio data can be represented by vectors, and by determining whether the numerical values of the two vectors are the same, it can be determined whether the two rhythm characteristics are the same.

[0022] The third rhythm of the second timbre represents information such as the average pitch, average pitch, average speed of the voice, and average intensity of the voice when reading various contents or singing a cappella with the second timbre. When reading the second text content or singing a cappella with the second timbre, the fact that the second rhythm of the second audio data is the same as the third rhythm of the second timbre means that information such as the average pitch at each moment, the average pitch at each moment, the average speed of the voice at each moment, and the average intensity of the voice at each moment when reading the second text content with a human voice or a robot voice in the second audio data or singing a cappella is the same as the rhythm of the second timbre.

[0023] As can be seen from the above, according to this embodiment, the first audio data can be processed into second audio data with the same content but a different timbre, and the effect of changing the timbre of the audio data can be achieved. Furthermore, compared to the prior art, which can only change the timbre of the audio data and cannot control the effect of changing the rhythm of the audio data, this embodiment allows for controllable changes to the rhythm of the audio data, the second rhythm of the second audio data can be made the same as the first rhythm of the first audio data, achieving the effect of changing the timbre while maintaining the rhythm, or the second rhythm of the second audio data can be made the same as the third rhythm of the second timbre, achieving the effect of changing the timbre while correspondingly changing the rhythm, thereby harmonizing and matching the timbre and rhythm of the second audio data.

[0024] Furthermore, in this embodiment, the language of the first audio data is not limited, and the second audio data can be obtained by processing the first audio data in any language, thereby realizing the effect of converting audio data between different languages.

[0025] In step S104 described above, the first audio data is processed by an audio processing model to obtain the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre. In this embodiment, an audio processing model for processing the first audio data into the second audio data is pre-trained. The audio processing model can obtain the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre to generate the second audio data.

[0026] In one embodiment, the second prosody of the second audio data is the same as the third prosody of the second timbre, and the steps of processing the first audio data with an audio processing model to obtain the content features and prosodic features of the second audio data are as follows: The audio encoding module in the audio processing model extracts content features from the first audio data as content features from the second audio data, The method includes the step of predicting the prosodic features of a second audio data based on the content features of a first audio data and the prosodic information of a third prosodic, using a prosodic prediction module in the audio processing model.

[0027] The content features of the second audio data include, but are not limited to, the text features of the text content at each time point in the second audio data. The prosodic features of the second audio data include, but are not limited to, the intonation, pitch, speech speed, and speech intensity at each time point in the second audio data.

[0028] Figure 2 is a schematic diagram of the inference process of an audio processing model according to one embodiment of the present disclosure. As shown in Figure 2, the trained audio processing model includes an audio coding module and a prosodic prediction module. In the process of processing first audio data by the audio processing model to generate second audio data, if the second prosodic of the second audio data is the same as the third prosodic of the second timbre, the spectral features of the first audio data are obtained and input to the audio coding module. The audio coding module processes the spectral features of the first audio data and extracts the content features of the first audio data as the content features of the second audio data.

[0029] In one example, a Fourier transform and Mel filter are performed on the first audio data to obtain the Mel spectrogram of the first audio data as the spectral features of the first audio data. The spectral features of the first audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the first audio data at the corresponding time and frequency. The spectral features of the first audio data are input to an audio coding module, which processes the spectral features of the first audio data to obtain the content features of the first audio data as the content features of the second audio data. The content features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature of the second audio data at the corresponding time, and is represented by a vector of size [1, 512].

[0030] In one example, the audio coding module may be an encoder in an ASR (Automatic Speech Recognition) model, and is composed of 18 Conformers (Convolution-augmented Transformers for Speech Recognition) connected in sequence, with the output of the last Conformer being the content features of the second audio data. The audio coding module can be trained using audio data in multiple languages ​​during training, thereby giving the module the ability to process audio data in multiple languages. The audio processing model can process the first audio data in any language to obtain the second audio data, achieving a speech conversion effect between audio data in different languages. In one example, by training the audio coding module using audio data in English and Chinese, the audio coding module can have the ability to process audio data in multiple languages ​​(including English and Chinese).

[0031] As shown in Figure 2, the trained audio processing model further includes a prosodic prediction module. In the process of processing the first audio data by the audio processing model to generate the second audio data, if the second prosodicity of the second audio data is the same as the third prosodicity of the second timbre, the content features of the second audio data output from the audio encoding module are input to the prosodic prediction module. The prosodic prediction module then predicts the prosodic features of the second audio data based on the input content features of the second audio data and the prosodic information of the third prosodicity stored in the prosodic prediction module.

[0032] As can be seen from the previous explanation, the prosodic information of the third tone of the second timbre includes information such as the average pitch, average speed, and average intensity of the voice when reading various contents aloud or singing a cappella with the second timbre. The prosodic features of the second audio data include information such as the pitch, pitch, speed, and intensity of the voice at each time when the second text content is read aloud or sung a cappella with a human or robot voice in the second audio data. Therefore, predicting the prosodic features of the second audio data based on the content features and prosodic information of the third tone is a process of predicting information such as the pitch, pitch, speed, and intensity of the voice at each time based on the specific content and the average pitch, pitch, speed, and intensity of the voice represented by the third tone of the second audio data.

[0033] To achieve the effect of predicting information such as the tone, pitch, speed, and intensity of the voice at each moment, the input to the prosodic prediction module includes the content features of the second audio data, and the prosodic information of the third prosody is learned by the prosodic prediction module during the model training process and stored in the prosodic prediction module as a model parameter. The prosodic prediction module takes the content features of the second audio data as input and, based on the content features of the second audio data and the prosodic information of the third prosody of the second timbre, can predict information such as the tone, pitch, speed, and intensity of the voice at each moment when the second text content is read aloud or sung a cappella in a human or robot voice in the second audio data, as prosodic features of the second audio data.

[0034] In one embodiment, the audio processing model learns the ability to convert the timbre of audio data to multiple preset timbres during the training process, and the prosodic prediction module learns the prosodic information of each preset timbre during training and stores the prosodic information of each preset timbre as a model parameter in the prosodic prediction module. Based on this, when a user uses the audio processing model in this embodiment to achieve a voice modification effect, they can input first audio data and specify the second timbre and second prosodic of the second audio data to be converted. If the second timbre is located in the above preset timbres and the audio processing model determines that the second prosodic is the same as the third prosodic of the second timbre, the prosodic prediction module in the audio processing model predicts the prosodic features of the second audio data based on the content features of the second audio data and the stored prosodic information of the third prosodic.

[0035] In one example, the prosodic features of the second audio data may be represented by a matrix of size [100,512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding prosodic feature of the second audio data at the corresponding time, and is represented by a vector of size [1,512].

[0036] As can be seen from the above, according to this embodiment, the audio processing model can implement a prosodic prediction function using a prosodic prediction module, and achieve the effect of changing the prosodicity of the first audio data using the prosodic prediction function.

[0037] In one embodiment, the second prosody of the second audio data is the same as the first prosody of the first audio data, and the steps of processing the first audio data with an audio processing model to obtain the content features and prosodic features of the second audio data are as follows: The audio encoding module in the audio processing model extracts content features from the first audio data as content features from the second audio data, The audio processing model includes the step of extracting prosodic features of the first audio data as prosodic features of the second prosodic data using an audio encoding module in the audio processing model.

[0038] The content features of the second audio data include, but are not limited to, the text features of the text content at each time point in the second audio data. The prosodic features of the second audio data include, but are not limited to, the intonation, pitch, speech speed, speech weight, etc., at each time point in the second audio data.

[0039] Referring to Figure 2 and the explanation above, the process by which the audio encoding module in the audio processing model extracts content features from the first audio data as content features from the second audio data is the same as described above and will not be repeated here.

[0040] In this embodiment, spectral features of the first audio data are acquired, the spectral features of the first audio data are input to an audio coding module, the audio coding module processes the spectral features of the first audio data, and extracts the prosodic features of the first audio data as the prosodic features of the second audio data.

[0041] In one example, a Fourier transform and Mel filter are performed on the first audio data to obtain the Mel spectrogram of the first audio data as the spectral features of the first audio data. The spectral features of the first audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the first audio data at the corresponding time and frequency. The spectral features of the first audio data are input to an audio coding module, which processes the spectral features of the first audio data to obtain the prosodic features of the first audio data as the prosodic features of the second audio data. The prosodic features of the second audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding prosodic features of the second audio data at the corresponding time, and is represented by a vector of size [1, 512].

[0042] In one example, the audio encoding module may be an encoder in an ASR model, consisting of 18 Conformers connected in sequence, where the output of the last layer is the content feature, and the outputs of the intermediate layers are the prosodic features; for example, the output of the 12th Conformer is the prosodic feature of the second audio data.

[0043] As can be seen from the above, according to this embodiment, the audio encoding module can perform both prosodic feature extraction and content feature extraction. This allows multiple functions to be implemented with the same module, eliminating the need to distribute modules for content feature extraction and prosodic feature extraction, thereby improving the efficiency of audio processing.

[0044] In one embodiment, the step of acquiring the timbre characteristics of the second timbre is: This includes the step of obtaining the timbre features of a second timbre from a timbre feature library of the audio processing model.

[0045] In this embodiment, the audio processing model learns the ability to convert the timbre of audio data to multiple preset timbres during the training process, the prosodic prediction module learns the prosodic information of each preset timbre during training, stores the prosodic information of each preset timbre as a model parameter in the prosodic prediction module, and after the training of the audio processing model is completed, the timbre features of each preset timbre are stored in the timbre feature library. Based on this, when a user uses the audio processing model in this embodiment to achieve a voice modification effect, they can input first audio data and specify the second timbre and second prosodic of the second audio data to be converted, and since the second timbre is located in the above preset timbres, the audio processing model extracts the timbre features of the second timbre from the timbre feature library. In this embodiment, the audio processing model may store the timbre features of various preset timbres in the timbre feature library during the training process.

[0046] As can be seen from the above, according to this embodiment, the timbre features of various timbres can be stored in the timbre feature library of the audio processing model, and in the audio processing process, the timbre features of the second timbre can be directly obtained from the timbre feature library, thus eliminating the process of extracting timbre features and improving the efficiency of audio processing.

[0047] Since the audio data includes three elements: timbre, content, and prosody, in step S106 above, the audio processing model further generates a second audio data based on the content features of the second audio data, the prosody features of the second audio data, and the timbre features of the second timbre.

[0048] In one embodiment, if the second prosody of the second audio data is the same as the third prosody of the second timbre, the step of generating the second audio data based on the content features of the second audio data, the prosody features of the second audio data, and the timbre features of the second timbre by the audio processing model is: The audio processing model includes the steps of generating spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the prosodic prediction module, and the timbre features of the second timbre, using the first timbre conversion module in the audio processing model. The method includes the step of generating a second audio data based on spectral features corresponding to the second audio data using a vocoder in the audio processing model.

[0049] In this embodiment, as shown in Figure 2, the audio processing model further includes a first timbre conversion module and a vocoder. When the second prosody of the second audio data is the same as the third prosody of the second timbre, the first timbre conversion module has the ability to synthesize audio data based on content features, prosodic features, and timbre features. Therefore, the first timbre conversion module generates spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the prosody prediction module, and the timbre features of the second timbre. The first timbre conversion module can obtain the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0050] In one example, the content features of the second audio data may be represented by a matrix of size [100,512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature of the second audio data at the corresponding time, and is represented by a vector of size [1,512], and the prosodic features of the second audio data may be represented by a matrix of size [100,512], where 100 represents the time dimension The first audio data is represented by a vector of size [1,512], where 512 represents the feature dimension, and each value in the matrix represents the corresponding prosodic feature at the corresponding time of the second audio data. The timbre features of the second timbre may also be represented by a matrix of size [100,256], where 100 represents the time dimension and 256 represents the feature dimension, and each value in the matrix represents the timbre feature at the corresponding time of the second timbre, and each value in the matrix represents the timbre feature at the corresponding time of the second timbre, and each value in the matrix represents the vector of size [1,256].

[0051] In this embodiment, feature concatenation is performed on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the prosodic prediction module, and the timbre features of the second timbre to obtain the concatenation result [100, 512 + 512 + 256]. This concatenation result is input to the first timbre conversion module, which generates spectral features corresponding to the second audio data based on the concatenation result. The spectral features corresponding to the second audio data are represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the second audio data at the corresponding time and frequency. The spectral features corresponding to the second audio data may also be represented by a Mel spectrogram.

[0052] The first timbre conversion module is connected to a vocoder, and the input to the vocoder includes spectral features corresponding to the second audio data output from the first timbre conversion module. The vocoder generates the second audio data based on the spectral features corresponding to the second audio data.

[0053] As can be seen from the above, according to this embodiment, when the second prosody of the second audio data is the same as the third prosody of the second timbre, the first timbre conversion module generates spectral features corresponding to the second audio data, and the vocoder generates the second audio data based on the spectral features corresponding to the second audio data. This allows the first audio data to be processed into the second audio data efficiently and quickly, thereby improving audio processing efficiency.

[0054] In one embodiment, if the second prosody of the second audio data is the same as the first prosody of the first audio data, the step of generating the second audio data based on the content features of the second audio data, the prosody features of the second audio data, and the timbre features of the second timbre by the audio processing model is: The audio processing model includes a step of generating spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the audio encoding module, and the timbre features of the second timbre, by a second timbre conversion module in the audio processing model. The method includes the step of generating a second audio data based on spectral features corresponding to the second audio data using a vocoder in the audio processing model.

[0055] In this embodiment, as shown in Figure 2, the audio processing model further includes a second timbre conversion module and a vocoder. When the second prosody of the second audio data is the same as the first prosody of the first audio data, the second timbre conversion module has the ability to synthesize audio data based on content features, prosodic features, and timbre features. Therefore, the second timbre conversion module generates spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the audio encoding module, and the timbre features of the second timbre. The second timbre conversion module can obtain the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0056] In one example, the content features of the second audio data may be represented by a matrix of size [100,512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature of the second audio data at the corresponding time, and is represented by a vector of size [1,512], and the prosodic features of the second audio data may be represented by a matrix of size [100,512], where 100 represents the time dimension The first audio data is represented by a vector of size [1,512], where 512 represents the feature dimension, and each value in the matrix represents the corresponding prosodic feature at the corresponding time of the second audio data. The timbre features of the second timbre may also be represented by a matrix of size [100,256], where 100 represents the time dimension and 256 represents the feature dimension, and each value in the matrix represents the timbre feature at the corresponding time of the second timbre, and each value in the matrix represents the timbre feature at the corresponding time of the second timbre, and each value in the matrix represents the vector of size [1,256].

[0057] In this embodiment, feature concatenation is performed on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the audio encoding module, and the timbre features of the second timbre to obtain the concatenation result [100, 512 + 512 + 256]. This concatenation result is input to the second timbre conversion module, which generates spectral features corresponding to the second audio data based on the concatenation result. The spectral features corresponding to the second audio data are represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension, and each value in the matrix represents the amplitude of the second audio data at the corresponding time and frequency. The spectral features corresponding to the second audio data may also be represented by a Mel spectrogram.

[0058] The second timbre conversion module is connected to a vocoder, and the input to the vocoder contains spectral features corresponding to the second audio data output from the second timbre conversion module. The vocoder generates the second audio data based on the spectral features corresponding to the second audio data.

[0059] As can be seen from the above, according to this embodiment, when the second prosody of the second audio data is the same as the first prosody of the first audio data, the second timbre conversion module generates spectral features corresponding to the second audio data, and the vocoder generates the second audio data based on the spectral features corresponding to the second audio data. This allows the first audio data to be processed into the second audio data efficiently and quickly, thereby improving audio processing efficiency.

[0060] As shown in Figure 2, the audio processing model further includes a switching module. In one example, when a user uses the audio processing model in this embodiment to achieve a voice modification effect, they can input first audio data and specify the second timbre and second prosody of the second audio data to be converted, with the second timbre being one of the preset timbres. After inputting the spectral features of the first audio data, the second timbre specified by the user, and the second prosody specified by the user into the audio processing model, the switching module can receive the second timbre and second prosody specified by the user. After determining that the second prosody is the same as the third prosody of the second timbre, the switching module activates the audio coding module, the prosody prediction module, the first timbre conversion module, and the vocoder. The content features output from the audio coding module are output to the prosody prediction module and the first timbre conversion module, respectively, and the second audio data is output to the user through the process described above. At this time, the switching module controls the audio coding module so as not to output the prosody features. After determining that the second prosody is the same as the first prosody, the switching module activates the audio encoding module, the second timbre conversion module, and the vocoder. It outputs the content features output from the audio encoding module to the second timbre conversion module, and outputs the second audio data to the user through the process described earlier. At this time, the switching module controls the audio encoding module to output the prosody features and content features.

[0061] The above describes in detail the audio processing process using the audio processing model. The training process for the audio processing model is described below. In one embodiment, the flow of the above method is: Steps include obtaining sample audio data that has a sample timbre, Steps include extracting content features of sample audio data, timbre features of sample timbres, and spectral features of sample audio data, The method further includes the step of training a neural network structure based on the content features of sample audio data, the timbre features of sample timbres, and the spectral features of sample audio data, and the trained neural network structure is used to construct an audio processing model.

[0062] First, sample audio data containing sample timbres is obtained. Sample timbres are the preset timbres mentioned above, i.e., target timbres that the audio processing model can convert to. For example, if a trained audio processing model is set to change audio data to timbres such as old man, male, and female, then these timbres are sample timbres. In one embodiment, sample audio data in multiple languages ​​is obtained for each sample timbre; for example, sample audio data in Chinese and sample audio data in English are obtained for each sample timbre.

[0063] Next, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data are extracted. The content features of the sample audio data include the text features of the text content at each time point in the sample audio data. The spectral features of the sample audio data may be represented by a Mel spectrogram.

[0064] Finally, a neural network structure is trained based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data. The trained neural network structure is then used to construct an audio processing model.

[0065] As can be seen from the above, according to this embodiment, sample audio data having a sample timbre can be obtained, and by training an audio processing model based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data, the trained audio processing model can process the timbre of the audio data into the sample timbre, thereby realizing a voice modification effect.

[0066] In one embodiment, the step of extracting content features from sample audio data is: The process includes a step of extracting content features from sample audio data using a pre-trained audio coding module, which is then used to construct an audio processing model.

[0067] In this embodiment, the audio coding module is the audio coding module shown in Figure 2. This audio coding module is obtained by pre-training it compared to other modules in the audio processing model. After the audio coding module has been trained, the spectral features of each sample audio data are obtained, and these spectral features are input to the audio coding module. The audio coding module then processes the spectral features of the sample audio data and extracts the content features of the sample audio data.

[0068] In one example, a Fourier transform and Mel filter are performed on sample audio data to obtain the Mel spectrogram of the sample audio data as the spectral features of the sample audio data. The spectral features of the sample audio data can be represented by a matrix of size [100, 80], where 100 represents the time dimension and 80 represents the frequency dimension, and each value in the matrix represents the amplitude at the corresponding frequency at the corresponding time of the sample audio data. The spectral features of the sample audio data are input to a trained audio coding module, which processes the spectral features of the sample audio data to obtain the content features of the sample audio data. The content features of the sample audio data can be represented by a matrix of size [100, 512], where 100 represents the time dimension and 512 represents the feature dimension, and each value in the matrix represents the corresponding content feature at the corresponding time of the sample audio data, and is represented by a vector of size [1, 512].

[0069] As mentioned above, the audio coding module may also be an encoder in the ASR model, and is composed of 18 Conformers connected in sequence, with the output of the last Conformer being the content features of the sample audio data. The audio coding module can be trained using audio data in multiple languages ​​during training, thereby giving the audio coding module the ability to process audio data in multiple languages. The audio processing model can process first audio data in any language to obtain second audio data, achieving a speech conversion effect between audio data in different languages. For example, by training the audio coding module using audio data in English and Chinese, the audio coding module can have the ability to process audio data in multiple languages ​​(including English and Chinese).

[0070] As can be seen from the above, according to this embodiment, the content features of sample audio data can be extracted efficiently and quickly using a pre-trained audio coding module.

[0071] In one embodiment, the audio data used to train the audio coding module may be sample audio data for training the audio processing model described above, or it may not be sample audio data for training the audio processing model described above. The audio data used to train the audio coding module may have multiple timbres and multiple languages, and the sample audio data may have multiple timbres and multiple languages.

[0072] In one embodiment, audio data is acquired to train an audio coding module, which has multiple timbres and multiple languages. The audio coding module is configured with 18 Conformers connected in sequence. During training, one fully connected layer is added to the last Conformer, the Mel spectrogram of the training audio data is extracted, and the Mel spectrogram is input to the audio coding module for processing. The training goal is for the fully connected layer to accurately predict the category of each phoneme in the audio data. After training is complete, the fully connected layer is removed, and the first audio data is processed using the audio coding module configured with 18 Conformers connected in sequence. Here, the output of the last Conformer is the content features of the first audio data, i.e., the content features of the second audio data, and the output of the intermediate layers is the prosodic features. For example, the output of the 12th Conformer is the prosodic features of the first audio data, i.e., the prosodic features of the second audio data.

[0073] Unlike conventional content coding modules and prosodic coding modules, this embodiment achieves dual functionality of content coding and prosodic coding using the same audio coding module. By training the audio coding module using multilingual data, the trained audio coding module can perform content extraction and prosodic extraction on multilingual audio data, and the audio processing model can process multilingual audio data.

[0074] In this embodiment, after the audio encoding module has been trained, the prosodic prediction module, the first timbre conversion module, and the second timbre conversion module shown in Figure 2 are trained. In one embodiment, the neural network structure for configuring the audio processing model includes a prosodic prediction module and a first timbre conversion module, and the step of training the neural network structure based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data is as follows: The prosodic prediction module performs the step of predicting the prosodic features of sample audio data, The first timbre conversion module generates spectral features of a third audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre. The process includes the step of training a prosodic prediction module and a first timbre conversion module based on the difference between the spectral features of a sample audio data and the spectral features of a third audio data.

[0075] Figure 3 is a schematic diagram of the training process of an audio processing model according to one embodiment of the present disclosure. As shown in Figure 3, the neural network structure for constituting the audio processing model includes a prosodic prediction module and a first timbre conversion module in the audio processing model. Based on this, the audio encoding module extracts and obtains content features of sample audio data, inputs the content features of the sample audio data to the prosodic prediction module, and the prosodic prediction module predicts the prosodic features of the sample audio data. Next, the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre are concatenated to obtain concatenated features. These concatenated features are input to the first timbre conversion module for processing to obtain spectral features of the third audio data. Finally, the prosodic prediction module and the first timbre conversion module are trained based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

[0076] In this embodiment, the neural network structure can pre-store the timbre features of various sample timbres. It is explained that when the difference between the spectral features of the sample audio data and the spectral features of the third audio data is smaller than the preset difference, the prosodic prediction module can accurately predict the prosodic features of the sample audio data. It is also explained that the first timbre conversion module can accurately generate the spectral features of the sample audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre. Thus, it is determined that the training of the prosodic prediction module and the first timbre conversion module is complete.

[0077] The representation and specific meaning of the prosodic features of the sample audio data, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the third audio data are similar to the representation and specific meaning of the prosodic features, content features, timbre features, and spectral features related when processing the first audio data. For example, the prosodic features include the intonation, pitch, speech speed, and speech weight at each time point in the audio data; the content features include the text features of the text content at each time point in the audio data; and the spectral features can be represented by a Mel spectrogram. For specifics, please refer to the previous explanation of the processing process for the first audio data, which will not be repeated here.

[0078] In this embodiment, since the trained prosodic prediction module can accurately predict the prosodic features of the sample audio data, it can be determined that the trained prosodic prediction module can accurately learn the prosodic information of various sample timbres. The trained first timbre conversion module can accurately generate spectral features of the sample audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre. Therefore, it can be determined that the trained first timbre conversion module has the ability to generate audio data based on prosodic features, content features, and timbre features. Thus, by using the prosodic prediction module and the first timbre conversion module together, the effect of changing the prosodicity of the first audio data to the prosodicity of the sample timbre is achieved.

[0079] As can be seen from the above, according to this embodiment, a prosodic prediction module and a first timbre conversion module can be accurately trained to obtain a prosodic prediction module and a first timbre conversion module can accurately learn the prosodic information of various sample timbres, and the trained first timbre conversion module has the ability to generate audio data based on prosodic features, content features and timbre features. Thus, by using the prosodic prediction module and the first timbre conversion module together, the effect of changing the prosodicity of the first audio data to the prosodicity of the sample timbre is realized.

[0080] In one embodiment, the neural network structure includes a second timbre conversion module, and an audio encoding module is further used to output the prosodic features of the sample audio data. The step of training the neural network structure based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data is as follows: The second timbre conversion module generates spectral features of a fourth audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the audio encoding module, and the timbre features of the sample timbre. This includes the step of training a second timbre conversion module based on the difference between the spectral features of a sample audio data and the spectral features of a fourth audio data.

[0081] As shown in Figure 3, the neural network structure for constructing the audio processing model further includes a second timbre conversion module. Based on this, the content features of the sample audio data are extracted in the last layer of the audio encoding module, and the prosodic features of the sample audio data are extracted in the intermediate layer of the audio encoder. Then, the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the audio encoding module, and the timbre features of the sample timbre are concatenated to obtain concatenated features. These concatenated features are input to the second timbre conversion module for processing to obtain spectral features of the fourth audio data. Finally, the second timbre conversion module is trained based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

[0082] In this embodiment, the timbre features of various sample timbres can be pre-stored in the neural network structure. If the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data is smaller than the preset difference, the second timbre conversion module can accurately generate the spectral features of the sample audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the audio encoding module, and the timbre features of the sample timbre. This determines that the training of the second timbre conversion module is complete.

[0083] The representation and specific meaning of the prosodic features of the sample audio data, the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the fourth audio data are similar to the representation and specific meaning of the prosodic features, content features, timbre features, and spectral features related when processing the first audio data. For example, the prosodic features include the intonation, pitch, speech velocity, and speech weight at each time point in the audio data; the content features include the text features of the text content at each time point in the audio data; and the spectral features can be represented by a Mel spectrogram. For specifics, refer to the previous explanation of the processing process for the first audio data, which is not repeated here.

[0084] In this embodiment, the trained second timbre conversion module can accurately generate spectral features of sample audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the audio encoding module, and the timbre features of the sample timbre. Therefore, it can be determined that the trained second timbre conversion module has the ability to generate audio data based on prosodic features, content features, and timbre features. This enables the use of the second timbre conversion module to preserve the prosody of the first audio data.

[0085] As can be seen from the above, according to this embodiment, a second timbre conversion module can be accurately trained to obtain a second timbre conversion module, and the trained second timbre conversion module has the ability to generate audio data based on prosodic features, content features and timbre features. As a result, the use of the second timbre conversion module achieves the effect of maintaining the prosody of the first audio data as is.

[0086] In one embodiment, both the first and second timbre conversion modules in the audio processing model are composed of multiple conformers connected together, and their parameters are updated separately during training. The prosodic prediction module is composed of multiple convolutional networks stacked on top of each other. The vocoder in the audio processing model can be obtained by pre-training it, similar to the audio processing module.

[0087] In one embodiment, as shown in Figure 3, the complete training flow of the audio processing model is as follows:

[0088] First, a vocoder and audio processing module are pre-trained to obtain a sample audio data with a sample timbre. Then, a pre-trained audio encoding module extracts the content features and prosodic features of the sample audio data. A prosodic prediction module predicts the prosodic features of the sample audio data. A first timbre conversion module generates spectral features of a third audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre. A second timbre conversion module then converts the sample audio data output from the audio encoding module. Based on the content features of the audio data, the prosodic features of the sample audio data output from the audio coding module, and the timbre features of the sample timbre, spectral features of the fourth audio data are generated. Based on the principle that the difference between the spectral features of the sample audio data, the spectral features of the third audio data, and the spectral features of the fourth audio data is minimized, the prosodic prediction module, the first timbre conversion module, and the second timbre conversion module are trained. An audio processing model is obtained by combining the pre-trained audio coding module, the trained prosodic prediction module, the first timbre conversion module, the second timbre conversion module, the pre-trained vocoder, and the switching module shown in Figure 2. The switching module can also be obtained by pre-training. Since the vocoder is obtained by pre-training, the vocoder is not shown in Figure 3.

[0089] In summary, the following technical effects can be achieved using the above audio processing model and method.

[0090] 1. The first audio data is processed into a second audio data with the same content but a different timbre, thereby achieving the effect of changing the timbre of the audio data.

[0091] 2. The second prosody of the second audio data can be made the same as the first prosody of the first audio data, or the same as the third prosody of the second timbre. This achieves the effect of changing the timbre, changing the prosody, or keeping the prosody as is, thereby enabling control over the prosody and achieving high-quality audio processing.

[0092] 3. A pre-trained audio coding module can be obtained by training it on data from multiple languages. This gives the audio coding module the ability to process audio data from multiple languages. The audio processing model can process first audio data from any language to obtain second audio data, achieving a speech conversion effect between audio data from different languages.

[0093] One embodiment of the present disclosure further provides an audio processing device for realizing the above audio processing method. Figure 4 is a schematic diagram of the structure of an audio processing device according to one embodiment of the present disclosure, and as shown in Figure 4, the device is A data acquisition unit 401 for acquiring first audio data having first text content, first timbre, and first prosody, A feature acquisition unit 402 for processing the first audio data by an audio processing model and obtaining content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre, The audio processing model includes an audio generation unit 403 for generating the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre.

[0094] Preferably, the second prosody is the same as the third prosody, and the feature acquisition unit 402 is used to extract the content features of the first audio data as content features of the second audio data by the audio coding module in the audio processing model, and to predict the prosodic features of the second audio data based on the content features of the second audio data and the prosodic information of the third prosody by the prosodic prediction module in the audio processing model.

[0095] Preferably, the second prosody is the same as the first prosody, and the feature acquisition unit 402 is used specifically to extract the content features of the first audio data as content features of the second audio data by the audio coding module in the audio processing model, and to extract the prosodic features of the first audio data as prosodic features of the second prosody by the audio coding module in the audio processing model.

[0096] Preferably, the feature acquisition unit 402 is used specifically to acquire the timbre features of the second timbre from the timbre feature library of the audio processing model.

[0097] Preferably, the audio generation unit 403 is used to generate spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the prosodic prediction module, and the timbre features of the second timbre, using the first timbre conversion module in the audio processing model, and to generate the second audio data based on the spectral features corresponding to the second audio data using the vocoder in the audio processing model.

[0098] Preferably, the audio generation unit 403 is used to generate spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the audio encoding module, and the timbre features of the second timbre, by the second timbre conversion module in the audio processing model, and to generate the second audio data based on the spectral features corresponding to the second audio data by the vocoder in the audio processing model.

[0099] Preferably, the system includes a training unit for acquiring sample audio data having a sample timbre, extracting content features of the sample audio data, timbre features of the sample timbre, and spectral features of the sample audio data, and training a neural network structure based on the content features of the sample audio data, timbre features of the sample timbre, and spectral features of the sample audio data. The trained neural network structure is used to constitute the audio processing model.

[0100] Preferably, the training unit is used specifically to extract content features of the sample audio data using a pre-trained audio coding module. The audio coding module is used to constitute the audio processing model.

[0101] Preferably, the neural network structure includes a prosodic prediction module and a first timbre conversion module, wherein the training unit is used to predict the prosodic features of the sample audio data using the prosodic prediction module, generate spectral features of a third audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre using the first timbre conversion module, and train the prosodic prediction module and the first timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

[0102] Preferably, the neural network structure includes a second timbre conversion module, the audio coding module is further used to output the prosodic features of the sample audio data, and the training unit is used to generate spectral features of a fourth audio data based on the content features of the sample audio data output from the audio coding module, the prosodic features of the sample audio data output from the audio coding module, and the timbre features of the sample timbre, and to train the second timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

[0103] The audio processing apparatus in the embodiments of this disclosure can implement each process of the above-described audio processing method embodiments and achieve the same effects and functions, and there is no overlap here.

[0104] One embodiment of the present disclosure further provides an electronic device, Figure 5 being a schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure, and as shown in Figure 5, the electronic device may vary considerably in configuration or performance and may include one or more processors 501 and memory 502, the memory 502 may store one or more application programs or data. The memory 502 may be a temporary or persistent storage device. The application program stored in the memory 502 may include one or more modules (not shown), each module may include a set of computer executable instructions in the electronic device. Furthermore, the processor 501 may be configured to communicate with the memory 502 and execute a set of computer executable instructions in the memory 502 in the electronic device. The electronic device may further include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input or output interfaces 505, and one or more keyboards 506, etc.

[0105] In one specific embodiment, the electronic device includes a processor and a memory configured to store computer-executable instructions, wherein, when the computer-executable instructions are executed, the processor, First audio data having first text content, first timbre, and first prosody is obtained. The audio processing model processes the first audio data to obtain content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre. The aforementioned audio processing model enables a workflow in which the second audio data is generated based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre.

[0106] The electronic devices in the embodiments of this disclosure can implement each process of the above-described audio processing method embodiments and achieve the same effects and functions, and there is no overlap here.

[0107] Another embodiment of the present disclosure further provides a computer-readable storage medium used to store computer-executable instructions, which, when executed by a processor, First audio data having first text content, first timbre, and first prosody is obtained. The audio processing model processes the first audio data to obtain content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre. The aforementioned audio processing model enables a workflow in which the second audio data is generated based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre.

[0108] The storage medium in the embodiments of this disclosure can implement each process of the above-described audio processing method embodiment and achieve the same effects and functions, and there is no overlap here.

[0109] In each embodiment of the present disclosure, the computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] In the 1990s, improvements to technology could be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to method flows). However, with technological advancements, many current method flow improvements can be considered direct improvements to hardware circuit structures. Most designers obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that method flow improvements cannot be realized in hardware entity modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by the user programming the device. Chip manufacturers do not need to design and manufacture dedicated integrated circuit chips; designers program and "integrate" a single digital system onto a single PLD.Currently, instead of manually manufacturing integrated circuit chips, such programming is almost entirely implemented using "logic compiler" software, similar to the software compilers used during program development and organization. The source code before compilation also needs to be organized in a specific programming language, known as a Hardware Description Language (HDL). There are not just one, but several types of HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. It is also clear to those skilled in the art that by simply programming a method flow logically in one of the above hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that realizes that logical method flow can be easily obtained.

[0111] A controller can be implemented in any suitable way, for example, a controller can take the form of a microprocessor or processor and a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, microcontrollers such as the ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and a memory controller may be implemented as part of the control logic of the memory. Those skilled in the art will also see that, in addition to implementing a controller in the form of pure computer-readable program code, a controller can achieve the same function in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the entire method step. Thus, such a controller can be considered a hardware component, and the devices for realizing the various functions contained therein can also be considered structures within the hardware component. Alternatively, devices that realize various functions can be considered both as software modules that implement methods and as structures within hardware components.

[0112] The systems, devices, modules, or units described in the above embodiments may be specifically implemented by computer chips or entities, or by products having certain functions. A typical implementing device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0113] For the sake of explanation, the above apparatus will be described by dividing it into various units according to their function. Of course, when implementing the embodiments of this disclosure, the functions of each unit can be realized with the same or multiple software and / or hardware.

[0114] Those skilled in the art will see that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware embodiments. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-readable program code.

[0115] This disclosure has been described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of this disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. By providing these computer program instructions to a processor of a general-purpose computer, a dedicated computer, an embedded processor, or other programmable data processing device, a device can be generated that realizes a function specified in one flow of a flowchart or one or more blocks of multiple flows and / or block diagrams by instructions executed by the processor of the computer or other programmable data processing device.

[0116] These computer program instructions may be stored in computer-readable memory that can guide a computer or other programmable data processing device to operate in a particular manner, thereby generating a product in which the instructions stored in the computer-readable memory generate an instruction unit that implements a function specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0117] These computer program instructions may be loaded into a computer or other programmable data processing device to perform a series of operational steps on the computer or other programmable device to generate processing that is implemented on the computer, thereby providing steps to implement a function specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0118] The terms “includes,” “incorporates,” or any other variation thereof are intended to encompass non-exclusive inclusions, meaning that a process, method, product, or apparatus containing a set of elements includes not only those elements but also other elements not explicitly listed, or further elements specific to such a process, method, product, or apparatus. Unless further limited, an element limited by the phrase “includes one…” does not preclude the presence of another identical element in a process, method, product, or apparatus containing such element.

[0119] One or more embodiments of this disclosure can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, a program module includes routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstraction data type. One or more embodiments of this disclosure may be implemented in a distributed computing environment, in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0120] Each embodiment in this disclosure is described using an incremental method, and any identical or similar parts between embodiments may be referenced to one another. Each embodiment is described with an emphasis on its differences from the other embodiments. In particular, the system embodiments are essentially similar to the method embodiments and therefore their description is simple; relevant sections may be referenced from parts of the description of the method embodiments.

[0121] The foregoing describes only examples of the present disclosure and is not intended to limit the present disclosure. To those skilled in the art, various modifications and changes are possible to this disclosure. All changes, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the claims of this disclosure.

Claims

1. A step of obtaining first audio data having first text content, first timbre, and first prosody, A step of processing the first audio data using an audio processing model to obtain content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre. An audio processing method comprising the step of generating the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre, using the audio processing model described above.

2. The second prosody is the same as the third prosody, and the steps of processing the first audio data by the audio processing model and obtaining the content features and prosodic features of the second audio data are as follows: The steps include: extracting content features of the first audio data as content features of the second audio data using the audio encoding module in the audio processing model; The method according to claim 1, comprising the step of predicting the prosodic features of the second audio data based on the content features of the second audio data and the prosodic information of the third prosody using a prosodic prediction module in the audio processing model.

3. The second prosody is the same as the first prosody, and the step of processing the first audio data by the audio processing model and obtaining the content features and prosodic features of the second audio data is: The steps include: extracting content features of the first audio data as content features of the second audio data using the audio encoding module in the audio processing model; The method according to claim 1, comprising the step of extracting the prosodic features of the first audio data as prosodic features of the second prosodic using an audio encoding module in the audio processing model.

4. The step of acquiring the timbre characteristics of the aforementioned second timbre is: The method according to claim 1, further comprising the step of obtaining the timbre features of the second timbre from the timbre feature library of the audio processing model.

5. The step of generating the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre using the aforementioned audio processing model is: The audio processing model includes the step of generating spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the prosodic prediction module, and the timbre features of the second timbre, by the first timbre conversion module. The method according to claim 2, comprising the step of generating second audio data based on spectral features corresponding to the second audio data using a vocoder in the audio processing model.

6. The step of generating the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre using the aforementioned audio processing model is: The audio processing model includes the step of generating spectral features corresponding to the second audio data based on the content features of the second audio data output from the audio encoding module, the prosodic features of the second audio data output from the audio encoding module, and the timbre features of the second timbre, The method according to claim 3, comprising the step of generating second audio data based on spectral features corresponding to the second audio data using a vocoder in the audio processing model.

7. Steps include obtaining sample audio data that has a sample timbre, The steps include extracting the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data, The method according to claim 1, further comprising the step of training a neural network structure based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data, wherein the trained neural network structure is used to constitute the audio processing model.

8. The above step of extracting content features from the sample audio data is: The method according to claim 7, comprising the step of extracting content features of the sample audio data using a pre-trained audio coding module, wherein the audio coding module is used to constitute the audio processing model.

9. The neural network structure includes a prosodic prediction module and a first timbre conversion module, and the step of training the neural network structure based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data is: The steps include: predicting the prosodic features of the sample audio data using the prosodic prediction module; The first timbre conversion module generates spectral features of a third audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the prosodic prediction module, and the timbre features of the sample timbre. The method according to claim 8, comprising the step of training the prosodic prediction module and the first timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the third audio data.

10. The neural network structure includes a second timbre conversion module, the audio encoding module is further used to output the prosodic features of the sample audio data, and the step of training the neural network structure based on the content features of the sample audio data, the timbre features of the sample timbre, and the spectral features of the sample audio data is: The second timbre conversion module generates spectral features of a fourth audio data based on the content features of the sample audio data output from the audio encoding module, the prosodic features of the sample audio data output from the audio encoding module, and the timbre features of the sample timbre. The method according to claim 8, comprising the step of training the second timbre conversion module based on the difference between the spectral features of the sample audio data and the spectral features of the fourth audio data.

11. A data acquisition unit for acquiring first audio data having first text content, first timbre, and first prosody, A feature acquisition unit for processing the first audio data using an audio processing model and obtaining content features of the second audio data, prosodic features of the second audio data, and timbre features of the second timbre, wherein the second text content of the second audio data is the same as the first text content, the second timbre of the second audio data is different from the first timbre, and the second prosody of the second audio data is the same as the first prosody or the same as the third prosody of the second timbre, An audio processing apparatus including an audio generation unit for generating the second audio data based on the content features of the second audio data, the prosodic features of the second audio data, and the timbre features of the second timbre, according to the audio processing model.

12. Processor and An electronic device comprising: a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor causes the processor to perform a step according to any one of claims 1 to 10.

13. A computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, realize steps according to any one of claims 1 to 10.

Images