Audio synthesis method, device, electronic device and computer-readable storage medium

By inputting phoneme information and audio features into a neural network model, the problem of poor sound quality in synthesized audio was solved, and the sound quality and sound details of synthesized songs were improved.

CN112164387BActive Publication Date: 2024-11-19TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011002477.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-22
Publication Date
2024-11-19
Estimated Expiration
2040-09-22

AI Technical Summary

Technical Problem

The sound quality of synthesized audio in existing technologies is poor, especially when the parameters are accurate, it is difficult to achieve high sound quality.

Method used

By acquiring the phoneme information and audio features of the target dry audio, including fundamental frequency features, energy features, and perceptual linear prediction features, the data are input into a trained neural network model to generate synthetic dry audio.

Benefits of technology

It improves the sound quality and sound details of synthesized audio, enhances the overall sound quality of synthesized songs, and strengthens the neural network model's ability to characterize the target dry audio.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112164387B_ABST
    Figure CN112164387B_ABST
Patent Text Reader

Abstract

The present application discloses an audio synthesis method, device, electronic device and computer-readable storage medium, the method comprising: obtaining target dry audio, and generating phoneme information corresponding to the target dry audio; extracting audio features from the target dry audio; wherein the audio features include any one or a combination of any several of fundamental frequency features, energy features and perceptual linear prediction features; inputting the target dry audio, phoneme information and audio features into a trained neural network model to obtain synthesized dry audio. It can be seen that the audio synthesis method provided by the present application embeds audio features into the synthesis process of the neural network model, improves the efficiency and accuracy of the neural network model in synthesizing dry audio, can enhance the neural network model's ability to characterize the target dry audio, and enables the trained neural network model to generate synthesized dry audio with better sound quality and richer sound details, thereby improving the sound quality of the final synthesized song.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of sound synthesis, and more specifically, to an audio synthesis method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of deep learning technology and audio signal processing technology, artificially synthesized singing voices have gradually become possible. People can use technology to generate dry voices, that is, pure human voices without music. These synthesized dry voices can be matched with accompaniment to get songs.

[0003] In the related art, a method based on parameter synthesis is used to synthesize dry sound. When the parameters are accurate, the effect of accurate pronunciation time and controllable rhythm can be achieved, but the sound quality is generally poor, which leads to poor sound quality of the synthesized audio. It can be seen that in the process of realizing the present invention, the inventors found that there are at least the following problems in the related art: the sound quality of the synthesized audio is poor.

[0004] Therefore, how to improve the sound quality of synthesized audio is a technical problem that those skilled in the art need to solve. Summary of the invention

[0005] The purpose of the present application is to provide an audio synthesis method, device, electronic device and computer-readable storage medium, which improve the sound quality of synthesized audio.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides an audio synthesis method, comprising:

[0007] Acquire target dry audio, and generate phoneme information corresponding to the target dry audio;

[0008] Extracting audio features from the target dry audio; wherein the audio features include any one or a combination of any several of fundamental frequency features, energy features, and perceptual linear prediction features;

[0009] The target dry audio, the phoneme information and the audio features are input into the trained neural network model to obtain synthesized dry audio.

[0010] To achieve the above-mentioned purpose, the second aspect of the present application provides an audio synthesis device, comprising:

[0011] An acquisition module, used to acquire target dry audio and generate phoneme information corresponding to the target dry audio;

[0012] An extraction module, used to extract audio features from the target dry audio; wherein the audio features include any one or a combination of any several of fundamental frequency features, energy features and perceptual linear prediction features;

[0013] An input module is used to input the target dry audio, the phoneme information and the audio features into the trained neural network model to obtain synthesized dry audio.

[0014] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:

[0015] Memory for storing computer programs;

[0016] A processor is used to implement the steps of the above-mentioned audio synthesis method when executing the computer program.

[0017] To achieve the above objectives, the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio synthesis method as described above are implemented.

[0018] It can be seen from the above scheme that an audio synthesis method provided by the present application includes: obtaining target dry sound audio and generating phoneme information corresponding to the target dry sound audio; extracting audio features from the target dry sound audio; wherein the audio features include any one or a combination of any several of fundamental frequency features, energy features and perceptual linear prediction features; inputting the target dry sound audio, the phoneme information and the audio features into a trained neural network model to obtain synthesized dry sound audio.

[0019] The audio synthesis method provided by the present application first obtains the target dry audio, which is the human voice singing sound without accompaniment, and then extracts the audio features of the target dry audio, which may include fundamental frequency features, energy features, and perceptual linear prediction features. The target dry audio, the corresponding phoneme information, and the audio features are used as the input of the trained neural network model to obtain the synthesized dry audio. Since the audio features are closely related to the sound quality of the target dry audio, these audio features, when determined by the neural network model, help the neural network model to better and faster model the vocal characteristics and sound quality details of the target dry audio, which is conducive to improving the sound quality of the synthesized dry audio. It can be seen that the audio synthesis method provided by the present application embeds the audio features into the synthesis process of the neural network model, improves the efficiency and accuracy of the neural network model in synthesizing dry audio, and can enhance the neural network model's ability to depict the target dry audio, so that the trained neural network model can generate a synthesized dry audio with better sound quality and richer sound details, thereby improving the sound quality of the final synthesized song. The present application also discloses an audio synthesis device, an electronic device, and a computer-readable storage medium, which can also achieve the above technical effects.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific implementation methods, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0022] Figure 1 An architecture diagram of an audio synthesis system provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of an audio synthesis method provided in an embodiment of the present application;

[0024] Figure 3 A flow chart of a method for extracting fundamental frequency characteristics of target dry audio provided in an embodiment of the present application;

[0025] Figure 4 A flow chart of a method for extracting energy characteristics of target dry audio provided in an embodiment of the present application;

[0026] Figure 5 A flow chart of a method for extracting perceptual linear prediction features of target dry audio provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of a training method for a neural network model provided in an embodiment of the present application;

[0028] Figure 7 A structural diagram of an audio synthesis device provided in an embodiment of the present application;

[0029] Figure 8 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In order to facilitate understanding of the audio synthesis method provided by the present application, the system used by the method is introduced below. Figure 1 , which shows an architecture diagram of an audio synthesis system provided by an embodiment of the present application, such as Figure 1 As shown, it includes an audio acquisition device 10 and a server 20.

[0032] The audio acquisition device 10 is used to acquire the target dry audio, and the server 20 is used to generate the phoneme information of the target dry audio and extract the audio features of the target dry audio. The target dry audio, the corresponding phoneme information and the audio features are used as inputs of the trained neural network model to obtain the synthesized dry audio.

[0033] Of course, the audio synthesis system can also have a client 30, which can include fixed terminals such as PC (Chinese full name: Personal Computer, English full name: Personal Computer) and mobile terminals such as mobile phones. A speaker can be set on the client 30 to output synthesized dry audio or songs synthesized based on the synthesized dry audio.

[0034] The embodiment of the present application discloses an audio synthesis method, which improves the sound quality of synthesized audio.

[0035] See also Figure 2 , a flowchart of an audio synthesis method provided in an embodiment of the present application, such as Figure 2 As shown, including:

[0036] S101: Acquire target dry audio, and generate phoneme information corresponding to the target dry audio;

[0037] The executor of this embodiment is the server in the audio synthesis system provided in the above embodiment, and its purpose is to synthesize dry audio. In this step, the audio acquisition device acquires the target dry audio and sends it to the server, and the server generates the phoneme information corresponding to the target dry audio. The target dry audio is a dry waveform file recorded by the user, and the audio format is WAV (Waveform Audio File Format). It should be noted that due to the lossy encoding methods such as MP3, the differences between different decoders will cause the actual read audio to have a time offset at the beginning or end of the audio. Therefore, in order to ensure the invariance of the dry waveform, it is saved in WAV format.

[0038] In a specific implementation, to obtain the lyric text corresponding to the target dry audio, one can directly obtain the lyric file corresponding to the target dry audio, or directly extract the lyric text from the target dry audio, that is, directly recognize the lyric text corresponding to the dry voice from the target dry audio, and no specific limitation is imposed here. It can be understood that since the target dry audio may contain noise during recording, resulting in inaccurate lyric recognition, the target dry audio can be denoised before recognizing the lyric text. Secondly, each lyric character in the target dry audio is obtained from the training lyric text, and the phonetic symbol of each lyric character is determined according to the character type of each lyric character. If the character type of the lyric character is Chinese, its corresponding phonetic symbol is pinyin. For example, for the lyric text "太,阳,当,空,照" (tai, yang, dang, kong, zhao), its corresponding pinyin is "tai yang dangkong zhao". If the character type of the lyric character is English, its corresponding phonetic symbol is the English phonetic symbol. The phoneme information of the training dry audio can be generated according to the phonetic symbol of each lyric character, and this phoneme information describes the start and end times of each phoneme in the target dry audio. For example, the start and end times of the phoneme "t" are 10ms and 30ms respectively, and the start and end times of the phoneme "ai" are 30ms and 60ms respectively. This embodiment does not limit the specific representation method of the phoneme information. For example, it can be represented in the form of info = [10, 30, t; 30, 60, ai; …], and the semicolon represents the end of a phoneme.

[0039] S102: Extract audio features from the target dry audio; wherein, the audio features include any one or a combination of several of fundamental frequency features, energy features, and perceptual linear prediction features;

[0040] The purpose of this step is to extract the audio features of the target dry audio, and these audio features are closely related to the vocal characteristics and sound quality of the target dry audio. The audio features here can include fundamental frequency features, energy features, and perceptual linear prediction features, etc. The fundamental frequency feature refers to the lowest vibration frequency of a segment of dry audio, which reflects the pitch of the dry audio. The larger the value of the fundamental frequency, the higher the pitch of the dry audio. The energy feature refers to the energy value of the dry audio. The perceptual linear prediction feature (Perceptual Linear Prediction, plp) has relatively good speech recognition accuracy and noise robustness. Those skilled in the art can flexibly select any of the above audio features during feature extraction, or select multiple audio features, and no specific limitation is imposed here.

[0041] S103: Input the target dry audio, the phoneme information, and the audio features into the trained neural network model to obtain the synthesized dry audio.

[0042] In this step, the audio features are embedded as features in the synthesis process of the dry audio, that is, the target dry audio, the corresponding phoneme information and the audio features are input into the trained neural network model to synthesize the target dry audio. The neural network model in this embodiment may include CNN (Chinese full name: Convolutional Neural Networks, English full name: Convolutional Neural Networks), DNN (Chinese full name: Deep Neural Networks, English full name: Deep Neural Networks), waveRNN (Chinese full name: Single-layer Recurrent Neural Network, English full name: wave Recurrent Neural Network), etc., which are not specifically limited here.

[0043] The audio synthesis method provided in the embodiment of the present application first obtains the target dry audio, which is the human voice singing sound without accompaniment, and then extracts the audio features of the target dry audio, which may include fundamental frequency features, energy features, and perceptual linear prediction features. The target dry audio, the corresponding phoneme information, and the audio features are used as the input of the trained neural network model to obtain the synthesized dry audio. Since the audio features are closely related to the sound quality of the target dry audio, these audio features, when determined by the neural network model, help the neural network model to better and faster model the vocal characteristics and sound quality details of the target dry audio, which is conducive to improving the sound quality of the synthesized dry audio. It can be seen that the audio synthesis method provided in the embodiment of the present application embeds the audio features into the synthesis process of the neural network model, improves the efficiency and accuracy of the neural network model in synthesizing dry audio, and can enhance the neural network model's ability to depict the target dry audio, so that the trained neural network model can generate a synthesized dry audio with better sound quality and richer sound details, thereby improving the sound quality of the final synthesized song.

[0044] On the basis of the above embodiments, as a preferred implementation mode, after synthesizing the target dry sound audio using the trained neural network model, it also includes: using an audio effect device to add sound effects to the synthesized dry sound audio; obtaining the accompaniment audio corresponding to the synthesized dry sound audio, and superimposing the accompaniment audio and the synthesized dry sound audio with the added sound effects in a preset manner to obtain the synthesized audio.

[0045] It is understandable that the synthesized dry audio can be combined with the accompaniment to generate the final song, and the synthesized song can be stored in the background of the server, output to the client or played through the speaker.

[0046] In a specific implementation, the synthesized dry audio can be processed by sound effect devices such as reverberators and equalizers to obtain dry audio with certain sound effects. There are many ways to choose from for the sound effect device here, such as processing through sound effect plug-ins, sound effect algorithms, etc., which are not specifically limited here. Since the synthesized dry audio is pure vocal audio and there is no instrumental sound, it is actually different from common songs in life. For example, it does not contain a prelude without vocals. If there is no accompaniment, the prelude is a silent section. Therefore, it is necessary to superimpose the synthesized dry audio with the added effect and the accompaniment audio in a preset manner to obtain the synthesized audio, that is, the song.

[0047] The specific superposition method is not limited here, and the technology in this field can be flexibly selected according to actual conditions. As a feasible implementation method, the accompaniment audio and the synthesized dry sound audio after adding sound effects are superimposed in a preset manner to obtain a synthesized audio, including: performing power normalization processing on the accompaniment audio and the synthesized dry sound audio after adding sound effects to obtain intermediate accompaniment audio and intermediate dry sound audio; superimposing the intermediate accompaniment audio and the intermediate dry sound audio according to a preset energy ratio to obtain the synthesized audio. In a specific implementation, the accompaniment audio and the synthesized dry sound audio after adding sound effects are power normalized respectively to obtain intermediate accompaniment audio accom and intermediate dry sound audio vocal, both of which are time domain waveforms. If the preset energy ratio is 0.6:0.4, the synthesized audio W=0.6×vocal+0.4×accom.

[0048] It can be seen that this embodiment designs a method for improving the sound quality of synthesized songs based on audio features. Taking advantage of the efficiency, robustness and accuracy of the algorithm, under the premise of a certain data set and neural network model, by providing more accurate sound feature information, the model's ability to model the sound quality of songs is improved, and ultimately the sound quality of the model's synthesized sound is improved. These synthesized songs with higher sound quality can not only enrich the existing user listening experience and improve user satisfaction, but also expand the number of existing high-quality songs and enhance the influence and competitiveness of platform providers.

[0049] The present application embodiment discloses a method for extracting fundamental frequency characteristics of target dry sound audio, specifically:

[0050] See also Figure 3 , a flow chart of a method for extracting fundamental frequency characteristics of target dry sound audio provided by an embodiment of the present application, such as Figure 3 As shown, including:

[0051] S201: Decomposing the target dry audio into multiple audio waveforms;

[0052] S202: extracting the fundamental frequency of each audio waveform within a preset frequency range using a filter;

[0053] S203: Determine a fundamental frequency feature of the target dry audio based on the fundamental frequencies of all the audio waveforms.

[0054] The purpose of this embodiment is to extract the fundamental frequency characteristics of the target dry audio. In a specific implementation, the target dry audio can be decomposed into multiple continuous audio waveforms. If the target dry audio audio = [y 1 ,y 2 ,y 3 ,…,y n ], then y n is the audio waveform of the nth segment, which contains multiple sampling points, and each sampling point is the value of the time domain waveform at each moment. The fundamental frequency feature can be extracted using a filter. During the calculation process, when the frequency of the filter matches the fundamental frequency to be extracted, the output energy is the largest. In general, the accuracy of fundamental frequency extraction affects the pitch of the final synthesis effect. The more accurate the fundamental frequency, the more accurate the pitch of the synthesized sound. It should be noted that in the process of extracting the fundamental frequency of an audio waveform, the preset frequency range that can be limited is 65Hz to 750Hz. This frequency range includes not only the range of common male voices, but also the range of common female voices, which is conducive to the accurate extraction of the fundamental frequency when singing high or low notes in actual scenarios. Through fundamental frequency extraction, the fundamental frequency feature A of the target dry audio can be obtained = [f 1 ,f 2 ,f 3 ,…,f n ], where f n To extract the nth audio waveform y n The fundamental frequency.

[0055] The present application embodiment discloses a method for extracting energy characteristics of target dry sound audio, specifically:

[0056] See also Figure 4 , a flow chart of a method for extracting energy characteristics of target dry sound audio provided by an embodiment of the present application, such as Figure 4 As shown, including:

[0057] S301: Decomposing the target dry audio into multiple audio waveforms;

[0058] S302: Calculate the Mel spectrum of each segment of the audio waveform, express the Mel spectrum in decibel values, and perform maximum value normalization processing on all the decibel values ​​to obtain energy data corresponding to each segment of the audio waveform;

[0059] S303: Adding energy data of each audio waveform within a preset frequency range to obtain energy characteristics of each audio waveform;

[0060] S304: Determine the energy characteristics of the target dry audio based on the energy characteristics of all the audio waveforms.

[0061] The purpose of this embodiment is to extract the energy characteristics of the target dry sound audio. In the specific implementation, the target dry sound audio is first decomposed into multiple continuous audio waveforms, and then the Mel spectrum is calculated for each audio waveform, recorded as b1, and then b1 is converted into data in decibels (dB), recorded as b2, and then b2 is normalized to the maximum value, that is, the maximum value is controlled at 0dB, to obtain b3, and finally the energy of the value of b3 within the specified frequency range is added to obtain the energy value within the preset frequency range, which is recorded as the energy characteristic of this audio waveform. The preset frequency range here can also be 65Hz to 750Hz.

[0062] For example, the nth waveform y n By calculating the Mel spectrum, decibel conversion and normalization, we can get b3 = [e 1 ,e 2 ,e 3 ,e m ], where e m is the energy at the mth frequency, in dB. If the preset frequency range is 65Hz to 750Hz, corresponding to the 10th and 50th frequencies respectively, the data in this interval is recorded as tmp=[e 10 ,e 11 ,e 12 ,…,e 5 0], then the nth waveform y n The corresponding energy characteristic B n The calculation formula is B n =10×log 10 (∑(10 tmp / 10 )), we can get the energy characteristic of the target dry sound audio B = [B 1 ,B 2 ,B 3 ,…,B n ].

[0063] The present application embodiment discloses a method for extracting perceptual linear prediction features of target dry audio, specifically:

[0064] See also Figure 5 , a flow chart of a method for extracting perceptual linear prediction features of target dry audio provided by an embodiment of the present application, such as Figure 5 As shown, including:

[0065] S401: Decomposing the target dry audio into multiple audio waveforms;

[0066] S402: extracting the perceptual linear prediction features of each segment of the audio waveform, and calculating the first-order difference mean and the second-order difference mean of the perceptual linear prediction features of each segment of the audio waveform;

[0067] S403: Determine the perceptual linear prediction features of the target dry audio based on the first-order difference means and the second-order difference means of the perceptual linear prediction features of all the audio waveforms.

[0068] The purpose of this embodiment is to extract the perceptual linear prediction features of the target dry sound audio. In the specific implementation, the target dry sound audio is first decomposed into multiple continuous audio waveforms, and then the first-order difference mean and the second-order difference mean of the perceptual linear prediction features of each audio waveform are calculated. The first-order difference refers to the subtraction of adjacent values ​​of the plp feature in the time dimension, and the second-order difference refers to the subtraction of adjacent values ​​of the first-order difference results in the time dimension. The first-order difference results and the second-order difference results are then used to calculate the mean of the frequency dimension to obtain the features of the time dimension.

[0069] For example, the nth waveform y n The plp characteristic is P n =[h 1 ,h 2 ,h 3 ,…,h k ], where h k YesP n The kth feature in the frequency dimension. After calculating the first-order difference and the second-order difference, we obtain the first-order difference result dPn of plp and the second-order difference result ddPn of plp respectively, and then calculate the mean of the frequency dimension, that is, calculate the mean of the first to kth values ​​in dPn, recorded as d1, and the mean of the first to kth values ​​in ddPn, recorded as d2, then C n = [d1, d2], training perceptual linear prediction features of dry audio C = [C 1 ,C 2 ,C 3 ,…,C n ].

[0070] The present application embodiment discloses a method for training a neural network model, specifically:

[0071] See also Figure 6 , a flowchart of a training method for a neural network model provided in an embodiment of the present application, such as Figure 6 As shown, including:

[0072] S501: Acquire training dry audio, and generate training phoneme information corresponding to the training dry audio;

[0073] S502: extracting training audio features from the training dry audio; wherein the training audio features include any one or a combination of any several of fundamental frequency features, energy features, and perceptual linear prediction features;

[0074] S503: Train a neural network model using the training dry audio, the training phoneme information, and the training audio features to obtain the trained neural network model.

[0075] In this embodiment, the training dry sound audio, the corresponding training phoneme information and the training audio features are input into the neural network model to obtain a trained neural network model. The neural network model here may include CNN, DNN, waveRNN, etc., which are not specifically limited here. The specific process of generating the training phoneme information and extracting the training audio features are similar to the process described in the above embodiment, and will not be repeated here.

[0076] Of course, some other necessary information can also be input during the training process, such as dry sound number, sound information, etc. Compared with the solution in the related art that directly uses the spectrum information of the training dry sound audio to train the neural network model, embedding the training process during the training process can effectively enhance the learning ability of the neural network model for the input data and improve the model's ability to model the sound quality details, so that in the synthesis stage, a better training model can be used to synthesize dry sound audio with higher sound quality.

[0077] In this embodiment, since the audio features of the training dry audio are embedded in the training process of the neural network model, the audio features are closely related to the sound quality of the dry audio. When these audio features are determined by the neural network model, they help the neural network model to better and faster model the vocal characteristics and sound quality details of the training dry audio. The dry audio synthesized by the trained neural network model has a higher sound quality, which effectively improves the sound quality and naturalness of the synthesized singing voice, thereby helping to improve user satisfaction, and also helping platform providers to obtain higher quality synthesized singing voices, and enhance the influence and competitiveness of platform providers. In addition, this embodiment can be processed by computer background or cloud processing, with high processing efficiency and fast running speed.

[0078] The training method of the neural network model provided in the embodiment of the present application first obtains the training dry audio, which is the human voice singing sound without accompaniment, and then extracts the audio features of the training dry audio, which may include fundamental frequency features, energy features and perceptual linear prediction features, etc. The training dry audio, the corresponding training phoneme information and the training audio features are used as the input of the neural network model for model training. Since the training audio features are closely related to the sound quality of the training dry audio, these training audio features, when determined by the neural network model, help the neural network model to better and faster model the vocal characteristics and sound quality details of the training dry audio, which is conducive to improving the sound quality of the neural network model synthesized singing. It can be seen that the training method of the neural network model provided in the embodiment of the present application embeds the audio features into the training process of the neural network model, improves the efficiency and accuracy of the neural network model training, and can enhance the neural network model's ability to depict the target dry audio, so that the trained neural network model can generate a target dry audio with better sound quality and richer sound details, thereby improving the sound quality of the final synthesized song.

[0079] An audio synthesis device provided in an embodiment of the present application is introduced below. The audio synthesis device described below and the audio synthesis method described above can be referenced to each other.

[0080] See also Figure 7 , a structural diagram of an audio synthesis device provided in an embodiment of the present application, such as Figure 7 As shown, including:

[0081] A first acquisition module 100 is used to acquire a target dry audio and generate phoneme information corresponding to the target dry audio;

[0082] A first extraction module 200 is used to extract audio features from the target dry audio; wherein the audio features include any one or a combination of any several of fundamental frequency features, energy features and perceptual linear prediction features;

[0083] The input module 300 is used to input the target dry audio, the phoneme information and the audio feature into the trained neural network model to obtain synthesized dry audio.

[0084] The audio synthesis device provided in the embodiment of the present application first obtains the target dry audio, which is the human voice singing sound without accompaniment, and then extracts the audio features of the target dry audio, which may include fundamental frequency features, energy features, and perceptual linear prediction features. The target dry audio, the corresponding phoneme information, and the audio features are used as the input of the trained neural network model to obtain the synthesized dry audio. Since the audio features are closely related to the sound quality of the target dry audio, these audio features, when determined by the neural network model, help the neural network model to better and faster model the vocal characteristics and sound quality details of the target dry audio, which is conducive to improving the sound quality of the synthesized dry audio. It can be seen that the audio synthesis device provided in the embodiment of the present application embeds the audio features into the synthesis process of the neural network model, improves the efficiency and accuracy of the neural network model in synthesizing dry audio, and can enhance the neural network model's ability to depict the target dry audio, so that the trained neural network model can generate a synthesized dry audio with better sound quality and richer sound details, thereby improving the sound quality of the final synthesized song.

[0085] Based on the above embodiment, as a preferred implementation, it also includes:

[0086] A second acquisition module is used to acquire training dry sound audio and generate training phoneme information corresponding to the training dry sound audio;

[0087] A second extraction module is used to extract training audio features from the training dry audio; wherein the training audio features include any one or a combination of any several of fundamental frequency features, energy features and perceptual linear prediction features;

[0088] A training module is used to train a neural network model using the training dry sound audio, the training phoneme information and the training audio features to obtain the trained neural network model.

[0089] Based on the above embodiment, as a preferred implementation, it also includes:

[0090] An adding module, used for adding sound effects to the synthesized dry audio using a sound effect device;

[0091] The superposition module is used to obtain the accompaniment audio corresponding to the synthesized dry audio, and superimpose the accompaniment audio and the synthesized dry audio with added sound effects in a preset manner to obtain the synthesized audio.

[0092] Based on the above embodiment, as a preferred implementation, the superposition module includes:

[0093] An acquisition unit, used to acquire the accompaniment audio corresponding to the synthesized dry voice audio;

[0094] A normalization unit, used for performing power normalization processing on the accompaniment audio and the synthesized dry audio with added sound effects to obtain an intermediate accompaniment audio and an intermediate dry audio;

[0095] A superposition unit is used to superimpose the intermediate accompaniment audio and the intermediate dry sound audio according to a preset energy ratio to obtain the synthesized audio.

[0096] Based on the above embodiment, as a preferred implementation, if the audio feature includes a fundamental frequency feature, the extraction module 200 includes:

[0097] A decomposition unit, used to decompose the target dry sound audio into multiple audio waveforms;

[0098] A first extraction unit, configured to extract the fundamental frequency of each audio waveform within a preset frequency range using a filter;

[0099] The first determining unit is used to determine the fundamental frequency characteristics of the target dry audio based on the fundamental frequencies of all the audio waveforms.

[0100] Based on the above embodiment, as a preferred implementation, if the audio feature includes an energy feature, the extraction module 200 includes:

[0101] A decomposition unit, used to decompose the target dry sound audio into multiple audio waveforms;

[0102] A first calculation unit is used to calculate the Mel spectrum of each segment of the audio waveform, express the Mel spectrum in decibel values, and perform maximum value normalization processing on all the decibel values ​​to obtain energy data corresponding to each segment of the audio waveform;

[0103] A second calculation unit, configured to add energy data of each segment of the audio waveform within a preset frequency range to obtain an energy feature of each segment of the audio waveform;

[0104] The second determining unit is configured to determine the energy feature of the target dry audio based on the energy features of all the audio waveforms.

[0105] Based on the above embodiment, as a preferred implementation, if the audio feature includes a perceptual linear prediction feature, the extraction module 200 includes:

[0106] A decomposition unit, used to decompose the target dry sound audio into multiple audio waveforms;

[0107] A second extraction unit, used to extract the perceptual linear prediction features of each segment of the audio waveform, and calculate the first-order difference mean and the second-order difference mean of the perceptual linear prediction features of each segment of the audio waveform;

[0108] The third determining unit is used to determine the perceptual linear prediction features of the target dry audio based on the first-order difference means and the second-order difference means of the perceptual linear prediction features of all the audio waveforms.

[0109] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0110] The present application also provides an electronic device, see Figure 8 , a structural diagram of an electronic device 80 provided in an embodiment of the present application, such as Figure 8 As shown, a processor 81 and a memory 82 may be included.

[0111] Among them, the processor 81 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 81 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 81 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 81 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 81 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0112] The memory 82 may include one or more computer-readable storage media, which may be non-transitory. The memory 82 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 82 is at least used to store the following computer program 821, wherein, after the computer program is loaded and executed by the processor 81, it can implement the relevant steps in the audio synthesis method performed by the server side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 82 may also include an operating system 822 and data 823, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 822 may include Windows, Unix, Linux, etc.

[0113] In some embodiments, the electronic device 80 may further include a display screen 83 , an input / output interface 84 , a communication interface 85 , a sensor 86 , a power source 87 , and a communication bus 88 .

[0114] certainly, Figure 8 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0115] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the audio synthesis method executed by the electronic device in any of the above embodiments are implemented.

[0116] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0117] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. An audio synthesis method, It is characterized in that include: Acquire target dry audio, and generate phoneme information corresponding to the target dry audio; Extracting audio features related to the sound quality of the target dry audio from the target dry audio; wherein the audio features include energy features and perceptual linear prediction features; Inputting the target dry audio, the phoneme information and the audio features into the trained neural network model, so that the trained neural network model models the sound quality of the target dry audio based on the audio features, thereby generating synthetic dry audio in combination with the phoneme information; The step of extracting audio features related to the sound quality of the target dry audio from the target dry audio includes: Decomposing the target dry sound audio into multiple audio waveforms; Calculating the Mel spectrum of each segment of the audio waveform, expressing the Mel spectrum in decibel values, and performing maximum value normalization processing on all the decibel values ​​to obtain energy data corresponding to each segment of the audio waveform; Adding the energy data of each audio waveform within a preset frequency range to obtain the energy characteristics of each audio waveform; Determine the energy characteristics of the target dry audio based on the energy characteristics of all the audio waveforms; Extracting perceptual linear prediction features of each segment of the audio waveform; wherein the perceptual linear prediction features of the audio waveform include features corresponding to multiple frequencies; Calculate the first-order difference result and the second-order difference result of the perceptual linear prediction feature of each audio waveform; wherein the first-order difference refers to the subtraction calculation of adjacent values ​​of the perceptual linear prediction feature in the time dimension, and the second-order difference refers to the subtraction calculation of adjacent values ​​of the first-order difference result in the time dimension; Calculate the mean of the first-order difference results of the perceptual linear prediction features of each segment of the audio waveform in the frequency dimension as the first-order difference mean, and calculate the mean of the second-order difference results in the frequency dimension as the second-order difference mean; The perceptual linear prediction features of the target dry audio are determined based on first-order difference means and second-order difference means of the perceptual linear prediction features of all the audio waveforms.

2. The audio synthesis method according to claim 1, It is characterized in that Also includes: Acquire training dry audio, and generate training phoneme information corresponding to the training dry audio; Extracting training audio features from the training dry audio; wherein the training audio features include energy features and perceptual linear prediction features; The neural network model is trained using the training dry audio, the training phoneme information and the training audio features to obtain the trained neural network model.

3. The audio synthesis method according to claim 1, It is characterized in that After inputting the target dry audio, the phoneme information and the audio feature into the trained neural network model to obtain synthesized dry audio, the method further includes: Using an audio effect device to add sound effects to the synthesized dry audio; Accompaniment audio corresponding to the synthesized dry audio is obtained, and the accompaniment audio and the synthesized dry audio with added sound effects are superimposed in a preset manner to obtain synthesized audio.

4. The audio synthesis method according to claim 3, It is characterized in that The accompaniment audio and the synthesized dry sound audio with added sound effects are superimposed in a preset manner to obtain a synthesized audio, including: Performing power normalization processing on the accompaniment audio and the synthesized dry audio with added sound effects to obtain intermediate accompaniment audio and intermediate dry audio; The intermediate accompaniment audio and the intermediate dry sound audio are superimposed according to a preset energy ratio to obtain the synthesized audio.

5. An audio synthesis device, It is characterized in that include: An acquisition module, used to acquire target dry audio and generate phoneme information corresponding to the target dry audio; An extraction module, configured to extract audio features related to the sound quality of the target dry audio from the target dry audio; wherein the audio features include energy features and perceptual linear prediction features; An input module, used for inputting the target dry audio, the phoneme information and the audio features into the trained neural network model, so that the trained neural network model models the sound quality of the target dry audio based on the audio features, thereby generating a synthetic dry audio in combination with the phoneme information; The extraction module is specifically used to: decompose the target dry sound audio into multiple audio waveforms; calculate the Mel spectrum of each audio waveform, express the Mel spectrum in decibel values, and perform maximum value normalization processing on all the decibel values ​​to obtain energy data corresponding to each audio waveform; add the energy data of each audio waveform within a preset frequency range to obtain the energy characteristics of each audio waveform; determine the energy characteristics of the target dry sound audio based on the energy characteristics of all the audio waveforms; extract the perceptual linear prediction features of each audio waveform; wherein the perceptual linear prediction features of the audio waveform include multiple frequency corresponding Features; calculating the first-order difference results and second-order difference results of the perceptual linear prediction features of each segment of the audio waveform; wherein the first-order difference refers to the subtraction of adjacent values ​​of the perceptual linear prediction features in the time dimension, and the second-order difference refers to the subtraction of adjacent values ​​of the first-order difference results in the time dimension; calculating the mean of the first-order difference results of the perceptual linear prediction features of each segment of the audio waveform in the frequency dimension as the first-order difference mean, and the mean of the second-order difference results in the frequency dimension as the second-order difference mean; determining the perceptual linear prediction features of the target dry sound audio based on the first-order difference mean and the second-order difference mean of the perceptual linear prediction features of all the audio waveforms.

6. An electronic device, It is characterized in that include: Memory for storing computer programs; A processor, configured to implement the steps of the audio synthesis method as claimed in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio synthesis method according to any one of claims 1 to 4 are implemented.