Singing voice synthesis method, device, computer equipment and storage medium
By encoding and feature extraction of song music data, using attention mechanism model and Fourier transform, the problem of the existing singing synthesis system requiring a large amount of training data is solved, the efficiency and applicability of singing synthesis are improved, and the live singing is generated close to real-life singing audio effects.
Patent Information
- Application Number
- CN202110837643.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-07-23
AI Technical Summary
The existing singing vocal synthesis system requires a large amount of training data and can only be synthesized for the same singer, resulting in low applicability and low synthesis efficiency.
By obtaining the music data of the target song for encoding, music encoding data is generated, including lyrics data, vocal score data, tone data and audio data, and using the attention mechanism model to extract timbre and style feature information and input it into the singing synthesis model for sound. Spectral synthesis, combined with Fourier transform to generate target song audio.
The efficiency and applicability of vocal synthesis are improved, so that the synthesized target song audio is close to the auditory effect of live singing, and a small amount of data is input into the trained model.
Smart Images

Figure CN113555001B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a singing synthesis method, device, computer equipment and storage medium. Background Art
[0002] With the development of computer technology, speech synthesis technology is becoming more mature and has been widely used in transportation, education, medical care and social fields. For example, traffic voice navigation, language learning pronunciation, medical voice control, etc., have facilitated people's work and life.
[0003] A further singing voice synthesis system (SVS) is a system that generates natural singing voices based on given musical scores and lyrics information. SVS requires that the duration and pitch of each syllable be controllable.
[0004] The current singing synthesis system uses a deep neural network approach, which uses the target singer's audio data as training data and inputs it into the neural network for training, so as to synthesize audio using the singer's voice.
[0005] However, this method requires a large amount of training data and can only be trained on the same singer to obtain the same singer's singing synthesized audio. This method results in the low applicability of the singing synthesis system and requires a large amount of training data, and the efficiency of synthesizing singing is not high. Summary of the invention
[0006] The invention provides a singing voice synthesis method, device, computer equipment and storage medium, which are used to improve the efficiency of singing voice synthesis.
[0007] A method for singing voice synthesis, comprising:
[0008] Acquire music data of a target song, encode the music data through a preset encoder, and generate music encoding data, wherein the music data includes lyrics data, spectrum data, pitch data, and audio data of a target singer, and the music encoding includes lyrics encoding, spectrum encoding, pitch encoding, and audio encoding;
[0009] Input the lyrics code and audio code into the first attention mechanism model for encoding to obtain timbre feature information;
[0010] Input the audio code, spectral code, and pitch code into the second attention mechanism model for encoding to obtain style feature information;
[0011] Inputting the timbre feature information and the style feature information into a singing voice synthesis model to perform spectrum synthesis to obtain first spectrum data;
[0012] The first sound spectrum data is decoded to obtain first linear spectrum data, and the first linear spectrum data is converted into target song audio through Fourier transform.
[0013] A singing voice synthesis device, comprising:
[0014] A data encoding module, used for acquiring music data of a target song, encoding the music data through a preset encoder, and generating music encoding data, wherein the music data includes lyrics data, spectrum data, pitch data, and audio data of a target singer, and the music encoding includes lyrics encoding, spectrum encoding, pitch encoding, and audio encoding;
[0015] A first feature information generation module is used to input the lyrics code and the audio code into the first attention mechanism model for encoding to obtain timbre feature information;
[0016] The second feature information generation module is used to input the audio code, the spectrum code, and the pitch code into the second attention mechanism model for encoding to obtain style feature information;
[0017] A spectrum data generation module, used for inputting the timbre feature information and the style feature information into the singing voice synthesis model to perform spectrum synthesis and obtain first spectrum data;
[0018] The target audio generation module is used to decode the first sound spectrum data to obtain the first linear spectrum data, and convert the first linear spectrum data into the target song audio through Fourier transform.
[0019] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned singing synthesis method when executing the computer program.
[0020] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned singing synthesis method when executed by a processor.
[0021] The singing voice synthesis method, device, computer equipment and storage medium provided by the present invention encode the music data of the target synthesized song through a preset encoder to generate music encoding data, wherein the music data includes lyrics data, spectrum data, tone data and audio data, and the music data encoding includes lyrics encoding, spectrum encoding, tone encoding and audio encoding; the lyrics encoding and the audio encoding are input into a first attention mechanism model for encoding to obtain timbre feature information, and the audio encoding, spectrum encoding and tone encoding are input into a second attention mechanism for encoding to obtain style feature information; based on the timbre feature information and the style feature information, the timbre feature information and the style feature information are input into a pre-trained singing voice synthesis model, thereby ensuring that the synthesized target song audio carries the timbre feature and the style feature, ensuring that the synthesized target song audio has an auditory effect close to that of a real person singing, and using a small amount of data to input into the trained singing voice synthesis model to improve the efficiency of synthesizing the target song audio. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0023] Figure 1 is a schematic diagram of an application environment of a singing voice synthesis method according to an embodiment of the present invention;
[0024] Figure 2 is a flow chart of a singing voice synthesis method in one embodiment of the present invention;
[0025] Figure 3 is a structural schematic diagram of a singing voice synthesis device in one embodiment of the present invention;
[0026] Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] The singing voice synthesis method provided in this application can be applied in Figure 1In an application environment, a terminal device communicates with a server through a network. The terminal device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers.
[0029] The system framework 100 may include a terminal device, a network and a server. The network is used to provide a medium for a communication link between the terminal device and the server. The network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0030] Users can use terminal devices to interact with the server through the network to receive or send messages, etc.
[0031] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III), MP4 (Moving Picture Eperts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0032] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0033] It should be noted that the singing voice synthesis method provided in the embodiment of the present invention is executed by a server, and accordingly, the singing voice synthesis device is arranged in the server.
[0034] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only illustrative. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices in the embodiment of the present invention may specifically correspond to application systems in actual production.
[0035] In one embodiment, if Figure 2 As shown, a singing voice synthesis method is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate, including the following steps S101 to S105:
[0036] S101, obtain music data of a target song, encode the music data through a preset encoder, and generate music encoding data, wherein the music data includes lyrics data, spectrum data, pitch data and audio data of a target singer, and the music encoding includes lyrics encoding, spectrum encoding, pitch encoding and audio encoding.
[0037] The target song refers to a synthesized song generated by the embodiment of the present invention, that is, the audio of a target singer singing a certain song is generated. For example, the audio of Jacky Cheung singing "Red Sun" is generated.
[0038] The music data of the target song refers to the lyrics data, the spectrum data, the pitch data of the target song, and the audio data of the target singer.
[0039] The lyrics data refers to the lyrics text of the song to be synthesized, and the spectrogram data refers to the spectrum data of the song data, which is specifically the mel spectrum of the song in this embodiment. The pitch data refers to the melody data of the song, corresponding to the pitch data of the song. The audio data of the target singer refers to the audio data of the target singer when he speaks or sings, and the purpose is to extract the voiceprint features of the target singer through the audio data.
[0040] Furthermore, in this embodiment, audio data of a target singer singing a song is selected to extract the singer's vocal characteristics, thereby simulating the singer's singing voice or singing style.
[0041] The preset encoder extracts features of the music data and forms a feature vector, wherein the preset encoder is composed of a bidirectional threshold cyclic unit, and the feature vector is encoded by the bidirectional threshold cyclic unit, the forward output of the bidirectional threshold cyclic unit is a hidden coding vector, and the backward output is a hidden coding vector, the hidden coding vectors of the forward and backward outputs are connected, and the hidden code vector of the music data is output, and the hidden code vector is used as the music data encoding, wherein the music data encoding includes lyrics encoding, spectrum encoding, pitch encoding and audio encoding.
[0042] As a preferred method, different encoders are used to encode the lyrics data, spectrum data, pitch data and audio data in the music data, extract the characteristic information of each music data, and generate a hidden code vector for each music data.
[0043] S102, input the lyrics code and audio code into the first attention mechanism model for encoding to obtain timbre feature information.
[0044] The lyrics data is input into the text encoder to generate a lyrics feature vector. The audio data is input into the singer identity encoder to generate an audio feature vector. The lyrics feature vector and the audio feature vector are input into the first attention mechanism model for encoding to generate a timbre feature code, which is used as timbre feature information.
[0045] Among them, the first attention mechanism model calculates the lyrics feature vector and the audio feature vector in each attention unit through the attention matrix calculation formula, generates the timbre feature matrix of each attention unit, and concatenates the timbre feature matrix of each attention unit to obtain the timbre feature code as the timbre feature information.
[0046] The timbre feature information specifically refers to data having the timbre features of a target singer and the lyrics features of a target synthesized song.
[0047] S103, input the audio code, the spectral code, and the pitch code into the second attention mechanism model for encoding to obtain style feature information.
[0048] The spectrum data is encoded by a Mel-spectrogram encoder to generate a Mel-spectrogram feature vector; the pitch data is encoded by a pitch encoder to generate a pitch feature vector.
[0049] The mel spectrum feature vector, pitch feature vector and audio feature vector are input into the second attention mechanism model for encoding to generate style feature encoding as style feature information.
[0050] Among them, the second attention mechanism model calculates the Mel-spectrogram feature vector, pitch feature vector and audio feature vector in each attention unit through the attention matrix calculation formula, generates the style feature matrix of each attention unit, and concatenates the style feature matrices of each attention unit to obtain the style feature encoding as the style feature information.
[0051] The style feature information specifically represents the expression form characteristics of a song, such as the pitch of the high and low notes of a song.
[0052] S104, input the timbre feature information and the style feature information into a singing synthesis model to perform spectrum synthesis to obtain first spectrum data.
[0053] Among them, a deep neural network for singing synthesis is selected as the initial singing synthesis model. Optionally, the deep neural network includes but is not limited to a WaveNet model, a Deep Voice model, etc., wherein the WaveNet model is an autoregressive deep neural network model that integrates a dilated convolutional neural network (Dilated CNN), a residual network, a gate structure, etc., and can be used for speech generation modeling. In the acoustic model modeling of speech synthesis, the WaveNet model can directly learn the mapping of the sampling value sequence, so it has a good synthesis effect.
[0054] Preferably, the present embodiment adopts the WaveNet model, and directly collects the mapping to the speech sequence based on the WaveNET model, so that the synthesized target song audio is closer to a real person.
[0055] The music data is used to train the initial singing synthesis model to obtain the singing synthesis model.
[0056] The music feature information and the style feature information are input into the singing synthesis model to obtain the first spectrum data.
[0057] S105, decoding the first sound spectrum data to obtain first linear spectrum data, and converting the first linear spectrum data into target song audio by Fourier transform.
[0058] A decoder is used to decode the first spectrum data to obtain first linear spectrum data, and then the first linear spectrum data is converted into target song audio through Fourier transform.
[0059] The singing voice synthesis method, device, computer equipment and storage medium provided by the embodiments of the present invention encode the music data of the target synthesized song through a preset encoder to generate music encoding data, wherein the music data includes lyrics data, spectrum data, tone data and audio data, and the music data encoding includes lyrics encoding, spectrum encoding, tone encoding and audio encoding; the lyrics encoding and audio encoding are input into the first attention mechanism model for encoding to obtain timbre feature information, and the audio encoding, spectrum encoding and tone encoding are input into the second attention mechanism for encoding to obtain style feature information; based on the timbre feature information and the style feature information, the first spectrum data is generated, the first spectrum data is decoded and the spectrum is transformed to obtain the target song audio, which simplifies the process of training the singing voice synthesis network and improves the efficiency of singing voice synthesis.
[0060] In some optional implementations of this embodiment, in step S101, the music data is encoded by a preset encoder, and the steps of generating music encoded data include the following:
[0061] S1011, obtaining the phoneme features of the lyrics data, and generating the lyrics code through the text encoder.
[0062] Among them, phoneme is the smallest speech unit divided from the perspective of timbre, such as the initial consonants and finals in Chinese.
[0063] The lyrics data refers to the lyrics part of the song, which is specifically expressed as the lyrics text.
[0064] For example, the current lyrics text includes "lyrics", where the pinyin of "song" is "ge" and the pinyin of "ci" is "ci". It can be concluded that the phoneme features of "song" are "g" and "e", and the phoneme features of "ci" are "c" and "i". The phoneme features of the lyrics are extracted to generate a phoneme sequence. The phoneme sequence is encoded by the text encoder to generate the lyrics code.
[0065] In this embodiment, the text encoder may be an encoder having a residual network structure, which is easy to optimize and can improve accuracy by increasing considerable depth.
[0066] S1012, obtaining the voiceprint features of the audio data, and generating an audio code of the audio data through the singer identity encoder.
[0067] The audio data can be identified by a pre-trained voiceprint recognition model to obtain a voiceprint feature vector in the audio data. In this embodiment, the singer identity encoder can be a combination of a 3-layer convolutional layer and a bidirectional GRU network, and the audio encoding is obtained by encoding the voiceprint feature vector, and the audio encoding includes feature information related to the timbre.
[0068] S1013, obtaining fundamental frequency characteristics of the spectrum data, and generating spectrum encoding of the spectrum data through the Mel-spectrogram encoder.
[0069] The fundamental frequency feature corresponds to the spectrum parameter of the target song, wherein each spectrum parameter corresponds to each lyric to form a spectrum sequence.
[0070] In this embodiment, the spectrum parameter may be a Mel spectrum and / or a linear spectrum, and the Mel spectrum may be transformed to generate a linear spectrum. The Mel spectrum encoder is used to generate the spectrum code.
[0071] S1014, acquiring the pitch feature of the tone data, and generating a tone code of the tone data through the tone encoder.
[0072] Among them, the pitch feature refers to the vibration frequency of the vocal cords when singing, and the pitch f can be converted according to the following formula to obtain the MIDI (Musical Instrument Digital Interface) feature p:
[0073]
[0074] Among them, 69 is the code of a middle note A in the MIDI protocol. The complete MIDI scale has 128 notes, represented by numbers 0 to 127. A complete octave contains 12 semitones. The ratio of the wavelength (period) between two adjacent semitones is the same, and the frequency between each octave is exactly doubled. For example, the frequency of middle note A (La) is 440Hz (also known as the international standard tone), the frequency of bass A is 220Hz, and the frequency of treble A is 880Hz.
[0075] Taking the middle tone as the benchmark, based on the above content, we can get the frequency corresponding to each tone.
[0076]
[0077] In this embodiment, the lyrics data, audio data, spectrum data and pitch code are encoded to generate lyrics code, audio code, spectrum code and pitch code, and the lyrics code and audio code are processed to generate timbre feature information, which reflects the characteristics of the singer of the song, that is, it means that the song is sung by a person with specific timbre characteristics; the audio code, spectrum code and pitch code are processed to generate style feature information, which reflects the singing style of the song, and by generating singer characteristics and singing style characteristics, the synthesized song audio for specific needs is realized, thereby expanding the applicability of singing synthesis.
[0078] In some optional implementations of this embodiment, in step S102, the lyrics code and the audio code are input into the first attention mechanism model for encoding, and the step of obtaining the timbre feature information includes:
[0079] S1021, through the first attention mechanism model, the lyrics encoding and audio encoding are processed to obtain the resonance peak mask vector.
[0080] Among them, the lyrics encoding and audio encoding are fused through the first attention mechanism to generate the pronunciation features of each lyric, generate the pronunciation features of all lyrics, and generate the resonance peak mask vector.
[0081] Resonance peaks refer to areas where energy is relatively concentrated in the sound spectrum. Resonance peaks appear as horizontal black bands on the spectrum graph and as amplitude peaks of specific frequencies on the linear spectrum. Resonance peak masks reflect the sound quality and timbre of the sound-emitting object. Resonance peak mask parameters include resonance peak frequency, bandwidth, and amplitude.
[0082] The formant mask parameters are extracted as features to generate a formant mask vector.
[0083] S1022, decoding the formant mask vector through a formant mask decoder to obtain timbre feature information.
[0084] The formant mask decoder is used to parse out the corresponding formant coefficient according to the corresponding spectrum feature vector, and reconstruct the formant to obtain a spectrum including timbre feature information.
[0085] The resonance peak mask reflects the voice characteristics of the target singer, namely the timbre characteristics and the processing characteristics of the lyrics part, so that the synthesized audio has personal characteristics and a very rich auditory enjoyment.
[0086] The timbre characteristic information specifically includes a feature vector for representing the timbre characteristics, which is specifically expressed as the waveform of the audio signal. Taking musical instruments as an example, the audio output by musical instruments with different timbres has different waveforms.
[0087] In this embodiment, the phoneme features and timbre features of the lyrics are fused to obtain a resonance peak mask vector, the resonance peak mask vector is decoded to generate timbre feature information, and the timbre features and lyrics are combined to perform audio synthesis according to the target timbre, so that the synthesized audio is more in line with the real human voice and has a multi-level auditory enjoyment, which makes the audio synthesis effect better.
[0088] In some optional implementations of this embodiment, in step S103, the step of inputting the audio code, the spectrum code, and the pitch code into the second attention mechanism model for encoding to obtain the style feature information includes:
[0089] S1031, through the second attention mechanism model, the audio coding, the spectral coding, and the pitch coding are processed to generate a pitch feature vector.
[0090] S1032: Decode the tone feature vector using a tone decoder to obtain style feature information.
[0091] Among them, the style feature information refers to the characteristics of the singing style of the song. The singing style refers to the pronunciation method and pronunciation skills of the singer when singing. The singing style can be associated with the market of phonemes and / or the fundamental frequency corresponding to the phonemes. Different singers may have different pronunciation time habits of initials or finals when singing, resulting in different pronunciation methods. If the singer uses pronunciation techniques such as vibrato, the corresponding fundamental frequency will reflect the corresponding characteristics.
[0092] Through the second attention mechanism, the audio code, the spectrum code and the pitch code are processed to produce a pitch feature vector, where the pitch feature vector includes the corresponding singing style features. Specifically, it can be understood as the frequency and amplitude corresponding to the linear spectrum at a specific pitch.
[0093] Among them, the pitch decoder is a Mel-spectrogram decoder. In this embodiment, the Mel-spectrogram decoder can be an autoregressive recursive neural network, which predicts a Mel-spectrogram from the pitch feature vector one frame at a time, and finally splices the Mel-spectrograms in the time sequence to generate style feature information.
[0094] In this embodiment, the style feature information reflects the singer's processing method when singing, so that the synthesized drink retains the singer's emotions when singing, making the synthesized audio more realistic.
[0095] In some optional implementations of this embodiment, before step S104, the following steps are also included:
[0096] a. Obtain sample data, the sample data includes sample lyrics encoding, sample spectrum encoding, sample pitch encoding and sample audio encoding.
[0097] Optionally, a Mel-spectrogram sequence is extracted from the sample data as a sample spectrogram code, which is used as acoustic feature information of the sample data; a manual labeling method is used to label the lyrics information with phoneme features to obtain a sample lyrics code, and the pitch is encoded to obtain a sample audio code; and resonance peak features of the sample data are collected as a sample audio code.
[0098] b. Input the sample data into the neural network model for training, and obtain the loss parameters based on the training results.
[0099] c. Based on the loss parameters, adjust the parameters of the neural network model to obtain the trained singing synthesis model.
[0100] The sample data is input into the neural network model for training, the loss parameter is determined according to the training result of the neural network model, the loss parameter is input into the neural network model for back propagation training, and the trained singing synthesis model is obtained.
[0101] In this embodiment, by acquiring sample data, extracting timbre features and singing style from the sample data, and training a singing synthesis model, the singing synthesis model can synthesize target song audio based on the music data, so that the target song audio carries timbre features and singing style features, and the synthesized target song audio has an auditory enjoyment close to that of real person singing, thereby improving the synthesis effect of the target song audio.
[0102] In some optional implementations of this embodiment, in step S104, the step of inputting the timbre feature information and the style feature information into the singing synthesis model for sound spectrum synthesis to obtain the first sound spectrum data includes:
[0103] S1041, using a time value encoder to encode the time value feature to obtain a time value feature vector.
[0104] S1042, input the time feature vector, timbre feature information and style feature information into a singing synthesis model, and generate first spectrum data.
[0105] Among them, duration refers to the duration of a note, which refers to the length of a note that should be sung as indicated on the score. The duration can be converted into duration features according to the following formula:
[0106]
[0107] For example, a score indicates that there are 234 beats in one minute, that is, the length of one beat is 60s / 234=256ms. The note sequence of the score is one beat (256ms) and half a beat (128ms). The corresponding time value features are 256 / 1500=0.1707, 128 / 1500=0.0853.
[0108] The time value feature is encoded by a time value encoder to obtain a time value feature vector.
[0109] The time feature vector, timbre feature information and style feature information are input into the singing synthesis model for encoding to generate the first spectrum data.
[0110] As an optional method, the duration of all notes in the phrase and the duration of each factor in the lyrics encoding are obtained, so that the sum of the pronunciation durations of the phoneme sequences of the phrase is equal to the sum of the durations of all notes in the phrase.
[0111] In some optional implementations of this embodiment, in step S105, the step of decoding the first spectrum data to obtain the first linear spectrum data, and converting the first linear spectrum data into the target song audio by Fourier transform includes:
[0112] S1051, decoding the first sound spectrum data by using a formant mask decoding method to obtain first linear spectrum data.
[0113] S1052: judge the first linear spectrum based on the linear spectrum discriminator to obtain a judgment result.
[0114] S1053: If the judgment result is that the first linear spectrum is linear spectrum data, the first linear spectrum is converted into the target song audio through Fourier transform.
[0115] The first spectrum data is expressed in the form of Mel spectrum, and the corresponding target song audio can be generated by converting the Mel spectrum.
[0116] The Mel spectrum is converted into a linear spectrum, and the input linear spectrum is judged by a linear spectrum discriminator, where the linear spectrum discriminator is a binary structure that determines whether the input data is a linear spectrum, and ensures the processing effect in the subsequent processing process, so that the target song audio effect is better.
[0117] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0118] In one embodiment, a singing voice synthesis device is provided, and the singing voice synthesis device corresponds to the singing voice synthesis method in the above embodiment. Figure 3 As shown, the singing voice synthesis device includes the following modules:
[0119] The data encoding module 31 is used to obtain the music data of the target song, encode the music data through a preset encoder, and generate music encoding data, wherein the music data includes lyrics data, spectrum data, pitch data and audio data of the target singer, and the music encoding includes lyrics encoding, spectrum encoding, pitch encoding and audio encoding.
[0120] The first feature information generating module 32 is used to input the lyrics code and the audio code into the first attention mechanism model for encoding to obtain timbre feature information.
[0121] The second feature information generating module 33 is used to input the audio code, the spectrum code and the pitch code into the second attention mechanism model for encoding to obtain style feature information.
[0122] The spectrum data generating module 34 is used to input the timbre feature information and the style feature information into the singing synthesis model to perform spectrum synthesis to obtain the first spectrum data.
[0123] The target audio generation module 35 is used to decode the first sound spectrum data to obtain the first linear spectrum data, and convert the first linear spectrum data into the target song audio through Fourier transformation.
[0124] In some optional implementations of this embodiment, the data encoding module 31 includes the following units:
[0125] The lyrics encoding unit is used to obtain the phoneme features of the lyrics data and generate the lyrics code through the text encoder.
[0126] The audio encoding unit is used to obtain the voiceprint features of the audio data and generate the audio code of the audio data through the singer identity encoder.
[0127] The tone encoding unit is used to obtain the fundamental frequency characteristics of the spectrum data and generate the tone encoding of the spectrum data through the Mel spectrum encoder.
[0128] The spectral encoding unit is used to obtain the pitch feature of the tone data and generate the spectral encoding of the tone data through the tone encoder.
[0129] In some optional implementations of this embodiment, the first feature information generating module 32 includes the following units:
[0130] The formant mask vector is used to process the lyrics encoding, audio encoding and time value feature vector through the first attention mechanism model to obtain the formant mask vector.
[0131] The timbre characteristic information generating unit is used to decode the formant mask vector through the formant mask decoder to obtain the timbre characteristic information.
[0132] In some optional implementations of this embodiment, the second feature information generating module 33 includes the following units:
[0133] The tone feature vector generating unit is used to process the audio coding, the spectral coding, the tone coding and the time value feature vector through the second attention mechanism model to generate the tone feature vector.
[0134] The style feature information generating unit is used to decode the tone feature vector through a tone decoder to obtain the style feature information.
[0135] In some optional implementations of this embodiment, the spectrum data generating module 34 includes the following units:
[0136] A time value feature vector generating unit, used for encoding the time value feature by using a time value encoder to obtain a time value feature vector;
[0137] The first spectrum data generating unit is used to input the time value feature vector, timbre feature information and style feature information into the singing synthesis model and generate the first spectrum data.
[0138] In some optional implementations of this embodiment, the target audio generation module 35 includes the following units:
[0139] The first linear spectrum generating unit is used to decode the first sound spectrum data by using a formant mask decoding method to obtain the first linear spectrum data.
[0140] The judging unit is used to judge the first linear spectrum based on the linear spectrum discriminator to obtain a judgment result.
[0141] The target song audio generation unit is used to convert the first linear spectrum into the target song audio by Fourier transform if the judgment result is that the first linear spectrum is linear spectrum data.
[0142] In some optional implementations of this embodiment, the singing voice synthesis device includes the following modules:
[0143] The sample data acquisition module is used to acquire sample data, wherein the sample data includes sample lyrics coding, sample spectrum coding, sample pitch coding and sample audio coding.
[0144] The loss parameter acquisition module is used to input sample data into the neural network model for training and obtain the loss parameters based on the training results.
[0145] The singing synthesis model acquisition module is used to adjust the parameters of the neural network model based on the loss parameters to obtain the trained singing synthesis model.
[0146] The meaning of "first" and "second" in the above modules / units is only to distinguish different modules / units, and is not used to define which module / unit has a higher priority or other limiting meanings. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in this application is only a logical division, and there may be other division methods when implemented in actual applications.
[0147] The specific definition of the singing voice synthesis device can be found in the definition of the singing voice synthesis method above, which will not be repeated here. The various modules in the above-mentioned singing voice synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0148] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the singing synthesis method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a singing synthesis method is implemented.
[0149] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the singing synthesis method in the above embodiment are implemented, for example Figure 2 Alternatively, when the processor executes the computer program, the functions of each module / unit of the singing voice synthesis device in the above embodiment are realized, for example, Figure 3 The functions of modules 31 to 35 are shown in Figure 3. To avoid repetition, they will not be described here.
[0150] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and various interfaces and lines are used to connect various parts of the entire computer device.
[0151] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.), etc.
[0152] The memory may be integrated into the processor or may be arranged separately from the processor.
[0153] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the steps of the singing synthesis method in the above embodiment are implemented, for example Figure 2 Alternatively, when the computer program is executed by the processor, the functions of each module / unit of the singing voice synthesis device in the above embodiment are realized, for example, Figure 3 The functions of modules 31 to 35 are shown in Figure 3. To avoid repetition, they will not be described here.
[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0155] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0156] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A singing voice synthesis method, characterized in that: include: Acquire music data of a target song, encode the music data by a preset encoder, and generate music encoding data, wherein the music data includes lyrics data, spectrum data, pitch data, and audio data of a target singer, and the music encoding includes lyrics encoding, spectrum encoding, pitch encoding, and audio encoding; Inputting the lyrics code and the audio code into the first attention mechanism model for encoding to obtain timbre feature information; Inputting the audio code, the spectral code, and the pitch code into the second attention mechanism model for encoding to obtain style feature information; Inputting the timbre characteristic information and the style characteristic information into a singing voice synthesis model to perform sound spectrum synthesis to obtain first sound spectrum data; The step of inputting the timbre characteristic information and the style characteristic information into the singing voice synthesis model to perform sound spectrum synthesis to obtain the first sound spectrum data comprises: A time value encoder is used to encode the time value feature to obtain a time value feature vector; Inputting the duration feature vector, the timbre feature information and the style feature information into the singing synthesis model, and generating the first spectrum data; The theoretical length of the musical score notes includes: obtaining the duration of all notes in the phrase and the duration of each factor in the lyrics encoding, so that the sum of the pronunciation duration of the phoneme sequence of the phrase is equal to the sum of the duration of all notes in the phrase; the unit of the theoretical length of the musical score notes is milliseconds; The first spectrum data is decoded to obtain first linear spectrum data, and the first linear spectrum data is converted into target song audio through Fourier transform.
2. The singing voice synthesis method according to claim 1, characterized in that: The step of encoding the music data by a preset encoder to generate music encoded data also includes: Acquire the phoneme features of the lyrics data, and generate the lyrics code through a text encoder; Acquire the voiceprint features of the audio data, and generate an audio code of the audio data through a singer identity encoder; Acquire the fundamental frequency characteristics of the spectrum data, and generate the spectrum encoding of the spectrum data through a Mel-spectrogram encoder; The pitch feature of the tone data is acquired, and a tone code of the tone data is generated through a tone encoder.
3. The singing voice synthesis method according to claim 1, characterized in that: The step of inputting the lyrics code and the audio code into the first attention mechanism model for encoding to obtain the timbre feature information includes: The lyrics encoding and the audio encoding are processed by the first attention mechanism model to obtain a formant mask vector; The formant mask vector is decoded by a formant mask decoder to obtain the timbre feature information.
4. The singing voice synthesis method according to claim 1, characterized in that: The step of inputting the audio code, the spectrum code, and the pitch code into the second attention mechanism model for encoding to obtain style feature information comprises: Processing the audio code, the spectral code, and the pitch code through the second attention mechanism model to generate a pitch feature vector; The tone feature vector is decoded by a tone decoder to obtain the style feature information.
5. The singing voice synthesis method according to claim 1, characterized in that: Before the step of inputting the timbre characteristic information and the style characteristic information into a singing voice synthesis model to perform spectrum synthesis to obtain first spectrum data, the method includes: Acquire sample data, wherein the sample data includes sample lyrics code, sample spectrum code, sample pitch code and sample audio code; Inputting the sample data into a neural network model for training, and obtaining a loss parameter according to the training result; Based on the loss parameters, the parameters of the neural network model are adjusted to obtain a trained singing synthesis model.
6. The singing voice synthesis method according to claim 1, characterized in that: The step of decoding the first spectrum data to obtain first linear spectrum data, and converting the first linear spectrum data into target song audio by Fourier transform comprises: Decoding the first sound spectrum data by using a formant mask decoding method to obtain first linear spectrum data; Judging the first linear spectrum based on a linear spectrum discriminator to obtain a judgment result; If the judgment result is that the first linear spectrum is linear spectrum data, the first linear spectrum is converted into the target song audio through Fourier transform.
7. A singing voice synthesis device, characterized in that: include: A data encoding module, used for acquiring music data of a target song, encoding the music data through a preset encoder to generate music encoding data, wherein the music data includes lyrics data, spectrum data, pitch data and audio data of a target singer, and the music encoding includes lyrics encoding, spectrum encoding, pitch encoding and audio encoding; A first feature information generating module is used to input the lyrics code and the audio code into the first attention mechanism model for encoding to obtain timbre feature information; A second feature information generating module is used to input the audio code, the spectrum code, and the pitch code into a second attention mechanism model for encoding to obtain style feature information; A spectrum data generation module, used for inputting the timbre characteristic information and the style characteristic information into a singing synthesis model to perform spectrum synthesis to obtain first spectrum data; The spectrum data generating module is further used for: A time value encoder is used to encode the time value feature to obtain a time value feature vector; Inputting the duration feature vector, the timbre feature information and the style feature information into the singing synthesis model, and generating the first spectrum data; The theoretical length of the musical score notes includes: obtaining the duration of all notes in the phrase and the duration of each factor in the lyrics encoding, so that the sum of the pronunciation duration of the phoneme sequence of the phrase is equal to the sum of the duration of all notes in the phrase; the unit of the theoretical length of the musical score notes is milliseconds; The target audio generation module is used to decode the first sound spectrum data to obtain first linear spectrum data, and convert the first linear spectrum data into target song audio through Fourier transform.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the singing voice synthesis method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the singing voice synthesis method according to any one of claims 1 to 6 are implemented.
