A highly expressive singing voice synthesis model training method, synthesis method and device

Abstract

This invention provides a training method, synthesis method, and apparatus for a high-expressive singing voice synthesis model. The method involves acquiring lyrics and a musical score sequence predicted from real singing voices; converting the lyrics into phoneme information; extracting and analyzing the physical information of the singing voice from the audio data to obtain intensity information; inputting the phoneme information and the attribute information of the musical score sequence into a text encoder to generate a text embedding sequence; using a duration predictor to determine the duration of each word; inputting the text embedding sequence and duration into a length constraint to generate a speech representation; inputting the speech representation into a fundamental frequency decoder and a Mel decoder to generate a fundamental frequency and a Mel spectrum; adding the fundamental frequency, Mel spectrum, and speech representation to obtain a hybrid embedding vector; inputting the hybrid embedding vector into a Transformer and a continuous speech unit representation module to generate a continuous speech unit vector; and using a vocoder to synthesize and predict the singing voice. The method provided by this invention performs fine-grained modeling of each attribute, avoiding coupling and synthesizing high-quality singing voices.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a highly expressive singing voice synthesis model training method, synthesis method and device. Background Art

[0002] Singing is not only a medium for people to convey information, but also an art form that conveys the thoughts of creators and singers. With the continuous development of artificial intelligence and computer technology, high-expressive singing synthesis technology is an indispensable research topic in artificial intelligence, and it is also one of the hot research fields in computer music.

[0003] The current singing synthesis system can synthesize natural singing by inputting the pitch and duration of lyrics and music scores. However, in the task of high-expressive singing synthesis, four aspects of singing need to be controlled to achieve high-expressive singing generation: melody, intensity, rhythm and timbre. The current high-expressive singing synthesis system lacks modeling and control of singing intensity. If users want to manipulate the above attributes of singing in a fine-grained manner, it is difficult.

[0004] The existing singing synthesis technology solution consists of two parts: acoustic model and vocoder. The acoustic model consists of text encoder, music score encoder, encoder, variable adapter, length constraint and Mel encoder. The text encoder and music score encoder read the lyrics, preliminarily model the pitch and duration, and the encoder further encodes the pitch and duration obtained by the modeling. The variable adapter controls the synthesized singing from four aspects: timbre, emotion, pitch, and energy. The modules corresponding to each attribute are connected through residuals. Subsequently, after passing through the length constraint, the Mel spectrum encoder predicts the Mel spectrum. The vocoder reads the Mel spectrum and synthesizes the final singing audio.

[0005] However, in the existing technical solutions, emotional labels need to be manually labeled, which is very costly. There are individual differences in manually labeled emotional labels and they cannot objectively reflect the singer’s emotions. In addition, the emotional labels of the existing technical solutions are only labeled at the sentence level, and the operation granularity is coarse. It is not possible to achieve fine-grained labeling and control at the word level, lacks control over details, and also causes a mismatch between emotional labels and target emotions during training. Furthermore, the existing technical solutions use emotional labels, lyrics, and music scores to guide singing synthesis. However, changes in the emotional attributes of singing will trigger changes in multiple attributes in the singing voice, which will lead to the coupling of multiple attributes in the synthesis of highly expressive singing voices, lack of control over individual attributes, and poor synthesis effect. Summary of the invention

[0006] In view of this, the embodiments of the present invention provide a highly expressive singing voice synthesis model training method, a synthesis method and a device to eliminate or improve one or more defects existing in the prior art, and solve the problems that the existing singing synthesis technology solutions require manual labeling of emotional labels, have coarse granularity, high cost and cannot objectively reflect emotional bias, lack modeling and control of singing intensity, and changes in singing emotional attributes will lead to the coupling of multiple attributes of highly expressive singing synthesis, lack of control over individual attributes, and poor synthesis effect.

[0007] In one aspect, the present invention provides a highly expressive singing voice synthesis model training method, the method comprising the following steps:

[0008] Obtain the lyrics text to be synthesized and the score sequence predicted from the real singing voice;

[0009] The lyrics text is converted into phoneme information, the physical information of the singing sound is extracted from the audio data of the real singing voice, and the physical information is analyzed to predict the intensity information of each note in the music score sequence, including: calculating the duration of each word in the lyrics text when it is actually sung, calculating the fundamental frequency of each word according to the duration, calculating the harmonic energy of each word according to the fundamental frequency, dividing the harmonic energy by the duration to obtain the time-averaged harmonic energy, pre-processing the time-averaged harmonic energy of each word and mapping it to a preset range to generate corresponding intensity information;

[0010] The phoneme information and the attribute information of the music score sequence are input into the initial singing synthesis model to synthesize the final predicted singing; wherein the initial singing synthesis model includes a text encoder, a duration prediction period, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module and a vocoder; the phoneme information and the attribute information are input into the text encoder to generate a text embedding sequence; the text embedding sequence is input into the duration predictor to determine the duration of each word; the text embedding sequence and the duration of each word are input into the length constraint to generate a speech representation; the speech representation is divided into The baseband decoder and the mel decoder are respectively input to generate the baseband and mel spectrum; the baseband, the mel spectrum and the speech representation are added to obtain a mixed embedding vector; the mixed embedding vector is sequentially input to the Transformer and the continuous speech unit representation module to generate a continuous speech unit vector; the continuous speech unit vector, the baseband and the harmonic energy are input to the signal processing module to generate a primary speech representation; the continuous speech unit vector is input to the vocoder, and the primary speech representation is used as the prior condition of the vocoder to synthesize the final predicted singing voice; the attribute information of the music score sequence includes at least pitch information and duration information;

[0011] The initial singing voice synthesis model is trained according to the lyrics text and the music score sequence until the preset performance requirements are met, thereby obtaining a final singing voice synthesis model.

[0012] In some embodiments of the present invention, lyrics text to be synthesized and a music score sequence predicted from real singing voice are obtained, and the music score sequence is in a Musical Instrument Digital Interface format.

[0013] In some embodiments of the present invention, the harmonic energy of each word is calculated according to its fundamental frequency, including:

[0014] The harmonic energy of a preset number of dimensions is calculated according to the fundamental frequency of each word, and the harmonic energy of all dimensions is added together to obtain the sum of the harmonic energy of each word.

[0015] In some embodiments of the present invention, the time average harmonic energy of each word is pre-processed and mapped to a preset range, including:

[0016] The time-averaged harmonic energy of each word is subjected to minmax normalization processing; and the processed result is mapped to a preset range to obtain the intensity corresponding to each note in the music score sequence.

[0017] In some embodiments of the present invention, after the mixed embedding vector is sequentially input into the Transformer and the continuous speech unit representation module to generate a continuous speech unit vector, the method further includes:

[0018] Input the audio data of the real singing voice into a preset HuBERT model to generate a discrete speech unit vector;

[0019] The continuous speech unit vector is linearly projected, and a cross entropy loss is calculated between the continuous speech unit vector and the discrete speech unit vector to optimize the speech representation quality.

[0020] In some embodiments of the present invention, the continuous speech unit vector, the fundamental frequency and the harmonic energy are input into a signal processing module to generate a primary speech representation, including:

[0021] Synthesizing the periodic part of the target singing voice according to the fundamental frequency and the harmonic energy, and synthesizing the non-periodic part of the target singing voice according to the continuous speech unit vector;

[0022] The periodic part and the non-periodic part of the target song are added together to obtain a primary speech representation of the target song.

[0023] On the other hand, the present invention also provides a method for synthesizing highly expressive singing voices, the method comprising the following steps:

[0024] Obtain the lyrics text to be synthesized and the music score sequence of the actual performance;

[0025] Extracting intensity information, pitch information and duration information from the music score sequence;

[0026] The intensity information, the pitch information and the duration information are input into a singing sound synthesis model trained by any of the highly expressive singing sound synthesis model training methods mentioned above to synthesize predicted singing; the attribute information of the music score sequence includes at least pitch information and duration information.

[0027] On the other hand, the present invention also provides a highly expressive singing sound synthesis device, which, when executed, implements the steps of any one of the highly expressive singing sound synthesis methods mentioned above, and the device comprises:

[0028] The singing voice acquisition module is used to acquire the lyrics text to be synthesized and the music score sequence of the actual performance; and extract the intensity information, pitch information and duration information from the music score sequence;

[0029] The singing voice synthesis module includes a text encoder, a duration prediction period, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module and a vocoder, which is used to automatically synthesize and predict singing according to the intensity information, the pitch information and the duration information.

[0030] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods mentioned above.

[0031] On the other hand, the present invention further provides a computer program product, comprising a computer program / instruction, which implements the steps of any one of the methods mentioned above when executed by a processor.

[0032] The beneficial effects of the present invention are at least:

[0033] The present invention provides a high-expressive singing voice synthesis model training method, synthesis method and device, including: obtaining lyrics text to be synthesized, and a music score sequence predicted by real singing voice; converting the lyrics text into phoneme information, extracting and analyzing the physical information of singing voice from the audio data of real singing voice, and obtaining the intensity information of each note in the music score sequence; inputting the phoneme information and the attribute information of the music score sequence into a text encoder to generate a text embedding sequence, determining the duration of each word via a duration predictor, inputting the text embedding sequence and the duration into a length constraint to generate a speech representation; inputting the speech representation into a baseband decoder and a Mel decoder to generate a baseband and a Mel spectrum; adding the baseband, Mel spectrum and speech representation to obtain a mixed embedding vector, inputting the mixed embedding vector into a Transformer and a continuous speech unit representation module to generate a continuous speech unit vector, and synthesizing the final predicted singing voice by a vocoder. The method provided by the present invention can fully automatically annotate the intensity information of the existing singing voice synthesis data set without increasing labor costs. At the same time, each attribute such as pitch, duration, intensity, etc. is accurately and finely modeled to avoid coupling between the attributes, and synthesize high-quality high-expressive singing voice.

[0034] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0035] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0037] Figure 1 Schematic diagram of the steps of a highly expressive singing voice synthesis method in one embodiment of the present invention.

[0038] Figure 2 4 is a flow chart of a method for synthesizing highly expressive singing voice in one embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0040] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0041] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0042] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0043] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0044] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0045] In order to solve the problems that the existing singing voice synthesis technology solutions require manual emotional labeling, have coarse granularity, high cost and cannot objectively reflect emotional bias, lack modeling and control of singing voice intensity, and the change of singing voice emotional attributes will lead to the coupling of multiple attributes of high-expressive singing voice synthesis, lack of control over individual attributes, and poor synthesis effect, the present invention proposes a high-expressive singing voice synthesis model training method, such as Figure 1 As shown, the method includes the following steps S101 to S104:

[0046] Step S101: obtaining lyrics text to be synthesized and a music score sequence predicted from real singing voice.

[0047] Step S102: converting the lyrics text into phoneme information, extracting the physical information of the singing sound from the audio data of the real singing voice, and analyzing the physical information to predict the intensity information of each note in the score sequence, specifically including: calculating the duration of each word in the lyrics text when it is actually sung, calculating the fundamental frequency of each word according to the duration, calculating the harmonic energy of each word according to the fundamental frequency, dividing the harmonic energy by the duration to obtain the time-averaged harmonic energy, pre-processing the time-averaged harmonic energy of each word and mapping it to a preset range to generate corresponding intensity information.

[0048] Step S103: input the phoneme information and the attribute information of the music score sequence into the initial singing synthesis model to synthesize the predicted singing.

[0049] Among them, the initial singing synthesis model includes a text encoder, a duration prediction period, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module and a vocoder; the phoneme information and attribute information are input into the text encoder to generate a text embedding sequence; the text embedding sequence is input into the duration predictor to determine the duration of each word; the text embedding sequence and the duration of each word are input into the length constraint to generate a speech representation; the speech representation is input into the baseband decoder and the Mel decoder respectively to generate the baseband and Mel spectrum; the baseband, Mel spectrum and speech representation are added to obtain a mixed embedding vector; the mixed embedding vector is input into the Transformer and the continuous speech unit representation module in turn to generate a continuous speech unit vector; the continuous speech unit vector, the baseband and the harmonic energy are input into the signal processing module to generate a primary speech representation; the continuous speech unit vector is input into the vocoder, and the primary speech representation is used as a priori condition for the vocoder to synthesize the final predicted singing.

[0050] Step S104: Train the initial singing voice synthesis model according to the lyrics text and the music score sequence until it reaches the preset performance requirements to obtain the final singing voice synthesis model.

[0051] In step S101, the lyrics text to be synthesized and the music score sequence predicted from the real singing voice are obtained.

[0052] In some embodiments, the music score sequence uses the Musical Instrument Digital Interface (MIDI) format. MIDI is a digital audio format and communication protocol designed to allow different electronic music devices, such as keyboards, synthesizers, computers, etc., to communicate and control each other. MIDI sequences and MIDI music scores refer to different applications and forms based on the MIDI protocol.

[0053] In step S102, the lyrics text is converted into phoneme information, the physical information of the singing voice is extracted from the audio data of the real singing voice, and the physical information is analyzed to predict the intensity information of each note in the score sequence. This solves the problem that the existing high-expressive singing voice synthesis system lacks modeling and control of singing voice intensity and improves the accuracy of the synthesis system.

[0054] Phonemes are the basic units in phonetics. They are the smallest sound segments in speech that can distinguish the meaning of a word. The process of converting lyrics text into phoneme information is called text-to-phoneme (TTP) conversion. This process involves mapping each word or syllable in the text to a corresponding phoneme sequence. Phoneme information is widely used in technical fields such as speech synthesis and automatic speech recognition to ensure correct pronunciation and naturalness of speech synthesis.

[0055] The present invention proposes a method for automatically marking intensity information according to the harmonic energy of singing voice, which specifically includes the following steps S1021-S1024:

[0056] Step S1021: Obtain lyrics text to be synthesized and music score predicted from real singing voice. Exemplarily, the music score is in MIDI format.

[0057] Step S1022: Determine the actual duration of singing each word in the lyrics text, and calculate the fundamental frequency of each word based on the duration.

[0058] In some embodiments, the lyrics text and audio are time-aligned through audio signal processing and text phoneme analysis, and each word in the lyrics text is mapped to a specific time period in the audio using an automatic alignment tool or manual annotation.

[0059] In some embodiments, according to the determined duration of each word, the corresponding audio segment is extracted from the entire audio, and the fundamental frequency of the audio segment can be directly estimated using the fundamental frequency extraction tool provided by the audio processing software, such as the tone analysis function in Praat, or the fundamental frequency can be calculated by drawing a waveform graph or a spectrum graph, observing the periodic vibration. Among them, the fundamental frequency (Fundamental Frequency, referred to as F0) refers to the lowest main frequency component in the sound, also commonly referred to as the pitch. In speech and music, the fundamental frequency determines the tone or pitch of the sound.

[0060] In some embodiments, when the musical score is a MIDI musical score, the pitch information of each note in the MIDI may be used as a reference value of the fundamental frequency.

[0061] Step S1023: Calculate the harmonic energy of each word according to its fundamental frequency, divide the harmonic energy by the start time, and obtain the time average harmonic energy.

[0062] In some embodiments, harmonic energies of a preset number (eg, 32) of dimensions are calculated based on the fundamental frequency of each word, and the harmonic energies of all dimensions are added together to obtain the sum of the harmonic energies of each word.

[0063] Step S1024: Map the time-averaged harmonic energy of each word to a preset range after preprocessing to generate corresponding intensity information.

[0064] In some embodiments, the time-averaged harmonic energy of each word is subjected to minmax normalization processing; the processed result is mapped to 0 to 127 to obtain a final intensity information label.

[0065] In step S103, the phoneme information and the attribute information of the music score sequence are input into a preset singing synthesis model to synthesize the predicted singing.

[0066] In some embodiments, the attribute information of the score sequence includes pitch information and duration information. Meanwhile, the pitch information and duration information are accurate and measurable in the score sequence, and can be understood as true values, and can be directly obtained and used.

[0067] The initial singing synthesis model includes a text encoder, a duration prediction stage, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module, and a vocoder. Figure 2 As shown, it is a schematic flow chart of the entire synthesis method, including the following steps S1031 to S1037.

[0068] Step S1031: Input the phoneme information and attribute information into the text encoder to generate a text embedding sequence.

[0069] In some embodiments, for each phoneme, pitch, duration and intensity information are combined to form a feature vector. For example, pitch, duration and intensity are used as different dimensions of features. A pre-trained text encoder such as BERT, GPT, etc. is used to map the feature vector of each phoneme to a text embedding sequence.

[0070] Step S1032: Input the text embedding sequence into the duration predictor to determine the duration of each word.

[0071] Step S1033: input the text embedding sequence and the duration of each word into the length constraint to generate a frame-level speech representation.

[0072] Step S1034: input the speech representation into the baseband decoder and the mel decoder respectively to generate the baseband and mel spectrum.

[0073] Step S1035: Add the fundamental frequency, the mel spectrum and the speech representation to obtain a mixed embedding vector.

[0074] Step S1036: After the mixed embedding vector is input into the feedforward Transformer structure, it continues to pass through a continuous speech unit representation module to generate a continuous speech unit vector.

[0075] In some embodiments, after generating the continuous speech unit vector, the method further includes:

[0076] The audio data of real singing is input into the preset HuBERT model to generate discrete speech unit vectors; the continuous speech unit vectors are linearly projected and the cross entropy loss is calculated between them and the discrete speech unit vectors to optimize the quality of speech representation.

[0077] Step S1037: Input the continuous speech unit vector, fundamental frequency and harmonic energy into the signal processing module to generate a primary speech representation; input the continuous speech unit vector into the vocoder, use the primary speech representation as a priori condition of the vocoder, and synthesize the final predicted singing voice.

[0078] In some embodiments, the primary speech representation includes a periodic part and a non-periodic part, and the generation of the primary speech representation includes the following steps:

[0079] Synthesizing the periodic part of the target singing voice according to the fundamental frequency and harmonic energy, and synthesizing the non-periodic part of the target singing voice according to the continuous speech unit vector;

[0080] The periodic part and the non-periodic part of the target song are added together to obtain the primary speech representation of the target song.

[0081] The present invention also provides a highly expressive singing voice synthesis method, which comprises the following steps S201-S203:

[0082] Step S201: Obtain lyrics text to be synthesized and a sequence of music scores actually performed.

[0083] Step S202: extracting intensity information, pitch information and duration information from the music score sequence.

[0084] Step S203: Input the intensity information, pitch information and duration information into the singing sound synthesis model trained based on the high-expressiveness singing sound synthesis model training method to synthesize the predicted singing voice, wherein the attribute information of the music score sequence includes at least the pitch information and duration information.

[0085] Corresponding to the highly expressive singing voice synthesis method provided by the present invention, the present invention also provides a highly expressive singing voice synthesis device, which can implement the steps of the highly expressive singing voice synthesis method when executed, and the device includes:

[0086] The singing acquisition module is used to obtain the lyrics text to be synthesized and the actual performance score sequence; and extract the intensity information, pitch information and duration information from the score sequence.

[0087] The singing voice synthesis module includes a text encoder, a duration prediction period, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module and a vocoder, which is used to automatically synthesize and predict singing according to intensity information, pitch information and duration information.

[0088] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a highly expressive singing voice synthesis model training method and / or a highly expressive singing voice synthesis method.

[0089] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0090] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0091] In summary, the present invention provides a highly expressive singing voice synthesis model training method, synthesis method and device, including: obtaining lyrics text to be synthesized, and a music score sequence predicted by real singing voice; converting the lyrics text into phoneme information, extracting and analyzing the physical information of the singing voice from the audio data of the real singing voice, and obtaining the intensity information of each note in the music score sequence; inputting the phoneme information and the attribute information of the music score sequence into a text encoder to generate a text embedding sequence, determining the duration of each word via a duration predictor, inputting the text embedding sequence and the duration into a length constraint to generate a speech representation; inputting the speech representation into a baseband decoder and a Mel decoder to generate a baseband and a Mel spectrum; adding the baseband, Mel spectrum and speech representation to obtain a mixed embedding vector, inputting the mixed embedding vector into a Transformer and a continuous speech unit representation module to generate a continuous speech unit vector, and synthesizing the final predicted singing voice by a vocoder. The method provided by the present invention can fully automatically label the intensity information of the existing singing voice synthesis data set without increasing labor costs. At the same time, precise and fine-grained modeling is performed on various attributes such as pitch, duration, and intensity to avoid coupling between the attributes and synthesize high-quality, expressive singing.

[0092] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0093] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0094] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A highly expressive singing voice synthesis model training method, characterized in that: The method comprises the following steps: Obtain the lyrics text to be synthesized and the MIDI score sequence predicted by the real singing voice; The lyrics text is converted into phoneme information, the physical information of the singing voice is extracted from the audio data of the real singing voice, and the physical information is analyzed to predict the intensity information of each note in the music score sequence, including: calculating the duration of each word in the lyrics text when it is actually sung, calculating the fundamental frequency of each word according to the duration, calculating the harmonic energy of each word according to the fundamental frequency, dividing the harmonic energy by the duration to obtain the time-averaged harmonic energy, pre-processing the time-averaged harmonic energy of each word and mapping it to a preset range to generate corresponding intensity information; The phoneme information and the attribute information of the music score sequence are input into the initial singing synthesis model to synthesize the predicted singing; wherein the initial singing synthesis model includes a text encoder, a duration prediction period, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module and a vocoder; the phoneme information and the attribute information are input into the text encoder to generate a text embedding sequence; the text embedding sequence is input into the duration predictor to determine the duration of each word; the text embedding sequence and the duration of each word are input into the length constraint to generate a speech representation; the speech representation is respectively input into the The baseband decoder and the mel decoder generate a baseband and a mel spectrum; the baseband, the mel spectrum and the speech representation are added to obtain a mixed embedding vector; the mixed embedding vector is sequentially input into the Transformer and the continuous speech unit representation module to generate a continuous speech unit vector; the continuous speech unit vector, the baseband and the harmonic energy are input into the signal processing module to generate a primary speech representation; the continuous speech unit vector is input into the vocoder, and the primary speech representation is used as the prior condition of the vocoder to synthesize and predict the singing voice; the attribute information of the music score sequence includes at least pitch information, duration information and the intensity information; The initial singing voice synthesis model is trained according to the lyrics text and the music score sequence until the preset performance requirements are met, thereby obtaining a final singing voice synthesis model.

2. The highly expressive singing voice synthesis model training method according to claim 1, characterized in that: Calculate the harmonic energy of each word based on its fundamental frequency, including: The harmonic energy of a preset number of dimensions is calculated according to the fundamental frequency of each word, and the harmonic energy of all dimensions is added together to obtain the sum of the harmonic energy of each word.

3. The highly expressive singing voice synthesis model training method according to claim 1, characterized in that: The time-averaged harmonic energy of each word is pre-processed and mapped to a preset range, including: The time-averaged harmonic energy of each word is subjected to minmax normalization processing; and the processed result is mapped to a preset range to obtain the intensity corresponding to each note in the music score sequence.

4. The highly expressive singing voice synthesis model training method according to claim 1, characterized in that: After the mixed embedding vector is sequentially input into the Transformer and the continuous speech unit representation module to generate a continuous speech unit vector, the method further includes: Input the audio data of the real singing voice into a preset HuBERT model to generate a discrete speech unit vector; The continuous speech unit vector is linearly projected, and a cross entropy loss is calculated between the continuous speech unit vector and the discrete speech unit vector to optimize the speech representation quality.

5. The highly expressive singing voice synthesis model training method according to claim 1, characterized in that: Inputting the continuous speech unit vector, the fundamental frequency and the harmonic energy into a signal processing module to generate a primary speech representation, comprising: Synthesizing the periodic part of the target singing voice according to the fundamental frequency and the harmonic energy, and synthesizing the non-periodic part of the target singing voice according to the continuous speech unit vector; The periodic part and the non-periodic part of the target song are added together to obtain a primary speech representation of the target song.

6. A highly expressive singing voice synthesis method, characterized in that: The method comprises the following steps: Obtain lyrics text and MIDI score sequence to be synthesized; Convert the lyrics text into phoneme information; extract intensity information, pitch information and duration information from the music score sequence; The phoneme information, the intensity information, the pitch information and the duration information are input into a singing sound synthesis model trained based on the highly expressive singing sound synthesis model training method according to any one of claims 1 to 5 to synthesize predicted singing.

7. A highly expressive singing voice synthesis device, characterized in that: When the device is executed, the steps of the highly expressive singing voice synthesis method as claimed in claim 6 are implemented, and the device comprises: The singing voice acquisition module is used to acquire the lyrics text to be synthesized and the MIDI music score sequence; convert the lyrics text into phoneme information; and extract intensity information, pitch information and duration information from the music score sequence; The singing voice synthesis module includes a text encoder, a duration prediction period, a length constraint, a baseband decoder, a Mel decoder, a Transformer, a continuous speech unit representation module, a signal processing module and a vocoder, which is used to automatically synthesize and predict singing according to the phoneme information, the intensity information, the pitch information and the duration information.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images