Acoustic model training method and device, medium and song synthesis method

Abstract

The application discloses an acoustic model training method and device, a medium and a singing voice synthesis method. The method comprises the following steps: using an encoder to encode spliced features corresponding to a phoneme number sequence and a pitch number sequence, and adding a singer variable and an emotion variable to obtain an encoded hidden vector; using a pitch prediction module to predict phoneme pitch according to the pitch number sequence and the encoded hidden vector, using a duration prediction module to predict phoneme duration according to a pitch duration number sequence and the encoded hidden vector, and using a phoneme energy prediction module to predict phoneme energy according to the phoneme number sequence to obtain decoder input; the prediction mode is residual prediction; using a decoder to reconstruct the decoder input into a corresponding predicted mel spectrum, and training each module of a to-be-trained acoustic model according to the predicted mel spectrum and a real mel spectrum. The application can make the prediction result of the trained acoustic model more accurate.

Description

Technical Field

[0001] This invention relates to the field of vocal synthesis technology, and in particular to an acoustic model training method, device, medium, and vocal synthesis method. Background Technology

[0002] Singing voice synthesis is a technology that generates singing voices based on musical scores, and it's a crucial step in creating virtual humans that can speak and sing. Compared to speech synthesis, singing voice synthesis requires generating correct pronunciation based on the musical score, while also adhering to the score's notation (such as pitch and rhythm). Furthermore, the range of acoustic features, such as fundamental frequency, that vary more widely, and the presence of singing techniques like vibrato, makes modeling singing voices significantly more challenging. Acoustic modeling is a critical part of singing voice synthesis; it primarily generates acoustic features based on information output from the text front-end. With the introduction of deep learning technology, acoustic modeling has made tremendous strides, resulting in increasingly realistic and expressive singing voices. However, it's still some distance from truly rivaling human singing.

[0003] Therefore, the aforementioned technical problems urgently need to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an acoustic model training method, device, medium, and singing voice synthesis method, which can make the prediction results of the trained acoustic model more accurate. The specific solution is as follows:

[0005] The first aspect of this application provides an acoustic model training method, comprising:

[0006] Obtain the score to be trained and the real Mel spectrum of the score to be trained, and preprocess the score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence and pitch duration digital sequence;

[0007] The real Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence are input into the acoustic model to be trained.

[0008] The encoder is used to encode the spliced ​​features corresponding to the phoneme digital sequence and the pitch digital sequence, and singer variables and emotion variables are added to obtain the encoded latent vector;

[0009] The phoneme pitch is predicted using a pitch prediction module based on the pitch digital sequence and the encoded latent vector; the phoneme duration is predicted using a duration prediction module based on the pitch duration digital sequence and the encoded latent vector; and the phoneme energy is predicted using a phoneme energy prediction module based on the phoneme digital sequence, resulting in a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information. The pitch prediction module and the duration prediction module use residual prediction for their prediction methods.

[0010] The decoder input is reconstructed into the corresponding predicted Mel spectrum using the decoder, and each module of the acoustic model to be trained is trained based on the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0011] Optionally, the preprocessing of the musical score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence, and pitch duration digital sequence includes:

[0012] The musical score to be trained is analyzed to obtain the corresponding phoneme sequence, pitch sequence, and pitch duration sequence;

[0013] The phoneme sequence is converted into a phoneme number sequence according to the phoneme set index, the pitch sequence is converted into a pitch number sequence according to the pitch set index, and the pitch duration sequence is converted into a pitch duration number sequence according to the pitch duration set index.

[0014] Optionally, before encoding the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence using the encoder, the method further includes:

[0015] The corresponding phoneme embedding features are extracted based on the phoneme digit sequence, and the corresponding pitch embedding features are extracted based on the pitch digit sequence.

[0016] The concatenated feature is obtained by concatenating the phoneme embedding feature and the pitch embedding feature, and then input to the encoder through a linear layer.

[0017] Optionally, the pitch prediction module includes a pitch prediction network and a pitch residual coding network;

[0018] The step of using a pitch prediction module to predict phoneme pitch based on the pitch number sequence and the encoded latent vector includes:

[0019] The pitch digital sequence and the encoded latent vector are input into the pitch prediction network, and the output of the pitch prediction network is input into the pitch residual coding network to obtain the corresponding phoneme pitch information.

[0020] Optionally, the duration prediction module includes a pitch duration feature extraction network and a pitch duration residual coding network;

[0021] The duration prediction module predicts phoneme duration based on the pitch duration digital sequence and the encoded latent vector, including:

[0022] The pitch duration digital sequence and the encoded latent vector are input into the pitch duration feature extraction network, and the output of the pitch duration feature extraction network is input into the pitch duration residual coding network to obtain the corresponding phoneme duration information.

[0023] Optionally, the phoneme energy prediction module includes an energy prediction network and an energy encoding network;

[0024] The phoneme energy prediction module predicts phoneme energy based on the phoneme digital sequence, including:

[0025] The phoneme digital sequence is input into the energy prediction network, and the output of the energy prediction network is input into the energy encoding network to obtain the corresponding phoneme energy information.

[0026] A second aspect of this application provides a method for synthesizing singing voices, based on the aforementioned trained acoustic model, including:

[0027] The target musical score is obtained and the target singer information and target emotional information to be synthesized are determined. The target musical score is preprocessed to obtain the corresponding target phoneme digital sequence, target pitch digital sequence and target pitch duration digital sequence.

[0028] The target phoneme digital sequence, the target pitch digital sequence, the target pitch duration digital sequence, the target singer information, and the target emotion information are input into the trained acoustic model to output the corresponding target Mel spectrum using the trained acoustic model;

[0029] The target Mel spectrum is input to a vocoder to output a synthesized song corresponding to the target Mel spectrum.

[0030] Optionally, the training process of the vocoder includes:

[0031] By employing a singer conditional discriminator and conducting conditional adversarial training, a vocoder capable of extracting and reconstructing the singer's identity from the Mel spectrum was obtained.

[0032] A third aspect of this application provides an acoustic model training apparatus, comprising:

[0033] The acquisition module is used to acquire the score to be trained and the real Mel spectrum of the score to be trained, and to preprocess the score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence and pitch duration digital sequence.

[0034] The input module is used to input the real Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence into the acoustic model to be trained;

[0035] The encoding module is used to encode the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence using the encoder and add singer variables and emotion variables to obtain the encoded latent vector;

[0036] The prediction module is used to predict the phoneme pitch using the pitch prediction module based on the pitch digital sequence and the encoded latent vector, predict the phoneme duration using the duration prediction module based on the pitch duration digital sequence and the encoded latent vector, and predict the phoneme energy using the phoneme energy prediction module based on the phoneme digital sequence, thereby obtaining a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information; wherein, the prediction method of the pitch prediction module and the duration prediction module is residual prediction;

[0037] The decoding module is used to reconstruct the decoder input into the corresponding predicted Mel spectrum using the decoder, and to train each module of the acoustic model to be trained based on the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0038] A fourth aspect of this application provides an electronic device comprising a processor and a memory; wherein the memory is used to store a computer program, the computer program being loaded and executed by the processor to implement the aforementioned acoustic model training method and singing voice synthesis method.

[0039] The fifth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned acoustic model training method and singing voice synthesis method.

[0040] In this application, the following steps are first taken: First, the musical score to be trained and its actual Mel spectrum are obtained. Then, the musical score is preprocessed to obtain corresponding phoneme digital sequences, pitch digital sequences, and pitch duration digital sequences. Next, the actual Mel spectrum, phoneme digital sequences, pitch digital sequences, and pitch duration sequences are input into the acoustic model to be trained. Then, an encoder is used to encode the splicing features corresponding to the phoneme digital sequences and pitch digital sequences, and singer variables and emotion variables are added to obtain encoded latent vectors. Based on this, a pitch prediction module is used to predict the pitch based on the pitch digital sequences and the encoded latent vectors. The process involves predicting phoneme pitch, using a duration prediction module to predict phoneme duration based on the pitch-duration digital sequence and the encoded latent vector, and using a phoneme energy prediction module to predict phoneme energy based on the phoneme digital sequence, resulting in a decoder input containing phoneme energy, pitch, and duration information. The pitch and duration prediction modules employ residual prediction. Finally, the decoder reconstructs the decoder input into a corresponding predicted Mel spectrum, and trains each module of the acoustic model based on the predicted and actual Mel spectra to obtain the trained acoustic model. This application reflects the singer's characteristics and emotional intensity by adding variances of energy, pitch, and pitch duration based on singer and emotion variables. Furthermore, it trains a multi-singer acoustic model capable of controlling emotional intensity using residual prediction, resulting in more accurate predictions from the trained acoustic model. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A hardware framework diagram of an acoustic model training method and / or a singing voice synthesis method provided in this application;

[0043] Figure 2 A hardware framework diagram of a specific acoustic model training method and / or singing voice synthesis method provided in this application;

[0044] Figure 3 A flowchart of an acoustic model training method provided in this application;

[0045] Figure 4 A flowchart of a specific acoustic model training method provided in this application;

[0046] Figure 5 An example diagram illustrating a specific phoneme digit sequence extraction process provided in this application;

[0047] Figure 6 A schematic diagram illustrating a specific preprocessing method provided in this application;

[0048] Figure 7 A flowchart of a specific acoustic model training method provided in this application;

[0049] Figure 8 This application provides a specific diagram of the acoustic model architecture to be trained.

[0050] Figure 9 A flowchart of a singing voice synthesis method provided in this application;

[0051] Figure 10 A specific vocoder architecture diagram is provided for this application;

[0052] Figure 11 Example diagram of a specific virtual singer singing solution provided in this application;

[0053] Figure 12 This is a schematic diagram of an acoustic model training device provided in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] In related technologies, with the introduction of deep learning technology, acoustic modeling has made great strides, making the synthesized singing voices increasingly realistic and expressive. However, it is still far from truly comparable to real human singing. To address these technical shortcomings, this application provides an acoustic model training scheme. This scheme reflects the singer's characteristics and emotional intensity by adding variances of energy, pitch, and pitch duration based on singer and emotion variables. Furthermore, it trains a multi-singer acoustic model capable of controlling emotional intensity through residual prediction, resulting in more accurate predictions after training. Based on the above acoustic model training scheme, this application also provides a corresponding singing voice synthesis scheme, which will not be elaborated upon here.

[0056] To facilitate understanding, the hardware framework applicable to the acoustic model training method and / or singing voice synthesis method of this application will be introduced first. See [link to relevant documentation]. Figure 1,in, Figure 1 It shows a schematic diagram of the hardware framework applicable to the acoustic model training method and / or singing voice synthesis method of this application.

[0057] Depend on Figure 1 It can be seen that the hardware framework may include: electronic device 10, wherein electronic device 10 may include: processor 11, memory 12, communication interface 13, multimedia component 14, input / output interface 15, and communication bus 16. Processor 11, memory 12, communication interface 13, multimedia component 14, and input / output interface 15 all communicate with each other through communication bus 16.

[0058] In this embodiment, the processor 11 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, an off-the-shelf programmable gate array, or other programmable logic devices. The processor can call programs stored in the memory 12. Specifically, the processor can perform the operations executed on the computer device side in the embodiments of the acoustic model training method and / or the singing synthesis method.

[0059] The memory 12 stores one or more programs, which may include program code, including computer operation instructions. The program code can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. In this embodiment, the memory stores at least a program for implementing the following functions:

[0060] Obtain the score to be trained and the real Mel spectrum of the score to be trained, and preprocess the score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence and pitch duration digital sequence;

[0061] The real Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence are input into the acoustic model to be trained.

[0062] The encoder is used to encode the spliced ​​features corresponding to the phoneme digital sequence and the pitch digital sequence, and singer variables and emotion variables are added to obtain the encoded latent vector;

[0063] The phoneme pitch is predicted using a pitch prediction module based on the pitch digital sequence and the encoded latent vector; the phoneme duration is predicted using a duration prediction module based on the pitch duration digital sequence and the encoded latent vector; and the phoneme energy is predicted using a phoneme energy prediction module based on the phoneme digital sequence, resulting in a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information. The pitch prediction module and the duration prediction module use residual prediction for their prediction methods.

[0064] The decoder input is reconstructed into the corresponding predicted Mel spectrum using the decoder, and each module of the acoustic model to be trained is trained based on the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0065] In one possible implementation, the memory 12 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as sound playback, image playback, etc.). The data storage area may store data created during the use of the electronic device, such as sheet music to be trained.

[0066] The communication interface 13 can be an interface for a communication module, such as the interface for a GSM module.

[0067] Multimedia component 14 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 12 or transmitted via communication interface 13. The audio component also includes at least one speaker for outputting audio signals.

[0068] Input / output interface 15 provides an interface between processor 11 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication interface 13 is used for wired or wireless communication between electronic device 10 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 4G, 5G, or 6G, or a combination thereof. Therefore, the corresponding communication component 105 may include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0069] Electronic device 10 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform acoustic model training methods and / or vocal synthesis methods.

[0070] certainly, Figure 1 The structure of the electronic device 10 shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 1 More or fewer components as shown, or combinations of certain components.

[0071] in, Figure 1 The electronic device 10 can be a terminal (such as a mobile terminal like a mobile phone or tablet computer, or a fixed terminal like a PC), a server, or a smart electronic device.

[0072] It is understood that the number of electronic devices is not limited in the embodiments of this application; multiple electronic devices may collaborate to complete the functions of the acoustic model training method and / or the singing voice synthesis method. In one possible scenario, please refer to... Figure 2 .Depend on Figure 2 It can be seen that the hardware framework may include: a first electronic device 101 and a second electronic device 102. The first electronic device 101 and the second electronic device 102 communicate with each other through a network 103.

[0073] In this embodiment, the hardware structure of the first electronic device 101 and the second electronic device 102 can be referred to Figure 1 The electronic device 10 in this embodiment can be understood as having two electronic devices 10, which interact with each other to realize acoustic model training and / or singing synthesis functions. Furthermore, this embodiment does not limit the form of the network 103; for example, the network 103 can be a wireless network (such as WIFI, Bluetooth, etc.) or a wired network.

[0074] In this design, the first electronic device 101 and the second electronic device 102 can be the same type of electronic device, such as both being servers; alternatively, they can be different types of electronic devices, such as the first electronic device 101 being a terminal or a smart electronic device, and the second electronic device 102 being a server. In another possible scenario, a server with high computing power can be used as the second electronic device 102 to improve data processing efficiency and reliability, thereby improving the training efficiency of the acoustic model and / or the efficiency of vocal synthesis. Simultaneously, a low-cost, widely applicable terminal or smart electronic device can be used as the first electronic device 101 to enable interaction between the second electronic device 102 and the user.

[0075] Understandably, the interaction process can be as follows: the terminal collects the target musical score, the target singer information to be synthesized, and the target emotional information and transmits them to the server. The server preprocesses the target musical score to obtain the corresponding target phoneme digital sequence, target pitch digital sequence, and target pitch duration digital sequence, and inputs the target phoneme digital sequence, the target pitch digital sequence, the target pitch duration digital sequence, the target singer information, and the target emotional information into the trained acoustic model, so as to output the corresponding target Mel spectrum using the trained acoustic model. The server then inputs the target Mel spectrum into the vocoder, so as to output the synthesized song corresponding to the target Mel spectrum using the vocoder, and outputs the synthesized song to the terminal.

[0076] Figure 3 A flowchart illustrating an acoustic model training method provided in this application embodiment. See also... Figure 3 As shown, the acoustic model training method includes:

[0077] S11: Obtain the score to be trained and the real Mel spectrum of the score to be trained, and preprocess the score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence and pitch duration digital sequence.

[0078] In this embodiment, before training, the score to be trained and its actual Mel spectrum are acquired. The score to be trained is then preprocessed to obtain the corresponding phoneme digital sequence, pitch digital sequence, and pitch duration digital sequence. The score to be trained is a pure musical score from the singing dataset, and the actual Mel spectrum can be obtained by extracting acoustic features using a vocoder.

[0079] In Chinese, the pronunciation of one character corresponds to one syllable, and one syllable can be divided into multiple phonemes. A phoneme is the smallest unit of phonetics. There are more than 400 syllables in Chinese, but only a few dozen phonemes. In order to enable the model to learn the rules of pronunciation more quickly, using phonemes as input is the best choice. Therefore, when turning a piece of music into a song, the lyrics need to be converted into syllables first, and then the syllables need to be converted into phoneme information.

[0080] S12: Input the real Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence into the acoustic model to be trained.

[0081] In this embodiment, the real Mel spectrum, the phoneme sequence, the pitch sequence, and the pitch duration sequence are input into the acoustic model to be trained. The acoustic model to be trained is an emotion-controlled multi-singer acoustic model, which is adapted from FastSpeech2. The inputs are the phoneme sequence, the pitch sequence, and the pitch duration sequence, representing what sound is produced, how high it is produced, and how long it is produced, respectively. These three inputs can guide the model training and inference.

[0082] S13: Encode the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence using the encoder, and add singer variables and emotion variables to obtain the encoded latent vector.

[0083] In this embodiment, after inputting the real Mel spectrum, the phoneme digit sequence, the pitch digit sequence, and the pitch digit duration sequence into the acoustic model to be trained, an encoder is used to encode the concatenated features corresponding to the phoneme digit sequence and the pitch digit sequence, and singer variables and emotion variables are added to obtain the encoded latent vector. The encoder is an encoder FFT Block. It should be noted that the encoder processes the concatenated features corresponding to the phoneme digit sequence and the pitch digit sequence, and also needs to add singer variables and emotion variables. The singer variable is a singer embedding, and the emotion variable is an emotion embedding. That is, different singer controls are controlled through singer embedding, and different emotion controls are controlled through emotion embedding. Specifically, the singer timbre can be controlled through singer embedding (lookup table), and the emotion can be controlled through emotion embedding (lookup table).

[0084] S14: The pitch prediction module predicts the phoneme pitch based on the pitch digital sequence and the encoded latent vector; the duration prediction module predicts the phoneme duration based on the pitch duration digital sequence and the encoded latent vector; and the phoneme energy prediction module predicts the phoneme energy based on the phoneme digital sequence, thereby obtaining a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information; wherein, the prediction method of the pitch prediction module and the duration prediction module is residual prediction.

[0085] In this embodiment, a pitch prediction module predicts phoneme pitch based on the pitch digital sequence and the encoded latent vector; a duration prediction module predicts phoneme duration based on the pitch-duration digital sequence and the encoded latent vector; and a phoneme energy prediction module predicts phoneme energy based on the phoneme digital sequence, resulting in a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information. The phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence are input to their respective modules for prediction. It is important to note that the inputs to the pitch prediction module are the pitch digital sequence and the encoded latent vector, and the inputs to the duration prediction module are the pitch-duration digital sequence and the encoded latent vector.

[0086] Furthermore, the pitch prediction module and the duration prediction module use residual prediction. By learning from the residual networks of the pitch digital duration sequence (note duration) and the pitch digital duration sequence (note pitch), more accurate duration and pitch predictions are achieved, ultimately resulting in a high-quality acoustic model capable of handling multiple singers and conveying multiple emotions. The duration of a syllable in a musical score is already defined, but when a syllable is divided into phonemes, how is the duration determined? This is where the duration model comes in. Phonemes can be divided into initials and finals, and finals are further divided into vowels and consonants. The duration prediction module plans the duration of each phoneme within a syllable based on the type of different phonemes and the context.

[0087] S15: The decoder input is reconstructed into the corresponding predicted Mel spectrum using the decoder, and each module of the acoustic model to be trained is trained based on the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0088] In this embodiment, the decoder input is reconstructed into a corresponding predicted Mel spectrum using a decoder. The modules of the acoustic model to be trained are then trained based on the predicted Mel spectrum and the true Mel spectrum to obtain the trained acoustic model. The decoder is a decoder FFT Block, which primarily encodes a Mel spectrum, i.e., the true Mel spectrum, into a latent vector containing "pitch information + phoneme information + energy information + emotion information + singer information." The trained acoustic model is a multi-singer acoustic model that controls emotional intensity and can express different levels of emotional intensity. During the synthesis of human voice based on lyrics, pitch, and musical duration, the characteristics and emotional intensity of the singer are reflected by adding variances of pitch, energy, and phoneme duration based on the singer and emotional intensity. Furthermore, by using residuals from predicted duration and residuals from predicted pitch, the model can more accurately predict duration and pitch, guiding it to learn how long and how high to sing each phoneme.

[0089] As can be seen, in this embodiment, the musical score to be trained and its actual Mel spectrum are first obtained, and the musical score is preprocessed to obtain the corresponding phoneme digital sequence, pitch digital sequence, and pitch duration digital sequence. Then, the actual Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch duration digital sequence are input into the acoustic model to be trained. Next, the encoder is used to encode the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence and add singer variables and emotion variables to obtain the encoded latent vector. Based on this, the pitch prediction module is used to predict the pitch digital sequence and the encoded latent vector. The process involves predicting phoneme pitch, using a duration prediction module to predict phoneme duration based on the pitch-duration digital sequence and the encoded latent vector, and using a phoneme energy prediction module to predict phoneme energy based on the phoneme digital sequence, resulting in a decoder input containing phoneme energy, pitch, and duration information. The pitch and duration prediction modules employ residual prediction. Finally, the decoder reconstructs the decoder input into a corresponding predicted Mel spectrum, and trains each module of the acoustic model based on the predicted and actual Mel spectra to obtain the trained acoustic model. This embodiment reflects the singer's characteristics and emotional intensity by adding variances of energy, pitch, and pitch duration based on singer and emotion variables. Furthermore, it trains a multi-singer acoustic model capable of controlling emotional intensity using residual prediction, resulting in more accurate predictions from the trained acoustic model.

[0090] Figure 4 A flowchart illustrating a specific acoustic model training method provided in this application embodiment. See also... Figure 4 As shown, the acoustic model training method includes:

[0091] S21: Obtain the score to be trained and the true Mel spectrum of the score to be trained.

[0092] In this embodiment, the specific process of step S21 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.

[0093] S22: Perform music score analysis on the music score to be trained to obtain the corresponding phoneme sequence, pitch sequence and pitch duration sequence, and convert the phoneme sequence into a phoneme number sequence according to the phoneme set index, convert the pitch sequence into a pitch number sequence according to the pitch set index, and convert the pitch duration sequence into a pitch duration number sequence according to the pitch duration set index.

[0094] In this embodiment, after obtaining the score to be trained, the score is first analyzed to obtain the corresponding phoneme sequence, pitch sequence, and pitch duration sequence. Then, the phoneme sequence is converted into a phoneme number sequence according to the phoneme set index, the pitch sequence is converted into a pitch number sequence according to the pitch set index, and the pitch duration sequence is converted into a pitch duration number sequence according to the pitch duration set index. The score analysis mainly extracts the lyric phoneme sequence and music theory features from the score. The music theory features include the pitch and pitch duration in the score, which are preprocessed into corresponding number sequences. For the phoneme number sequence, the Chinese characters are first converted into a phoneme sequence of initials and finals, and then converted into the corresponding number sequence through the phoneme set index, such as... Figure 5 Similarly, pitch and pitch duration are also converted into corresponding numerical sequences, and the overall process is as follows: Figure 6 As shown.

[0095] S23: Input the real Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence into the acoustic model to be trained.

[0096] In this embodiment, the specific process of step S23 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.

[0097] S24: Extract the corresponding phoneme embedding features based on the phoneme digit sequence, and extract the corresponding pitch embedding features based on the pitch digit sequence.

[0098] S25: The phoneme embedding feature and the pitch embedding feature are spliced ​​together to obtain the spliced ​​feature, and the spliced ​​feature is input to the encoder through the linear layer.

[0099] S26: Encode the spliced ​​features using the encoder and add singer variables and emotion variables to obtain the encoded latent vector.

[0100] In this embodiment, after the digital sequence is input, the corresponding phoneme embedding features are first extracted from the phoneme digital sequence. These phoneme embedding features are phone embeddings. Then, the corresponding pitch embedding features are extracted from the pitch digital sequence. These pitch embedding features are note embeddings. Next, the phoneme embedding features and the pitch embedding features are concatenated to obtain the concatenated features, which are then input to the encoder through a linear layer. Here, the phoneme digital sequence is converted to phone embeddings, and the pitch digital sequence is converted to note embeddings. The concatenation is performed instead of addition, and then fed into the linear layer before being input to the encoder. Finally, the encoder encodes the concatenated features and adds singer and emotion variables to obtain the encoded latent vector.

[0101] S27: The pitch prediction module predicts the phoneme pitch based on the pitch digital sequence and the encoded latent vector; the duration prediction module predicts the phoneme duration based on the pitch duration digital sequence and the encoded latent vector; and the phoneme energy prediction module predicts the phoneme energy based on the phoneme digital sequence, thereby obtaining a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information; wherein, the pitch prediction module and the duration prediction module use residual prediction as their prediction method.

[0102] S28: The decoder input is reconstructed into the corresponding predicted Mel spectrum using the decoder, and each module of the acoustic model to be trained is trained based on the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0103] In this embodiment, the specific processes of steps S27 and S28 can be referred to the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0104] Figure 7 A flowchart illustrating a specific acoustic model training method provided in this application embodiment. See also... Figure 7 As shown, the acoustic model training method includes:

[0105] S31: Obtain the score to be trained and the real Mel spectrum of the score to be trained, and preprocess the score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence and pitch duration digital sequence.

[0106] S32: Input the real Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch digital duration sequence into the acoustic model to be trained.

[0107] S33: Encode the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence using the encoder, and add singer variables and emotion variables to obtain the encoded latent vector.

[0108] In this embodiment, the specific processes of steps S31 to S33 can be referred to the corresponding content disclosed in the foregoing embodiments, and will not be repeated here. Figure 8 The diagram shown is a specific architecture diagram of the acoustic model to be trained provided in this embodiment.

[0109] S34: Input the pitch digital sequence and the encoded latent vector into the pitch prediction network, and input the output of the pitch prediction network into the pitch residual coding network to obtain the corresponding phoneme pitch information.

[0110] In this embodiment, the pitch digital sequence is fed into the pitch prediction module, and prediction is performed using residual prediction. The pitch prediction module includes a pitch prediction network and a pitch residual encoding network. The pitch prediction network is a residual pitch predictor, and the pitch residual encoding network is a residual pitch encoder. When using the pitch prediction module for prediction, the pitch digital sequence and the encoded latent vector are first input into the pitch prediction network, and then the output of the pitch prediction network is input into the pitch residual encoding network to obtain the corresponding phoneme pitch information. Determining the pitch of each phoneme is still a typical residual structure; the input is the latent vector output from the encoder plus the note pitch, which first passes through a residual pitch predictor network and then through a residual pitch encoder.

[0111] S35: Input the pitch duration digital sequence and the encoded latent vector into the pitch duration feature extraction network, and input the output of the pitch duration feature extraction network into the pitch duration residual coding network to obtain the corresponding phoneme duration information.

[0112] In this embodiment, the pitch duration digital sequence is fed separately into the duration prediction module, and prediction is performed using residual prediction. The duration prediction module includes a pitch duration feature extraction network and a pitch duration residual coding network. The pitch duration feature extraction network is a note duration embedding, and the pitch duration residual coding network is a residual duration predictor. When using the duration prediction module for prediction, the pitch duration digital sequence and the encoded latent vector are first input into the pitch duration feature extraction network. Then, the output of the pitch duration feature extraction network is input into the pitch duration residual coding network to obtain the corresponding phoneme duration information. The input for each phoneme is the phoneme sequence, and the output is the frame length, i.e., how many frames each phoneme occupies. However, it is important to note that the residual duration predictor is a residual network. The input to the residual duration predictor includes the latent vector output from the encoder plus the note duration.

[0113] S36: Input the phoneme digital sequence into the energy prediction network, and input the output of the energy prediction network into the energy encoding network to obtain the corresponding phoneme energy information.

[0114] In this embodiment, the energy prediction network can predict the energy of each phoneme, and its input and output are relatively simple. The phoneme energy prediction module includes an energy prediction network and an energy encoding network. The energy prediction network is an energy predictor, and the energy encoding network is an energy encoder. When using the energy prediction network for prediction, the phoneme digital sequence is first input into the energy prediction network, and then the output of the energy prediction network is input into the energy encoding network to obtain the corresponding phoneme energy information.

[0115] S37: The decoder input containing the phoneme energy information, the phoneme pitch information and the phoneme duration information is reconstructed into the corresponding predicted Mel spectrum using the decoder, and each module of the acoustic model to be trained is trained according to the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0116] In this embodiment, the specific process of step S37 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.

[0117] Figure 9 A flowchart illustrating a singing voice synthesis method provided in an embodiment of this application. See also... Figure 9As shown, this singing voice synthesis method is based on the aforementioned trained acoustic model and includes:

[0118] S41: Obtain the target musical score and determine the target singer information and target emotional information to be synthesized. Preprocess the target musical score to obtain the corresponding target phoneme digital sequence, target pitch digital sequence and target pitch duration digital sequence.

[0119] In this embodiment, the target musical score is acquired, and the target singer information and target emotional information to be synthesized are determined. The target musical score is preprocessed to obtain the corresponding target phoneme digital sequence, target pitch digital sequence, and target pitch duration digital sequence. The target singer information is a singer variable, such as singer id, which can be determined by looking up a table. The target emotional information is determined in the same way.

[0120] Singing voice synthesis (SVS) refers to the technology of converting standardized musical scores containing lyrics into singing voices using computers. As an important branch of speech synthesis technology, singing voice synthesis has a long research history. In recent years, with the breakthroughs brought by deep learning in speech synthesis tasks such as duration modeling, acoustic modeling, and vocoders, singing voice synthesis has gradually moved towards practical application, showing great potential in scenarios such as virtual idols, music creation, and game soundtracks. Speech synthesis technology is internally divided into a front-end and a back-end. The front-end is mainly responsible for the language parsing and processing of the target musical score, including language identification, word segmentation, part-of-speech prediction, polyphonic character processing, prosody prediction, and sentiment prediction. After predicting all this information about the speech on the target musical score, the information is sent to the back-end system, including acoustic models and vocoders. The back-end system fuses this information, converting the content into speech acoustic features, and then converting them into time-domain waveforms through the vocoder, thus realizing the conversion of content into speech. The backend has a long history of development, from the first generation of speech splicing and synthesis to the second generation of speech parameter synthesis, and then to the third generation of end-to-end synthesis. The intelligence level of the backend acoustic system has gradually increased, and the level of detail required for training annotation, the difficulty, and the training steps have been gradually reduced.

[0121] In this embodiment, the front end converts the Chinese characters of the target musical score into a phonetic number sequence, and the back end's trained acoustic model generates acoustic features based on the information output by the front end, such as mapping the phonetic number sequence to the Mel spectrum. The back end's vocoder uses acoustic features such as the Mel spectrum to generate speech sample points and reconstruct time-domain waveforms, such as restoring the Mel spectrum to the corresponding singing voice.

[0122] S42: Input the target phoneme digital sequence, the target pitch digital sequence, the target pitch duration digital sequence, the target singer information, and the target emotion information into the trained acoustic model, so as to output the corresponding target Mel spectrum using the trained acoustic model.

[0123] S43: Input the target Mel spectrum into the vocoder to output a synthesized song corresponding to the target Mel spectrum.

[0124] In this embodiment, the target phoneme digital sequence, the target pitch digital sequence, the target pitch duration digital sequence, the target singer information, and the target emotional information are input into the trained acoustic model to output the corresponding target Mel spectrum. Finally, the target Mel spectrum is input into a vocoder to output a synthesized song corresponding to the target Mel spectrum.

[0125] This embodiment does not limit the network structure of the vocoder described above; it can use an existing vocoder network structure. Specifically, the training process of the vocoder includes: employing a singer conditional discriminator and performing conditional adversarial training to obtain a vocoder capable of acquiring and reconstructing the singer's identity from the Mel spectrum. That is, the vocoder in this embodiment is mainly a fast multi-singer vocoder based on generative adversarial networks. Specifically, it uses WAVNet blocks and PQMF multiple subbands to accelerate the training and inference process, employs a singer conditional discriminator and conditional adversarial training objective, and acquires and reconstructs the singer's identity from the Mel spectrum. The overall framework of the vocoder is as follows: Figure 10 As shown, the basic steps are as follows:

[0126] 1. Prepare feature data. For the audio WAV corresponding to different singers, slice it (5-second segments are optional), and then extract the Mel spectrum of the corresponding segments. This forms a triple (wav, mel spectrum, singer ID), where the mel spectrum and singer ID are the inputs and the wav is the output.

[0127] 2. Construct the generator and discriminator for the vocoder respectively;

[0128] 3. Use GAN for training. The loss functions for the Generator and Discriminator in GAN are as follows:

[0129]

[0130]

[0131] The following example, using a virtual singer, illustrates the vocal synthesis solution of this application. In recent years, with technological advancements, more and more things that were previously unimaginable have become possible. Perhaps one day in the future, everyone will be able to have their own virtual avatar, fulfilling their childhood dreams in a virtual world. Virtual singers primarily utilize the voice of a virtual person, endowing the virtual human with singing skills, promoting and publicizing their work, and even launching their own careers. In the past, while music was one of humanity's oldest and most natural forms of self-expression, the creative threshold was extremely high. For ordinary people, music creation was a huge challenge, and the creation / consumption ratio was far lower than other media. Today, virtual singers allow ordinary people to create music, play music, and even express themselves through music alongside AI. This has also fostered many interesting and unique AI music styles, and the enormous potential of AGC (AI Generated Content, i.e., content production methods that automatically generate content through artificial intelligence technology) in the music field is gradually becoming apparent. The vocal synthesis technology in this embodiment can create a personalized virtual singer, building an AI vocal synthesis platform where users can upload their own compositions to have the virtual singer sing them. Figure 11 The image shows a complete virtual singer singing solution. Creators can create sheet music files, upload them, select a virtual singer (or emotion), and after training, the acoustic model and vocoder can synthesize the corresponding audio.

[0132] See Figure 12 As shown in the embodiments, this application also discloses an acoustic model training device, including:

[0133] The acquisition module 21 is used to acquire the score to be trained and the real Mel spectrum of the score to be trained, and to preprocess the score to be trained to obtain the corresponding phoneme digital sequence, pitch digital sequence and pitch duration digital sequence.

[0134] Input module 22 is used to input the real Mel spectrum, the phoneme digital sequence, the pitch digital sequence and the pitch digital duration sequence into the acoustic model to be trained;

[0135] Encoding module 23 is used to encode the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence using an encoder and add singer variables and emotion variables to obtain the encoded latent vector;

[0136] The prediction module 24 is used to predict the phoneme pitch using the pitch prediction module based on the pitch digital sequence and the encoded latent vector, predict the phoneme duration using the duration prediction module based on the pitch duration digital sequence and the encoded latent vector, and predict the phoneme energy using the phoneme energy prediction module based on the phoneme digital sequence, to obtain a decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information; wherein, the prediction method of the pitch prediction module and the duration prediction module is residual prediction;

[0137] The decoding module 25 is used to reconstruct the decoder input into the corresponding predicted Mel spectrum using the decoder, and to train each module of the acoustic model to be trained based on the predicted Mel spectrum and the real Mel spectrum to obtain the trained acoustic model.

[0138] As can be seen, in this embodiment, the musical score to be trained and its actual Mel spectrum are first obtained, and the musical score is preprocessed to obtain the corresponding phoneme digital sequence, pitch digital sequence, and pitch duration digital sequence. Then, the actual Mel spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch duration digital sequence are input into the acoustic model to be trained. Next, the encoder is used to encode the splicing features corresponding to the phoneme digital sequence and the pitch digital sequence and add singer variables and emotion variables to obtain the encoded latent vector. Based on this, the pitch prediction module is used to predict the pitch digital sequence and the encoded latent vector. The process involves predicting phoneme pitch, using a duration prediction module to predict phoneme duration based on the pitch-duration digital sequence and the encoded latent vector, and using a phoneme energy prediction module to predict phoneme energy based on the phoneme digital sequence, resulting in a decoder input containing phoneme energy, pitch, and duration information. The pitch and duration prediction modules employ residual prediction. Finally, the decoder reconstructs the decoder input into a corresponding predicted Mel spectrum, and trains each module of the acoustic model based on the predicted and actual Mel spectra to obtain the trained acoustic model. This embodiment reflects the singer's characteristics and emotional intensity by adding variances of energy, pitch, and pitch duration based on singer and emotion variables. Furthermore, it trains a multi-singer acoustic model capable of controlling emotional intensity using residual prediction, resulting in more accurate predictions from the trained acoustic model.

[0139] In some specific embodiments, the acquisition module 21 is further configured to perform music score analysis on the music score to be trained to obtain the corresponding phoneme sequence, pitch sequence and pitch duration sequence, and convert the phoneme sequence into the phoneme number sequence according to the phoneme set index, convert the pitch sequence into the pitch number sequence according to the pitch set index, and convert the pitch duration sequence into the pitch duration number sequence according to the pitch duration set index.

[0140] In some specific embodiments, the acoustic model training device further includes:

[0141] An embedding feature extraction module is used to extract corresponding phoneme embedding features based on the phoneme digit sequence and to extract corresponding pitch embedding features based on the pitch digit sequence.

[0142] The feature splicing module is used to splice the phoneme embedding feature and the pitch embedding feature to obtain the spliced ​​feature, and input the spliced ​​feature to the encoder through a linear layer.

[0143] In some specific embodiments, the pitch prediction module in the acoustic model training device includes a pitch prediction network and a pitch residual coding network;

[0144] The prediction module 24 is further configured to input the pitch digital sequence and the encoded latent vector into the pitch prediction network, and input the output of the pitch prediction network into the pitch residual encoding network to obtain the corresponding phoneme pitch information.

[0145] In some specific embodiments, the duration prediction module in the acoustic model training device includes a pitch duration feature extraction network and a pitch duration residual coding network;

[0146] The prediction module 24 is further configured to input the pitch duration digital sequence and the encoded latent vector into the pitch duration feature extraction network, and input the output of the pitch duration feature extraction network into the pitch duration residual coding network to obtain the corresponding phoneme duration information.

[0147] In some specific embodiments, the phoneme energy prediction module in the acoustic model training device includes an energy prediction network and an energy encoding network;

[0148] The prediction module 24 is further configured to input the phoneme digital sequence into the energy prediction network and input the output of the energy prediction network into the energy encoding network to obtain the corresponding phoneme energy information.

[0149] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the acoustic model training method and / or singing voice synthesis method steps disclosed in any of the foregoing embodiments.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0151] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] The acoustic model training method, apparatus, device, medium, and singing voice synthesis method provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An acoustic model training method, characterized in that, The method comprises: acquiring a to-be-trained musical score and a real mel-frequency spectrum of the to-be-trained musical score, and pre-processing the to-be-trained musical score to obtain a corresponding phoneme digital sequence, a pitch digital sequence, and a pitch duration digital sequence; inputting the real mel-frequency spectrum, the phoneme digital sequence, the pitch digital sequence, and the pitch duration digital sequence into a to-be-trained acoustic model; encoding, by using an encoder, spliced features corresponding to the phoneme digital sequence and the pitch digital sequence, adding a singer variable and an emotion variable, and obtaining an encoded hidden vector; predicting, by using a pitch prediction module, phoneme pitch according to the pitch digital sequence and the encoded hidden vector, predicting, by using a duration prediction module, phoneme duration according to the pitch duration digital sequence and the encoded hidden vector, predicting, by using a phoneme energy prediction module, phoneme energy according to the phoneme digital sequence, and obtaining decoder input containing phoneme energy information, phoneme pitch information, and phoneme duration information; wherein the prediction mode of the pitch prediction module and the duration prediction module is residual prediction; reconstructing, by using a decoder, the decoder input into a corresponding predicted mel-frequency spectrum, and training, according to the predicted mel-frequency spectrum and the real mel-frequency spectrum, each module of the to-be-trained acoustic model to obtain a trained acoustic model.

2. The acoustic model training method of claim 1, wherein, The pre-processing of the to-be-trained musical score to obtain the corresponding phoneme digital sequence, the pitch digital sequence, and the pitch duration digital sequence comprises: performing score analysis on the to-be-trained musical score to obtain a corresponding phoneme sequence, a pitch sequence, and a pitch duration sequence; converting the phoneme sequence into the phoneme digital sequence according to a phoneme set index, converting the pitch sequence into the pitch digital sequence according to a pitch set index, and converting the pitch duration sequence into the pitch duration digital sequence according to a pitch duration set index.

3. The acoustic model training method of claim 1, wherein, Before the encoding, by using the encoder, of the spliced features corresponding to the phoneme digital sequence and the pitch digital sequence, the method further comprises: extracting corresponding phoneme embedding features according to the phoneme digital sequence, and extracting corresponding pitch embedding features according to the pitch digital sequence; splicing the phoneme embedding features and the pitch embedding features to obtain the spliced features, and inputting the spliced features into the encoder through a linear layer.

4. The acoustic model training method of claim 1, wherein, The pitch prediction module comprises a pitch prediction network and a pitch residual coding network. The predicting, by using the pitch prediction module, of the phoneme pitch according to the pitch digital sequence and the encoded hidden vector comprises: inputting the pitch digital sequence and the encoded hidden vector into the pitch prediction network, and inputting the output of the pitch prediction network into the pitch residual coding network to obtain corresponding phoneme pitch information.

5. The acoustic model training method of claim 1, wherein, The duration prediction module comprises a pitch duration feature extraction network and a pitch duration residual coding network. The predicting, by using the duration prediction module, of the phoneme duration according to the pitch duration digital sequence and the encoded hidden vector comprises: The pitch duration digital sequence and the encoded latent vector are input into the pitch duration feature extraction network, and an output of the pitch duration feature extraction network is input into the pitch duration residual coding network to obtain corresponding phoneme duration information.

6. The acoustic model training method of claim 1, wherein, The phoneme energy prediction module comprises an energy prediction network and an energy coding network. The phoneme energy prediction module comprises an energy prediction network and an energy coding network. The phoneme energy prediction module comprises an energy prediction network and an energy coding network.

7. A singing voice synthesis method characterized by, The post-training acoustic model based on any one of claims 1 to 6 comprises: Obtaining a target score and determining target singer information and target emotion information to be synthesized, and preprocessing the target score to obtain corresponding target phoneme digital sequence, target pitch digital sequence and target pitch duration digital sequence; The target phoneme digital sequence, the target pitch digital sequence, the target pitch duration digital sequence, the target singer information and the target emotion information are input into the post-training acoustic model to output corresponding target mel-frequency spectrum using the post-training acoustic model. The target mel-frequency spectrum is input into a vocoder to output a synthesized song corresponding to the target mel-frequency spectrum using the vocoder.

8. The singing synthesis method according to claim 7, wherein, The training process of the vocoder comprises: A singer condition discriminator is used to perform conditional adversarial training to obtain a vocoder capable of obtaining and reconstructing the identity of a singer from a mel-frequency spectrum.

9. An electronic device, comprising: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, A computer executable instruction is used to store the computer executable instruction, and the computer executable instruction is loaded and executed by the processor to implement the method of any one of claims 1 to 8.

Images