Singing beautifying method and system based on streaming matching

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a song beautifying method and system based on streaming matching, and the method comprises the steps: obtaining song data and music score data; extracting timbre characteristics and a phoneme posterior probability graph from the singing data; generating a multi-dimensional song expressive force sequence according to the music score data and the phoneme posterior probability graph; generating a voice Mel spectrum according to the multi-dimensional song expressive force sequence, the phoneme posterior probability graph and the timbre characteristics; and inputting the voice Mel spectrum into a vocoder for conversion processing to obtain beautified singing voice. According to the invention, the output singing voice can be optimized in intonation, timbre and expressive force, the expressive force and naturalness of the singing voice can be obviously improved, and the generated singing voice is higher in quality, smoother in listening feeling and rich in expressive force.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a singing beautification method and system based on streaming matching. Background Art

[0002] With the rapid development of deep learning technology, singing voice beautification technology has gradually been applied to improve audio quality. It can correct the shortcomings of the input singing voice and make it closer to the singing style of professional singers. This type of technology provides a new solution for the automatic correction and beautification of singing voices, especially in pitch correction. It has achieved certain results and can effectively solve problems such as pitch deviation, thereby improving audio quality and making singing voices sound more accurate and pleasant.

[0003] However, most of the existing singing beautification technologies only focus on the correction of a single dimension, namely pitch, and fail to comprehensively model and optimize the expressiveness of singing. Expressiveness includes multi-dimensional parameters such as pitch, singing voice, tension, energy, etc., which together determine the natural fluency and emotional expression of singing. Due to the neglect of these dimensions, although the generated singing voice is accurate in pitch, it lacks the expressiveness and emotional transmission of professional singers. In addition, during the generation process of the existing technology, the dynamic relationship between the expressiveness parameters and the time step of the singing content lacks consistency, resulting in the beautified singing voice appearing stiff and unnatural in emotional expression, and the overall expressiveness is low, which makes it difficult to meet the demand for high-quality singing in actual application scenarios. Summary of the invention

[0004] The present invention is to improve the defects of low vocal expression and quality in the existing vocal beautification technology, and proposes the following technical solutions: In a first aspect, the present invention proposes a singing voice beautification method based on streaming matching, comprising: Acquire singing data and music score data; Extracting timbre features and phoneme posterior probability maps from singing voice data; Generate a multi-dimensional singing expressiveness sequence based on the score data and the phoneme posterior probability map; Generate speech mel spectrogram based on multi-dimensional singing expressiveness sequence, phoneme posterior probability map and timbre characteristics; The speech Mel spectrum is input into the vocoder for conversion processing to obtain the beautified singing voice.

[0005] As a preferred technical solution, the multi-dimensional singing expressiveness sequence includes a pitch frame level sequence, a singing frame level sequence, an energy frame level sequence and a tension frame level sequence.

[0006] As a preferred technical solution, a multi-dimensional singing expression sequence is generated according to the score data and the phoneme posterior probability map, including: Build and train a streaming matching network; Perform dimension expansion processing on the phoneme posterior probability map; Based on the dimensionally expanded phoneme posterior probability map and music score data, the trained streaming matching network is used to perform feature mapping to generate a pitch frame-level sequence. According to the score data and pitch frame-level sequence, the trained streaming matching network is used to perform feature mapping to generate the singing frame-level sequence; According to the score data, pitch frame-level sequence and phoneme posterior probability map, the trained streaming matching network is used to perform feature mapping to generate energy frame-level sequence; According to the score data, pitch frame-level sequence, energy frame-level sequence prediction and phoneme posterior probability map, the trained streaming matching network is used for feature mapping to generate the target tension frame-level sequence.

[0007] As a preferred technical solution, a streaming matching network is constructed and feature mapping is performed using the streaming matching network, including: The mapping relationship between modeling input distribution and target distribution is expressed as follows:

[0008] in, Indicates j The expressiveness parameter is in the time step t The distribution of time, Indicates j The target distribution of expressive parameters, Indicates the j The input distribution of the expressive parameters; By integral solution, the input distribution Mapping to target distribution ,Will Output as the final vocal expressive sequence , whose expression is as follows: ,

[0009] in:

[0010]

[0011]

[0012]

[0013] In the formula, For the streaming matching network in learning j The vector velocity obtained in the expressive parameter stage, For the time step t The state distribution of Indicates that the streaming matching network is learning j Model parameters for expressiveness parameters, is the input score data, is the phoneme posterior probability map, It is the phoneme posterior probability map after dimension expansion.

[0014] As a preferred technical solution, when the streaming matching network is used for feature mapping, the method further includes predicting the vector velocity of the streaming matching network, including: Get phoneme posterior probability maps, time steps, and expressiveness parameters in real time; Extracting music score features including rhythm, style and range from music score data; The time step is position-encoded by a sinusoidal position encoding module to generate position encoding information; The music score features, phoneme posterior probability map and position encoding information are input as input features into the dilated convolution module, where the dilated convolution module consists of multiple dilated convolution blocks, each of which performs the following operations: Perform dilated convolution operation on the input features; The input features are activated through the tanh activation function and the sigmoid activation function; The output of the dilated convolutional block is added to the input features through a residual connection; The output results of all dilated convolution blocks are accumulated and dimensionally integrated to generate the final vector velocity.

[0015] As a preferred technical solution, training a streaming matching network includes: Use music score data and singing data to build a training set; Extracting music score feature parameters including notes, singing style and note duration from the music score data, and extracting expressive parameters including energy, pitch and tension from the singing voice data; The music score feature parameters, expressiveness parameters, phoneme posterior probability maps and timbre features are used as training labels, and the training set data is used to minimize the objective function. Train a streaming matching network.

[0016] As a preferred technical solution, a speech mel spectrum is generated according to a multi-dimensional singing voice expressiveness sequence, a phoneme posterior probability map and timbre characteristics, including: The phoneme posterior probability map is used as the query matrix Q, and the multi-dimensional singing voice expressiveness sequence is used as the key-value matrix K and V. The features of the multi-dimensional singing voice expressiveness sequence and the phoneme posterior probability map are fused through the multi-head attention mechanism to obtain the multi-attention fusion feature, which is expressed as follows:

[0017] =softmax ( * / )* ,

[0018] in, Represents a splicing operation, Represents the attention head j Attention calculation, is the output weight matrix in the multi-head attention mechanism, express Dimensions; Generate speech mel-spectrogram based on multi-attention fusion features, phoneme posterior probability map and timbre features.

[0019] As a preferred technical solution, a speech mel spectrum is generated based on multi-attention fusion features, phoneme posterior probability maps and timbre features, including: After dimensional integration of the multi-attention fusion features and the phoneme posterior probability map, they are added to the timbre features to obtain the fused input feature matrix; The input feature matrix is ​​processed by four DITBlocks in turn to obtain a high-dimensional feature matrix, where each DITBlock calculates the scaling parameters through a layer perceptron. , With bias parameters , and perform two layers of AdaLN operations on the input feature matrix to perform feature adjustment according to the following formula:

[0020]

[0021] In the formula, x is the input feature matrix; The high-dimensional feature matrix is ​​compressed through a one-dimensional convolution operation to generate a speech mel spectrum.

[0022] As a preferred technical solution, each DITBlock calculates the scaling parameters and bias parameters through a layer perceptron according to the introduced conditional matrix; wherein the conditional matrix of the first DITBlock is a time step information matrix, which is used to align the input feature matrix with the time step information, and the conditional matrices of the last three DITBlocks are timbre feature matrices, which are used to adaptively adjust the timbre features of the input feature matrix; Among them, the steps of obtaining the time step information matrix include: Extract time step information from the phoneme posterior probability map and construct a time step information matrix; The timbre feature parameters are extracted from the timbre features and a timbre feature matrix is ​​constructed.

[0023] In a second aspect, the present invention further proposes a singing voice beautification system based on streaming matching, which is applied to the singing voice beautification method based on streaming matching as described in any scheme of the first aspect, comprising: An acquisition module is used to acquire singing data and music score data; An extraction module, used for extracting timbre features and phoneme posterior probability maps from singing voice data; The first generation module is used to generate a multi-dimensional singing voice expressiveness sequence according to the music score data and the phoneme posterior probability map; The second generation module is used to generate speech mel spectrum according to the multi-dimensional singing expressiveness sequence, the phoneme posterior probability map and the timbre characteristics; The conversion module is used to input the speech Mel spectrum into the vocoder for conversion processing to obtain the beautified singing voice.

[0024] The beneficial effects of the present invention include at least: The present invention can accurately capture the basic timbre information and speech content characteristics of singing voice by extracting timbre features and phoneme posterior probability maps; generate a multi-dimensional singing voice expressiveness sequence through music score data and phoneme posterior probability maps, and effectively model the expressiveness parameters of singing voice; generate speech mel-spectrograms in combination with these features to ensure the coordination and unity of pitch, timbre and expressiveness parameters; finally, the vocoder converts the speech mel-spectrograms to optimize the output singing voice in terms of pitch, timbre and expressiveness, which can significantly improve the expressiveness and naturalness of singing voice, and make the generated singing voice higher in quality, more fluent and expressive to listen to. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart of a method for beautifying singing voice based on streaming matching provided in an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of a method for generating a multi-dimensional vocal expressiveness sequence based on music score data and a phoneme posterior probability map provided in an embodiment of the present invention.

[0027] Figure 3 Schematic diagram of the improved WaveNet architecture in an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of a method for generating a beautified singing voice according to a multi-dimensional singing voice expressiveness sequence, a phoneme posterior probability map and a timbre feature provided in an embodiment of the present invention.

[0029] Figure 5 A schematic diagram of a streaming matching decoder based on a DIT architecture provided in an embodiment of the present invention.

[0030] Figure 6 A pitch graph of an input singing voice provided by an embodiment of the present invention.

[0031] Figure 7 A pitch graph of a beautified singing voice output provided by an embodiment of the present invention.

[0032] Figure 8 The present invention provides an architecture diagram of a singing voice beautification system based on streaming matching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0034] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0035] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0036] Example 1 This embodiment proposes a singing voice beautification method based on streaming matching, such as Figure 1 As shown, Figure 1 A flow chart of a method for beautifying singing voice based on streaming matching provided by an embodiment of the present invention, the method comprising the following steps: S1: Acquire singing data and music score data; S2: Extracting timbre features and phoneme posterior probability maps from singing data; S3: Generate a multi-dimensional singing expressiveness sequence based on the score data and the phoneme posterior probability map; S4: Generate speech mel spectrogram based on multi-dimensional singing expressiveness sequence, phoneme posterior probability map and timbre characteristics; S5: Input the speech Mel spectrum into the vocoder for conversion processing to obtain the beautified singing voice.

[0037] As an exemplary explanation, in the specific implementation process, a user-recorded singing voice is used as input, combined with the corresponding song score data, and firstly, the input singing voice data is feature extracted to obtain timbre features (such as timbre stability and clarity) and phoneme posterior probability map (PPG, a feature map representing the singing content and time step information); then, the input score data and the phoneme posterior probability map are used to generate a multi-dimensional singing voice expressiveness sequence, including multiple expressiveness dimensions such as pitch, timbre, tension and energy, to ensure the modeling of the rich expressiveness of the singing voice; then, the generated multi-dimensional singing voice expressiveness sequence, phoneme posterior probability map and timbre features are fused and processed to generate the corresponding speech mel-spectrogram; finally, the generated speech mel-spectrogram is input into a vocoder (such as NSF-HIFIGAN vocoder), and combined with the pitch parameter sequence, it is converted to obtain a beautified singing voice.

[0038] It can be understood that by extracting timbre features and phoneme posterior probability maps, the basic timbre information and speech content characteristics of the singing voice can be accurately captured; a multi-dimensional singing voice expressiveness sequence is generated through score data and phoneme posterior probability maps, effectively modeling the expressiveness parameters of the singing voice; these features are combined to generate a speech mel-spectrogram to ensure the coordination and unity of pitch, timbre and expressiveness parameters; finally, the vocoder converts the speech mel-spectrogram to optimize the output singing voice in terms of pitch, timbre and expressiveness, which can significantly improve the expressiveness and naturalness of the singing voice, making the generated singing voice of higher quality, smoother and more expressive.

[0039] Example 2 This embodiment makes improvements on the singing voice beautification method based on streaming matching proposed in Embodiment 1.

[0040] In this embodiment, the multi-dimensional singing expression sequence includes a pitch frame level sequence, a singing frame level sequence, an energy frame level sequence and a tension frame level sequence.

[0041] In this embodiment, a multi-dimensional singing voice expressiveness sequence is generated according to the music score data and the phoneme posterior probability map, including: Build and train a streaming matching network; Perform dimension expansion processing on the phoneme posterior probability map; Based on the dimensionally expanded phoneme posterior probability map and music score data, the trained streaming matching network is used to perform feature mapping to generate a pitch frame-level sequence. According to the score data and pitch frame-level sequence, the trained streaming matching network is used to perform feature mapping to generate the singing frame-level sequence; According to the score data, pitch frame-level sequence and phoneme posterior probability map, the trained streaming matching network is used to perform feature mapping to generate energy frame-level sequence; According to the score data, pitch frame-level sequence, energy frame-level sequence prediction and phoneme posterior probability map, the trained streaming matching network is used for feature mapping to generate the target tension frame-level sequence.

[0042] In this embodiment, a streaming matching network is constructed and feature mapping is performed using the streaming matching network, including: The mapping relationship between modeling input distribution and target distribution is expressed as follows:

[0043] in, Indicates j The expressiveness parameter is in the time step t The distribution of time, Indicates j The target distribution of expressive parameters, Indicates the j The input distribution of the expressive parameters; By integral solution, the input distribution Mapping to target distribution ,Will Output as the final vocal expressive sequence , whose expression is as follows: ,

[0044] in:

[0045]

[0046]

[0047]

[0048] In the formula, For the streaming matching network in learning j The vector velocity obtained in the expressive parameter stage, For the time step t The state distribution of Indicates that the streaming matching network is learning j Model parameters for expressiveness parameters, is the input score data, is the phoneme posterior probability map, is the phoneme posterior probability map after dimension expansion, , and They are all combined multidimensional matrices connected on their input dimensions.

[0049] As an exemplary explanation, in the specific implementation process, if Figure 2 As shown, Figure 2 The principle diagram of generating a multi-dimensional singing voice expressiveness sequence according to the music score data and the phoneme posterior probability map provided by the embodiment of the present invention, taking the input singing voice data and the music score data as the initial data source, extracting the timbre features and the phoneme posterior probability map from the singing voice data, and obtaining the multi-dimensional singing voice expressiveness sequence according to the music score data and the phoneme posterior probability map through the dimension expansion operation . The score data is processed by the score encoder to generate implicit features of the score that represent information such as melody, style, and range. A streaming matching network is constructed, and corresponding predictors (pitch predictor, singing predictor, energy predictor, and tension predictor) are designed based on different expressive parameters (pitch, singing style, energy, and tension), and the pitch predictor, singing style predictor, energy predictor, and tension predictor are used to construct.

[0050] Among them, the pitch predictor learns the pitch frame-level sequence based on the dimensionally expanded phoneme posterior probability map and the score data; the singing predictor learns the singing frame-level sequence based on the score data and the pitch frame-level sequence; the energy predictor learns the energy frame-level sequence based on the score data, the pitch frame-level sequence and the phoneme posterior probability map; the tension predictor learns the target tension frame-level sequence based on the score data, the pitch frame-level sequence, the energy frame-level sequence prediction and the phoneme posterior probability map. Match the time step of each expressive parameter with the input data to ensure a synchronization relationship in the time series dimension. In the streaming matching network, the input distribution is solved by integral. Mapping to each target distribution , generating the corresponding multi-dimensional expressive parameter sequence.

[0051] In this embodiment, when the streaming matching network is used for feature mapping, the method further includes predicting the vector velocity of the streaming matching network, including: Get phoneme posterior probability maps, time steps, and expressiveness parameters in real time; Extracting music score features including rhythm, style and range from music score data; The time step is position-encoded by a sinusoidal position encoding module to generate position encoding information; The music score features, phoneme posterior probability map and position encoding information are input as input features into the dilated convolution module, where the dilated convolution module consists of multiple dilated convolution blocks, each of which performs the following operations: Perform dilated convolution operation on the input features; The input features are activated through the tanh activation function and the sigmoid activation function; The output of the dilated convolutional block is added to the input features through a residual connection; The output results of all dilated convolution blocks are accumulated and dimensionally integrated to generate the final vector velocity.

[0052] As an exemplary illustration, this embodiment uses a modified WaveNet architecture to predict vector velocity, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the improved WaveNet architecture in the embodiment of the present invention. In the specific implementation process, the phoneme posterior probability map is processed by the first convolutional network (Conv1d) to generate a high-dimensional PPG feature matrix [B, 256, 16000]. The input time step t is processed by the sine position encoding module (SinPosEmb) and the multi-layer linear network (Linear), and the time step feature is mapped to the time coding information of [B, 256, 1]. The input score features and expressiveness parameters are input into the dilated convolution module through a stacking operation (stack). The dilated convolution module consists of 20 layers of dilated convolution (dilated_conv) and gated activation units (gatedactivation). Each layer of dilated convolution realizes multi-scale time feature extraction and matching through different dilation rates to obtain a feature matrix of [B, 512, 16000]. The features output by each layer of dilated convolution are accumulated through addition operations to achieve layer-by-layer feature fusion and form a score expressiveness feature matrix of [B, 320, 16000].

[0053] Among them, the gate activation unit , using tanh Functions and sigmoid The function multiplies, l is the length of the input channel, the first half of the input channel is used as the tanh input, and the second half is used as sigmoid enter. tanh The function is the hyperbolic tangent function, tanh and sigmoid All of them are saturation functions. tanh Functions and sigmoid Function, which can solve the gradient disappearance problem.

[0054] The final output of the dilated convolution module is processed by the second convolutional network (Conv1d) to generate a feature matrix of [B, 256, 16000]. Multiple feature matrices are summed to achieve the final fusion and dimension integration of features. Finally, the fused features are further processed through the output channel to generate vector velocity.

[0055] In this embodiment, training the streaming matching network includes: Use music score data and singing data to build a training set; Extracting music score feature parameters including notes, singing style and note duration from the music score data, and extracting expressive parameters including energy, pitch and tension from the singing voice data; The music score feature parameters, expressiveness parameters, phoneme posterior probability maps and timbre features are used as training labels, and the training set data is used to minimize the objective function. Train a streaming matching network.

[0056] As an exemplary explanation, for energy extraction, the root mean square of speech data is directly calculated by framing. For example, the speech signal is a discrete signal x[n], and the length of each frame after framing is N. Then the root mean square energy calculation formula of the kth frame is: .in, is the RMS energy of the kth frame, For the k The speech signal sample value of the frame, N is the number of samples per frame, n is the sample index.

[0057] For pitch extraction, a pitch extraction neural network is used to extract pitch directly from the audio by inputting speech.

[0058] For the extraction of tension, according to the formula Can be obtained, among which r is a ratio whose calculation formula is for , where is a full harmonic, is a semi-harmonic, RMS is the root mean square of the data.

[0059] In this embodiment, the speech mel spectrum is generated according to the multi-dimensional singing voice expressiveness sequence, the phoneme posterior probability map and the timbre characteristics, including: The phoneme posterior probability map is used as the query matrix Q, and the multi-dimensional singing voice expressiveness sequence is used as the key-value matrix K and V. The features of the multi-dimensional singing voice expressiveness sequence and the phoneme posterior probability map are fused through the multi-head attention mechanism to obtain the multi-attention fusion feature, which is expressed as follows:

[0060] =softmax ( * / )* ,

[0061] in, Represents a splicing operation, Represents the attention head j Attention calculation, is the output weight matrix in the multi-head attention mechanism, express The dimension of express Take the square root of the dimension to prevent the formula value from being too large; Generate speech mel-spectrogram based on multi-attention fusion features, phoneme posterior probability map and timbre features.

[0062] In this embodiment, a speech mel spectrum is generated based on the multi-attention fusion features, the phoneme posterior probability map and the timbre features, including: After dimensional integration of the multi-attention fusion features and the phoneme posterior probability map, they are added to the timbre features to obtain the fused input feature matrix; The input feature matrix is ​​processed by four DITBlocks in turn to obtain a high-dimensional feature matrix, where each DITBlock calculates the scaling parameters through a layer perceptron. , With bias parameters , and perform two layers of AdaLN operations on the input feature matrix to perform feature adjustment according to the following formula:

[0063]

[0064] In the formula, x is the input feature matrix; The high-dimensional feature matrix is ​​compressed through a one-dimensional convolution operation to generate a speech mel spectrum.

[0065] In this embodiment, each DITBlock calculates the scaling parameter and the bias parameter through the layer perceptron according to the introduced conditional matrix; wherein the conditional matrix of the first DITBlock is the time step information matrix, which is used to align the input feature matrix with the time step information, and the conditional matrices of the last three DITBlocks are the timbre feature matrix, which is used to adaptively adjust the timbre features of the input feature matrix; Among them, the steps of obtaining the time step information matrix include: Extract time step information from the phoneme posterior probability map and construct a time step information matrix; The timbre feature parameters are extracted from the timbre features and a timbre feature matrix is ​​constructed.

[0066] As an exemplary illustration, Figure 4 As shown, Figure 4 The principle diagram of generating beautified singing voice according to the multi-dimensional singing voice expressiveness sequence, phoneme posterior probability map and timbre features provided by the embodiment of the present invention, in the specific implementation process, the phoneme posterior probability map and the multi-dimensional expressiveness sequence are input into the expressiveness multi-head attention mechanism, and the multi-head attention mechanism has a total of four heads, namely, singing head, energy head, tension head and pitch head. In the process of generating speech mel spectrum, a streaming matching decoder based on the DIT architecture is used, such as Figure 5 As shown, Figure 5 The schematic diagram of the streaming matching decoder based on the DIT architecture provided in the embodiment of the present invention is that the timbre feature ([B, 160, 1]) is processed by the Linear layer to generate an intermediate feature matrix with a dimension of [B, 256, 1]. The phoneme posterior probability map and the multi-attention fusion feature are concatenated into an input matrix ([B, 160, 16000]), and the features are extracted by the Conv1d layer, and the output matrix dimension is [B, 256, 16000]. The time step information ([B, 1]) is encoded by SinPosEmb (sine position encoding), and the output matrix dimension is [B, 256, 1].

[0067] After dimension integration, the input matrix and the timbre representation are added and input into four DITBlocks. The structure of each DITBlock is the same. First, the conditional matrix is ​​passed through the multi-layer perceptron MLP to generate six parameters. , , used for AdaLN operation in DIT network, for The AdaLN calculation uses for The AdaLN calculation uses The DITBlock can embed the information of the condition matrix into the input matrix. In DIT streaming matching, the condition matrix of the first DITBlock is the time step t , used to embed the time step information, and the conditional matrix of the following three consecutive DITBlocks is the timbre representation, which is used to embed the timbre information and enhance the timbre similarity.

[0068] The vocoder selected is the NSF-HIFIGAN vocoder that supports pitch input. Compared with ordinary vocoders, it has more information on the pitch dimension and is more suitable for singing. The NSF-HIFIGAN vocoder synthesizes highly expressive singing voices that include pitch, singing, energy, and tension based on the input Mel spectrum and combined with the pitch parameter sequence.

[0069] As an exemplary illustration, Figure 6 As shown, Figure 6 The pitch graph of the input singing voice provided in the embodiment of the present invention, in the specific implementation process, first accepts the singing voice input and the song score input, and extracts the PPG, timbre representation and note sequence therein. Taking a score sung by a user as an example, the notes are C4, D4, E4, G4, Am4, F4, C4 and G4, and the duration of each note is 0.7 seconds. This melody starts from C4, gradually rises to the high point G4, and then drops slightly. The overall melody line is peaceful and warm. The singing content input by the user is "You ask me how deep I love you". Under the premise of not changing the singing content, by analyzing the PPG and the note sequence, it is input into the singing expression generator to generate a multi-dimensional expression sequence including pitch, singing, energy, and tension. In the specific operation, the pitch is dynamically predicted. At the beginning of the song, an upward glissando is predicted to express the emotion of the song; at the climax of the melody (G4), a vibrato is predicted to show a high and tense emotional expression; at the end of the song, a downward glissando is predicted to create a warm and soft emotional ending. At the same time, the singing voice is carefully modeled, and the chest pronunciation is more inclined in the bass segment to show the heaviness; the head pronunciation is more inclined in the high-pitched segment to show the singer's difficult singing skills. In addition, the energy output is dynamically adjusted according to different parts of the melody. The energy of the high-pitched part is enhanced to reflect the strength, while the energy of the bass part is appropriately reduced to appear soft. For the tension parameter, the system increases the tension in the emotionally deep part to further express a deeper emotional expression. Then, the streaming matching network and the attention mechanism are used to model the temporal relationship between the expressiveness parameter and the PPG to ensure the consistency of the expressiveness parameter and the singing content in the temporal dimension. This modeling method enables the system to automatically adjust the expressiveness, so that the generated singing voice is not only accurate in pitch, but also more natural and smooth in emotional expression.

[0070] like Figure 6 As shown in the pitch graph of the input singing voice, the melody line is relatively flat and lacks technical expression. Figure 7 As shown, Figure 7 The pitch graph of the song after outputting the beautification provided by the embodiment of the present invention is obtained from Figure 7 It can be seen that the pitch curve is obviously more layered, and the singing is more vivid and expressive by adding singing techniques such as vibrato and glissando. The high-quality open source dataset OpenCpop is used as the training and test set, and the average length of each score is about 3 minutes. The test results show that the beautified singing generated by the system performs well in pitch accuracy, with no more than 5% of the out-of-tune parts, and these out-of-tune parts are mainly the transition parts generated by the system, rather than prediction errors.

[0071] In summary, the present invention generates a beautified singing voice with accurate pitch and rich expressiveness by dynamically predicting expressiveness parameters such as pitch, singing voice, energy and tension, and combining the streaming matching network to model the time relationship without changing the singing content. This method effectively solves the limitation of traditional singing voice beautification technology that only focuses on pitch correction and lacks expressiveness, and provides users with singing output with more emotional expression and professional singing skills.

[0072] Example 3 like Figure 8 As shown, this embodiment proposes a singing beautification system based on streaming matching, which is applied to the singing beautification method based on streaming matching as described in the above embodiment, including: an acquisition module 100, an extraction module 200, a first generation module 300, a second generation module 400 and a conversion module 500.

[0073] Among them, the acquisition module 100 is used to obtain singing data and music score data; the extraction module 200 is used to extract timbre features and phoneme posterior probability maps from the singing data; the first generation module 300 is used to generate a multi-dimensional singing expressiveness sequence based on the music score data and the phoneme posterior probability map; the second generation module 400 is used to generate a speech mel-spectrogram based on the multi-dimensional singing expressiveness sequence, the phoneme posterior probability map and the timbre features; the conversion module 500 is used to input the speech mel-spectrogram into the vocoder for conversion processing to obtain a beautified singing voice.

[0074] It should be noted that the above explanation of the embodiment of the singing voice beautification method based on streaming matching is also applicable to the singing voice beautification system based on streaming matching in this embodiment, which will not be repeated here.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0076] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0077] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0078] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0079] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0080] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A singing voice beautification method based on streaming matching, characterized in that: include: Acquire singing data and music score data; Extracting timbre features and phoneme posterior probability maps from singing voice data; Generate a multi-dimensional singing expressiveness sequence based on the score data and the phoneme posterior probability map; Generate speech mel spectrogram based on multi-dimensional singing expressiveness sequence, phoneme posterior probability map and timbre characteristics; The speech Mel spectrum is input into the vocoder for conversion processing to obtain the beautified singing voice.

2. The method for beautifying singing voice based on streaming matching according to claim 1, characterized in that: The multi-dimensional singing expressiveness sequence includes pitch frame level sequence, singing frame level sequence, energy frame level sequence and tension frame level sequence.

3. The method for beautifying singing voice based on streaming matching according to claim 2, characterized in that: According to the score data and the phoneme posterior probability map, a multi-dimensional singing expression sequence is generated, including: Build and train a streaming matching network; Perform dimension expansion processing on the phoneme posterior probability map; Based on the dimensionally expanded phoneme posterior probability map and music score data, the trained streaming matching network is used to perform feature mapping to generate a pitch frame-level sequence. According to the score data and pitch frame-level sequence, the trained streaming matching network is used to perform feature mapping to generate the singing frame-level sequence; According to the score data, pitch frame-level sequence and phoneme posterior probability map, the trained streaming matching network is used to perform feature mapping to generate energy frame-level sequence; According to the score data, pitch frame-level sequence, energy frame-level sequence prediction and phoneme posterior probability map, the trained streaming matching network is used for feature mapping to generate the target tension frame-level sequence.

4. The method for beautifying singing voice based on streaming matching according to claim 3, characterized in that: Construct a streaming matching network and use it for feature mapping, including: The mapping relationship between modeling input distribution and target distribution is expressed as follows: in, Indicates j The expressiveness parameter is in the time step t The distribution of time, Indicates j The target distribution of expressive parameters, Indicates the j The input distribution of the expressive parameters; By integral solution, the input distribution Mapping to target distribution ,Will Output as the final vocal expressive sequence , whose expression is as follows: , in: In the formula, For the streaming matching network in learning j The vector velocity obtained in the expressive parameter stage, For the time step t The state distribution of Indicates that the streaming matching network is learning j Model parameters for expressiveness parameters, is the input score data, is the phoneme posterior probability map, It is the phoneme posterior probability map after dimension expansion.

5. The method for beautifying singing voice based on streaming matching according to claim 4, characterized in that: When the streaming matching network is used for feature mapping, the method also includes predicting a vector velocity of the streaming matching network, including: Get phoneme posterior probability maps, time steps, and expressiveness parameters in real time; Extracting music score features including rhythm, style and range from music score data; The time step is position-encoded by a sinusoidal position encoding module to generate position encoding information; The music score features, phoneme posterior probability map and position encoding information are input as input features into the dilated convolution module, where the dilated convolution module consists of multiple dilated convolution blocks, each of which performs the following operations: Perform dilated convolution operation on the input features; The input features are activated through the tanh activation function and the sigmoid activation function; The output of the dilated convolutional block is added to the input features through a residual connection; The output results of all dilated convolution blocks are accumulated and dimensionally integrated to generate the final vector velocity.

6. The method for beautifying singing voice based on streaming matching according to claim 4, characterized in that: Training a streaming matching network, including: Use music score data and singing data to build a training set; Extracting music score feature parameters including notes, singing style and note duration from the music score data, and extracting expressive parameters including energy, pitch and tension from the singing voice data; The music score feature parameters, expressiveness parameters, phoneme posterior probability maps and timbre features are used as training labels, and the training set data is used to minimize the objective function. Train a streaming matching network.

7. The method for beautifying singing voice based on streaming matching according to claim 4, characterized in that: Generate speech mel spectrogram based on multi-dimensional singing expressiveness sequence, phoneme posterior probability map and timbre characteristics, including: The phoneme posterior probability map is used as the query matrix Q, and the multi-dimensional singing voice expressiveness sequence is used as the key-value matrix K and V. The features of the multi-dimensional singing voice expressiveness sequence and the phoneme posterior probability map are fused through the multi-head attention mechanism to obtain the multi-attention fusion feature, which is expressed as follows: =softmax ( * / )* , in, Represents a splicing operation, Represents the attention head j Attention calculation, is the output weight matrix in the multi-head attention mechanism, express Dimensions; Generate speech mel-spectrogram based on multi-attention fusion features, phoneme posterior probability map and timbre features.

8. The method for beautifying singing voice based on streaming matching according to claim 7, characterized in that: Generate speech mel spectrum based on multi-attention fusion features, phoneme posterior probability map and timbre features, including: After dimensional integration of the multi-attention fusion features and the phoneme posterior probability map, they are added to the timbre features to obtain the fused input feature matrix; The input feature matrix is ​​processed by four DITBlocks in turn to obtain a high-dimensional feature matrix, where each DITBlock calculates the scaling parameters through a layer perceptron. , With bias parameters , and perform two layers of AdaLN operations on the input feature matrix to perform feature adjustment according to the following formula: In the formula, x is the input feature matrix; The high-dimensional feature matrix is ​​compressed through a one-dimensional convolution operation to generate a speech mel spectrum.

9. The method for beautifying singing voice based on streaming matching according to claim 8, characterized in that: Each DITBlock calculates the scaling parameters and bias parameters through the layer perceptron according to the introduced conditional matrix; the conditional matrix of the first DITBlock is the time step information matrix, which is used to align the input feature matrix with the time step information, and the conditional matrices of the last three DITBlocks are the timbre feature matrix, which is used to adaptively adjust the timbre features of the input feature matrix; Among them, the steps of obtaining the time step information matrix include: Extract time step information from the phoneme posterior probability map and construct a time step information matrix; The timbre feature parameters are extracted from the timbre features and a timbre feature matrix is ​​constructed.

10. A singing voice beautification system based on streaming matching, characterized in that: include: An acquisition module is used to acquire singing data and music score data; An extraction module, used for extracting timbre features and phoneme posterior probability maps from singing data; The first generation module is used to generate a multi-dimensional singing voice expressiveness sequence according to the music score data and the phoneme posterior probability map; The second generation module is used to generate speech mel spectrum according to the multi-dimensional singing expressiveness sequence, the phoneme posterior probability map and the timbre characteristics; The conversion module is used to input the speech Mel spectrum into the vocoder for conversion processing to obtain the beautified singing voice.

Images