Song synthesis method and device, computer device and storage medium

Abstract

The application discloses a song synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a to-be-processed music score; performing music score vector coding on the to-be-processed music score to obtain a music score hidden vector; performing recursive decoding on the music score hidden vector by using an SU-net network constructed based on a stripe pooling layer to obtain a target frequency spectrum; and performing conversion synthesis on the target frequency spectrum to obtain a target synthesized song. The method uses the SU-net network constructed based on the stripe pooling layer to simplify the modeling method of song synthesis, converts the song synthesis into a translation task between a music score coding vector and a frequency spectrum, makes the target frequency spectrum converted into the corresponding target synthesized song by a vocoder more in line with the actual situation, and improves the song synthesis effect. The stripe pooling layer can enhance the extraction of stripe structure information in the frequency spectrum, expand the frequency spectrum information, enhance the pitch information in the song, and guarantee the song synthesis effect.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a method, apparatus, computer device, and storage medium for synthesizing singing voices. Background Technology

[0002] Vocal synthesis is a generative task similar to speech synthesis. Speech synthesis synthesizes audio from text, while vocal synthesis synthesizes audio from musical score. Specifically, vocal synthesis involves modeling and synthesizing multi-dimensional information about singing, including but not limited to lyrics, pitch, duration, and the singer's timbre and singing techniques. Speech synthesis simply synthesizes audio from text, while the musical score in vocal synthesis contains much more information than text, such as lyrics, pitch, and duration. This makes vocal synthesis more complex; the same lyrics can correspond to multiple audio representations, while the speech representation of the same text remains relatively consistent. Existing vocal synthesis processes, similar to speech synthesis, collect fewer parameters, resulting in coarse and poor-quality synthesized vocal results. Summary of the Invention

[0003] This invention provides a method, apparatus, computer device, and storage medium for synthesizing singing voices, in order to solve the problem of poor singing voice synthesis effects in existing methods.

[0004] A method for synthesizing singing voices, comprising:

[0005] Obtain the sheet music to be processed;

[0006] The musical score to be processed is encoded into a musical score vector to obtain the latent vector of the musical score;

[0007] The SU-net network, based on striped pooling layers, is used to recursively decode the latent vectors of the musical score to obtain the target spectrum.

[0008] The target spectrum is converted and synthesized to obtain the target synthesized vocals.

[0009] A singing voice synthesis device, comprising:

[0010] The unprocessed sheet music acquisition module is used to acquire unprocessed sheet music.

[0011] The music score latent vector acquisition module is used to encode the music score to be processed into music score vectors and obtain the music score latent vectors.

[0012] The target spectrum acquisition module uses a SU-net network built on striped pooling layers to recursively decode the latent vectors of the musical score and obtain the target spectrum.

[0013] The target synthesized vocal acquisition module is used to convert and synthesize the target spectrum to obtain the target synthesized vocal.

[0014] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described singing voice synthesis method.

[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described singing voice synthesis method.

[0016] The aforementioned singing synthesis method, apparatus, computer, and storage medium encode the musical score into a musical score vector to obtain the latent musical score vector, thereby converting the musical score into a vector form that can be recognized by the SU-net network. This helps ensure the feasibility of singing synthesis. The SU-net network architecture simplifies the modeling method for singing synthesis, transforming it into a translation task between the musical score encoded vector and the spectrum. This makes the output target spectrum more accurate, and the target spectrum, when converted by a vocoder, more realistic in synthesizing the corresponding target singing voice, thus improving the singing synthesis effect. Furthermore, the SU-net network, built based on fringe pooling layers, enhances the extraction of fringe structure information in the spectrum, expands the spectral information, and enhances the pitch information in the singing voice, ensuring the singing synthesis effect. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an application environment for a singing voice synthesis method according to an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of a singing voice synthesis method according to an embodiment of the present invention;

[0020] Figure 3 This is another flowchart of a singing voice synthesis method in one embodiment of the present invention;

[0021] Figure 4 This is another flowchart of a singing voice synthesis method in one embodiment of the present invention;

[0022] Figure 5 This is another flowchart of a singing voice synthesis method in one embodiment of the present invention;

[0023] Figure 6 This is another flowchart of a singing voice synthesis method in one embodiment of the present invention;

[0024] Figure 7 This is another flowchart of a singing voice synthesis method in one embodiment of the present invention;

[0025] Figure 8 This is another flowchart of a singing voice synthesis method in one embodiment of the present invention;

[0026] Figure 9 This is a schematic diagram of a singing voice synthesis device according to an embodiment of the present invention;

[0027] Figure 10 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The singing voice synthesis method provided in this embodiment of the invention can be applied to, for example... Figure 1 The application environment is shown. Specifically, this singing voice synthesis method is applied in a singing voice synthesis system, which includes, for example, […]. Figure 1 The diagram shows a client and server. The client and server communicate over a network to perform vocal synthesis. The client, also known as the user terminal, is the program that provides local services to the user, corresponding to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers.

[0030] In one embodiment, such as Figure 2 As shown, a method for synthesizing singing voices is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0031] S201: Obtain the sheet music to be processed;

[0032] S202: Encode the musical score into a musical score vector to obtain the latent vector of the musical score;

[0033] S203: The SU-net network based on striped pooling layers is used to recursively decode the latent vectors of the musical score to obtain the target spectrum;

[0034] S204: Convert and synthesize the target spectrum to obtain the target synthesized vocals.

[0035] The sheet music to be processed refers to the sheet music that needs to be processed. Generally, each sheet music to be processed includes information such as lyrics, duration, and notes. Specifically, the sheet music to be processed can be in various formats, including but not limited to MIDI format, as long as it can be machine-readable. The sheet music to be processed can be in Chinese or foreign languages, and its presentation is unrestricted.

[0036] As an example, in step S201, the server can obtain the sheet music to be processed, which is a sheet music that can be parsed and recognized by the machine, so as to ensure the feasibility of the subsequent singing synthesis process.

[0037] In this example, the server receives the original sheet music input by the user, determines the sheet music format corresponding to the original sheet music, and judges whether the sheet music format is a system-preset standard format. Here, the standard format can be understood as a format that the system can parse and recognize. If the sheet music format is a standard format, the original sheet music is determined as the sheet music to be processed. If the sheet music format is not a standard format, a format mismatch prompt message is output. Alternatively, based on the sheet music format and the standard format, the corresponding format conversion logic is determined, and the original sheet music is converted using the format conversion logic to obtain the sheet music to be processed corresponding to the standard format, so as to ensure the feasibility of the sheet music to be processed for subsequent vocal synthesis.

[0038] Among them, the latent vector of the musical score refers to the latent vector related to the musical score information formed by encoding the musical score vector.

[0039] As an example, in step S202, after obtaining the score to be processed, the server performs score vector encoding on the score to obtain the latent score vector. In this example, the server first performs content parsing on the score to be processed. This content parsing includes reading the score information such as lyrics, duration, and notes, specifically parsing out the corresponding lyrics, the notes corresponding to each lyric, and the start and end information (i.e., duration) of each note. Then, feature extraction and vector encoding are performed on the score information obtained from the content parsing to obtain the latent score vector. In essence, performing score vector encoding on the score to be processed converts the score information into a latent score vector represented by a vector, making it input into the SU-net network for processing and ensuring the feasibility of processing.

[0040] The SU-net network is a U-shaped semantic segmentation network built upon fringe pooling layers. The fringe pooling layer, located within the SU-net network, enhances the fringe information by employing fringe pooling to improve the SU-net network's learning of fringe information.

[0041] As an example, the SU-net network consists of three convolutional layers, striped pooling layers, and three transposed convolutional layers. The convolutional layers are composed of ConvDown layers (compression layers) and skip connections, and the transposed convolutional layers are composed of ConvUp layers (amplification layers). A striped pooling layer is added after each ConvDown layer to expand the data by rows or columns, improve the information feedback from the data, and ensure the accuracy of the output results.

[0042] Recursive decoding refers to the decoding process of processing the input information of the current frame and the output information of the previous frame to obtain the output result of the current frame.

[0043] As an example, in step S203, the server can use a SU-net network built on a striped pooling layer to recursively decode the latent vector of the musical score and obtain the target spectrum. Specifically, the latent vector of the musical score of the current frame and the output information of the previous frame output by the SU-net network built on a striped pooling layer can be fused together and determined as the input information of the current frame. This input is then fed into the SU-net network built on a striped pooling layer for processing to obtain the output information of the current frame, thereby achieving the effect of recursive decoding.

[0044] In one example, the output information of the SU-net network built based on striped pooling layers is the target spectrum. The recursive decoding process of the latent vector of the musical score by the server specifically includes: First, obtaining the latent vector of the musical score of the current frame; second, obtaining and processing the target spectrum of the previous frame to obtain the latent vector corresponding to the target spectrum of the previous frame; third, processing the latent vector of the musical score of the current frame and the latent vector corresponding to the target spectrum of the previous frame to obtain the input information of the current frame; finally, inputting the obtained input information of the current frame into the SU-net network built based on striped pooling layers to complete the recursive decoding of the latent vector of the musical score and output the target spectrum corresponding to the current frame.

[0045] For example, the SU-net network includes three ConvDown layers (compression layers), skip connection layers, and three ConvUp layers (amplification layers). After each ConvDown layer (compression layer), a striped pooling layer is used to expand the spectral information of the input striped pooling layer with striped structure information, thereby enhancing the amount of information represented by its output result and helping to ensure the synthesis effect of the subsequent target synthesized singing voice.

[0046] In this example, the SU-net network architecture simplifies the modeling method for vocal synthesis, transforming vocal synthesis into a translation task between musical score encoding vectors and the spectrum, making the output target spectrum more accurate. Moreover, by using the SU-net network built on striped pooling layers, the extraction of striped structure information in the spectrum can be enhanced, expanding the spectral information and enhancing the pitch information in the vocals, thus ensuring the vocal synthesis effect.

[0047] As an example, in step S204, the server converts and synthesizes the target spectrum to obtain the target synthesized vocal track. In this example, the server uses a vocoder to convert and synthesize the target spectrum of the current frame into the corresponding audio. After converting and synthesizing the target spectrum of the entire musical score, the corresponding audio file is formed, which is the target synthesized vocal track. Understandably, the server may use, but is not limited to, vocoders such as Griffin-Lim, Hifi-GAN, and PWG to convert and synthesize the target spectrum to obtain the target synthesized vocal track.

[0048] In the singing synthesis method provided in this embodiment, the music score to be processed is first encoded into a music score vector to obtain the latent music score vector, so as to convert the music score into a vector form that can be recognized by the SU-net network, which helps to ensure the feasibility of singing synthesis. The SU-net network architecture simplifies the modeling method of singing synthesis, transforming singing synthesis into a translation task between the music score encoded vector and the spectrum, making the output target spectrum more accurate, and making the target spectrum more realistic in synthesizing the corresponding target singing voice through the vocoder conversion, thus improving the singing synthesis effect. Moreover, the SU-net network based on the striped pooling layer is used to enhance the extraction of stripe structure information in the spectrum, expand the spectrum information, enhance the pitch information in the singing voice, and ensure the singing synthesis effect.

[0049] In one embodiment, such as Figure 3 As shown, step S202, which involves encoding the musical score into a musical score vector to obtain the latent vector, specifically includes the following steps:

[0050] S301: Extract features from the musical score to be processed, and obtain lyric features, note features and duration features;

[0051] S302: Perform feature encoding on lyric features, note features, and duration features to obtain the latent vector of the musical score.

[0052] As an example, in step S301, the server first performs content parsing on the score to be processed to obtain lyrics information, duration information, and note information; then, it performs feature extraction on the lyrics information, note information, and duration information respectively to obtain lyrics features, note features, and duration features respectively.

[0053] As an example, in step S302, the server performs feature encoding on the lyrics features, note features, and duration features to obtain the latent music score vector. The feature encoding process is as follows: First, the lyrics features and note features are respectively combined with the duration features for feature encoding, resulting in lyrics vectors and note vectors, respectively. That is, feature encoding is performed based on the singing features and duration features to obtain the lyrics vector, and feature encoding is performed on the note features and duration features to obtain the note vector. Second, the server concatenates the lyrics vector and note vector to obtain the concatenated vector. Third, the server inputs the concatenated vector into the music score encoder, and the output of the music score encoder is determined as the latent music score vector.

[0054] In the singing synthesis method provided in this embodiment, the score to be processed is encoded into a score vector, which converts the score information of the score to be processed into a latent score vector in vector form, so that it can be input into the SU-net network for processing, thus ensuring the feasibility of the processing method.

[0055] In one embodiment, such as Figure 4 As shown, step S302, which involves feature encoding of the lyrics features, note features, and duration features to obtain the latent vector of the musical score, specifically includes the following steps:

[0056] S401: Obtain lyrics vectors by extending vector encoding based on lyrics features and duration features;

[0057] S402: Obtain note vectors by extending vector encoding based on note features and duration features;

[0058] S403: Obtain the hidden vector of the musical score based on the lyric vector and note vector.

[0059] As an example, in step S401, the server performs vector encoding expansion based on lyric features and duration features to obtain lyric vectors. Specifically, each lyric feature in the score to be processed is vector encoded to obtain a vector corresponding to each lyric feature. At the same time, during the vector encoding process of lyric features, the duration feature corresponding to each lyric feature is considered, and the vector corresponding to each lyric feature is expanded in terms of duration. Finally, the lyric vector of the score to be processed is obtained.

[0060] In one embodiment, the server encodes each lyric feature in the score to be processed, obtaining a vector corresponding to each lyric feature. For example, the vector corresponding to a certain lyric feature is [2.0, 3.1, 4.2, 0.6], which is a multi-dimensional vector. Understandably, the target synthesized singing voice is a temporal audio signal. In computer storage, audio signals are typically stored using one-dimensional data. If the sampling frequency of the audio signal is 16000Hz, then one second of audio signal consists of 16000 data points. When using the spectrum as the model output, the audio signal will be framed in terms of temporal features, and then the short-time Fourier transform of each frame will be calculated to obtain the spectral representation of each frame. Here, a frame is usually 100ms or shorter, such as 50ms; while a lyric feature may typically last 0.5s or longer during performance, therefore, the lyric feature needs to be... To expand the spectrum corresponding to the lyrics of the musical score in terms of time characteristics, the expansion is performed according to the duration features. In this embodiment, a copying method is used. For example, if the vector corresponding to a certain lyric feature is [2.0,3.1,4.2,0.6], which corresponds to 30 frames of spectrum in terms of duration features, then the vector [2.0,3.1,4.2,0.6] corresponding to the lyric feature is copied 30 times to complete the expansion in terms of duration features. The server performs the above duration feature expansion process on the vector corresponding to each lyric feature in the musical score to be processed, and finally obtains the lyric vector of the musical score to be processed.

[0061] As an example, in step S402, the server performs vector encoding extension based on note features and duration features to obtain note vectors. Specifically, each note feature in the score to be processed is vector encoded to obtain the vector corresponding to each note feature. At the same time, in the process of vector encoding the note features, the duration feature corresponding to each note feature is considered, and the vector corresponding to each note feature is extended in terms of duration. Finally, the note vector of the score to be processed is obtained.

[0062] In one embodiment, the server encodes each note feature in the musical score to be processed, obtaining a vector corresponding to each note feature. For example, the vector corresponding to a certain note feature is [2.6, 3.1, 4.2, 1.2], which is a multi-dimensional vector. Understandably, the target synthesized singing voice is a temporal audio signal. In computer storage, audio signals are typically stored using one-dimensional data. If the sampling frequency of the audio signal is 16000Hz, then one second of audio signal consists of 16000 data points. When using the spectrum as the model output, the audio signal will be framed in terms of temporal features, and then the short-time Fourier transform of each frame will be calculated to obtain the spectral representation of each frame. Here, the frame is usually 100ms or shorter, such as 50ms; while a note feature may typically last for 0.5s or longer during singing, therefore the note features need to be... To expand the spectrum corresponding to the notes in the musical score, the duration features need to be expanded. In this embodiment, a copying method is used. For example, if the vector corresponding to a certain note feature is [2.6,3.1,4.2,1.2], which corresponds to 20 frames of spectrum in terms of duration features, then the vector [2.6,3.1,4.2,1.2] corresponding to the note feature is copied 20 times to complete the expansion in terms of duration features. The server completes the above duration feature expansion process for the vector corresponding to each note feature in the musical score to be processed, and finally obtains the note vector of the musical score to be processed.

[0063] As an example, in step S403, the server obtains the latent musical score vector based on the lyrics vector and the note vector. Specifically, the lyrics vector and the note vector can be input into the encoder, and the encoder can be used to encode the lyrics vector and the note vector to obtain the latent musical score vector.

[0064] In the singing synthesis method provided in this embodiment, the lyric features are extended and encoded based on the duration features to obtain lyric vectors, and the note features are extended and encoded based on the duration features to obtain note vectors. This not only ensures the information content of the lyric vectors and note vectors, but also enables the lyric vectors and note vectors to achieve time alignment, making it feasible for the encoder to encode the lyric vectors and note vectors.

[0065] In one embodiment, such as Figure 5 As shown, step S403, which involves obtaining the latent vector of the musical score based on the lyrics vector and the note vector, specifically includes the following steps:

[0066] S501: Concatenate the lyrics vector and the note vector to obtain the first concatenated vector;

[0067] S502: The first spliced ​​vector is encoded using a music score encoder to obtain the hidden music score vector.

[0068] As an example, in step S501, the server concatenates the lyrics vector and note vector to obtain a first concatenated vector. Specifically, this includes concatenating the lyrics vector and note vector corresponding to the same frame based on the lyrics vector corresponding to each lyrics feature and the note vector corresponding to each note feature, to determine the first concatenated vector for the same frame. Understandably, since each lyrics vector and note vector parsed from the audio to be processed is time-related, meaning the lyrics vector and note vector of the same frame need to be time-aligned, concatenating the lyrics vector and note vector of the same frame allows the obtained first concatenated vector to comprehensively reflect the lyrics and note information corresponding to the same moment, helping to ensure the information content of the latent musical score vector obtained in the final encoding.

[0069] As an example, in step S502, the server uses a music score encoder to encode the first spliced ​​vector to obtain the music score latent vector. Specifically, this includes: inputting the first spliced ​​vector into the music score encoder in chronological order for encoding; processing the first spliced ​​vector of the current frame in the intermediate layer of the music score encoder; and using the latent vector output by the intermediate layer as the music score latent vector of the current frame.

[0070] In this example, the music score encoder uses a two-layer one-dimensional convolutional network structure. More layers of convolutional networks can also be used, or an LSTM temporal network layer and an attention-based transformer structure can be employed. In this embodiment, the experimental results of the two-layer one-dimensional convolutional network structure show that, compared to other network structures, it achieves better performance while reducing model parameters.

[0071] In the singing synthesis method provided in this embodiment, the server concatenates the lyrics vector and the note vector accordingly, making the first concatenated vector of the encoded output more accurate and more consistent with the lyrics and note features expressed in the musical score to be processed, which is in line with the actual situation, and makes the latent vector of the musical score output by the encoder for the current frame more accurate.

[0072] In one embodiment, such as Figure 6 As shown, step S203, which involves using a SU-net network built on striped pooling layers to recursively decode the latent vectors of the musical score and obtain the target spectrum, specifically includes the following steps:

[0073] S601: Concatenate the latent musical score vector corresponding to the current frame and the latent spectrum vector corresponding to the previous frame to obtain the second concatenated vector corresponding to the current frame.

[0074] S602: The SU-net network based on striped pooling layers is used to process the second splicing vector corresponding to the current frame to obtain the target spectrum corresponding to the current frame.

[0075] The latent vector corresponding to the previous frame is the latent vector obtained by the spectrum encoder processing the target spectrum output from the previous frame.

[0076] As an example, in step S601, the server concatenates the latent musical score vector corresponding to the current frame and the latent spectral vector corresponding to the previous frame to obtain the second concatenated vector corresponding to the current frame. Specifically, the server obtains the target spectrum of the previous frame output by the SU-net network and inputs it into the spectrum encoder; then, it obtains the latent vector of the target spectrum of the previous frame in the intermediate layer of the spectrum encoder, which is the latent spectral vector corresponding to the previous frame; the latent musical score vector corresponding to the current frame and the latent spectral vector corresponding to the previous frame are concatenated, and the concatenation result is used as the second concatenated vector corresponding to the current frame.

[0077] In this example, the target spectrum output in the previous frame is the target spectrum at time t-1. Its latent vector, obtained through the spectrum encoder, is represented by audio features, equivalent to the audio features at time t-1, rather than musical notation features. The spectrum encoder can be an encoder constructed using a four-layer two-dimensional convolutional network structure. This four-layer convolutional network structure can be adjusted, changing the number of layers and the number of kernels in each layer; the convolutional network can also be replaced, for example, with a transformer structure. The specific effect after modification requires parameter optimization and training. This embodiment uses four two-dimensional convolutional layers with certain parameter adjustments and optimizations, achieving superior results without significantly increasing the number of parameters.

[0078] As an example, in step S602, the server inputs the second splicing vector corresponding to the current frame into the SU-net network and outputs the target spectrum corresponding to the current frame.

[0079] In this example, the SU-net network consists of three convolutional layers, striped pooling layers, and three transposed convolutional layers. Understandably, the SU-net network includes three ConvDown layers (compression layers), skip connections, and three ConvUp layers (amplification layers). The three convolutional layers are composed of three ConvDown layers (compression layers). A striped pooling layer is added after each ConvDown layer (compression layer) to expand the data in rows or columns, improving the information feedback from the data and ensuring the accuracy of the output results. Skip connections are established between the corresponding transposed convolutional layers after the striped pooling layers to input the high-dimensional features from the lower layers into the higher layers. The three transposed convolutional layers are composed of three ConvUp layers (amplification layers). The specific working principle of ConvDown and ConvUp layers is as follows: During the downward compression process of ConvDown layer, the intermediate results of each layer are continuously compressed at a ratio of 1 / 2. During the upward expansion process of ConvUp layer, the intermediate results of each layer are enlarged at a ratio of 2. The intermediate results of the lower layer and the intermediate results of the higher layer of the same size pass information through cross-layer connection. That is, the intermediate results are not only input to the adjacent next layer, but also passed to the corresponding layer that is separated by a certain distance.

[0080] In the singing synthesis method provided in this embodiment, since music is temporally related and there is a contextual relationship between the previous frame and the current frame, a second splicing vector is obtained through splicing processing. This second splicing vector is then used as the input to the SU-net network, which can effectively improve the prediction of spectral features and reduce the difficulty of translating from musical score to spectrum.

[0081] In one embodiment, such as Figure 7 As shown, step S602, which involves using a SU-net network constructed based on striped pooling layers to process the second concatenation vector corresponding to the current frame and obtain the target spectrum corresponding to the current frame, specifically includes the following steps:

[0082] S701: The SU-net network, based on striped pooling layers, is used to split and expand the second splicing vector of the current frame to obtain the target row spectral features and the target column spectral features.

[0083] S702: Perform convolution and activation processing on the target row spectral features and target column spectral features to obtain the activation parameter values ​​of the current frame;

[0084] S703: Perform a dot product between the second concatenation vector of the current frame and the activation parameter value to obtain the target spectrum corresponding to the current frame.

[0085] As an example, in step S701, the striped pooling layer splits the second splicing vector corresponding to the current frame into row vectors and column vectors respectively; then, it performs one-dimensional convolution on the row vectors and column vectors respectively to obtain the row vector convolution result and the column vector convolution result; finally, it expands the obtained row vector convolution result according to the number of rows to obtain the target row spectral feature, and expands the obtained column vector convolution result according to the number of columns to obtain the target column spectral feature.

[0086] As an example, in step S702, the striped pooling layer concatenates the target row spectral features and the target column spectral features, performs a two-dimensional convolution on the concatenation result, and then performs activation processing on the two-dimensional convolution result to obtain activation parameter values.

[0087] As an example, in step S703, the striped pooling layer obtains the activation parameter value, performs a dot product between the activation parameter value and the second concatenation vector corresponding to the current frame, and obtains the target spectrum corresponding to the current frame.

[0088] In the singing synthesis method provided in this embodiment, the striped pooling layer splits, expands, and splices the second splicing vector corresponding to the current frame, performs convolution and activation processing on the splicing result, and then performs a dot product between the processing result and the second splicing vector corresponding to the current frame to obtain the target spectrum corresponding to the current frame, making the target spectrum output by the SU-net network more accurate; moreover, by using the SU-net network built based on the striped pooling layer, the striped pooling layer enhances the extraction of stripe structure information in the spectrum, ensuring that the song synthesis effect is more in line with reality.

[0089] In one embodiment, such as Figure 8 As shown, step S701, which involves splitting and expanding the second concatenation vector of the current frame to obtain the target row spectral features and target column spectral features, specifically includes the following steps:

[0090] S801: Split the second concatenation vector of the current frame to obtain the original row spectral features and the original column spectral features;

[0091] S802: Based on the number of rows in the second concatenation vector, expand the original row spectrum features to obtain the target row spectrum features;

[0092] S803: Based on the number of columns in the second splicing vector, the original column spectrum features are expanded to obtain the target column spectrum features.

[0093] The original row spectral features are row vectors obtained by splitting the second concatenation vector of the current frame by rows in the striped pooling layer. The original column spectral features are column vectors obtained by splitting the second concatenation vector of the current frame by columns in the striped pooling layer.

[0094] As an example, in step S801, the striped pooling layer splits the second splicing vector corresponding to the current frame according to the row vector to obtain the original row spectral features, and splits the second splicing vector corresponding to the current frame according to the column vector to obtain the original column spectral features.

[0095] In this example, in the second splicing vector region of size H*W, the horizontal row vector of the second splicing vector, i.e., 1*W, is obtained, which is the original row spectrum feature. The vertical column vector of the second splicing vector, i.e., H*1, is obtained, which is the original column spectrum feature.

[0096] The target row spectral feature is a multidimensional matrix obtained by processing the original row spectral feature through the striped pooling layer.

[0097] As an example, in step S802, the striped pooling layer performs a one-dimensional convolution on the original row spectral features. Then, the one-dimensional convolution result of the original row spectral features is expanded according to the number of rows to obtain the target row spectral features.

[0098] In this example, the original row spectral feature is 1*W. A one-dimensional convolution is performed on 1*W to obtain the convolved vector. Then, the one-dimensional convolved vector is copied and expanded according to the row size H to obtain the corresponding result, which is an H*W dimensional matrix, that is, the target row spectral feature.

[0099] The target column spectrum feature is a multidimensional matrix obtained by processing the original column spectrum feature through the striped pooling layer.

[0100] As an example, in step S803, the striped pooling layer performs a one-dimensional convolution on the original column spectrum features. Subsequently, the one-dimensional convolution result of the original column spectrum features is expanded according to the number of columns to obtain the target column spectrum features.

[0101] In this example, the original column spectral feature is H*1. A one-dimensional convolution is performed on H*1 to obtain the convolved vector. Then, the one-dimensional convolved vector is copied and expanded according to the column size W to obtain the corresponding result, which is an H*W dimensional matrix, that is, the target column spectral feature.

[0102] In the singing synthesis method provided in this embodiment, the striped pooling layer splits and expands the second splicing vector of the current frame to obtain the target row spectral features and the target column spectral features. The striped pooling layer is then used to expand the spectral information, enhance the pitch information in the singing, and ensure the final singing synthesis effect.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0104] In one embodiment, a singing voice synthesis device is provided, which corresponds one-to-one with the singing voice synthesis method described in the above embodiments. For example... Figure 9 As shown, the singing voice synthesis device includes a music score acquisition module 901, a music score latent vector acquisition module 902, a target spectrum acquisition module 903, and a target synthesized singing voice acquisition module 904. Detailed descriptions of each functional module are as follows:

[0105] The unprocessed musical score acquisition module 901 is used to acquire unprocessed musical scores.

[0106] The music score latent vector acquisition module 902 is used to encode the music score to be processed into a music score vector and obtain the music score latent vector.

[0107] The target spectrum acquisition module 903 uses a SU-net network built based on striped pooling layers to recursively decode the latent vectors of the musical score and acquire the target spectrum.

[0108] The target synthesized vocal acquisition module 904 is used to convert and synthesize the target spectrum to obtain the target synthesized vocal.

[0109] In one embodiment, the music score latent vector acquisition module 902 includes:

[0110] The feature extraction submodule is used to extract features from the musical score to be processed, and to obtain lyric features, note features, and duration features.

[0111] The latent vector acquisition submodule is used to encode the features of lyrics, notes, and duration to obtain the latent vector of the musical score.

[0112] In one embodiment, the music score latent vector acquisition submodule includes:

[0113] The lyrics vector acquisition unit is used to perform vector encoding expansion based on lyrics features and duration features to obtain lyrics vectors;

[0114] The note vector acquisition unit is used to perform vector encoding expansion based on note features and duration features to obtain note vectors;

[0115] The music score latent vector acquisition unit is used to acquire the music score latent vector based on the lyrics vector and the note vector.

[0116] In one embodiment, the music score latent vector acquisition unit includes:

[0117] The first splicing vector acquisition sub-unit is used to splice the lyrics vector and the note vector to obtain the first splicing vector;

[0118] The sub-unit for obtaining the latent vector of the musical score uses a musical score encoder to encode the first concatenated vector to obtain the latent vector of the musical score.

[0119] In one embodiment, the target spectrum acquisition module 903 includes:

[0120] The second splicing vector acquisition submodule is used to splice the latent musical score vector corresponding to the current frame and the latent spectrum vector corresponding to the previous frame to obtain the second splicing vector corresponding to the current frame.

[0121] The target spectrum acquisition submodule uses a SU-net network based on striped pooling layers to process the second splicing vector corresponding to the current frame to obtain the target spectrum corresponding to the current frame.

[0122] The latent vector corresponding to the previous frame is the latent vector obtained by the spectrum encoder processing the target spectrum output from the previous frame.

[0123] In one embodiment, the target spectrum acquisition submodule includes:

[0124] The target spectral feature acquisition unit uses a SU-net network built based on striped pooling layers to split and expand the second splicing vector of the current frame to obtain the target row spectral features and the target column spectral features.

[0125] The activation parameter value acquisition unit is used to perform convolution and activation processing on the target row spectral features and the target column spectral features to obtain the activation parameter values ​​of the current frame.

[0126] The target spectrum acquisition unit is used to perform a dot product between the second concatenation vector of the current frame and the activation parameter value to obtain the target spectrum corresponding to the current frame.

[0127] In one embodiment, the target spectral feature acquisition unit includes:

[0128] The original spectral feature acquisition subunit is used to split the second splicing vector of the current frame to obtain the original row spectral features and the original column spectral features;

[0129] The target row spectrum feature acquisition subunit is used to expand the original row spectrum features based on the number of rows in the second concatenation vector to obtain the target row spectrum features;

[0130] The target column spectrum feature acquisition subunit expands the original column spectrum features based on the number of columns in the second concatenation vector to obtain the target column spectrum features.

[0131] Specific limitations regarding the vocal synthesis device can be found in the limitations of the vocal synthesis method described above, and will not be repeated here. Each module in the aforementioned vocal synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0132] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data used or generated during the execution of the vocal synthesis method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a vocal synthesis method.

[0133] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the singing voice synthesis method described in the above embodiment, for example... Figure 2 As shown in S201-S204, or Figures 3 to 8 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of the singing voice synthesis device, for example... Figure 9 The functions of the unprocessed musical score acquisition module 901, the musical score latent vector acquisition module 902, the target spectrum acquisition module 903, and the target synthesized singing voice acquisition module 904 shown are not described again here to avoid repetition.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the singing voice synthesis method described above, for example... Figure 2 As shown in S201-S204, or Figures 3 to 8 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the singing voice synthesis device, for example... Figure 9The functions of the unprocessed music score acquisition module 901, the music score latent vector acquisition module 902, the target spectrum acquisition module 903, and the target synthesized vocal acquisition module 904, as shown, will not be described again here to avoid repetition. The computer-readable storage medium may be non-volatile or volatile.

[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0137] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A singing voice synthesis method characterized by, The method comprises the following steps: acquiring a to-be-processed score; performing score vector coding on the to-be-processed score to acquire a score hidden vector; performing splicing processing on the score hidden vector corresponding to a current frame and a spectrum hidden vector corresponding to a previous frame to acquire a second splicing vector corresponding to the current frame; performing processing on the second splicing vector corresponding to the current frame by using an SU-net network constructed based on a stripe pooling layer to acquire a target spectrum corresponding to the current frame; wherein the stripe pooling layer is a structure in the SU-net network and has an enhancement effect on stripe information; the spectrum hidden vector corresponding to the previous frame is a hidden vector obtained by performing processing on a target spectrum output by a spectrum encoder for the previous frame; performing conversion and synthesis on the target spectrum to acquire a target synthesized singing voice.

2. The singing voice synthesis method according to claim 1, wherein The step of performing score vector coding on the to-be-processed score to acquire a score hidden vector comprises the following steps: performing feature extraction on the to-be-processed score to acquire a lyric feature, a note feature and a duration feature; performing feature coding on the lyric feature, the note feature and the duration feature to acquire a score hidden vector.

3. The singing voice synthesis method according to claim 2, wherein The step of performing feature coding on the lyric feature, the note feature and the duration feature to acquire a score hidden vector comprises the following steps: performing vector coding expansion based on the lyric feature and the duration feature to acquire a lyric vector; performing vector coding expansion based on the note feature and the duration feature to acquire a note vector; acquiring a score hidden vector according to the lyric vector and the note vector.

4. The singing voice synthesis method according to claim 3, wherein The step of acquiring a score hidden vector according to the lyric vector and the note vector comprises the following steps: performing splicing processing on the lyric vector and the note vector to acquire a first splicing vector; performing coding processing on the first splicing vector by using a score encoder to acquire a score hidden vector.

5. The singing voice synthesis method according to claim 1, wherein The step of performing processing on the second splicing vector corresponding to the current frame by using an SU-net network constructed based on a stripe pooling layer to acquire a target spectrum corresponding to the current frame comprises the following steps: performing split expansion on the second splicing vector of the current frame by using the SU-net network constructed based on the stripe pooling layer to acquire a target row spectrum feature and a target column spectrum feature; performing convolution and activation processing on the target row spectrum feature and the target column spectrum feature to acquire an activation parameter value of the current frame; performing dot multiplication on the second splicing vector of the current frame and the activation parameter value to acquire a target spectrum corresponding to the current frame.

6. The singing voice synthesis method according to claim 5, wherein The step of performing split expansion on the second splicing vector of the current frame to acquire a target row spectrum feature and a target column spectrum feature comprises the following steps: performing split on the second splicing vector of the current frame to acquire an original row spectrum feature and an original column spectrum feature; performing expansion on the original row spectrum feature according to the number of rows of the second splicing vector to acquire a target row spectrum feature; performing expansion on the original column spectrum feature according to the number of columns of the second splicing vector to acquire a target column spectrum feature.

7. A singing voice synthesis apparatus characterized by comprising: The method comprises the following steps: a to-be-processed score acquiring module is configured to acquire a to-be-processed score; a score hidden vector acquiring module is configured to perform score vector coding on the to-be-processed score to acquire a score hidden vector; The target spectrum acquisition module splices the music score hidden vector corresponding to the current frame and the spectrum hidden vector corresponding to the previous frame to obtain a second spliced vector corresponding to the current frame; an SU-net network constructed based on a stripe pooling layer is used to process the second spliced vector corresponding to the current frame to obtain a target spectrum corresponding to the current frame; wherein the stripe pooling layer is a structure in the SU-net network that has an enhancement effect on stripe information; the spectrum hidden vector corresponding to the previous frame is a hidden vector obtained by processing the target spectrum output by the previous frame by a spectrum encoder; The target synthesized singing voice acquisition module is configured to convert and synthesize the target spectrum to obtain a target synthesized singing voice.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the singing voice synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the singing voice synthesis method according to any one of claims 1 to 6.

Images