A noise-controllable song synthesis method, device, equipment and medium

Abstract

The present application relates to the technical field of artificial intelligence, and more particularly to a song synthesis method and device based on controllable noise, equipment and medium. The method is applied to the medical field, random noise values are added to the corresponding pixels of the pre-defined two-dimensional image to obtain a random noise image, the emotional encoding features and rhythm encoding features are added to the random noise image, the random noise values in the corresponding pixels of the random noise image are fused, the fused noise image is noise predicted, and the predicted noise image is denoised to obtain a denoised noise image. The denoised noise image is determined as the to-be-synthesized mel-frequency spectrum corresponding to the to-be-synthesized song. By fusing the pre-defined emotional encoding features and pre-defined rhythm encoding features into the noise image, the emotion and rhythm in the song synthesis are controlled, thereby improving the effect of song synthesis. In the medical field, controllable diversified music with good effect can be synthesized, which is helpful for patient rehabilitation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a song synthesis method and device based on controllable noise, equipment and medium. BACKGROUND

[0002] In the medical field, music therapy is a non-invasive treatment method, which is popular among patients because it has no side effects in treating sleep disorders, anxiety and lack of concentration. In music therapy, diverse music is generally generated by song synthesis. The existing song synthesis methods can be roughly divided into two types: the first type is to give the lyrics and melody, professional attributes such as tone, which are fitted by a parameterized generator to generate the song with these attributes. The second type is to learn the melody, tone and other characteristics by the model itself under the condition of given lyrics according to a large amount of labeled training data, and to synthesize the corresponding song according to the learned model. However, these two methods have some shortcomings. The first method is that professional music workers are very rare and it is difficult to meet the given lyrics of the song. The second method is that the effect of the synthesized song is poor due to the dependence on training data. Therefore, in the music therapy process in the medical field, how to improve the effect of song synthesis has become a problem to be solved. SUMMARY

[0003] Therefore, it is necessary to provide a song synthesis method and device based on controllable noise, equipment and storage medium to solve the problem of poor song synthesis effect in the music therapy process in the medical field.

[0004] The first aspect of the embodiment of the present application provides a song synthesis method based on controllable noise, which comprises:

[0005] obtaining a preset emotion signal, performing emotion feature coding on the preset emotion signal to obtain emotion coding features corresponding to the preset emotion signal, obtaining a preset rhythm signal, and performing rhythm feature coding on the preset rhythm signal to obtain rhythm coding features corresponding to the preset rhythm signal;

[0006] generating a random noise value according to a preset random function, adding the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise image;

[0007] adding the emotion coding features and the rhythm coding features to the random noise image, and fusing the random noise value in the corresponding pixel of the random noise image to obtain a fused noise image;

[0008] using a trained neural network to perform noise prediction on the fused noise image to obtain a predicted noise image containing noise points and noise values corresponding to the noise points.

[0009] de-noising the predicted noise map to obtain a de-noised noise map, and determining the de-noised noise map as a to-be-synthesized mel-frequency spectrum, wherein a horizontal coordinate in the de-noised noise map is determined as a frame number in the to-be-synthesized mel-frequency spectrum, and a vertical coordinate in the de-noised noise map is determined as a frequency in the to-be-synthesized mel-frequency spectrum;

[0010] synthesizing a corresponding song voice using a preset vocoder based on the to-be-synthesized mel-frequency spectrum.

[0011] A second aspect of the embodiment of the present application provides a song voice synthesis device based on controllable noise, which comprises:

[0012] an acquisition module, configured to acquire a preset emotion signal, encode emotion features of the preset emotion signal to obtain emotion encoding features corresponding to the preset emotion signal, acquire a preset rhythm signal, and encode rhythm features of the preset rhythm signal to obtain rhythm encoding features corresponding to the preset rhythm signal;

[0013] a generation module, configured to generate a random noise value according to a preset random function, and add the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map;

[0014] an adding module, configured to add the emotion encoding features and the rhythm encoding features to the random noise map, and fuse the random noise value in the corresponding pixel in the random noise map to obtain a fused noise map;

[0015] a prediction module, configured to use a trained neural network to perform noise prediction on the fused noise map to obtain a predicted noise map containing noise points and noise values corresponding to the noise points;

[0016] a de-noising module, configured to de-noise the predicted noise map to obtain a de-noised noise map, and determine the de-noised noise map as a to-be-synthesized mel-frequency spectrum, wherein a horizontal coordinate in the de-noised noise map is determined as a frame number in the to-be-synthesized mel-frequency spectrum, and a vertical coordinate in the de-noised noise map is determined as a frequency in the to-be-synthesized mel-frequency spectrum;

[0017] a synthesis module, configured to synthesize a corresponding song voice using a preset vocoder based on the to-be-synthesized mel-frequency spectrum.

[0018] In a third aspect, the embodiment of the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the song voice synthesis method based on controllable noise according to the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the noise-controllable song synthesis method according to the first aspect.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] According to the preset random function, a random noise value is generated, the random noise value is added to a corresponding pixel of a predefined two-dimensional image to obtain a random noise image, the emotional encoding feature and the rhythm encoding feature are added to the random noise image, and the random noise value in the corresponding pixel in the random noise image is fused to obtain a fused noise image. The trained neural network is used to predict the noise of the fused noise image to obtain a predicted noise image containing noise points and noise values corresponding to the noise points, and the predicted noise image is denoised to obtain a denoised noise image. The denoised noise image is determined as a to-be-synthesized mel-frequency spectrum corresponding to a to-be-synthesized song, and the to-be-synthesized mel-frequency spectrum is synthesized into a corresponding song using a preset vocoder. In the present application, the predefined emotional encoding feature and the predefined rhythm encoding feature are fused into the noise image to control the emotion and rhythm in the song synthesis, to realize controllable song synthesis, thereby improving the effect of song synthesis. In the music therapy process in the medical field, diversified music with good controllable effect can be synthesized, which is helpful for patient rehabilitation. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is an application environment schematic diagram of a noise-controllable song synthesis method provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of a noise-controllable song synthesis method provided by an embodiment of the present application;

[0025] Figure 3 is a structure schematic diagram of a noise-controllable song synthesis device provided by an embodiment of the present application;

[0026] Figure 4 is a structure schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0028] It should be understood that the term "comprising" as used in the specification and the appended claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0029] It should also be understood that the term "and / or" as used in the specification and the appended claims indicates any combination of one or more of the associated listed items and all possible combinations of the items.

[0030] As used in the specification and the appended claims, the term "if' can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted to mean "once determined" or "in response to a determination" or "once detected [the described condition or event]" or "in response to a detection [the described condition or event]" depending on the context.

[0031] In addition, in the description of the specification and the appended claims, the terms "first", "second", "third", and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0032] In the present specification, the reference "one embodiment" or "some embodiments" and the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in yet some embodiments", and the like appearing in various places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically noted. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically noted.

[0033] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving environment, acquiring knowledge and using knowledge to obtain optimal results.

[0034] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0035] It should be understood that the size of the serial number of each step in the following embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0036] In order to illustrate the technical scheme of the present application, the following will be described by specific embodiments.

[0037] An embodiment of the present application provides a song synthesis method based on controllable noise, which can be applied in an application environment such as Figure 1 , wherein the client and the server communicate. The client includes but is not limited to a palmtop computer, a desktop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) and the like. The server can be realized by an independent server or a server cluster generated by multiple servers.

[0038] Referring to Figure 2 , it is a flowchart of a song synthesis method based on controllable noise provided by an embodiment of the present application, which can be applied to a server in Figure 1 , the server is connected with a corresponding client, as shown in Figure 2 , the song synthesis method based on controllable noise can include the following steps.

[0039] S201: acquiring a preset emotion signal, encoding emotion features of the preset emotion signal to obtain emotion encoding features corresponding to the preset emotion signal, acquiring a preset rhythm signal, encoding rhythm features of the preset rhythm signal to obtain rhythm encoding features corresponding to the preset rhythm signal.

[0040] In step S201, the preset emotion signal is a speech signal containing emotion information, and the preset rhythm signal is a speech rhythm signal corresponding to a song. The emotion feature of the preset emotion signal is encoded to obtain an emotion encoding feature corresponding to the preset emotion signal. The rhythm feature of the preset rhythm signal is encoded to obtain a rhythm encoding feature corresponding to the preset rhythm signal. The encoding features in the preset emotion signal and the preset rhythm signal are extracted to facilitate the fusion of the encoding features and the randomly generated noise graph.

[0041] In this embodiment, the emotion signal corresponding to the emotion information commonly used in music therapy in the medical field and the rhythm signal corresponding to the rhythm information are obtained, so that the song synthesized according to the emotion signal and the rhythm signal is helpful for the treatment of patients. First, the preset emotion signal and the preset rhythm signal are converted into corresponding mel-frequency spectra. This conversion can scale the spectral features of the signals to the coordinate system of human ear perception features, facilitating better analysis and processing. Since the convolutional neural network model has specific size requirements for the input spectral features, the size of the extracted transformed mel-frequency spectral features needs to be scaled to obtain the mel-frequency spectral features corresponding to the human voice frequency.

[0042] The emotion encoder is used to extract the emotion encoding features of the preset emotion signal corresponding to the mel-frequency spectrum. For example, the emotion encoder can be composed of two layers of convolutional neural network layers and two layers of bidirectional long short-term memory networks. The convolution kernel sizes of the two layers of convolutional neural network layers are 7*7 and 20*7 respectively. After the convolution layer, a batch normalization layer, a ReLU nonlinear activation layer and a maximum pooling layer are connected in sequence. The kernel sizes of the maximum pooling layer are 2*2 and 1*5 respectively. Through convolution operation, an intermediate emotion representation sequence M=[m1, m2,..., mn,..., mN] of 74*128 dimensions is obtained. mn is the feature vector at the nth position of the feature. The emotion-related features in the FBank acoustic features are extracted through the two layers of convolutional neural network layers, and the features are used as the input features of the long short-term memory network layer to output the corresponding emotion features. In the emotion encoder, two layers of bidirectional long short-term memory networks are used to model the time sequence relationship of the input intermediate sequence features M. The hidden vector representations of BiLSTM come from forward and reverse LSTM (using and representing), respectively. The number of hidden layer nodes of each LSTM layer is 128. At the same time, the use of nonlinear activation can obtain the output sequence of N time steps of the final hidden vector, which together constitutes the emotion encoding features.

[0043] The rhythm encoder is used to extract the rhythm encoding features of the preset rhythm signal corresponding to the mel-frequency spectrum. For example, the rhythm encoder can include a group of 5*1 convolution layers, a group normalization layer or a group normalization layer, a BiLSTM layer and a down-sampling layer.

[0044] It should be noted that the emotion encoder and the rhythm encoder are respectively added with corresponding information bottleneck layers. The information bottleneck layer forces the encoder to only transmit the features or information that cannot be provided by other encoders and discard other features. The size of the information bottleneck layer added by each encoder can be different, and can be adjusted according to the corresponding rule setting.

[0045] In the information bottleneck layer, a multi-layer perceptron neural network MLP and two linear models connected to the multi-layer perceptron neural network MLP can be included. The linear model has an activation function, and the hidden layer in the linear model can include K nodes, where the node is a neuron in the neural network. The emotion encoding signal corresponding to the feature vector after mel spectrum processing passes through a multi-layer perceptron neural network MLP, and then passes through two linear models to obtain the output mean and variance. The information bottleneck layer also includes a corresponding emotion encoding feature generation module and a rhythm encoding feature generation module. The expectation value and the variance enter the corresponding generation module, and the output corresponding feature is output, that is, the emotion encoding feature and the rhythm encoding feature. The emotion encoding feature and the rhythm encoding feature are a K-dimensional Gaussian distribution, which corresponds to the K nodes in the hidden layer of the linear model.

[0046] Optionally, the preset emotion signal is encoded with emotion features to obtain emotion encoding features corresponding to the preset emotion signal, including:

[0047] The preset emotion signal is converted into a corresponding Fourier spectrum, and the corresponding mel spectrum is determined according to the Fourier spectrum.

[0048] The preset emotion encoder is used to encode the mel spectrum with emotion features to obtain emotion encoding features corresponding to the preset emotion signal.

[0049] In this embodiment, a first preset function in the librosa library is used to convert the preset emotion signal into a corresponding Fourier transform spectrum, and then a second preset function is used to convert it into a mel spectrum that is more consistent with human ear hearing. In this way, a one-dimensional time sequence signal that is difficult to process can be converted into a two-dimensional frequency domain data that is easy to process and has more information. Then the preset emotion encoder is used to encode the mel spectrum with emotion features to obtain emotion encoding features corresponding to the preset emotion signal.

[0050] S202: According to a preset random function, a random noise value is generated, and the random noise value is added to a corresponding pixel of a predefined two-dimensional image to obtain a random noise image.

[0051] In step S202, the random noise image can be a random noise image satisfying a certain specific distribution, such as a random noise image satisfying a uniform distribution, and can also be a random noise image satisfying a Gaussian distribution, and the like.

[0052] In this embodiment, a preset random function is used to generate a random noise value, where the preset random function can be a mean distribution function or a Gaussian distribution function. A noise image obtained by using the mean distribution function is a random noise image satisfying a uniform distribution, and a noise image obtained by using the Gaussian distribution function is a random noise image satisfying a Gaussian distribution.

[0053] S203: Add the emotion encoding feature and the rhythm encoding feature to the random noise image, and fuse the random noise value in the corresponding pixel in the random noise image to obtain a fused noise image.

[0054] In step S203, the emotion encoding feature and the rhythm encoding feature are two-dimensional feature vectors. In a two-dimensional coordinate, adding the emotion encoding feature and the rhythm encoding feature to the random noise image can fuse the random noise value in the corresponding pixel in the random noise image to obtain a fused noise image.

[0055] In this embodiment, the emotion encoding feature and the rhythm encoding feature are aligned with the random noise image. The emotion encoding feature and the rhythm encoding feature are scaled according to the size of the random noise image, so that the scaled emotion encoding feature and the rhythm encoding feature are equal to the size of the random noise image. The emotion encoding feature and the rhythm encoding feature are added to the random noise image, and the random noise value in the corresponding pixel in the random noise image is fused to obtain a fused noise image.

[0056] Optionally, adding the emotion encoding feature and the rhythm encoding feature to the random noise image, and fusing the random noise value in the corresponding pixel in the random noise image to obtain a fused noise image, includes:

[0057] According to the emotion encoding feature and the rhythm encoding feature, determine the emotion encoding feature value and the rhythm encoding feature value corresponding to each pixel in the random noise image;

[0058] Add the emotion encoding feature value, the rhythm encoding feature value, and the random noise value corresponding to each pixel in the random noise image to obtain a fused value of each pixel. According to the fused value of each pixel, obtain a fused noise image.

[0059] In this embodiment, according to the emotion encoding feature and the rhythm encoding feature, the feature vectors in the emotion encoding feature and the rhythm encoding feature are determined, the feature values of each element in the feature vectors are extracted, the elements in the emotion encoding feature and the rhythm encoding feature are aligned with the pixels in the random noise map, the emotion encoding feature and the rhythm encoding feature are scaled according to the size in the random noise map, so that the scaled emotion encoding feature and the rhythm encoding feature are equal to the size of the random noise map, so as to facilitate the fusion of the emotion encoding feature, the rhythm encoding feature and the random noise map. After aligning the elements in the emotion encoding feature and the rhythm encoding feature with the pixels in the random noise map, the feature values of the elements in the same position are added and fused with the random noise values of the pixels, to obtain the fusion value of each pixel. The fusion value of each pixel is taken as the new random noise value of the pixel in the random noise map, to obtain the fusion noise map.

[0060] In step S204, the trained neural network can perform noise prediction on the fusion noise map to predict the noise values corresponding to the noise points in the fusion noise map, so as to remove the noise at the predicted noise point position.

[0061] In step S204, the trained neural network can perform noise prediction on the fusion noise map to predict the noise values corresponding to the noise points in the fusion noise map, so as to remove the noise at the predicted noise point position.

[0062] In this embodiment, the trained neural network can include two 4-layer two-dimensional convolution structures. In the first 4-layer two-dimensional convolution structure, down-sampling based on maximum pooling is used between each layer to down-sample the hidden sequence by one time. In the second 4-layer two-dimensional convolution structure, up-sampling based on nearest neighbor interpolation is used between each layer to up-sample the hidden sequence by one time. In the up-sampling calculation of each layer, the current result is connected in parallel with the down-sampled hidden sequence of the corresponding resolution in the time dimension. Finally, a linear layer is used to map the output dimension to the dimension of the mel spectrum.

[0063] Optionally, before using the trained neural network to perform noise prediction on the fusion noise map to obtain the prediction noise map containing the noise points and the noise values corresponding to the noise points, the method further includes:

[0064] The training data set and the initial neural network are obtained, the training data set includes the original song, the pre-defined emotion signal and the pre-defined rhythm signal;

[0065] The mel spectrum of the original song is extracted to obtain a first mel spectrum corresponding to the original song. Noise data is added to the first mel spectrum to obtain a second mel spectrum containing noise data. The second mel spectrum is determined as a first noise map.

[0066] encode the pre-defined emotion signal to obtain original emotion encoding features, encode the pre-defined rhythm signal to obtain original rhythm encoding features, and add the original emotion encoding features and the original rhythm encoding features to the first noise map to obtain a second noise map;

[0067] perform noise prediction on the second noise map using the initial neural network to obtain an original predicted noise map containing noise points and noise values corresponding to the noise points;

[0068] train the initial neural network according to the first noise map and the original predicted noise map, calculate a loss value between the first noise map and the original predicted noise map, and stop training when the loss value meets a preset condition to obtain a trained neural network.

[0069] In the embodiment, when the initial neural network is trained, a raw song is obtained, a mel spectrum of the raw song is extracted, a first mel spectrum corresponding to the raw song is obtained, noise data is added to the first mel spectrum to obtain a second mel spectrum containing the noise data, and the second mel spectrum is determined as the first noise map. The mel spectrum of the raw song is taken as a true value, noise data is added to the first mel spectrum to obtain a second mel spectrum containing the noise data, the second mel spectrum is determined as the first noise map, the first noise map is taken as an added true noise map, the noise data can be a random value generated by a random function and added to the true mel spectrum to obtain the second mel spectrum containing the noise data. The pre-defined emotion signal and the pre-defined rhythm signal are added to the corresponding first noise map to obtain the second noise map, the initial neural network is used to perform noise prediction on the second noise map to obtain the original predicted noise map, and the original predicted noise map is a noise map containing the emotion signal and the rhythm signal. In order to facilitate the initial neural network to learn the contribution of the emotion signal and the rhythm signal to the model during training.

[0070] The loss value is calculated according to the original predicted noise map predicted by the initial neural network and the first noise map corresponding to the second mel spectrum, and the training is stopped when the loss value meets a preset condition to obtain a trained neural network. The preset condition can be that the loss value is less than a preset threshold or that the loss value converges.

[0071] Optionally, the loss value between the first noise map and the original predicted noise map includes:

[0072] The mean square error loss between the first noise map and the original predicted noise map is calculated to obtain a first loss value, and the cross-entropy loss between the first noise map and the original predicted noise map is calculated to obtain a second loss value.

[0073] The target loss function is constructed according to the first loss value and the second loss value, and the loss value between the first noise map and the original predicted noise map is calculated according to the target loss function.

[0074] In the embodiment, when calculating the loss value between the first noise map and the third mel spectrum, the sum of the mean square error loss between the first noise map and the original predicted noise map and the cross entropy loss between the first noise map and the original predicted noise map is taken as the target loss value, so as to improve the speed and accuracy of model training when training the initial neural network.

[0075] S205: The predicted noise map is denoised to obtain a denoised noise map, and the denoised noise map is determined as the to-be-synthesized mel spectrum.

[0076] In step S205, the horizontal coordinates in the denoised noise map are determined as the frame numbers in the to-be-synthesized mel spectrum, and the vertical coordinates in the denoised noise map are determined as the frequencies in the to-be-synthesized mel spectrum. The trained neural network can determine the mapping relationship between the random noise map and the predicted noise map according to the learned knowledge, and map the predicted noise map according to the mapping relationship, so as to remove the noise in the predicted noise map.

[0077] In the embodiment, according to the position of the noise point in the predicted noise map and the noise value of the noise point, the noise value at the noise point is removed through denoising to obtain a denoised noise map without noise pollution, and the denoised noise map is determined as the to-be-synthesized mel spectrum, wherein the horizontal coordinates in the denoised noise map are determined as the frame numbers in the to-be-synthesized mel spectrum, and the vertical coordinates in the denoised noise map are determined as the frequencies in the to-be-synthesized mel spectrum.

[0078] Optionally, the denoising of the predicted noise map to obtain the denoised noise map comprises:

[0079] The noise value corresponding to the noise point in the predicted noise map is mapped according to the preset mapping relationship in the trained neural network to obtain a mapped noise value corresponding to the noise point;

[0080] The mapped noise value is filled into the corresponding noise point to obtain a filled noise map, and the filled noise map is determined as the denoised noise map.

[0081] In this embodiment, the predicted noise map is mapped according to the preset mapping relationship in the trained neural network to remove the noise values ​​at the corresponding noise points. For example, if the mapping relationship between the random noise map and the predicted noise map is a function F(x), then the pixels in the predicted noise map are mapped using the function F(x), the noise values ​​in the noise points in the predicted noise map are mapped using the function F(x), and the mapped noise values ​​are filled into the corresponding noise points to obtain the filled noise map. The function F(x) can be a linear function or a nonlinear function.

[0082] In another embodiment, the noise values ​​in the noise points can be mapped multiple times step by step. For example, the first mapping function is used to perform the first mapping process on the noise values ​​to obtain the first mapping result, and the second mapping function is used to perform the second mapping on the first mapping to obtain the second mapping result, and so on, and multiple mappings can be performed. The result of each mapping is the original data for the next mapping. The mapping function for each mapping can be the same or different, and can be a linear function or a non-linear function. In this embodiment, a cross mapping of linear and non-linear functions is used to prevent the corresponding noise from not being completely removed when performing a single mapping on the noise points with large noise values.

[0083] S206: Use a preset vocoder to synthesize the corresponding singing voice from the Mel spectrum to be synthesized.

[0084] In step S206, a preset vocoder is used to synthesize the corresponding singing voice from the Mel spectrum to be synthesized. The preset vocoder can be a neural network vocoder, and the corresponding singing voice is a singing voice with emotion and rhythm obtained by adding the emotion signal and rhythm signal to the random noise graph.

[0085] In this embodiment, a preset vocoder can be used for vocal synthesis. The vocoder in this embodiment can be a neural network vocoder, including but not limited to autoregressive neural network vocoders (such as waveRNN vocoders) and Gann network-based neural network vocoders (such as meluGan vocoders and hifiGan vocoders).

[0086] According to a preset random function, a random noise value is generated, the random noise value is added to a corresponding pixel of a predefined two-dimensional image, a random noise image is obtained, the emotional encoding feature and the rhythm encoding feature are added to the random noise image, and the random noise value in the corresponding pixel of the random noise image is fused to obtain a fused noise image. The trained neural network is used for noise prediction on the fused noise image to obtain a predicted noise image containing noise points and noise values corresponding to the noise points, and the predicted noise image is denoised to obtain a denoised noise image. The denoised noise image is determined as a to-be-synthesized mel spectrum corresponding to a to-be-synthesized song. The preset vocoder is used to synthesize the to-be-synthesized mel spectrum into a corresponding song. In the present application, by fusing the predefined emotional encoding feature and the predefined rhythm encoding feature into the noise image, the control of the emotion and the rhythm in the song synthesis is realized, the controllable song synthesis is realized, and thus the effect of the song synthesis is improved. In the music therapy process in the medical field, controllable diversified music with good effect can be synthesized, which is helpful for patient rehabilitation.

[0087] Referring to Figure 3 , Figure 3 The structure block diagram of the song synthesis device based on noise control provided by the embodiments of the present application is shown, and the song synthesis device based on noise control is applied to the server. For ease of illustration, only the parts related to the embodiments of the present application are shown. Referring to Figure 3 The song synthesis device 30 includes an acquisition module 31, a generation module 32, an addition module 33, a prediction module 34, a denoising module 35, and a synthesis module 36.

[0088] The acquisition module 31 is configured to acquire a preset emotional signal, encode emotional features of the preset emotional signal to obtain emotional encoding features corresponding to the preset emotional signal, acquire a preset rhythm signal, and encode rhythm features of the preset rhythm signal to obtain rhythm encoding features corresponding to the preset rhythm signal.

[0089] The generation module 32 is configured to generate a random noise value according to a preset random function, add the random noise value to a corresponding pixel of a predefined two-dimensional image, and obtain a random noise image.

[0090] The addition module 33 is configured to add the emotional encoding features and the rhythm encoding features to the random noise image, fuse the random noise value in the corresponding pixel of the random noise image, and obtain a fused noise image.

[0091] The prediction module 34 is configured to use a trained neural network to perform noise prediction on the fused noise image to obtain a predicted noise image containing noise points and noise values corresponding to the noise points.

[0092] The denoising module 35 is configured to denoise the predicted noise map to obtain a denoised noise map, and determine the denoised noise map as the to-be-synthesized mel spectrum, wherein a horizontal coordinate in the denoised noise map is determined as a frame number in the to-be-synthesized mel spectrum, and a vertical coordinate in the denoised noise map is determined as a frequency in the to-be-synthesized mel spectrum.

[0093] The synthesizing module 36 is configured to synthesize the to-be-synthesized mel spectrum into a corresponding song voice by using a preset vocoder.

[0094] Optionally, the adding module 33 includes:

[0095] The determining unit is configured to determine, according to the emotion encoding feature and the rhythm encoding feature, an emotion encoding feature value and a rhythm encoding feature value corresponding to each pixel in the random noise map.

[0096] The fusing unit is configured to add and fuse the emotion encoding feature value, the rhythm encoding feature value, and the random noise value corresponding to each pixel in the random noise map to obtain a fusion value of each pixel, and obtain a fusion noise map according to the fusion value of each pixel.

[0097] Optionally, the obtaining module 31 includes:

[0098] The transforming unit is configured to transform the preset emotion signal into a corresponding Fourier spectrum, and determine a corresponding mel spectrum according to the Fourier spectrum.

[0099] The encoding unit is configured to use a preset emotion encoder to perform emotion feature encoding on the mel spectrum to obtain an emotion encoding feature corresponding to the preset emotion signal.

[0100] Optionally, the song voice synthesizing apparatus 30 further includes:

[0101] The training data obtaining module is configured to obtain a training data set and an initial neural network, and the training data set includes an original song voice, a predefined emotion signal, and a predefined rhythm signal.

[0102] The extracting module is configured to extract a mel spectrum of the original song voice to obtain a first mel spectrum corresponding to the original song voice, add noise data to the first mel spectrum to obtain a second mel spectrum containing the noise data, and determine the second mel spectrum as a first noise map.

[0103] The obtaining module is configured to perform emotion feature encoding on the predefined emotion signal to obtain original emotion encoding features, perform rhythm feature encoding on the predefined rhythm signal to obtain original rhythm encoding features, and add the original emotion encoding features and the original rhythm encoding features to the first noise map to obtain a second noise map.

[0104] The original prediction module is used to perform noise prediction on the second noise map using the initial neural network, and obtain the original predicted noise map containing noise points and the noise values ​​corresponding to the noise points.

[0105] The training module is used to train the initial neural network based on the first noise map and the original predicted noise map, calculate the loss value between the first noise map and the original predicted noise map, and stop training when the loss value meets the preset conditions to obtain the trained neural network.

[0106] Optionally, the above training module includes:

[0107] The calculation unit is used to calculate the mean squared error loss between the first Mel spectrum and the third Mel spectrum to obtain the first loss value, and to calculate the cross-entropy loss between the first Mel spectrum and the third Mel spectrum to obtain the second loss value.

[0108] The construction unit is used to construct a target loss function based on the first loss value and the second loss value, and to calculate the loss value between the first Mel spectrum and the third Mel spectrum based on the target loss function.

[0109] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0110] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 4 As shown, the terminal device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executing the computer program, implements the steps in any of the above embodiments of the noise-controlled singing synthesis method.

[0111] The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. A terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as network interfaces, displays, and input devices.

[0112] The processor can be a CPU, and can also be other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0113] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the internal memory of the terminal device, and the internal memory provides an environment for the operation of the operating system and the computer readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the terminal device, and in other embodiments, can also be the external storage device of the terminal device, for example, the plug-in hard disk, the Smart Media Card (SMC), the Secure Digital (SD) card, the Flash Card, etc. equipped on the terminal device. Further, the memory can include both the internal storage unit of the terminal device and the external storage device. The memory is used to store the operating system, the application program, the BootLoader, the data, and other programs, such as the program code of the computer program, etc. The memory can also be used to temporarily store the data that has been output or will be output.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electric carrier signal and telecommunication signal.

[0115] The above-mentioned embodiment methods can also be completed by a computer program product, when the computer program product runs on the terminal device, so that the terminal device executes the steps to realize the above-mentioned method embodiments.

[0116] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0117] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the apparatus / terminal device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0119] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0120] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for synthesizing singing voice based on noise controllability, characterized in that, The singing voice synthesis method includes: A preset emotional signal is acquired, and emotional feature encoding is performed on the preset emotional signal to obtain the emotional coding feature corresponding to the preset emotional signal. A preset rhythm signal is acquired, and rhythm feature encoding is performed on the preset rhythm signal to obtain the rhythm coding feature corresponding to the preset rhythm signal. Random noise values ​​are generated according to a preset random function, and these random noise values ​​are added to the corresponding pixels of a predefined two-dimensional image to obtain a random noise map. The emotion encoding feature and the rhythm encoding feature are added to the random noise map and fused with the random noise value in the corresponding pixel of the random noise map to obtain a fused noise map; The trained neural network is used to perform noise prediction on the fused noise map to obtain a predicted noise map containing noise points and the noise values ​​corresponding to the noise points. The predicted noise map is denoised to obtain a denoised noise map. The denoised noise map is then determined as the Mel spectrum to be synthesized. The horizontal coordinate in the denoised noise map is determined as the frame number in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise map is determined as the frequency in the Mel spectrum to be synthesized. The Mel spectrum to be synthesized is used to synthesize the corresponding singing voice using a preset vocoder.

2. The singing voice synthesis method as described in claim 1, characterized in that, The step of adding the emotion encoding feature and the rhythm encoding feature to the random noise map and fusing them with the random noise values ​​in the corresponding pixels of the random noise map to obtain a fused noise map includes: Based on the emotion coding features and the rhythm coding features, determine the emotion coding feature value and the rhythm coding feature value corresponding to each pixel in the random noise map; The emotion coding feature value and rhythm coding feature value corresponding to each pixel in the random noise map are added and fused with the random noise value to obtain the fused value of each pixel. Based on the fused value of each pixel, the fused noise map is obtained.

3. The singing voice synthesis method as described in claim 1, characterized in that, The step of encoding the preset emotional signal with emotional features to obtain the emotional encoding features corresponding to the preset emotional signal includes: The preset emotional signal is converted into a corresponding Fourier spectrum, and the corresponding Mel spectrum is determined based on the Fourier spectrum; Using a preset emotion encoder, the Mel spectrum is encoded with emotion features to obtain the emotion coding features corresponding to the preset emotion signal.

4. The singing voice synthesis method as described in claim 1, characterized in that, The step of denoising the predicted noise map to obtain the denoised noise map includes: Based on the preset mapping relationship in the trained neural network, the noise value corresponding to the noise point in the predicted noise map is mapped to obtain the mapped noise value corresponding to the noise point. The mapped noise values ​​are filled into the corresponding noise points to obtain the filled noise map, and the filled noise map is determined as the denoised noise map.

5. The singing voice synthesis method as described in claim 1, characterized in that, Before using the trained neural network to perform noise prediction on the fused noise map to obtain a predicted noise map containing noise points and their corresponding noise values, the method further includes: Acquire a training dataset and an initial neural network, wherein the training dataset includes the original singing voice, predefined emotional signals, and predefined rhythm signals; Extract the Mel spectrum of the original singing voice to obtain the first Mel spectrum corresponding to the original singing voice. Add noise data to the first Mel spectrum to obtain the second Mel spectrum containing noise data. Determine the second Mel spectrum as the first noise map. The predefined emotional signal is encoded with emotional features to obtain the original emotional coding features. The predefined rhythm signal is encoded with rhythm features to obtain the original rhythm coding features. The original emotional coding features and the original rhythm coding features are added to the first noise map to obtain the second noise map. The initial neural network is used to perform noise prediction on the second noise map to obtain an original predicted noise map containing noise points and the noise values ​​corresponding to the noise points. The initial neural network is trained based on the first noise map and the original predicted noise map. The loss value between the first noise map and the original predicted noise map is calculated. When the loss value meets a preset condition, the training is stopped, and the trained neural network is obtained.

6. A singing voice synthesis device based on noise controllable, characterized in that, The singing voice synthesis device includes: The acquisition module is used to acquire a preset emotional signal, encode the preset emotional signal for emotional features to obtain the emotional coding features corresponding to the preset emotional signal, acquire a preset rhythm signal, encode the preset rhythm signal for rhythm features to obtain the rhythm coding features corresponding to the preset rhythm signal. The generation module is used to generate random noise values ​​according to a preset random function, and add the random noise values ​​to the corresponding pixels of a predefined two-dimensional image to obtain a random noise map; An addition module is used to add the emotion encoding features and the rhythm encoding features to the random noise map, and fuse them with the random noise values ​​in the corresponding pixels of the random noise map to obtain a fused noise map; The prediction module is used to perform noise prediction on the fused noise map using a trained neural network to obtain a predicted noise map containing noise points and the noise values ​​corresponding to the noise points. A denoising module is used to denoise the predicted noise map to obtain a denoised noise map, and to determine the denoised noise map as the Mel spectrum to be synthesized. The horizontal coordinate in the denoised noise map is determined as the frame number in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise map is determined as the frequency in the Mel spectrum to be synthesized. The synthesis module is used to synthesize the corresponding singing voice from the Mel spectrum to be synthesized using a preset vocoder.

7. The singing voice synthesis device as described in claim 6, characterized in that, The added module includes: The determining unit is configured to determine the emotion coding feature value and the rhythm coding feature value corresponding to each pixel in the random noise map based on the emotion coding feature and the rhythm coding feature; The fusion unit is used to add and fuse the emotion coding feature value and rhythm coding feature value corresponding to each pixel in the random noise map with the random noise value to obtain the fusion value of each pixel, and obtain the fused noise map based on the fusion value of each pixel.

8. A terminal device, characterized in that, The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the noise-controllable singing voice synthesis method as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the noise-controllable singing voice synthesis method as described in any one of claims 1 to 5.

Images