Singing synthesis method and device based on noise controllability, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a song synthesis method and device based on noise controllability, equipment and a medium. The method is applied to the medical field and comprises the following steps of: adding random noise values into corresponding pixels of a predefined two-dimensional image to obtain a random noise graph, adding emotion coding features and rhythm coding features into the random noise graph, fusing the emotion coding features and the rhythm coding features with the random noise values in the corresponding pixels in the random noise graph, and performing noise prediction on the fused noise graph to obtain the emotion coding feature and the rhythm coding feature. The method comprises the following steps: acquiring a predicted noise map, denoising the predicted noise map to obtain a denoised noise map, determining the denoised noise map as a to-be-synthesized Mel spectrum corresponding to a to-be-synthesized song, and fusing a predefined emotion coding feature and a predefined rhythm coding feature into the noise map to realize emotion and rhythm control in song synthesis. Therefore, the singing synthesis effect is improved. In the medical field, controllable diversified music with a good effect can be synthesized, and rehabilitation of a patient is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a singing voice synthesis method, device, equipment and medium based on controllable noise. Background Art

[0002] In the medical field, music therapy is a non-invasive treatment method. It is popular among patients for the treatment of sleep disorders, anxiety, and inattention because it has no side effects. In music therapy, a variety of music is generally generated by singing synthesis. In the prior art, there are roughly two methods for singing synthesis: the first is to fit the lyrics of a given song, as well as professional attributes such as melody and pitch, and generate singing with these attributes by a parameterized generator. The second method uses a large amount of labeled training data. Given the lyrics, the model itself learns the melody, pitch and other features, and synthesizes the corresponding singing according to the learned model. However, these two methods have some disadvantages to a greater or lesser extent. In the first method, professional music workers are very rare, and it is difficult to meet the lyrics of a given song. In the second method, the effect of the synthesized singing is poor due to reliance on training data. Therefore, in the process of music therapy in the medical field, how to improve the effect of singing synthesis has become an urgent problem to be solved. Summary of the invention Based on this, it is necessary to provide a noise-controllable singing synthesis method, device, equipment and storage medium to address the above-mentioned technical problems, so as to solve the problem of poor singing synthesis effect in the process of music therapy in the medical field.

[0003] A first aspect of an embodiment of the present application provides a method for synthesizing singing voice based on controllable noise, the method comprising: Acquire a preset emotion signal, perform emotion feature encoding on the preset emotion signal, and obtain emotion encoding features corresponding to the preset emotion signal; acquire a preset rhythm signal, perform rhythm feature encoding on the preset rhythm signal, and obtain rhythm encoding features corresponding to the preset rhythm signal; Generate a random noise value according to a preset random function, and add the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map; Adding the emotion coding feature and the rhythm coding feature to the random noise map, and fusing them with the random noise values ​​in corresponding pixels in the random noise map to obtain a fused noise map; Using a trained neural network to perform noise prediction on the fused noise map, to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points; Denoising the predicted noise image to obtain a denoised noise image, and determining the denoised noise image as a Mel spectrum to be synthesized, wherein the horizontal coordinate in the denoised noise image is determined as the frame number in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise image is determined as the frequency in the Mel spectrum to be synthesized; The Mel spectrum to be synthesized is synthesized into a corresponding singing voice using a preset vocoder.

[0004] A second aspect of an embodiment of the present application provides a singing voice synthesis device based on controllable noise, the singing voice synthesis device comprising: An acquisition module is used to acquire a preset emotion signal, perform emotion feature encoding on the preset emotion signal, obtain emotion encoding features corresponding to the preset emotion signal, acquire a preset rhythm signal, perform rhythm feature encoding on the preset rhythm signal, and obtain rhythm encoding features corresponding to the preset rhythm signal; A generating module, used for generating a random noise value according to a preset random function, and adding the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map; An adding module, used for adding the emotion coding feature and the rhythm coding feature to the random noise map, and fusing them with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map; A prediction module, used to perform noise prediction on the fused noise map using a trained neural network to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points; A denoising module, configured to perform denoising on the predicted noise image to obtain a denoised noise image, and determine the denoised noise image as a Mel spectrum to be synthesized, wherein the horizontal coordinate in the denoised noise image is determined as the number of frames in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise image is determined as the frequency in the Mel spectrum to be synthesized; The synthesis module is used to use a preset vocoder to synthesize the to-be-synthesized Mel-spectrum into corresponding singing voice.

[0005] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the noise-controllable singing synthesis method as described in the first aspect is implemented.

[0006] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the noise-controllable singing synthesis method as described in the first aspect is implemented.

[0007] Compared with the prior art, the present invention has the following beneficial effects: According to a preset random function, a random noise value is generated, and the random noise value is added to the corresponding pixel of a predefined two-dimensional image to obtain a random noise map, and the emotion coding feature and the rhythm coding feature are added to the random noise map, and the random noise value in the corresponding pixel in the random noise map is fused to obtain a fused noise map, and the fused noise map is predicted using a trained neural network to obtain a predicted noise map containing noise points and noise values ​​corresponding to the noise points, and the predicted noise map is denoised to obtain a denoised noise map, and the denoised noise map is determined as the mel spectrum to be synthesized corresponding to the song to be synthesized, and the mel spectrum to be synthesized is synthesized into the corresponding song using a preset vocoder. In the present application, by fusing the predefined emotion coding feature and the predefined rhythm coding feature into the noise map, the control of emotion and rhythm in the song synthesis is achieved, and controllable song synthesis is achieved, thereby improving the effect of song synthesis. In the process of music therapy in the medical field, controllable and effective diversified music can be synthesized, which is helpful for patient rehabilitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0009] Figure 1 Schematic diagram of an application environment of a noise-controllable singing voice synthesis method provided in an embodiment of the present application; Figure 2 It is a flow chart of a method for synthesizing singing voice based on controllable noise provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a singing voice synthesis device based on controllable noise provided in an embodiment of the present application; Figure 4 It is a structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0011] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0012] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0013] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0014] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0015] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0016] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0017] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0018] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0019] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0020] A method for synthesizing singing voice based on controllable noise provided by an embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDA, desktop computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA) and other terminal devices. The server can be implemented by an independent server or a server cluster generated by multiple servers.

[0021] See also Figure 2 , is a flow chart of a method for synthesizing singing voice based on controllable noise provided by an embodiment of the present invention. The method for synthesizing singing voice based on controllable noise can be applied to Figure 1 The server in the above connects to the corresponding client, such as Figure 2 As shown, the noise-controllable singing voice synthesis method may include the following steps.

[0022] S201: Obtain a preset emotion signal, encode the emotion feature of the preset emotion signal, and obtain the emotion coding feature corresponding to the preset emotion signal; obtain a preset rhythm signal, encode the rhythm feature of the preset rhythm signal, and obtain the rhythm coding feature corresponding to the preset rhythm signal.

[0023] In step S201, the preset emotion signal is a speech signal containing emotion information, and the preset rhythm signal is a speech rhythm signal corresponding to the song. The emotion feature of the preset emotion signal is encoded to obtain the emotion coding feature corresponding to the preset emotion signal. The rhythm feature of the preset rhythm signal is encoded to obtain the rhythm coding feature corresponding to the preset rhythm signal. The coding features in the preset emotion signal and the preset rhythm signal are extracted to facilitate the fusion of the coding features with the randomly generated noise map.

[0024] In this embodiment, the emotional signal corresponding to the emotional information and the rhythm signal corresponding to the rhythm information commonly used in music therapy in the medical field are obtained, so that the singing voice synthesized according to the emotional signal and the rhythm signal can help treat the patient. First, the preset emotional signal and the preset rhythm signal are converted into corresponding Mel spectrum. This transformation method can scale the spectral characteristics of the signal to a coordinate system similar to the human ear perception characteristics, which is convenient for better analysis and processing. Since the convolutional neural network model has specific size requirements for the input spectral characteristics, it is necessary to scale the size of the extracted transformed Mel spectrum features to obtain the Mel spectrum features corresponding to the human voice audio.

[0025] The emotion encoder is used to extract the emotion encoding features of the Mel spectrum corresponding to the preset emotion signal. For example, the emotion encoder can be composed of two layers of convolutional neural network layers and two layers of bidirectional long short-term memory networks. The convolution kernel sizes of the two layers of convolutional neural network layers are 7×7 and 20×7 respectively. The batch normalization layer, ReLU nonlinear activation layer and maximum pooling layer are connected in sequence after the convolution layer. The kernel size of the maximum pooling layer is 2×2 and 1×5 respectively. Through the convolution operation, a 74×128-dimensional intermediate emotion representation sequence M=[m1,m2,...,mn,...,mN] is obtained, where mn is the feature vector of the nth position of the feature. The emotion-related features in the FBank acoustic features are extracted through two layers of convolutional neural network layers, and the features are used as the input features of the long short-term memory network layer to output the corresponding emotion features. In the emotion encoder, a two-layer bidirectional long short-term memory network is used to model the temporal relationship of the input intermediate sequence feature M. The latent vector representation of BiLSTM comes from the forward and backward LSTM (represented by and ), and the number of hidden layer nodes in each LSTM layer is 128. At the same time, nonlinear activation can be used to obtain the final latent vector. The output sequence of N time steps together constitutes the emotion coding feature.

[0026] A rhythm encoder is used to extract rhythm encoding features of the Mel spectrum corresponding to a preset rhythm signal. For example, the rhythm encoder may include a set of 5*1 convolutional layers, a group normalization layer or a group normalization layer, a BiLSTM layer and a downsampling layer.

[0027] It should be noted that corresponding information bottleneck layers are added to the emotion encoder and the rhythm encoder respectively. The information bottleneck layer forces the encoder to only pass features or information that other encoders cannot provide and discard other features. The size of the information bottleneck layer added by each encoder may be different and will be adjusted according to the corresponding rule settings. Among them, the information bottleneck layer may include a multi-layer perceptron neural network MLP (Multi-layer Perceptron Neural Network) and two linear models connected to the multi-layer perceptron neural network MLP, the linear model has an activation function, and the hidden layer in the linear model may include K nodes, where the nodes are neurons in the neural network. The representation vector of the emotion coding signal corresponding to the Mel spectrum processing passes through a multi-layer perceptron neural network MLP, and then passes through two linear models to obtain the output mean and variance. The information bottleneck layer also includes a corresponding emotion coding feature generation module and a rhythm coding feature generation module. After the expected value and variance enter the corresponding generation module, the corresponding representation features, i.e., the emotion coding feature and the rhythm coding feature, are output. The emotion coding feature and the rhythm coding feature are a K-dimensional Gaussian distribution, which corresponds to the K nodes in the hidden layer of the linear model.

[0028] Optionally, encoding the preset emotion signal with emotion features to obtain emotion encoding features corresponding to the preset emotion signal includes: The preset emotion signal is converted into a corresponding Fourier spectrum, and the corresponding Mel spectrum is determined according to the Fourier spectrum; Use the preset emotion encoder to encode the emotion features of the Mel spectrum to obtain the emotion coding features corresponding to the preset emotion signal.

[0029] In this embodiment, the first preset function in the librosa library is used to convert the preset emotion signal into a corresponding Fourier transform spectrum, and then the second preset function is used to convert it into a Mel spectrum that is more in line with human hearing, so that the one-dimensional time series signal that is difficult to process can be converted into two-dimensional frequency domain data that is easy to process and has richer information. Then, the preset emotion encoder is used to encode the Mel spectrum for emotion features, and the emotion coding features corresponding to the preset emotion signal are obtained.

[0030] S202: Generate a random noise value according to a preset random function, and add the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map.

[0031] In step S202, the random noise image may be a random noise image satisfying a certain specific distribution, such as a random noise image satisfying a uniform distribution, or a random noise image satisfying a Gaussian distribution, etc.

[0032] In this embodiment, a preset random function is used to generate a random noise value, wherein the preset random function may be a mean distribution function or a Gaussian distribution function. The noise image obtained using the mean distribution function is a random noise image that satisfies a uniform distribution, and the noise image obtained using the Gaussian distribution function is a random noise image that satisfies a Gaussian distribution.

[0033] S203: Add the emotion coding features and the rhythm coding features to the random noise map, and fuse them with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map.

[0034] In step S203, the emotion coding features and the rhythm coding features are two-dimensional feature vectors. In the two-dimensional coordinates, the emotion coding features and the rhythm coding features are added to the random noise map and can be fused with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map.

[0035] In this embodiment, the emotion coding features and the rhythm coding features are aligned with the random noise map, and the emotion coding features and the rhythm coding features are scaled according to their size in the random noise map, so that the scaled emotion coding features and the rhythm coding features are equal in size to the random noise map, so as to facilitate adding the emotion coding features and the rhythm coding features to the random noise map and fusing them with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map.

[0036] Optionally, the emotion coding feature and the rhythm coding feature are added to the random noise map, and fused with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map, including: Determine the emotion coding feature value and the rhythm coding feature value corresponding to each pixel in the random noise image according to the emotion coding feature and the rhythm coding feature; The emotion coding feature value, rhythm coding feature value and random noise value corresponding to each pixel in the random noise map are added and fused to obtain the fusion value of each pixel, and the fusion noise map is obtained according to the fusion value of each pixel. In this embodiment, based on the emotion coding feature and the rhythm coding feature, the feature vectors in the emotion coding feature and the rhythm coding feature are determined, the feature value of each element in the feature vector is extracted, the elements in the emotion coding feature and the rhythm coding feature are aligned with the pixels in the random noise map, and the emotion coding feature and the rhythm coding feature are scaled according to the size in the random noise map so that the scaled emotion coding feature and the rhythm coding feature are equal in size to the random noise map, so as to facilitate the fusion of the emotion coding feature, the rhythm coding feature and the random noise map. After aligning the elements in the emotion coding feature and the rhythm coding feature with the pixels in the random noise map, the feature values ​​of the elements in the same position are added and fused with the random noise value of the pixel to obtain the fusion value of each pixel, and the fusion value of each pixel is used as the new random noise value of the pixel in the random noise map to obtain a fused noise map.

[0037] S204: Use the trained neural network to perform noise prediction on the fused noise map to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points.

[0038] In step S204, the trained neural network can perform noise prediction on the fused noise map, and predict the noise points and noise values ​​corresponding to the noise points in the corresponding fused noise map, so as to remove the noise at the predicted noise point positions.

[0039] In this embodiment, the trained neural network may include two four-layer two-dimensional convolution structures. In the first four-layer two-dimensional convolution structure, maximum pooling-based downsampling is used between each layer to downsample the hidden sequence by one time. In the second four-layer two-dimensional convolution structure, nearest neighbor interpolation-based upsampling is used between each layer to upsample the hidden sequence by one time. In the upsampling calculation of each layer, the current result and the downsampled hidden sequence of the corresponding resolution are connected in parallel in the time dimension. Finally, a linear layer is used to map the output dimension to the dimension of the Mel spectrum.

[0040] Optionally, before using the trained neural network to perform noise prediction on the fused noise map to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points, the method further includes: Obtaining a training data set and an initial neural network, the training data set including original singing, a predefined emotion signal, and a predefined rhythm signal; Extracting the Mel spectrum of the original singing voice to obtain a first Mel spectrum corresponding to the original singing voice, adding noise data to the first Mel spectrum to obtain a second Mel spectrum containing the noise data, and determining the second Mel spectrum as a first noise map; Performing emotion feature encoding on a predefined emotion signal to obtain an original emotion encoding feature, performing rhythm feature encoding on a predefined rhythm signal to obtain an original rhythm encoding feature, and adding the original emotion encoding feature and the original rhythm encoding feature to the first noise map to obtain a second noise map; Using the initial neural network to perform noise prediction on the second noise map, obtaining an original predicted noise map including noise points and noise values ​​corresponding to the noise points; The initial neural network is trained according to the first noise map and the original predicted noise map, and the loss value between the first noise map and the original predicted noise map is calculated. When the loss value meets the preset condition, the training is stopped to obtain a trained neural network.

[0041] In this embodiment, when the initial neural network is trained, the original singing voice is obtained, the Mel spectrum of the original singing voice is extracted, the Mel spectrum of the original singing voice is extracted, the first Mel spectrum corresponding to the original singing voice is obtained, noise data is added to the first Mel spectrum, a second Mel spectrum containing noise data is obtained, and the second Mel spectrum is determined as a first noise map, wherein the Mel spectrum of the original singing voice is used as the true value, the noise data is added to the first Mel spectrum, a second Mel spectrum containing noise data is obtained, the second Mel spectrum is determined as a first noise map, the first noise map is used as the added true noise map, the noise data can be a random value generated by a random function, added to the true Mel spectrum, and a second Mel spectrum containing noise data is obtained. A predefined emotional signal and a predefined rhythm signal are added to the corresponding first noise map to obtain a second noise map, and the initial neural network is used to perform noise prediction on the second noise map to obtain an original predicted noise map, which is a noise map containing emotional signals and rhythm signals, so that the initial neural network can learn the contribution of emotional signals and rhythm signals to the model when training.

[0042] The loss value is calculated based on the original predicted noise image predicted by the initial neural network and the first noise image corresponding to the second mel spectrum. When the loss value meets the preset condition, the training is stopped to obtain a trained neural network. The preset condition may be that the loss value is less than a preset threshold, or that the loss value converges.

[0043] Optionally, calculating a loss value between the first noise map and the original predicted noise map includes: Calculate the mean square error loss between the first noise map and the original predicted noise map to obtain a first loss value, and calculate the cross entropy loss between the first noise map and the original predicted noise map to obtain a second loss value; A target loss function is constructed according to the first loss value and the second loss value, and a loss value between the first noise map and the original predicted noise map is calculated according to the target loss function.

[0044] In this embodiment, when calculating the loss value between the first noise map and the third mel spectrum, the sum of the mean square error loss between the first noise map and the original predicted noise map and the cross entropy loss between the first noise map and the original predicted noise map is used as the target loss value, so as to improve the speed and accuracy of model training when training the initial neural network.

[0045] S205: De-noising the predicted noise image to obtain a de-noised noise image, and determining the de-noised noise image as the Mel spectrum to be synthesized.

[0046] In step S205, the horizontal coordinate in the denoised noise map is determined as the frame number in the to-be-synthesized Mel spectrum, and the vertical coordinate in the denoised noise map is determined as the frequency in the to-be-synthesized Mel spectrum. The trained neural network can determine the mapping relationship between the random noise map and the predicted noise map based on the learned knowledge, and map the predicted noise map based on the mapping relationship, thereby removing the noise in the predicted noise map.

[0047] In this embodiment, according to the position of the noise point in the predicted noise map and the noise value of the noise point, the noise value at the noise point is removed through denoising processing to obtain a denoised noise map without noise pollution, and the denoised noise map is determined as the Mel spectrum to be synthesized, wherein the horizontal coordinate in the denoised noise map is determined as the frame number in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise map is determined as the frequency in the Mel spectrum to be synthesized.

[0048] Optionally, denoising the predicted noise map to obtain a denoised noise map includes: According to the preset mapping relationship in the trained neural network, the noise values ​​corresponding to the noise points in the predicted noise map are mapped to obtain the mapped noise values ​​corresponding to the noise points; The mapped noise values ​​are filled into the corresponding noise points to obtain a filled noise map, and the filled noise map is determined as a denoised noise map.

[0049] In this embodiment, the predicted noise map is mapped according to the preset mapping relationship in the trained neural network, and the noise value at the corresponding noise point is removed. For example, the mapping relationship between the random noise map and the predicted noise map is a function F(x), then the pixel points in the predicted noise map are mapped by the function F(x), the noise values ​​in the noise points in the predicted noise map are mapped by the function F(x), and the mapped noise values ​​are filled into the corresponding noise points to obtain the filled noise map, wherein the function F(x) can be a linear function or a nonlinear function.

[0050] In another embodiment, the noise value in the noise point can also be mapped multiple times in steps. For example, the first mapping function is used to perform a first mapping process on the noise value to obtain a first mapping result, and the second mapping function is used to perform a second mapping on the first mapping to obtain a second mapping result, and so on. Multiple mappings can be performed, and the result of each mapping is the original data of the next mapping, wherein the mapping function of each mapping can be the same or different, and can be a linear function or a nonlinear function. In this embodiment, linear functions and nonlinear functions are used for cross-mapping to prevent the corresponding noise from being unable to be completely removed when a single mapping is performed in a noise point with a large noise value.

[0051] S206: Using a preset vocoder to synthesize the Mel-spectrogram to be synthesized into a corresponding singing voice.

[0052] In step S206, a preset vocoder is used to synthesize the mel spectrum to be synthesized into a corresponding singing voice, wherein the preset vocoder may be a neural network vocoder, and the corresponding singing voice is a singing voice with emotion and rhythm obtained by adding an emotion signal and a rhythm signal to a random noise graph.

[0053] In this embodiment, a preset vocoder can be used for singing synthesis. The vocoder in the embodiment of the present application can be a neural network vocoder, which can include but is not limited to an autoregressive-based neural network vocoder (such as a waveRNN vocoder) and a Gan network-based neural network vocoder (such as a melGan vocoder and a hifiGan vocoder).

[0054] According to a preset random function, a random noise value is generated, and the random noise value is added to the corresponding pixel of a predefined two-dimensional image to obtain a random noise map, and the emotion coding feature and the rhythm coding feature are added to the random noise map, and the random noise value in the corresponding pixel in the random noise map is fused to obtain a fused noise map, and the fused noise map is predicted using a trained neural network to obtain a predicted noise map containing noise points and noise values ​​corresponding to the noise points, and the predicted noise map is denoised to obtain a denoised noise map, and the denoised noise map is determined as the mel spectrum to be synthesized corresponding to the song to be synthesized, and the mel spectrum to be synthesized is synthesized into the corresponding song using a preset vocoder. In the present application, by fusing the predefined emotion coding feature and the predefined rhythm coding feature into the noise map, the control of emotion and rhythm in the song synthesis is achieved, and controllable song synthesis is achieved, thereby improving the effect of song synthesis. In the process of music therapy in the medical field, controllable and effective diversified music can be synthesized, which is helpful for patient rehabilitation.

[0055] See also Figure 3 , Figure 3 The structural block diagram of the noise-controllable singing voice synthesis device provided in the embodiment of the present application is shown. The noise-controllable singing voice synthesis device is applied to the above-mentioned server. For the convenience of explanation, only the part related to the embodiment of the present application is shown. Figure 3 The singing voice synthesis device 30 includes: an acquisition module 31, a generation module 32, an addition module 33, a prediction module 34, a denoising module 35, and a synthesis module 36.

[0056] The acquisition module 31 is used to acquire a preset emotion signal, perform emotion feature encoding on the preset emotion signal, obtain emotion encoding features corresponding to the preset emotion signal, acquire a preset rhythm signal, perform rhythm feature encoding on the preset rhythm signal, and obtain rhythm encoding features corresponding to the preset rhythm signal; The generating module 32 is used to generate a random noise value according to a preset random function, and add the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map.

[0057] The adding module 33 is used to add the emotion coding features and the rhythm coding features to the random noise map, and fuse them with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map.

[0058] The prediction module 34 is used to perform noise prediction on the fused noise map using the trained neural network to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points.

[0059] The denoising module 35 is used to denoise the predicted noise image to obtain a denoised noise image, and determine the denoised noise image as the Mel spectrum to be synthesized, wherein the horizontal coordinate in the denoised noise image is determined as the frame number in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise image is determined as the frequency in the Mel spectrum to be synthesized.

[0060] The synthesis module 36 is used to use a preset vocoder to synthesize the to-be-synthesized Mel-spectrogram into corresponding singing voice.

[0061] Optionally, the adding module 33 includes: The determination unit is used to determine the emotion coding feature value and the rhythm coding feature value corresponding to each pixel in the random noise image according to the emotion coding feature and the rhythm coding feature.

[0062] The fusion unit is used to add and fuse the emotion coding feature value, rhythm coding feature value and random noise value corresponding to each pixel in the random noise map to obtain the fusion value of each pixel, and obtain the fusion noise map according to the fusion value of each pixel.

[0063] Optionally, the acquisition module 31 includes: The transformation unit is used to transform the preset emotion signal into a corresponding Fourier spectrum, and determine the corresponding Mel spectrum according to the Fourier spectrum.

[0064] The encoding unit is used to use a preset emotion encoder to encode the emotion feature of the Mel spectrum to obtain the emotion coding feature corresponding to the preset emotion signal.

[0065] Optionally, the singing voice synthesis device 30 further includes: The training data acquisition module is used to obtain a training data set and an initial neural network. The training data set includes original singing, predefined emotional signals, and predefined rhythm signals.

[0066] The extraction module is used to extract the Mel spectrum of the original singing voice, obtain a first Mel spectrum corresponding to the original singing voice, add noise data to the first Mel spectrum, obtain a second Mel spectrum containing the noise data, and determine the second Mel spectrum as a first noise map.

[0067] A module is obtained, which is used to encode the emotion feature of a predefined emotion signal to obtain the original emotion encoding feature, encode the rhythm feature of a predefined rhythm signal to obtain the original rhythm encoding feature, and add the original emotion encoding feature and the original rhythm encoding feature to the first noise map to obtain the second noise map.

[0068] The original prediction module is used to use the initial neural network to perform noise prediction on the second noise map to obtain an original predicted noise map including noise points and noise values ​​corresponding to the noise points.

[0069] The training module is used to train the initial neural network according to the first noise map and the original predicted noise map, calculate the loss value between the first noise map and the original predicted noise map, and stop the training when the loss value meets the preset conditions to obtain the trained neural network.

[0070] Optionally, the training module includes: The calculation unit is used to calculate the mean square error loss between the first Mel spectrum and the third Mel spectrum to obtain a first loss value, and calculate the cross entropy loss between the first Mel spectrum and the third Mel spectrum to obtain a second loss value.

[0071] A construction unit is used to construct a target loss function according to the first loss value and the second loss value, and calculate the loss value between the first Mel spectrum and the third Mel spectrum according to the target loss function.

[0072] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0073] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 4 As shown, the terminal device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the noise-controllable singing synthesis method are implemented.

[0074] The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4It is only an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0075] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0076] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of the terminal device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of the terminal device, and in other embodiments, it may also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0077] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0078] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device implements the steps in the above-mentioned method embodiment when executing the computer program product.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0081] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for synthesizing singing voice based on controllable noise, characterized in that: The singing voice synthesis method comprises: Acquire a preset emotion signal, perform emotion feature encoding on the preset emotion signal, and obtain emotion encoding features corresponding to the preset emotion signal; acquire a preset rhythm signal, perform rhythm feature encoding on the preset rhythm signal, and obtain rhythm encoding features corresponding to the preset rhythm signal; Generate a random noise value according to a preset random function, and add the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map; Adding the emotion coding feature and the rhythm coding feature to the random noise map, and fusing them with the random noise values ​​in corresponding pixels in the random noise map to obtain a fused noise map; Using a trained neural network to perform noise prediction on the fused noise map, to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points; Denoising the predicted noise image to obtain a denoised noise image, and determining the denoised noise image as a Mel spectrum to be synthesized, wherein the horizontal coordinate in the denoised noise image is determined as the frame number in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise image is determined as the frequency in the Mel spectrum to be synthesized; The Mel spectrum to be synthesized is synthesized into a corresponding singing voice using a preset vocoder.

2. The singing voice synthesis method according to claim 1, wherein: The step of adding the emotion coding feature and the rhythm coding feature to the random noise map and fusing them with the random noise values ​​of corresponding pixels in the random noise map to obtain a fused noise map includes: Determine, according to the emotion coding feature and the rhythm coding feature, an emotion coding feature value and a rhythm coding feature value corresponding to each pixel in the random noise image; The emotion coding feature value and the rhythm coding feature value corresponding to each pixel in the random noise map are added and fused with the random noise value to obtain a fusion value of each pixel, and a fused noise map is obtained according to the fusion value of each pixel.

3. The singing voice synthesis method according to claim 1, characterized in that: The step of encoding the preset emotion signal with emotion features to obtain emotion coding features corresponding to the preset emotion signal includes: Converting the preset emotion signal into a corresponding Fourier spectrum, and determining a corresponding Mel spectrum according to the Fourier spectrum; Using a preset emotion encoder, the Mel spectrum is encoded with emotion features to obtain emotion coding features corresponding to the preset emotion signal.

4. The singing voice synthesis method according to claim 1, wherein: The step of performing denoising on the predicted noise map to obtain a denoised noise map includes: Mapping the noise values ​​corresponding to the noise points in the predicted noise map according to the preset mapping relationship in the trained neural network to obtain the mapped noise values ​​corresponding to the noise points; The mapped noise values ​​are filled into corresponding noise points to obtain a filled noise map, and the filled noise map is determined as a denoised noise map.

5. The singing voice synthesis method according to claim 1, characterized in that: Before using the trained neural network to perform noise prediction on the fused noise map to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points, the method further includes: Acquire a training data set and an initial neural network, wherein the training data set includes an original singing voice, a predefined emotion signal, and a predefined rhythm signal; Extracting the Mel spectrum of the original singing voice to obtain a first Mel spectrum corresponding to the original singing voice, adding noise data to the first Mel spectrum to obtain a second Mel spectrum containing the noise data, and determining the second Mel spectrum as a first noise map; Performing emotion feature encoding on the predefined emotion signal to obtain an original emotion encoding feature, performing rhythm feature encoding on the predefined rhythm signal to obtain an original rhythm encoding feature, and adding the original emotion encoding feature and the original rhythm encoding feature to the first noise map to obtain a second noise map; Using the initial neural network to perform noise prediction on the second noise map, to obtain an original predicted noise map including noise points and noise values ​​corresponding to the noise points; The initial neural network is trained according to the first noise map and the original predicted noise map, and a loss value between the first noise map and the original predicted noise map is calculated. When the loss value meets a preset condition, the training is stopped to obtain a trained neural network.

6. The singing voice synthesis method according to claim 5, characterized in that: The calculating the loss value between the first Mel spectrum and the third Mel spectrum includes: Calculate the mean square error loss between the first Mel spectrum and the third Mel spectrum to obtain a first loss value, and calculate the cross entropy loss between the first Mel spectrum and the third Mel spectrum to obtain a second loss value; A target loss function is constructed according to the first loss value and the second loss value, and a loss value between the first Mel spectrum and the third Mel spectrum is calculated according to the target loss function.

7. A singing voice synthesis device based on controllable noise, characterized in that: The singing voice synthesis device comprises: An acquisition module is used to acquire a preset emotion signal, perform emotion feature encoding on the preset emotion signal, obtain emotion encoding features corresponding to the preset emotion signal, acquire a preset rhythm signal, perform rhythm feature encoding on the preset rhythm signal, and obtain rhythm encoding features corresponding to the preset rhythm signal; A generating module, used for generating a random noise value according to a preset random function, and adding the random noise value to a corresponding pixel of a predefined two-dimensional image to obtain a random noise map; An adding module, used for adding the emotion coding feature and the rhythm coding feature to the random noise map, and fusing them with the random noise values ​​in the corresponding pixels in the random noise map to obtain a fused noise map; A prediction module, used to perform noise prediction on the fused noise map using a trained neural network to obtain a predicted noise map including noise points and noise values ​​corresponding to the noise points; A denoising module, configured to perform denoising on the predicted noise image to obtain a denoised noise image, and determine the denoised noise image as a Mel spectrum to be synthesized, wherein the horizontal coordinate in the denoised noise image is determined as the number of frames in the Mel spectrum to be synthesized, and the vertical coordinate in the denoised noise image is determined as the frequency in the Mel spectrum to be synthesized; The synthesis module is used to use a preset vocoder to synthesize the to-be-synthesized Mel-spectrogram into corresponding singing voice.

8. The singing voice synthesizing device according to claim 7, characterized in that: The adding module comprises: A determination unit, used to determine the emotion coding feature value and the rhythm coding feature value corresponding to each pixel in the random noise image according to the emotion coding feature and the rhythm coding feature; The fusion unit is used to add and fuse the emotion coding feature value and rhythm coding feature value corresponding to each pixel in the random noise map with the random noise value to obtain the fusion value of each pixel, and obtain the fused noise map according to the fusion value of each pixel.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the noise-controllable singing synthesis method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the noise-controllable singing synthesis method according to any one of claims 1 to 6 is implemented.

Images