Audio synthesis method, apparatus, device, and storage medium

Abstract

The application discloses an audio synthesis method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring speech audio features of a target object, sound attribute features of the target object, and score features of a target song; inputting the speech audio features of the target object, the sound attribute features of the target object and the score features of the target song into a preset audio synthesis model to perform audio synthesis, and outputting target audio; wherein the audio synthesis model is constructed based on a feature migration model and a combination model. That is, in the application, singing audio data of the target object does not need to be collected, the collection difficulty of input data is reduced, and the audio synthesis efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an audio synthesis method and device, equipment and a storage medium. BACKGROUND

[0002] At present, a method for synthesizing audio based on deep learning includes: collecting a large amount of music score / lyrics data and matching singing audio data as training data, training to obtain an audio synthesis model, inputting music score / lyrics data and singing audio data of a target object to the obtained audio synthesis model, and synthesizing singing audio. The above method for synthesizing audio based on deep learning takes singing audio data as one of the input data, but the singing audio data is recorded by a professional singer in a professional recording studio, and the recorded audio is annotated to obtain the singing audio data, that is, the singing audio data is difficult to obtain, which reduces the audio synthesis efficiency. SUMMARY

[0003] The main purpose of the present application is to provide an audio synthesis method, device, equipment and storage medium, which aims to solve the technical problem of low audio synthesis efficiency of the existing audio synthesis method.

[0004] To achieve the above purpose, the present application provides an audio synthesis method, which comprises the following steps:

[0005] obtaining speech audio features of a target object, sound attribute features of the target object and music score features of a target music piece;

[0006] inputting the speech audio features of the target object, the sound attribute features of the target object and the music score features of the target music piece into a preset audio synthesis model for audio synthesis, and outputting target audio;

[0007] The audio synthesis model is constructed based on a feature transfer model and a combination model.

[0008] Optionally, the step of inputting the speech audio features of the target object, the sound attribute features of the target object and the music score features of the target music piece into the preset audio synthesis model for audio synthesis, and outputting target audio, comprises:

[0009] inputting the speech audio features of the target object into a feature transfer model for feature transfer, and outputting singing audio features;

[0010] merging the singing audio features and the sound attribute features to obtain merged features;

[0011] inputting the merged features and the music score features into a combination model for audio synthesis, and outputting target audio.

[0012] Optionally, the feature transfer model is a multi-layer convolutional neural network model, and the step of inputting the speech audio features of the target object into the feature transfer model for feature transfer and outputting singing audio features includes:

[0013] The speech audio features of the target object are input into a feature transfer model for processing, and the weight matrix parameters of the fully connected layer in the feature transfer model are extracted to form singing audio features.

[0014] Optionally, the audio synthesis method further includes an audio synthesis model training process, which includes:

[0015] Acquire first training data and second training data, wherein the first training data includes speech audio features and singing audio features, and the second training data includes voice attribute features and musical score features;

[0016] The first preset model is trained using the first training data to obtain the feature transfer model;

[0017] A second preset model is trained using the first output data from the feature transfer model and the second training data to obtain a combined model.

[0018] An audio synthesis model is constructed based on the feature transfer model and the combined model.

[0019] Optionally, the step of training a first preset model using the first training data to obtain a feature transfer model includes:

[0020] The speech audio features are input into the first preset model to obtain the first output data output by the first preset model;

[0021] Based on the singing audio features and the first output data, determine the loss value of the loss function corresponding to the first preset model;

[0022] When the loss value exceeds a preset first loss value threshold, the parameters of the first preset model are corrected according to a preset correction method until the loss value is less than the first loss value threshold, at which point training stops and the feature transfer model is obtained.

[0023] Optionally, the first preset model is a multi-layer convolutional neural network model, and the step of inputting the speech audio features into the first preset model to obtain the first output data output by the first preset model includes:

[0024] The speech audio features are input into a multi-layer convolutional neural network model to train the multi-layer convolutional neural network model;

[0025] Obtain the weight matrix parameters of the fully connected layer of the multi-layer convolutional neural network model as the first output data.

[0026] Optionally, the step of training a second preset model using the first output data and the second training data from the feature transfer model to obtain a combined model includes:

[0027] The first output data and the sound attribute features are merged to obtain the first merged feature;

[0028] The first merged feature and the musical score feature are input into the second preset model, and the second preset model is iteratively trained to obtain the combined model.

[0029] Furthermore, to achieve the above objectives, this application also provides an audio synthesis apparatus, the audio synthesis apparatus comprising:

[0030] The feature acquisition module is used to acquire the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target piece;

[0031] An audio synthesis model is used to input the speech audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model to synthesize audio and output the target audio.

[0032] The audio synthesis model is constructed based on a feature transfer model and a combination model.

[0033] In addition, to achieve the above objectives, this application also provides an audio synthesis device, the device comprising: a memory, a processor, and an audio synthesis program stored in the memory and executable on the processor, the audio synthesis program being configured to implement the steps of the audio synthesis method as described above.

[0034] In addition, to achieve the above objectives, this application also provides a storage medium storing an audio synthesis program, which, when executed by a processor, implements the steps of the audio synthesis method described above.

[0035] This application discloses an audio synthesis method, apparatus, device, and storage medium. Compared with the low efficiency of existing audio synthesis technologies, this application acquires the speech audio features, sound attribute features, and musical score features of a target object; inputs these features into a preset audio synthesis model for audio synthesis, and outputs the target audio; wherein the audio synthesis model is constructed based on a feature transfer model and a combination model. In other words, this application eliminates the need to collect the singing audio data of the target object, reducing the difficulty of input data collection and improving audio synthesis efficiency. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the structure of an audio synthesis device in the hardware operating environment involved in the embodiments of this application;

[0039] Figure 2 This is a flowchart illustrating the first embodiment of the audio synthesis method of this application;

[0040] Figure 3 This is a flowchart illustrating the second embodiment of the audio synthesis method of this application;

[0041] Figure 4 This is a functional module diagram of the first embodiment of the audio synthesis device of this application.

[0042] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0044] Reference Figure 1 , Figure 1 This is a schematic diagram of the audio synthesis device structure of the hardware operating environment involved in the embodiments of this application.

[0045] like Figure 1As shown, the audio synthesis device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0046] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on audio synthesis equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0047] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an audio synthesis program. The operating system is a program that manages and controls the hardware and software resources of the audio synthesis device, supporting the operation of the audio synthesis program and other software or programs.

[0048] exist Figure 1 In the audio synthesis device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the audio synthesis device of this application calls the audio synthesis program stored in the memory 1005 through the processor 1001 and executes the audio synthesis method provided in the embodiment of this application.

[0049] This application provides an audio synthesis method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the audio synthesis method of this application.

[0050] In this embodiment, the audio synthesis method includes the following steps:

[0051] Step S10: Obtain the speech audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song.

[0052] This embodiment acquires the speech and audio data of the target object, and based on this data, obtains the speech and audio features and sound attribute features of the target object. The text information in the acquired speech and audio data of the target object can be any text.

[0053] In this embodiment, the speech audio features of the target object can be the speech audio Mel spectrum of the target object. The process of obtaining the speech audio Mel spectrum of the target object includes:

[0054] The method for obtaining the Mel spectrum of speech audio can be as follows: convert the speech audio data of the target object into the Mel spectrum of the target object's speech audio using a Mel filter.

[0055] Another method for obtaining the Mel spectrum of the target object's speech audio can be:

[0056] The sample data is input into the acoustic model to be trained to obtain the predicted Mel spectrum corresponding to the sample data; the predicted Mel spectrum is compared with the actual Mel spectrum corresponding to the sample data, and the acoustic model is updated in reverse based on the comparison result.

[0057] The speech audio data is input into the trained acoustic model to obtain the speech audio Mel spectrum.

[0058] It should be noted that the sample data are speech audio data with known Mel spectrum, and there are no specific restrictions on their source. They can be a segment of speech audio data from broadcast information, a segment of speech audio data from television / movies, or a segment of speech audio data recorded by a recording device.

[0059] It should be noted that the acoustic model includes one of the following: Hidden Markov Model (HMM), ChariNet model, and Tacotron2 model.

[0060] In this embodiment, the sound attribute features of the target object can be the timbre features or phoneme features of the target object.

[0061] The method for obtaining the timbre features (phoneme features) of the target object is as follows:

[0062] The sample data is input into the encoder to be trained to obtain the predicted timbre features (phoneme features) corresponding to the sample data; the predicted timbre features (phoneme features) are compared with the actual timbre features (phoneme features) corresponding to the sample data, and the encoder is updated in reverse based on the comparison results;

[0063] Input the speech audio data of the target object into the encoder that has completed training to obtain the timbre feature (phoneme feature) of the target object.

[0064] It should be noted that the sample data is speech audio data with known corresponding timbre features (phoneme features). There is no specific limitation on the source of the speech audio data. It can be a segment of speech audio data in broadcast information, a segment of speech audio data in TV / movies, or a segment of speech audio data recorded by a recording device.

[0065] It should be noted that the encoder to be trained includes one of the encoders in the tactron model, the encoder in the fastspeech model, and the encoder in the deepVoice model.

[0066] In this embodiment, the score features of the target track include lyric information features and melody features. Among them, the method for obtaining the lyric information features is as follows:

[0067] Extract the phoneme information corresponding to each character according to the pronunciation of each character in the lyrics of the target track;

[0068] Perform feature transformation on the phoneme information to generate lyric information features.

[0069] A phoneme is the smallest speech unit divided according to the natural attributes of speech. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, and "dài" has three phonemes.

[0070] To perform feature transformation on the phoneme information, the corresponding feature vector of the phoneme information can be obtained by querying a preset feature element-phoneme information mapping table, and lyric information features are generated based on the feature vectors corresponding to each phoneme information.

[0071] Among them, the method for obtaining the melody feature is as follows:

[0072] Extract the notes, note durations, beats, slurs, and sustains in the score of the target track;

[0073] Perform feature transformation on the notes, the note durations, the beats, the slurs, and the sustains to generate melody features.

[0074] Feature transformation is performed on the notes, note values, beats, legato, and sustains. This can be achieved by querying a pre-defined feature vector mapping table to obtain the feature vectors corresponding to each of the notes, note values, beats, legato, and sustains. Based on these feature vectors, melodic feature information is generated. For example, notes correspond to a pre-defined feature element-note mapping table, and note values ​​correspond to a pre-defined feature element-note value mapping table, etc.

[0075] Step S20: Input the speech audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model to synthesize audio and output the target audio. The audio synthesis model is constructed based on a feature transfer model and a combination model.

[0076] Specifically, the steps of inputting the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model for audio synthesis and outputting the target audio include:

[0077] Step S21: Input the speech audio features of the target object into the feature transfer model for feature transfer, and output the singing audio features.

[0078] The feature transfer model can be a multi-layer convolutional neural network model. When the feature transfer model is a multi-layer convolutional neural network model, the feature transfer model consists of a multi-layer convolutional submodule and a fully connected layer.

[0079] When the feature transfer model is a multi-layer convolutional neural network model, the steps of inputting the speech audio features of the target object into the feature transfer model for feature transfer and outputting the singing audio features include:

[0080] The speech audio features of the target object are input into a feature transfer model for processing, and the weight matrix parameters of the fully connected layer in the feature transfer model are extracted to form singing audio features.

[0081] For example, the speech audio features (speech audio Mel spectrum) of the target object are input into a feature transfer model for convolution processing. Then, the weight matrix parameters of the fully connected layers of the feature transfer model are extracted to obtain the singing audio features. The singing audio features can be represented as an n-dimensional vector, where n can be set according to the specific application, with a default value of 256.

[0082] Step S22: Merge the singing audio features and the sound attribute features of the target object to obtain merged features.

[0083] It should be noted that in this embodiment, the singing audio features are represented in the form of an n-dimensional vector. To simplify the merging calculation of the singing audio features and the sound attribute features, in this embodiment, the sound attribute features are also represented in the form of an n-dimensional vector. Therefore, merging the singing audio features and the sound attribute features is simply adding two n-dimensional vectors.

[0084] Step S23: Input the merged features and the score features of the target track into the combination model for audio synthesis and output the target audio.

[0085] Compared to existing technologies with low audio synthesis efficiency, this application acquires the speech audio features, sound attribute features, and musical score features of the target object; it then inputs these features into a preset audio synthesis model to synthesize the target audio. The audio synthesis model is constructed based on a feature transfer model and a combination model. In other words, this application eliminates the need to collect the target object's singing audio data, reducing the difficulty of input data acquisition and improving audio synthesis efficiency.

[0086] Furthermore, if the deep learning-based audio synthesis method described in the background technique is used to synthesize user A's singing audio, the singing audio data in the training data is user A's singing audio data. Moreover, the audio synthesis model trained based on user A's singing audio data can only be applied to user A and not to other users, resulting in poor versatility. Therefore, the audio synthesis method of this application only needs to obtain the speech audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song, and input them into the audio synthesis model to obtain the target audio of the target object. It is not limited to a single target object, thus improving the versatility of the audio synthesis model.

[0087] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the audio synthesis method of this application.

[0088] Based on the first embodiment described above, in this embodiment, the audio synthesis method further includes an audio synthesis model training process, which includes:

[0089] Step A10: Obtain first training data and second training data, wherein the first training data includes speech audio features and singing audio features, and the second training data includes sound attribute features and musical score features.

[0090] It should be noted that, in this embodiment, the speech audio feature is the speech audio Mel spectrum, and the singing audio feature is the singing audio Mel spectrum.

[0091] The method for obtaining the Mel spectrum of the speech audio can be as follows: convert the speech audio data into a Mel spectrum using a Mel filter. The source of the speech audio data is not specifically limited; it can be a segment of speech audio data from a broadcast, a segment of speech audio data from a television / movie, or a segment of speech audio data recorded by a recording device.

[0092] Other methods for obtaining the Mel spectrum of speech audio include:

[0093] The sample data is input into the acoustic model to be trained to obtain the predicted Mel spectrum corresponding to the sample data; the predicted Mel spectrum is compared with the actual Mel spectrum corresponding to the sample data, and the acoustic model is updated in reverse based on the comparison result.

[0094] The speech audio data is input into the trained acoustic model to obtain the speech audio Mel spectrum.

[0095] It should be noted that the sample data are speech audio data with known Mel spectrum.

[0096] It should be noted that the acoustic model includes one of the following: Hidden Markov Model (HMM), ChariNet model, and Tacotron2 model.

[0097] Similarly, the method for obtaining the Mel spectrum of singing audio can be as follows: convert the singing audio data into a Mel spectrum using a Mel filter. The source of the singing audio data is not specifically limited; it can be a segment of singing audio data from broadcast information, a segment of singing audio data from television / movies, or a segment of singing audio data recorded by a recording device.

[0098] Other methods for obtaining the Mel spectrum of singing audio include:

[0099] The sample data is input into the acoustic model to be trained to obtain the predicted Mel spectrum corresponding to the sample data; the predicted Mel spectrum is compared with the actual Mel spectrum corresponding to the sample data, and the acoustic model is updated in reverse based on the comparison result.

[0100] The singing audio data is input into the trained acoustic model to obtain the Mel spectrum of the singing audio.

[0101] It should be noted that the sample data are singing audio data with known Mel spectrum.

[0102] It should be noted that the acoustic model includes one of the following: Hidden Markov Model (HMM), ChariNet model, and Tacotron2 model.

[0103] It should be noted that, in this embodiment, the sound attribute features can be timbre features or phoneme features.

[0104] The method for obtaining timbre features (phoneme features) is the same as the method for obtaining the timbre features (phoneme features) of the target object in step S10, and will not be repeated here.

[0105] It should be noted that, in this embodiment, the musical score features of the track include lyric information features and melody features. The methods for obtaining lyric information features and melody features are the same as those in step S10, and will not be repeated here.

[0106] Step A20: Train the first preset model using the first training data to obtain the feature transfer model.

[0107] Specifically, the step of training a first preset model using the first training data to obtain a feature transfer model includes:

[0108] Step A21: Input the speech audio features into the first preset model to obtain the first output data output by the first preset model.

[0109] When the first preset model is a multi-layer convolutional neural network model, the speech audio features are input into the multi-layer convolutional neural network model to train the multi-layer convolutional neural network model; the weight matrix parameters of the fully connected layer of the multi-layer convolutional neural network model are obtained as the first output data.

[0110] It should be noted that in this embodiment, the weight matrix parameters are represented as an n-dimensional vector. The value of n can be set according to the specific application, and the default value is 256.

[0111] Step A22: Based on the singing audio features and the first output data, determine the loss value of the loss function corresponding to the first preset model.

[0112] It should be noted that, in this embodiment, the loss function is the cross-entropy loss function.

[0113] Step A23: When the loss value exceeds the preset first loss value threshold, the parameters of the first preset model are corrected according to the preset correction method until the loss value is less than the first loss value threshold, and then training is stopped to obtain the feature transfer model.

[0114] It should be noted that in this embodiment, when the first preset model is a multi-layer convolutional neural network model, the feature transfer model obtained after training is also a multi-layer convolutional neural network model.

[0115] Step A30: Use the first output data and the second training data output by the feature transfer model to train the second preset model to obtain the combined model.

[0116] Specifically, the step of training a second preset model using the first output data and the second training data from the feature transfer model to obtain a combined model includes:

[0117] Step A31: Merge the first output data and the sound attribute features to obtain the first merged feature.

[0118] It should be noted that when the first preset model is a multi-layer convolutional neural network model, the feature transfer model obtained after training is also a multi-layer convolutional neural network model, and the first output data of the feature transfer model is an n-dimensional vector.

[0119] To simplify the merging calculation of the first output data and the sound attribute features, in this embodiment, the sound attribute features are also represented as n-dimensional vectors. Therefore, merging the first output data and the sound attribute features is simply adding two n-dimensional vectors.

[0120] Step A32: Input the first merged feature and the musical score feature into the second preset model, and perform iterative training on the second preset model to obtain the combined model.

[0121] It should be noted that, in this embodiment, the second preset model can be one of the tactron model, the fastspeech model, or the deepVoice model, and all of the above second preset models include an encoding module and a decoding module.

[0122] Step A40: Construct an audio synthesis model based on the feature transfer model and the combined model.

[0123] It should be noted that in this embodiment, the feature transfer model and the combination model are trained simultaneously.

[0124] This application also provides an audio synthesis apparatus, referring to... Figure 4 , Figure 4 This is a functional module diagram of the first embodiment of the audio synthesis device of this application.

[0125] In this embodiment, the audio synthesis device includes:

[0126] The feature acquisition module 10 is used to acquire the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song.

[0127] Audio synthesis model 20 is used to input the speech audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model to synthesize audio and output the target audio.

[0128] The audio synthesis model is constructed based on a feature transfer model and a combination model.

[0129] Optionally, the audio synthesis model includes:

[0130] The singing audio feature generation subunit is used to input the speech audio features of the target object into the feature transfer model for feature transfer and output the singing audio features.

[0131] The feature merging subunit is used to merge the singing audio features and the sound attribute features of the target object to obtain merged features;

[0132] The audio synthesis subunit is used to input the merging features and the musical score features of the target track into the merging model for audio synthesis and output the target audio.

[0133] Optionally, the feature transfer model is a multi-layer convolutional neural network model, and the singing audio feature generation subunit is used to implement:

[0134] The speech audio features of the target object are input into a feature transfer model for processing, and the weight matrix parameters of the fully connected layer in the feature transfer model are extracted to form singing audio features.

[0135] Optionally, the audio synthesis apparatus further includes a model training module, the model training module comprising:

[0136] The training data acquisition unit is used to acquire first training data and second training data, wherein the first training data includes speech audio features and singing audio features, and the second training data includes sound attribute features and musical score features;

[0137] The first model training unit is used to train a first preset model using the first training data to obtain a feature transfer model.

[0138] The second model training unit is used to train a second preset model using the first output data and the second training data output by the feature transfer model to obtain a combined model.

[0139] The model building unit is used to build an audio synthesis model based on the feature transfer model and the combined model.

[0140] Optionally, the first model training unit is used to implement:

[0141] The speech audio features are input into the first preset model to obtain the first output data output by the first preset model;

[0142] Based on the singing audio features and the first output data, determine the loss value of the loss function corresponding to the first preset model;

[0143] When the loss value exceeds a preset first loss value threshold, the parameters of the first preset model are corrected according to a preset correction method until the loss value is less than the first loss value threshold, at which point training stops and the feature transfer model is obtained.

[0144] Optionally, the first preset model is a multi-layer convolutional neural network model, and the first model training unit is further used to implement:

[0145] The speech audio features are input into a multi-layer convolutional neural network model to train the multi-layer convolutional neural network model;

[0146] Obtain the weight matrix parameters of the fully connected layer of the multi-layer convolutional neural network model as the first output data.

[0147] Optionally, the second model training unit is used to: merge the first output data and the sound attribute features to obtain a first merged feature;

[0148] The first merged feature and the musical score feature are input into the second preset model, and the second preset model is iteratively trained to obtain the combined model.

[0149] The specific implementation of the audio synthesis device in this application is basically the same as the embodiments of the audio synthesis method described above, and will not be repeated here.

[0150] This application embodiment also provides a storage medium storing an audio synthesis program, which, when executed by a processor, implements the steps of the audio synthesis method described above.

[0151] The specific implementation of the storage medium in this application is basically the same as the embodiments of the audio synthesis method described above, and will not be repeated here.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0153] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0155] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An audio synthesis method, characterized in that, The audio synthesis method includes the following steps: Obtain the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target piece; The speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song are input into a preset audio synthesis model to synthesize audio and output the target audio. The audio synthesis model is constructed based on a feature transfer model and a combination model. The step of inputting the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model for audio synthesis and outputting the target audio includes: The speech audio features of the target object are input into a feature transfer model for feature transfer, and the singing audio features are output. The singing audio features and the sound attribute features are merged to obtain the merged features; The merging features and the musical score features are input into the combination model for audio synthesis, and the target audio is output.

2. The audio synthesis method as described in claim 1, characterized in that, The feature transfer model is a multi-layer convolutional neural network model. The step of inputting the speech audio features of the target object into the feature transfer model for feature transfer and outputting singing audio features includes: The speech audio features of the target object are input into a feature transfer model for processing, and the weight matrix parameters of the fully connected layer in the feature transfer model are extracted to form singing audio features.

3. The audio synthesis method as described in claim 1, characterized in that, The audio synthesis method further includes an audio synthesis model training process, which includes: Acquire first training data and second training data, wherein the first training data includes speech audio features and singing audio features, and the second training data includes voice attribute features and musical score features; The first preset model is trained using the first training data to obtain the feature transfer model; A second preset model is trained using the first output data from the feature transfer model and the second training data to obtain a combined model. An audio synthesis model is constructed based on the feature transfer model and the combined model.

4. The audio synthesis method as described in claim 3, characterized in that, The step of training a first preset model using the first training data to obtain a feature transfer model includes: The speech audio features are input into the first preset model to obtain the first output data output by the first preset model; Based on the singing audio features and the first output data, determine the loss value of the loss function corresponding to the first preset model; When the loss value exceeds a preset first loss value threshold, the parameters of the first preset model are corrected according to a preset correction method until the loss value is less than the first loss value threshold, at which point training stops and the feature transfer model is obtained.

5. The audio synthesis method as described in claim 4, characterized in that, The first preset model is a multi-layer convolutional neural network model. The step of inputting the speech audio features into the first preset model and obtaining the first output data output by the first preset model includes: The speech audio features are input into a multi-layer convolutional neural network model to train the multi-layer convolutional neural network model; Obtain the weight matrix parameters of the fully connected layer of the multi-layer convolutional neural network model as the first output data.

6. The audio synthesis method as described in claim 3, characterized in that, The step of training a second preset model using the first output data and the second training data from the feature transfer model to obtain a combined model includes: The first output data and the sound attribute features are merged to obtain the first merged feature; The first merged feature and the musical score feature are input into the second preset model, and the second preset model is iteratively trained to obtain the combined model.

7. An audio synthesis device, characterized in that, The audio synthesis device includes: The feature acquisition module is used to acquire the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target piece; An audio synthesis model is used to input the speech audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model to synthesize audio and output the target audio. The audio synthesis model is constructed based on a feature transfer model and a combination model. The step of inputting the speech and audio features of the target object, the sound attribute features of the target object, and the musical score features of the target song into a preset audio synthesis model for audio synthesis and outputting the target audio includes: The speech audio features of the target object are input into a feature transfer model for feature transfer, and the singing audio features are output. The singing audio features and the sound attribute features are merged to obtain the merged features; The merging features and the musical score features are input into the combination model for audio synthesis, and the target audio is output.

8. An audio synthesis device, characterized in that, The device includes: a memory, a processor, and an audio synthesis program stored in the memory and executable on the processor, the audio synthesis program being configured to implement the steps of the audio synthesis method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores an audio synthesis program, which, when executed by a processor, implements the steps of the audio synthesis method as described in any one of claims 1 to 6.

Images