Audio synthesis method, electronic device and readable storage medium

Abstract

The present application discloses an audio synthesis method, device and computer-readable storage medium, the method comprising: obtaining song audio to be processed and corresponding song information; performing vocal separation processing on the song audio to be processed to obtain vocal audio; determining target timbre information from multiple candidate timbre information based on the vocal audio; obtaining a text template, and generating a text to be processed based on the text template and song information; performing speech synthesis processing on the text to be processed based on the target timbre information to obtain audio to be synthesized; performing synthesis processing on the audio to be synthesized and the song audio to be processed to obtain synthesized audio; the quality of the synthesized audio obtained by this method is not limited to the level of manual broadcasting, so that the quality of the synthesized audio is high and the playback effect is better.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio synthesis method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the advancement of artificial intelligence, speech synthesis technology has now enabled the synthesis of text into highly realistic audio. Synthesized audio is commonly used in scenarios such as news broadcasts, book readings, and human-computer dialogue. Furthermore, users are already accustomed to listening to a wide variety of music. However, how can we combine music and speech to create a new audio format that allows people to enjoy music while also hearing soothing or inspiring voices? The current solution is radio broadcasting, where the host alternates between speaking and playing predetermined songs for the audience. However, this approach requires human interaction, and its effectiveness is limited by the host's voice skills. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide an audio synthesis method, an electronic device and a computer-readable storage medium, which are not limited by the level of manual broadcasting and have better playback effects.

[0004] To solve the above technical problems, in a first aspect, the present application provides an audio synthesis method, comprising:

[0005] Get the audio of the song to be processed and the corresponding song information;

[0006] Performing vocal separation processing on the song audio to be processed to obtain vocal audio;

[0007] Determining target timbre information from a plurality of candidate timbre information based on the human voice audio;

[0008] Acquire a text template, and generate a text to be processed based on the text template and the song information;

[0009] Based on the target timbre information, performing speech synthesis processing on the text to be processed to obtain audio to be synthesized;

[0010] The audio to be synthesized and the song audio to be processed are synthesized to obtain synthesized audio.

[0011] Optionally, determining target timbre information from a plurality of candidate timbre information based on the human voice audio includes:

[0012] Extracting a voiceprint feature vector corresponding to the human voice audio;

[0013] Generating cosine similarities between the voiceprint feature vector and each candidate voiceprint feature vector; each candidate voiceprint feature vector corresponds to each candidate timbre information;

[0014] The candidate timbre information corresponding to the maximum cosine similarity is determined as the target timbre information.

[0015] Optionally, determining target timbre information from a plurality of candidate timbre information based on the human voice audio includes:

[0016] Inputting the human voice audio into a timbre matching model trained based on candidate audios to obtain similarity probabilities between the human voice audio and each of the candidate audios;

[0017] The candidate timbre information of the candidate audio corresponding to the maximum similarity probability is determined as the target timbre information.

[0018] Optionally, the song information includes multiple pieces of song sub-information, and generating the text to be processed based on the text template and the song information includes:

[0019] Based on the information type of each song sub-information, each song sub-information is inserted into a corresponding position in the text template to obtain the text to be processed.

[0020] Optionally, performing speech synthesis processing on the text to be processed based on the target timbre information to obtain the audio to be synthesized includes:

[0021] Inputting the target timbre information and the text to be processed into a feature coding model to obtain a text code;

[0022] The text code is input into a vocoder model to obtain the audio to be synthesized.

[0023] Optionally, the training process of the feature coding model and the vocoder model includes:

[0024] Obtaining training data corresponding to each candidate timbre information; wherein the training data includes training text and corresponding candidate audio;

[0025] Inputting the training text into an initial feature coding model to obtain a training code;

[0026] Inputting the training code into an initial vocoder model to obtain training output audio;

[0027] Extracting a label code of the same type as the training code for the candidate audio, and generating a first loss value using the label code and the training code;

[0028] generating a second loss value using the candidate audio and the training output audio;

[0029] Using the first loss value to adjust parameters of the initial feature encoding model, and using the second loss value to adjust features of the initial vocoder model;

[0030] When it is detected that the training completion condition is met, the initial feature coding model after parameter adjustment is determined as the feature coding model, and the initial vocoder model after parameter adjustment is determined as the vocoder model.

[0031] Optionally, the synthesizing process of the audio to be synthesized and the song audio to be processed includes:

[0032] Determine the accompaniment time interval corresponding to the song audio to be processed;

[0033] Determining a synthesis position type based on the template information, and screening the accompaniment time interval using the synthesis position type to obtain a target accompaniment time interval;

[0034] If the length of the target accompaniment time interval is not less than the time length of the audio to be synthesized, synthesizing the audio to be synthesized with the portion of the song audio to be processed corresponding to the target accompaniment time interval;

[0035] and / or,

[0036] If the length of the target accompaniment time interval is less than the time length of the audio to be synthesized, the audio to be synthesized and the song audio to be processed are spliced ​​based on the type of the target accompaniment time interval.

[0037] Optionally, the splicing processing of the audio to be synthesized and the song audio to be processed based on the type of the target accompaniment time interval includes:

[0038] If the type of the target accompaniment time interval is a prelude, splicing the audio to be synthesized before the audio of the song to be processed;

[0039] If the type of the target accompaniment time interval is an ending, the audio to be synthesized is spliced ​​after the audio of the song to be processed.

[0040] Optionally, determining the accompaniment time interval corresponding to the song audio to be processed includes:

[0041] Use the voice detection model to detect the voice of the song audio to be processed and obtain multiple starting time points and ending time points;

[0042] Remove adjacent end time points and start time points whose distance is less than a preset threshold to obtain the target start time point and target end time point;

[0043] The target start time point and the target end time point are used to determine the accompaniment time interval.

[0044] In a second aspect, the present application further provides an electronic device, including a memory and a processor, wherein:

[0045] The memory is used to store computer programs;

[0046] The processor is used to execute the computer program to implement the above-mentioned audio synthesis method.

[0047] In a third aspect, the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above-mentioned audio synthesis method when executed by a processor.

[0048] The audio synthesis method provided by the present application obtains the audio of the song to be processed and the corresponding song information; performs vocal separation processing on the audio of the song to be processed to obtain vocal audio; based on the vocal audio, determines the target timbre information from multiple candidate timbre information; obtains a text template, and generates the text to be processed based on the text template and the song information; performs speech synthesis processing on the text to be processed based on the target timbre information to obtain the audio to be synthesized; and performs synthesis processing on the audio to be synthesized and the audio of the song to be processed to obtain the synthesized audio.

[0049] It can be seen that the method separates the human voice and uses the obtained human voice audio to determine the target timbre information from multiple candidate timbre information. The target timbre information is timbre information that is similar to the timbre of the human voice part in the song audio to be processed. After the text to be processed is generated, the content of the human voice in the audio to be synthesized based on the target timbre information is the same as the text to be processed, and the timbre is similar to the timbre of the singer of the song audio to be processed. In the synthesized audio obtained by synthesizing the audio to be synthesized and the song audio to be processed, the timbre of the human voice part remains similar before and after, and has a good listening experience. At the same time, the quality of the audio to be synthesized is high and is not limited by the level of manual broadcasting, so that the quality of the synthesized audio is high and the playback effect is better, which solves the problem of requiring manual participation and the effect being limited by the broadcasting level of the anchor.

[0050] In addition, the present application also provides an electronic device and a computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0052] Figure 1 A schematic diagram of a hardware composition framework applicable to an audio synthesis method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a hardware composition framework applicable to another audio synthesis method provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of an audio synthesis method provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of an audio synthesis position provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the entire audio synthesis process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] For ease of understanding, the hardware composition framework used in the solution corresponding to the audio synthesis method provided in the embodiment of the present application is first introduced. Figure 1 , Figure 1 Schematic diagram of a hardware component framework applicable to an audio synthesis method provided in an embodiment of the present application. The electronic device 100 may include a processor 101 and a memory 102, and may further include one or more of a multimedia component 103, an information input / output (I / O) interface 104, and a communication component 105.

[0059] The processor 101 is used to control the overall operation of the electronic device 100 to complete all or part of the steps in the audio synthesis method; the memory 102 is used to store various types of data to support the operation of the electronic device 100. Such data may include, for example, instructions for any application or method operating on the electronic device 100, as well as application-related data. The memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. In this embodiment, the memory 102 stores at least programs and / or data for implementing the following functions:

[0060] Get the audio of the song to be processed and the corresponding song information;

[0061] Perform vocal separation processing on the song audio to be processed to obtain vocal audio;

[0062] Determining target timbre information from a plurality of candidate timbre information based on human voice audio;

[0063] Obtaining a text template, and generating a text to be processed based on the text template and song information;

[0064] Based on the target timbre information, speech synthesis processing is performed on the text to be processed to obtain the audio to be synthesized;

[0065] The audio to be synthesized and the song audio to be processed are synthesized to obtain synthesized audio.

[0066] The multimedia component 103 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 102 or sent through the communication component 105. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 104 provides an interface between the processor 101 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 105 is used for wired or wireless communication between the electronic device 100 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 105 may include: a Wi-Fi component, a Bluetooth component, an NFC component.

[0067] The electronic device 100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the audio synthesis method.

[0068] certainly, Figure 1 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device 100 may include Figure 1 More or fewer components than shown, or combinations of certain components.

[0069] It is understandable that the number of electronic devices in the present embodiment is not limited, and multiple electronic devices may work together to complete the audio synthesis method. Figure 2 , Figure 2 Schematic diagram of the hardware composition framework applicable to another audio synthesis method provided in the embodiment of the present application. Figure 2It can be seen that the hardware composition framework may include: a first electronic device 11 and a second electronic device 12 , which are connected via a network 13 .

[0070] In the embodiment of the present application, the hardware structure of the first electronic device 11 and the second electronic device 12 can refer to Figure 1 In other words, the electronic device 100 in this embodiment has two electronic devices 100, and the two devices exchange data. Furthermore, the embodiment of the present application does not limit the form of the network 13, that is, the network 13 can be a wireless network (such as WIFI, Bluetooth, etc.) or a wired network.

[0071] Among them, the first electronic device 11 and the second electronic device 12 can be the same electronic device, such as the first electronic device 11 and the second electronic device 12 are both servers; or they can be different types of electronic devices, for example, the first electronic device 11 can be a smartphone or other smart terminal, and the second electronic device 12 can be a server. In one possible implementation, a server with strong computing power can be used as the second electronic device 12 to improve data processing efficiency and reliability, thereby improving the processing efficiency of audio synthesis. At the same time, a smartphone with low cost and wide application range is used as the first electronic device 11 to realize the interaction between the second electronic device 12 and the user. It can be understood that the interaction process can be: the smartphone obtains song information and sends the song information to the server, and the server generates the synthesized audio. The server sends the synthesized audio to the smartphone, and the smartphone plays the synthesized audio.

[0072] For details, please refer to Figure 3 , Figure 3 This is a flow chart of an audio synthesis method provided in an embodiment of the present application. The method in this embodiment includes:

[0073] S101: Acquire the audio of the song to be processed and the corresponding song information.

[0074] Song information refers to information used to identify a song, which may include the song title, singer name, etc. In addition, song information can be used to generate the audio to be synthesized. To enrich the content of the synthesized audio, song information may also include additional information, such as the background and purpose of the song's creation.

[0075] The song audio to be processed refers to the original song audio that needs to be synthesized to obtain a broadcasting effect. The broadcasting effect is the effect obtained after inserting song-related voice into the song, which can make the song sound better when played.

[0076] Depending on the content of the song information, the method for obtaining the song audio to be processed varies. For example, if the song information itself includes the song audio to be processed, the song audio to be processed can be directly filtered from the song information. If the song information does not include the song audio to be processed, the identity of the song audio to be processed can be uniquely determined based on information such as the song title and singer name, and the song audio to be processed can be obtained from a preset song audio file library.

[0077] S102: Perform vocal separation processing on the song audio to be processed to obtain vocal audio.

[0078] In order to make the vocal timbre in the audio to be synthesized close to the vocal timbre of the singer of the song audio to be processed, and thus to keep the vocal timbre in the entire synthesized audio relatively consistent and improve the playback effect, this application takes the vocals of the singer of the song audio to be processed as the basis, selects the most similar one from the multiple candidate timbres, and generates the audio to be synthesized. To achieve this purpose, it is first necessary to determine the vocal timbre of the singer of the song audio to be processed. Specifically, the song audio to be processed is subjected to vocal separation processing to obtain vocal audio, which is the effect of the singer singing a cappella without instrumental accompaniment in terms of auditory perception. This embodiment does not limit the specific method of vocal separation processing. For example, a trained vocal separation model can be used for processing, and the vocal separation model can be trained using clear vocal audio and clean accompaniment audio. When performing vocal separation processing, the song audio to be processed is input into the vocal separation model, and the vocal separation model splits and outputs the vocal audio and accompaniment audio. There is no limitation on the specific type of the human voice separation model. For example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), or a complex domain UNET model can be selected.

[0079] S103: Determine target timbre information from a plurality of candidate timbre information based on the human voice audio.

[0080] Candidate timbre information refers to the identification information corresponding to multiple candidate voice timbre candidates. The target timbre information is the one that most closely matches the singer's timbre in the vocal audio. This target timbre information is used to select the vocal timbre to be used during speech synthesis, ensuring that the synthesized audio's vocal timbre closely resembles the singer's actual timbre. The specific form of timbre information is not limited; for example, it can be a candidate voice number or name.

[0081] This embodiment does not limit the specific method for determining the target timbre information. Specifically, features that characterize timbre in the human voice audio can be extracted and matched with the timbre-characterizing features corresponding to each candidate timbre information. The closest match is determined as the target timbre information. In one embodiment, a voiceprint feature vector corresponding to the human voice audio can be extracted. The specific method for extracting the voiceprint feature vector is not limited, as long as it can characterize the timbre of the human voice audio. Furthermore, candidate voiceprint feature vectors are pre-generated, each corresponding to a respective candidate timbre information. These vectors are obtained by extracting voiceprint features from the candidate speech audio corresponding to each candidate timbre information. It is understood that all voiceprint feature vectors are extracted using the same method, for example, an I-Vector model or an X-Vector model can be used for feature extraction. Cosine similarities are calculated between the voiceprint feature vectors and each candidate voiceprint feature vector, and the candidate timbre information corresponding to the maximum cosine similarity is determined as the target timbre information. It is understood that the maximum cosine similarity indicates the closest match between the two.

[0082] In another embodiment, the human voice audio can be input into a timbre matching model obtained based on candidate audio training to obtain the similarity probability between the human voice audio and each candidate audio. Specifically, the timbre matching model is obtained by training each candidate speech audio. After the human voice audio is input into it, the timbre matching model can output the similarity probability between the human voice audio and each candidate audio (i.e., candidate speech audio). The similarity probability refers to the probability that the matching model believes that the timbre of the human voice audio is the same as that of the candidate audio. The candidate timbre information of the candidate audio corresponding to the maximum similarity probability is determined as the target timbre information. The maximum similarity probability indicates that the possibility that the human voice audio and the candidate audio have the same timbre is the greatest, that is, the two are most similar.

[0083] S104: Acquire a text template, and generate a text to be processed based on the text template and song information.

[0084] The text to be processed refers to the content text corresponding to the audio to be synthesized. It is generated based on song information and a text template. The text to be processed based on the song information has a strong correlation with the song audio to be processed, thereby making the audio to be synthesized related to the song audio to be processed. Typically, song information contains a lot of content, for example, it may include multiple song sub-information items. In this case, each song sub-information item can be inserted into the corresponding position in the text template based on its information type to obtain the text to be processed. For example, song information includes three song sub-information items: song title, singer name, and additional information. The text template is: "Welcome to listen to song B sung by A. This song wants C." Where A is the singer name, B is the song title, and C is the song's purpose, for example, "Encourage everyone to maintain a positive attitude in difficult situations," which is additional information. Text templates can have different types. Through this text organization method, song information and the text content to be expressed are integrated to obtain a variety of different types of text to be processed, including but not limited to encouragement, straightforward narrative, happy sharing, and humorous types.

[0085] This embodiment does not limit the specific method of obtaining the text template. In one embodiment, the text template can be obtained based on the template information. Template information refers to information used to describe the text template for generating the text to be processed corresponding to the audio to be synthesized. It can specifically be an identifier that identifies the text template, such as a serial number or a name. This embodiment does not limit the specific method of obtaining the template information and song information. For example, in one embodiment, it is possible to interact with the user through data interaction components, such as a keyboard, a mouse, a touch screen, etc., and obtain any one of the above two types of information based on the data transmitted by the user through the data interaction components. In another embodiment, data interaction can be performed with other electronic devices, or song information and / or template information can be obtained from a specified storage path.

[0086] It should be noted that the number of texts to be processed can be one or more, that is, for a song audio to be processed, it can correspond to multiple audios to be synthesized, and the content of the text to be processed corresponding to each audio to be synthesized can be different.

[0087] S105: Based on the target timbre information, perform speech synthesis processing on the text to be processed to obtain the audio to be synthesized.

[0088] The target timbre information can be used to specify the vocal timbre of the audio to be synthesized, and the text to be processed can be used to determine the vocal content of the audio to be synthesized. Therefore, based on the target timbre information, the text to be processed is subjected to speech synthesis processing to obtain the audio to be synthesized. This embodiment does not limit the specific method of speech synthesis processing. In a feasible implementation method, the target timbre information and the text to be processed are input into a feature coding model to obtain a text encoding. The feature coding model is used to determine an encoding method based on the target timbre information. The encoding method is related to the vocal timbre, and the text to be processed is encoded using this encoding method. The vocoder model is used to output the corresponding audio according to the text encoding. The text encoding is input into the vocoder model to obtain the audio to be synthesized.

[0089] It is understandable that the way the vocoder model recognizes text encoding should match the feature encoding model. To ensure this, the vocoder model and the feature encoding model can be trained together. During the training process, first obtain the training data corresponding to each candidate timbre information; wherein, the training data includes the training text and the corresponding candidate audio, and each candidate timbre corresponding to the candidate audio is consistent with the candidate timbre of the candidate audio in step S102, and can be obtained by several candidates reciting the same different content. The specific content of the training text data is not limited, for example, it can include text, Chinese pinyin data, pause identification data, etc. The pause identification data is used to indicate where the text is broken or paused. The training text is input into the initial feature encoding model, and the initial feature encoding model extracts the corresponding code based on the text, for example, it can be a Mel spectrum code extraction to obtain the training code.

[0090] The training code is input into the initial vocoder model, and the initial vocoder model recognizes the training code to train the output audio. The label code of the same type as the training code is extracted for the candidate audio. For example, when the initial feature coding model extracts the Mel spectrum code, the Mel spectrum code is extracted for the candidate audio to obtain the label code, and the label code and the training code are used to generate a first loss value. Correspondingly, the candidate audio is used as another label, and the second loss value is generated using the candidate audio and the training output audio. The parameters of the initial feature coding model are adjusted using the first loss value, and the features of the initial vocoder model are adjusted using the second loss value, so that the initial feature coding model and the initial vocoder model learn related content. When it is detected that the training completion conditions are met, the initial feature coding model after parameter adjustment is determined as the feature coding model, and the initial vocoder model after parameter adjustment is determined as the vocoder model. The number and specific content of the training completion conditions can be set as needed, for example, it can be an accuracy condition, a training time condition, or a training round condition.

[0091] It should be noted that there is no restriction on the model types of the vocoder model and the feature encoding model. For example, the vocoder model can adopt a model structure such as Tacotron and FastSpeech.

[0092] S106: Perform synthesis processing on the audio to be synthesized and the song audio to be processed to obtain synthesized audio.

[0093] By synthesizing the two, a synthesized audio with high quality, similar timbre throughout and good listening experience can be obtained. It is understandable that in order to maintain the integrity of the audio of the song to be processed, the audio to be synthesized can be synthesized at a position where there is no vocal in the audio of the song to be processed. Specifically, the accompaniment time interval corresponding to the audio of the song to be processed can be determined. The accompaniment time interval refers to a time interval with no vocals but only accompaniment. The template information can characterize its adapted position. For example, it can usually introduce the song to be played before the song starts, introduce the purpose of the song after the song is played, and introduce the background of the song during the song playing process, etc., which can be specifically set. The synthesis position type is determined based on the template information, and the accompaniment time interval is screened using the synthesis position type to obtain the target accompaniment time interval. Each audio to be synthesized can correspond to one or more target accompaniment time intervals.

[0094] If the length of the target accompaniment time interval is not less than the time length of the audio to be synthesized, it means that the audio to be synthesized can be played completely within the target accompaniment time interval. In this case, the audio to be synthesized can be synthesized with the part of the audio of the song to be processed that corresponds to the target accompaniment time interval, so that the audio to be synthesized can be played while the accompaniment is playing. If the length of the target accompaniment time interval is less than the time length of the audio to be synthesized, in order to maintain the integrity of the audio of the song to be processed, the audio to be synthesized and the audio of the song to be processed can be spliced ​​based on the type of the target accompaniment time interval, that is, the two will not overlap in time, and the duration of singing the song to be processed. Specifically, if the type of the target accompaniment time interval is a prelude, the audio to be synthesized will be spliced ​​before the audio of the song to be processed. If the type of the target accompaniment time interval is an ending, the audio to be synthesized will be spliced ​​after the audio of the song to be processed. Please refer to Figure 4 , Figure 4 A schematic diagram of audio synthesis positions provided in an embodiment of the present application shows that if the target accompaniment time interval is a prelude, the audio to be synthesized (i.e., speech) can be spliced ​​to the front of the song audio; if it is an outro, it can be spliced ​​to the back of the song audio. This embodiment is not limited to the processing method for interludes. For example, splicing to the front or back can be randomly selected, or splicing can be specified to a specific position, such as the back.

[0095] In addition, when synthesizing the audio to be synthesized with the part corresponding to the target accompaniment time interval in the song audio to be processed, in order to obtain a better sense of hearing, the volume of the part corresponding to the audio to be synthesized and the target accompaniment time interval can be adjusted. When the part corresponding to the target accompaniment time interval is a prelude, the volume of the prelude can be set from small to large, and finally restored to normal volume, and the volume of the audio to be synthesized can be set from large to small, which has the effect of gradually introducing the song. If the part corresponding to the target accompaniment time interval is an ending, the volume of the ending can be set from large to small, and the volume of the audio to be synthesized can be set from small to reach, which has the effect of gradually leading out the song. If the part corresponding to the target accompaniment time interval corresponds to an interlude, the volume of the interlude can be set to be always small, smaller than the volume of the audio to be synthesized, or the same setting method as the prelude can be adopted.

[0096] This embodiment does not limit the specific method for obtaining the accompaniment time interval. In one embodiment, the lyrics corresponding to the song audio to be processed can be obtained, and the accompaniment time interval can be determined based on the timestamp of the lyrics. For example, for Jay Chou's song "Qilixiang", through lyric analysis, the first line "The sparrows outside the window are chattering on the telephone poles" begins at approximately 28 seconds, so 0 to 28 seconds is the prelude; the line "The rain falls all night, my love overflows like rain" begins at approximately 162 seconds, and the previous line ends at approximately 135 seconds, with approximately 27 seconds of unvoiced time in between, which is the interlude; the last line ends at approximately 270 seconds, and the song ends at 299 seconds, so this 29 seconds of unvoiced time is the outro. There are three accompaniment time intervals: from 0 to 28 seconds, from 135 to 162 seconds, and from 270 to 299 seconds, and these three accompaniment time intervals correspond to different synthesis position types: front, middle, and back.

[0097] In another embodiment, if there are no lyrics, a vocal detection model can be used to perform vocal detection on the song audio to be processed to obtain multiple starting time points and ending time points. Since there is also a gap between two adjacent lyrics when the singer sings, the audio to be synthesized should not be inserted into the gap. In order to avoid the shorter audio to be synthesized being inserted between two adjacent lyrics, the adjacent ending time points and starting time points whose distance is less than a preset threshold can be removed to obtain the target starting time point and the target ending time point, where the distance refers to the time interval distance, and the specific size of the preset threshold is not limited. The target starting time point and the target ending time point can be used to determine the time interval of the singing part, and the accompaniment time interval can also be determined.

[0098] Please refer to Figure 5 , Figure 5A schematic diagram of the entire audio synthesis process provided by an embodiment of the present application. The text represents the aforementioned text template. Using song information, the song audio to be processed can be determined, and then the song audio to be processed can be used for voice separation and timbre recognition processing to obtain target timbre information. After generating the text to be processed using the text and song information, speech synthesis technology is used to perform speech synthesis on the text to be processed based on the target timbre information to obtain synthesized speech, i.e., the aforementioned audio to be synthesized. Automatic aliasing technology is then used to perform audio synthesis on the synthesized speech and the song audio to be processed corresponding to the song information to obtain a long audio, i.e., the synthesized audio.

[0099] Applying the audio synthesis method provided in the embodiment of the present application, through voice separation processing, and using the obtained human voice audio to determine the target timbre information from multiple candidate timbre information, the target timbre information is timbre information that is similar to the timbre of the human voice part in the song audio to be processed. After the text to be processed is generated, the content of the human voice in the audio to be synthesized based on the target timbre information is the same as the text to be processed, and the timbre is similar to the timbre of the singer of the song audio to be processed. In the synthesized audio obtained by synthesizing the audio to be synthesized and the song audio to be processed, the timbre of the human voice part remains similar before and after, and has a good listening experience. At the same time, the quality of the audio to be synthesized is high and is not limited by the level of manual broadcasting, so that the quality of the synthesized audio is high and the playback effect is better, which solves the problem of requiring manual participation and the effect being limited by the broadcasting level of the anchor.

[0100] The computer-readable storage medium provided in the embodiments of the present application is introduced below. The computer-readable storage medium described below and the audio synthesis method described above can be referenced to each other.

[0101] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned audio synthesis method are implemented.

[0102] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0104] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0105] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0106] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms include, comprise, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0107] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An audio synthesis method, characterized in that: include: Get the audio of the song to be processed and the corresponding song information; Performing vocal separation processing on the song audio to be processed to obtain vocal audio; Determining target timbre information from a plurality of candidate timbre information based on the human voice audio; Obtaining a text template, and generating a to-be-processed text based on the text template and the song information; the type of the text template includes one or more of an encouraging type, a straightforward type, a happy sharing type, and a funny type; Based on the target timbre information, performing speech synthesis processing on the text to be processed to obtain audio to be synthesized; The audio to be synthesized and the song audio to be processed are synthesized to obtain synthesized audio.

2. The audio synthesis method according to claim 1, wherein: The determining target timbre information from a plurality of candidate timbre information based on the human voice audio includes: Extracting a voiceprint feature vector corresponding to the human voice audio; Generating cosine similarities between the voiceprint feature vector and each candidate voiceprint feature vector; each candidate voiceprint feature vector corresponds to each candidate timbre information; The candidate timbre information corresponding to the maximum cosine similarity is determined as the target timbre information.

3. The audio synthesis method according to claim 1, wherein: The determining target timbre information from a plurality of candidate timbre information based on the human voice audio includes: Inputting the human voice audio into a timbre matching model trained based on candidate audios to obtain similarity probabilities between the human voice audio and each of the candidate audios; The candidate timbre information of the candidate audio corresponding to the maximum similarity probability is determined as the target timbre information.

4. The audio synthesis method according to claim 1, wherein: The song information includes multiple song sub-information, and the generating of the to-be-processed text based on the text template and the song information includes: Based on the information type of each song sub-information, each song sub-information is inserted into a corresponding position in the text template to obtain the text to be processed.

5. The audio synthesis method according to claim 1, wherein: The step of performing speech synthesis processing on the text to be processed based on the target timbre information to obtain the audio to be synthesized includes: Inputting the target timbre information and the text to be processed into a feature coding model to obtain a text code; The text code is input into a vocoder model to obtain the audio to be synthesized.

6. The audio synthesis method according to claim 5, characterized in that The training process of the feature coding model and the vocoder model includes: Obtaining training data corresponding to each candidate timbre information; wherein the training data includes training text and corresponding candidate audio; Inputting the training text into an initial feature coding model to obtain a training code; Inputting the training code into an initial vocoder model to obtain training output audio; Extracting a label code of the same type as the training code for the candidate audio, and generating a first loss value using the label code and the training code; generating a second loss value using the candidate audio and the training output audio; Using the first loss value to adjust parameters of the initial feature encoding model, and using the second loss value to adjust features of the initial vocoder model; When it is detected that the training completion condition is met, the initial feature coding model after parameter adjustment is determined as the feature coding model, and the initial vocoder model after parameter adjustment is determined as the vocoder model.

7. The audio synthesis method according to claim 1, characterized in that The synthesizing process of the audio to be synthesized and the song audio to be processed comprises: Determine the accompaniment time interval corresponding to the song audio to be processed; Determining a synthesis position type based on the template information, and screening the accompaniment time interval using the synthesis position type to obtain a target accompaniment time interval; If the length of the target accompaniment time interval is not less than the time length of the audio to be synthesized, synthesizing the audio to be synthesized with the portion of the song audio to be processed corresponding to the target accompaniment time interval; and / or, If the length of the target accompaniment time interval is less than the time length of the audio to be synthesized, the audio to be synthesized and the song audio to be processed are spliced ​​based on the type of the target accompaniment time interval.

8. The audio synthesis method according to claim 7, characterized in that: The step of splicing the audio to be synthesized and the song audio to be processed based on the type of the target accompaniment time interval includes: If the type of the target accompaniment time interval is a prelude, splicing the audio to be synthesized before the audio of the song to be processed; If the type of the target accompaniment time interval is an ending, the audio to be synthesized is spliced ​​after the audio of the song to be processed.

9. The audio synthesis method according to claim 7, characterized in that: Determining the accompaniment time interval corresponding to the song audio to be processed includes: Use the voice detection model to detect the voice of the song audio to be processed and obtain multiple starting time points and ending time points; Remove adjacent end time points and start time points whose distance is less than a preset threshold to obtain the target start time point and target end time point; The accompaniment time interval is determined using the target start time point and the target end time point.

10. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the audio synthesis method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the audio synthesis method according to any one of claims 1 to 9 is implemented.

Images