Singing synthesis method and device and computer readable storage medium
By splitting the vocal data according to a fixed frame length on edge devices and synthesizing the singing voice, the long waiting time problems caused by tight GPU computing power and poor network are solved, and the effect of fast broadcasting and saving GPU inference costs is achieved.
Patent Information
- Application Number
- CN202411153076.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-08
AI Technical Summary
In the case of tight GPU computing power and poor network, the singing synthesis time is long and the broadcast waiting time is long.
By splitting the vocal data by a fixed frame length, a vocal task is obtained, and the target tone is inferred and synthesized in turn with the vocal task. Silent tasks are inserted into the target tone vocal data to realize singing vocal synthesis and use the edge The device performs inference operations to reduce GPU dependency and wait time.
It effectively reduces the waiting time for starting broadcasts, shortening from about 8 seconds to 1-3 seconds, saving the cost of background GPU inference, and improving the efficiency and effect of singing synthesis.
Smart Images

Figure CN118918912A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a singing synthesis method, device and computer-readable storage medium. Background Art
[0002] Singing synthesis is a technology that can replace the user's voice on the target audio. With the development of technology, singing synthesis has evolved from the early stiff and mechanical sounds to the increasingly natural and realistic AI (Artificial Intelligence) sounds. This technology allows users to replace their own voices with selected music tracks, thus achieving a personalized singing experience.
[0003] Because AI singing requires GPU (Graphics Processing Unit) device support for model loading and reasoning, the initial solution was to perform reasoning and synthesis in the background. The user initiates a request on the mobile phone, and the background combines the currently available device resources to schedule and queue, and then sends it to the user after synthesis. Considering the current background of tight GPU computing power, the complete background playback link almost always takes more than 8 seconds. When the network condition is poor or there are a large number of users, the waiting time is even longer.
[0004] Therefore, when using GPU for singing synthesis, if the GPU computing power is insufficient and the network is not good, there are technical problems such as long singing synthesis time and long waiting time for starting broadcasting. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a singing synthesis method, device, equipment and computer-readable storage medium, which solves the technical problem of long waiting time for starting broadcasting when computing power is tight and network is poor in the prior art.
[0006] To solve the above technical problems, the present invention provides a singing synthesis method, comprising:
[0007] The vocal data is split according to a fixed frame length to obtain a vocal subtask; wherein the fixed frame length is less than a preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is data of the vocal part of the song;
[0008] Inferring and synthesizing the target timbre with the vocal subtasks in sequence to obtain the target timbre vocal data;
[0009] A silence task is inserted between each segment of the human voice data in the target timbre human voice data to obtain a target synthesized singing voice.
[0010] Optionally, before splitting the vocal data according to the fixed frame length to obtain the vocal subtasks, the method further includes:
[0011] The resource file is trimmed according to the audio features to obtain the vocal part;
[0012] The vocal parts are merged to obtain the vocal data.
[0013] Optionally, after splitting the vocal data according to the fixed frame length to obtain the vocal subtask, the method further includes:
[0014] Merging the vocal subtasks to obtain merged vocal data;
[0015] Performing fade-in and fade-out processing on the spliced parts in the combined vocal data to obtain a target vocal subtask;
[0016] Accordingly, the target timbre is sequentially inferred and synthesized with the vocal subtasks to obtain the target timbre vocal data, including:
[0017] The target timbre is sequentially inferred and synthesized with the target vocal subtask to obtain the target timbre vocal data.
[0018] Optionally, the step of sequentially inferring and synthesizing the target timbre with the vocal subtasks to obtain the target timbre vocal data includes:
[0019] Inferring and synthesizing the target timbre with each of the vocal subtasks in sequence to obtain a plurality of initial segmented timbre data;
[0020] According to the timestamp information of the vocal subtask, the initial segmented timbre data is subjected to RMS processing until the RMS processing of all the vocal subtasks is completed to obtain an inferred RMS; wherein the RMS processing is to square the vocal subtasks element by element according to a preset duration, then sum them, and then take the square root;
[0021] According to the comparison result between the target root mean square and the inferred root mean square, the multiple initial segmented timbre data are adjusted to obtain the target timbre vocal data; wherein the target root mean square is the root mean square corresponding to the original singing voice data.
[0022] Optionally, adjusting the plurality of initial segmented timbre data according to a comparison result between the target root mean square and the inferred root mean square to obtain the target timbre vocal data includes:
[0023] Determine a scaling factor according to the target root mean square and the inferred root mean square;
[0024] The loudness equalization adjustment is performed on the plurality of initial segmented timbre data according to the proportional factor to obtain the target timbre vocal data.
[0025] Optionally, after splitting the vocal data according to the fixed frame length to obtain the vocal subtask, the method further includes:
[0026] Each of the human voice subtasks is connected using a circular double linked list to obtain a human voice subtask double linked list.
[0027] Optionally, the method further includes:
[0028] When a file pointer movement operation occurs, the file pointer is moved to the corresponding target vocal subtask in the double linked list of vocal subtasks according to the timestamp; wherein the file pointer is a variable identifying the timestamp position, each subtask in the double linked list of vocal subtasks corresponds to a timestamp, and the timestamp is a number marking the time node at which the vocal subtask is located.
[0029] Optionally, splitting the vocal data according to the fixed frame length to obtain the vocal subtasks includes:
[0030] The human voice data in the half-precision floating point data format is split according to the fixed frame length to obtain the human voice subtask.
[0031] Optionally, the fixed frame length is greater than or equal to a preset minimum frame length.
[0032] The present application also provides a singing voice synthesis device, which is applied to an edge device, including:
[0033] A vocal data splitting module, used to split the vocal data according to a fixed frame length to obtain vocal subtasks; wherein the fixed frame length is less than a preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song;
[0034] A successive reasoning and synthesis module, used for performing reasoning and synthesis on the target timbre and the vocal subtasks in sequence to obtain the target timbre vocal data;
[0035] The silent task inserting module is used to insert silent tasks between each segment of vocal data in the target timbre vocal data to obtain the target synthesized singing voice.
[0036] The present application also provides a singing voice synthesis device, comprising:
[0037] Memory for storing computer programs;
[0038] A processor is used to implement the steps of the singing synthesis method as described above when executing the computer program.
[0039] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the singing synthesis method as described above are implemented.
[0040] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned singing synthesis method when executed by a processor.
[0041] It can be seen that the present invention obtains vocal subtasks by splitting the vocal data according to a fixed frame length; wherein the fixed frame length is less than the preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song; the target timbre is inferred and synthesized with the vocal subtask in sequence to obtain the target timbre vocal data; a silent task is inserted between each segment of vocal data in the target timbre vocal data to obtain the target synthesized singing. The present application divides the vocal data into multiple vocal subtasks according to the fixed frame length. Since the fixed frame length is less than the preset frame length maximum threshold, the memory peak value is lower than the threshold of the highest memory peak value of the edge device, so that the reasoning of the entire singing synthesis can be moved to the edge device for implementation, and since the target timbre is inferred and synthesized with the vocal subtask in sequence, it can be played after the reasoning synthesis is completed for one subtask, that is, it is achieved while synthesizing and playing, which effectively reduces the waiting time for starting broadcasting and improves the starting speed.
[0042] In addition, the present invention also provides a singing synthesis device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0044] Figure 1 A flow chart of a singing voice synthesis method provided by an embodiment of the present invention;
[0045] Figure 2 A comparison chart of inference results provided by an embodiment of the present invention;
[0046] Figure 3 A schematic diagram of a fade-in and fade-out process provided by an embodiment of the present invention;
[0047] Figure 4 A flowchart of a singing voice synthesis method provided by an embodiment of the present invention;
[0048] Figure 5 A schematic diagram of merging a vocal part determined in an embodiment of the present invention;
[0049] Figure 6 A structural schematic diagram of a singing voice synthesis device provided by an embodiment of the present invention;
[0050] Figure 7 A structural schematic diagram of a singing voice synthesis device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] Some of the terms that appear in the description of the embodiments of the present application are subject to the following interpretation:
[0053] SVC: Singing Voice Conversion, singing voice conversion algorithm;
[0054] Limiter: compression processing;
[0055] VUV: voice unvoiced, speech and silence;
[0056] RMS: Root Mean Square, root mean square;
[0057] Rate: Rate is the ratio of rms_infer, indicating the proportional factor;
[0058] FP16: A half-precision floating point data format.
[0059] Because model loading and reasoning for singing synthesis require GPU device support, the initial solution was to perform synthesis in the background. The user initiates a request on the mobile phone, and the background schedules and queues based on the currently available device resources, and then sends it to the user after synthesis. Considering the current background of tight GPU computing power, performing reasoning operations on edge devices can effectively release this part of the dependence on GPU device computing power and reduce costs.
[0060] At the same time, the complete background playback link almost always takes more than 8 seconds. When the network conditions are poor or there are a large number of users, the waiting time will be longer.
[0061] User initiates request->background scheduling->inference synthesis->user downloads work->plays.
[0062] When the entire inference process is performed on the edge device, the waiting time from user click to playback can be compressed to around 1-3 seconds;
[0063] User initiates request->downloads resource->infers the first segment->plays [continues inference in the background].
[0064] The main difficulty in performing reasoning and synthesis on edge devices (devices close to data sources that can perform local computing and data processing) is that the computing power of edge devices is much lower than that of GPUs, and they cannot perform multi-path and multi-task computing at once like the backend. At the same time, the resource files that edge device reasoning relies on are proportional to the length of the song. The template of a single song is as long as tens of megabytes. If it is downloaded directly, the bandwidth cost introduced will be much higher than the saved GPU device cost.
[0065] Please refer to Figure 1 , Figure 1 A flow chart of a singing voice synthesis method provided by an embodiment of the present invention. The method may include:
[0066] S101, split the vocal data according to the fixed frame length to obtain the vocal subtask; wherein the fixed frame length is less than the preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song.
[0067] The execution subject of this embodiment is an edge device, which is a device at the edge of the network, close to the data source, and can perform local computing and data processing. The edge device corresponds to edge computing, which is to analyze the data collected from the terminal directly in the local device or network close to the data generation, without transmitting the data to the cloud data processing center. This embodiment does not limit the specific method of obtaining the human voice data, and the human voice data can be obtained according to the audio feature, and the audio feature is a feature that can distinguish the human voice from other parts. For example, this embodiment can obtain the human voice data according to the pitch; or this embodiment can also obtain the human voice data according to the loudness. The human voice subtask in this embodiment refers to the task of dividing the continuous human voice signal into short time periods (frames) of fixed length when processing the human voice data. It should be noted that in order to facilitate the traversal of the subtasks, the data structures storing the sequence such as arrays and linked lists can be used to connect the subtasks. This embodiment can convert the human voice data into a half-precision floating point data format after splitting it according to a fixed frame length; or the human voice data can be directly converted into a half-precision floating point data format, and then split according to a fixed frame length to reduce the resource volume. The fixed frame length in this embodiment may be greater than the minimum frame length threshold, which is the duration that can ensure the stability of the singing inference synthesis effect on the edge device. The fixed frame length in this embodiment needs to be less than the preset frame length maximum threshold, which is the longest singing synthesis time supported by the edge device. The vocal data in this embodiment is the data of the vocal part of the song. For example, the preset frame length maximum threshold in this embodiment is 800 frames (8s). The fixed frame length in this embodiment is greater than or equal to the preset minimum frame length. This embodiment does not limit the specific preset minimum frame length. For example, the preset minimum frame length in this embodiment may be 300 frames; or the preset minimum frame length in this embodiment may also be 400 frames.
[0068] It should be noted that in this embodiment, splitting the vocal data according to a fixed frame length is task decomposition, and frame-by-frame reasoning and progressive merging are used to achieve a streaming effect. The background reasoning scheme is to split the song into 25s, and then splice it into a whole song after reasoning and synthesis. The edge device cannot directly replicate the corresponding logic because if 25s of data are reasoned at one time, the peak memory overhead of the edge device can reach more than 1.8G, and the CPU (Central Processing Unit) increment is far more than 50%, which is far beyond the performance level of the online line. Considering that the reasoning overhead of the model is proportional to the input tensor length, reasoning is performed by splitting a single long task into multiple subtasks. There is a problem here. The split subtasks are synthesized separately, and direct splicing has mutations and introduces noise. One solution is to refer to the boundary of vuv (voice and silence), split it at the granularity of the sentence, and ensure that each subtask is distinguished by silence. In this way, the two subtasks can be directly merged without worrying about the introduction of noise in the connection part. The reasons for finally giving up the choice are: a. The existing resource files cannot be accurately segmented by sentence, and there are errors in the annotation of the time axis. b. The length of the sentence is uncontrollable. The length of a conventional single sentence is about 6-10s, but some sentences with long vowels at the end may be longer. Combined with the requirements of performance testing, the edge device currently supports a maximum of 8s of content for a single inference. The 8s in this embodiment refers to the maximum duration of a single inference of human voice, excluding the insertion of silence. Post-processing such as compression is performed on the data after inferring the human voice, and there is no duration requirement. Therefore, the final choice is to directly segment according to the fixed frame length. In the real scenario of the user, from clicking the button to starting the broadcast, assuming that all resource files are ready, the waiting time is approximately equal to the inference time of the first subtask. In theory, the shorter the length of a single task, the faster the start-up speed. But in actual scenarios, one problem encountered is that when the subtask is too short, the inference effect is not stable. For ease of understanding, please refer to Figure 2 , Figure 2 A comparison chart of inference results provided for an embodiment of the present invention, the first row is the effect of one-time inference, the second row is the effect of inference once every 100 frames, the third row is the effect of inference once every 200 frames, and the fourth row is the effect of inference once every 300 frames. From the difference comparison of the spectrogram in the last column, it can be seen that the longer the duration, the more stable the effect. After comparative testing, it was found that the sentence length of a single inference should not be shorter than 300 frames (3s), that is, greater than or equal to 300 frames.
[0069] It should be further explained that, in order to reduce the size of resource files, the above-mentioned splitting of vocal data according to fixed frame lengths to obtain vocal subtasks may include: splitting vocal data in half-precision floating point data format according to fixed frame lengths to obtain vocal subtasks. The vocal data in this embodiment may include a high-pitched file for singing synthesis and a phoneme-to-phoneme mapping file. In this embodiment, the resource files that each song depends on for singing synthesis are two files, ppg (Phonetic Posteriorgram, posterior probability of speech) / pitch (pitch). It can be understood that the entire resource file is quantized, and float32 (32-bit floating point) and float16 (FP16 (float16) is a half-precision floating point data type that uses 16 bits (2 bytes) to store values) are used for experiments. After randomly sampling 100 songs in the background for statistics, it was found that the range of float32 is between [-64,64]. If you want to do fixed-point conversion, you need to reserve at least 7 bits of integer positions, fixed-point conversion: [s:1bit][int:6bit][float:9bit]; float16 is better or equal to fixed-point conversion, and float16 will have higher precision. After changing to float16, the effect does not change much, and the amplitude statistics and frequency statistics are basically consistent, and the spectrum comparison is not much different. The current compression method can compress 15%-25% by segmentation, and changing to float16 can compress it by another 50% on this basis. The original 50M ppg can be compressed to 15-20M.
[0070] It should be further explained that, in order to improve the effect of determining the human voice data, before the human voice data is split according to the fixed frame length and the human voice subtask is obtained, it can also include: cutting the resource file according to the audio feature to obtain the human voice part; merging the human voice part to obtain the human voice data. It can be understood that the use of pitch extraction is to use the difference in frequency between the human voice and background music or other sounds. This embodiment does not limit the specific audio features, as long as the human voice part can be cut according to the audio features. For example, the audio feature in this embodiment can be pitch, and the pitch in this embodiment refers to various sounds with different pitches, that is, the height of the sound, a basic feature of the sound. In this embodiment, cutting the resource file according to the pitch means distinguishing the human voice part and the silent part according to the pitch, thereby cutting out the human voice part. The advantages of this method include: improving clarity: by extracting sounds in a specific pitch range, the human voice can be highlighted and the influence of other interfering background sounds can be reduced, thereby improving the auditory clarity of the audience to the human voice. Maintaining sound quality: Unnecessary frequency components can be removed or weakened without damaging the quality and details of the original human voice recording. Flexibility: Pitch extraction can be adjusted for different audio materials, adapting to various sound scenes and music styles, and has strong flexibility. Save time: Compared with manual editing and separation of vocals, using software to automatically extract the vocal part based on pitch can greatly save time and labor. In summary, by extracting the vocal part through pitch, the effect of determining the vocal data can be improved because it can improve clarity and maintain printing.
[0071] It should be further explained that in order to improve the effect of singing synthesis, after the vocal data is split according to the fixed frame length to obtain the vocal subtask, it can also include: merging the vocal subtasks to obtain the merged vocal data; fading in and out the spliced parts in the merged vocal data to obtain the target vocal subtask; correspondingly, the target timbre is inferred and synthesized with the vocal subtasks in sequence to obtain the target timbre vocal data, which can include: inferring and synthesizing the target timbre with the target vocal subtasks in sequence to obtain the target timbre vocal data. The splicing part in this embodiment refers to the connection between the two vocal subtasks. It can be understood that a certain overlap is left at the splicing place for fade-in and fade-out. It can be understood that there will be a certain length of overlap between the two vocals before and after. After fading in and out the overlapping part, the connection will be more natural, otherwise the vocals will have noise. There are two modes of fade-in and fade-out: soft (smooth) and hard (sudden). Assume that the length of fadeIn (transition effect from transparent to opaque) and fadeOut (transition effect from opaque to transparent) is N, where N is a non-negative integer. i=[0,N]. For ease of understanding, please refer to Figure 3 , Figure 3A schematic diagram of a fade-in and fade-out process provided by an embodiment of the present invention, assuming that the length of fadeIn (transition effect from transparent to opaque, i.e. fade-in) and fadeOut (transition effect from opaque to transparent, i.e. fade-out) is N, where N is a non-negative integer. i=[0,N]. The Soft calculation formula is: ; ;Hard calculation formula is: ; , where fadeIn represents the coefficient array for fade-in, fadeIn[i] represents the i-th value, fadeOut represents fade-out, and N represents the length of the fade-in and fade-out array, that is, the number of them. Pi is This embodiment improves the naturalness of transition by performing fade-in and fade-out processing on the spliced parts, thereby improving the effect of singing synthesis.
[0072] It should be further explained that in order to improve the efficiency of traversing the singing synthesis part, after splitting the vocal data according to the fixed frame length and obtaining the vocal subtasks, it can also include: connecting each vocal subtask using a circular double linked list to obtain a double linked list of vocal subtasks. The circular double linked list in this embodiment is a bidirectional linked list with a special structure, in which the next of the tail node points to the head node, and the prev of the head node points to the tail node to form a closed loop. The specific features and operations are as follows: Due to its bidirectionality, the linked list can be traversed from two directions, thereby improving the traversal efficiency.
[0073] It should be further explained that in order to improve the efficiency of locating the target vocal subtask, the above-mentioned singing synthesis method may also include: when a file pointer moving operation occurs, the file pointer is moved to the corresponding target vocal subtask in the double linked list of vocal subtasks according to the timestamp; wherein the file pointer is a variable identifying the timestamp position, each subtask in the double linked list of vocal subtasks corresponds to a timestamp, and the timestamp is a number marking the time node where the vocal subtask is located. The file pointer in this embodiment is a variable used to identify the timestamp position. The moving operation in this embodiment means that the position of the file pointer has changed. The timestamp in this embodiment is a mark with time significance, which can mark the time node where the vocal subtask is located. This embodiment can move the file pointer to the corresponding target vocal subtask in the double linked list of vocal subtasks according to the timestamp, thereby improving the efficiency of locating the target vocal subtask.
[0074] S102, inferring and synthesizing the target timbre and the vocal subtasks in sequence to obtain the target timbre vocal data.
[0075] The target timbre in this embodiment is the timbre parameter that needs to be determined to infer and synthesize the timbre with the corresponding vocal part of the song, which refers to the characteristics and quality of the sound, which is determined by factors such as the frequency, amplitude, waveform and time course of the sound. This embodiment does not limit the specific source of the target timbre. For example, the target timbre in this embodiment can be derived from reference audio; or it can be derived from specified timbre parameters; or the target timbre can also be generated by a machine learning model. This embodiment infers and synthesizes the target timbre with the vocal subtasks in sequence, so that the subtasks can be inferred and played in segments. The inference synthesis in this embodiment, that is, the preset frame length maximum threshold mentioned above is the longest inference synthesis frame length supported by the edge device.
[0076] It should be further explained that, in order to optimize the effect of singing synthesis, the above-mentioned inference synthesis of the target timbre and the vocal subtask in sequence to obtain the target timbre vocal data may include:
[0077] S1021, inferring and synthesizing the target timbre with each vocal subtask in sequence to obtain a plurality of initial segmented timbre data;
[0078] S1022, performing RMS processing on the initial segmented timbre data according to the timestamp information of the vocal subtask until the RMS processing of all the vocal subtasks is completed to obtain an inferred RMS; wherein the RMS processing is to square the vocal subtasks element by element according to a preset duration, then sum them, and then take the square root;
[0079] S1023, according to the comparison result of the target root mean square and the inference root mean square, the multiple initial segmented timbre data are adjusted to obtain the target timbre vocal data; wherein the target root mean square is the root mean square corresponding to the original singing voice data.
[0080] The overall steps of this embodiment include: Inference synthesis: In this stage, the target timbre is combined with each vocal subtask in turn, and synthesized through a certain inference model (possibly a deep learning model) to obtain multiple initial segmented timbre data. Each segment corresponds to a vocal subtask. Root mean square processing: According to the timestamp information of the vocal subtask, the initial segmented timbre data is subjected to root mean square (RMS) processing. Timestamp information refers to the data at the specific time point when the data is generated or the event occurs. The timestamp can be obtained by dividing the serial number of the sampling point by the sampling rate. This processing is to calculate the energy level of each segment and may be used to adjust the loudness of the segment to match the energy distribution of the entire song. Element-by-element square: Square each sample value in the segment. Sum: Add up all the squared sample values to obtain a sum, which represents the total energy of the segment. Square root: Square the sum to obtain the RMS value of the segment. This process continues until all vocal subtasks have been RMS processed to obtain a series of inference root mean square values. Adjusting the initial segmented timbre data: According to the comparison results of the target root mean square and the inference root mean square, multiple initial segmented timbre data are adjusted. The target root mean square is the root mean square corresponding to the original singing data, which represents the energy level that the synthesized singing voice hopes to achieve. Comparison: Compare the target root mean square and the inference root mean square to determine whether each segment needs to be gained (increased) or reduced (reduced) to better match the target. Adjustment: According to the comparison results, adjust the loudness of each segment to make it closer to the target root mean square, so as to maintain a uniform energy distribution throughout the song. Ultimately, this process will produce a vocal accompaniment data with adjusted energy levels, which should have consistent loudness and energy distribution throughout the song, thereby improving the naturalness and overall auditory quality of the synthesized singing voice, that is, improving the effect of the synthesized singing voice.
[0081] It should be further explained that, in order to improve the user experience, the above adjustment of multiple initial segmented timbre data according to the comparison result of the target root mean square and the inferred root mean square to obtain the target timbre vocal data may include: determining a proportional factor according to the target root mean square and the inferred root mean square; performing loudness equalization adjustment on multiple initial segmented timbre data according to the proportional factor to obtain the target timbre vocal data. The proportional factor in this embodiment can calculate the difference between the target root mean square and the inferred root mean square, and determine the proportion according to the proportion. Rate (proportional factor) is a number between 0-1, representing whether it is based on the original vocal RMS or the target vocal RMS. 0 is completely based on the original, and 1 is completely based on the target vocal. In this embodiment, loudness equalization adjustment refers to the difference of the template segmented timbre data corresponding to each initial segmented timbre data, calculating the proportional factor, and multiplying the difference corresponding to each segmented data by the proportional factor to obtain the loudness equalization adjustment parameter, thereby performing loudness equalization adjustment based on the loudness equalization adjustment parameter. For example, the loudness corresponding to the initial segmented audio data is 3, the loudness corresponding to the target template segmented audio data (audio corresponding to the target root mean square) is 5, and the difference is 2. At this time, 2 is multiplied by the calculated proportional factor to obtain the loudness adjustment parameter, and the loudness equalization adjustment is performed based on the loudness adjustment parameter. Through the loudness equalization adjustment in this embodiment, the user can obtain a consistent listening experience on different devices, whether in headphones, car audio or home theater systems, they can enjoy similar sound intensity and quality, so the user experience can be improved.
[0082] It should be further explained that, in order to improve the quality of the vocal accompaniment data, the above-mentioned inference synthesis of the target accompaniment with the vocal subtask in sequence to obtain the vocal accompaniment data may include: inference synthesis of the target accompaniment with the vocal subtask in sequence, and compression processing to obtain the vocal accompaniment data. It is understandable that the compression processing can compress the dynamic range of the audio signal, that is, reduce the gap between the maximum amplitude and the minimum amplitude. Doing so can prevent the audio signal from exceeding the maximum volume that the system can withstand at the peak, thereby preventing distortion and improving the quality of the vocal accompaniment data.
[0083] S103, inserting a silence task between each segment of the target timbre vocal data to obtain a target synthesized singing voice.
[0084] The silent task in this embodiment refers to a segment without vocals. The audio signal corresponding to the time segment corresponding to the silent task is set to 0 to obtain the silent task. The silent part in this embodiment is directly filled with silent data without model synthesis, which improves the efficiency of singing. It can be understood that there is a certain proportion of silent parts in the whole song, and this part of the data can actually be directly filled with silent data without model synthesis. Directly use the existing pitch resource file to do boundary detection, and set the threshold thresholdDb (delimited threshold) to segment the pitch sequence. The segmented segments are very fragmented, basically the area where the vowels are located. On this basis, small segments are merged. After such compression, most songs can be compressed by 15%-25% of the volume. For ease of understanding, please refer to Table 1, which is an example table of the volume of songs after the separation of silence and vocals provided in an embodiment of the present invention, wherein npy is a file format that saves ppg / pitch information, ppg represents the posterior probability of speech, and pitch represents pitch information.
[0085] Table 1 An example table of song volume after separation of silence and vocals
[0086] Song title npy(M) After split (ppg + pitch) Proportion after split A 47.3 33.7 + 66kb 71.2% B 53.9 40.1 + 78kb 74.3% C 40.1 33 + 64kb 82.5% D 47.3 39.9 + 78kb 84.3% E 52.2 40.3 + 79kb 77.2% F 61.7 45.9 + 90kb 74.4% G 53 43.7 + 85kb 82.4%
[0087] A singing voice synthesis method provided by an embodiment of the present invention may include: S101, splitting vocal data according to a fixed frame length to obtain vocal subtasks; wherein the fixed frame length is less than a preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song; S102, inferring and synthesizing the target timbre with the vocal subtasks in sequence to obtain the target timbre vocal data; S103, inserting a silent task between each segment of the vocal data in the target timbre vocal data to obtain the target synthesized singing voice. The present application divides the vocal data into multiple vocal subtasks according to a fixed frame length. Since the fixed frame length is greater than the preset frame length threshold, the vocal synthesis can be kept stable, and the memory peak is lower than the threshold of the highest memory peak of the edge device, so that the reasoning of the entire vocal synthesis can be moved to the edge device for implementation, and since the target timbre is sequentially reasoned and synthesized with the vocal subtasks, it can be played after the reasoning and synthesis of a subtask is completed, that is, it can be played while synthesizing, effectively reducing the waiting time for starting the broadcast, increasing the starting speed, and saving the background GPU reasoning cost. In addition, this embodiment splits the vocal data by a fixed frame length of not less than 300 frames, so that singing synthesis can be performed on the edge device; and, the vocal data is converted into a half-precision floating point data format, reducing the resource volume; and, the vocal part is extracted by pitch, thereby improving the effect of vocal data determination; and, this embodiment improves the naturalness of the transition by fading in and out the spliced part, thereby improving the effect of singing synthesis; and, the file pointer can be moved to the corresponding target vocal subtask in the double linked list of the vocal subtask according to the timestamp, thereby improving the target vocal subtask The efficiency of positioning; and, according to the comparison result of the target root mean square and the inference root mean square, multiple initial segmented timbre data are adjusted to improve the effect of singing synthesis; and, through loudness equalization adjustment, this embodiment allows users to obtain a consistent listening experience on different devices, whether in headphones, car audio or home theater systems, they can enjoy similar sound intensity and quality, so the user experience can be improved; and, this implementation will perform compression processing to prevent the audio signal from exceeding the maximum volume that the system can withstand at the peak, thereby preventing distortion and improving the quality of vocal accompaniment data.
[0088] In order to make the present invention easier to understand, please refer to Figure 4 , Figure 4 A flow chart of a singing voice synthesis method provided by an embodiment of the present invention may specifically include:
[0089] S401, cutting the resource file according to pitch to obtain the vocal part, and merging the vocal part to obtain the vocal data.
[0090] The resource files in this embodiment are ppg (Phonetic Posteriorgram) and pitch (pitch) files. It should be noted that the original model in the background is inferred as a whole. Before landing on the end side, it is expected that the internal uncoupled parts will be split. Three sub-modules can be split, namely NSF (Neural source filter), VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech, Conditional Variational Autoencoder for speech synthesis with adversarial learning), and Decoder. There are several main purposes for doing this: a. Avoid the existence of operators that do not support hardware acceleration in the entire model, reducing the overall operating efficiency. Usually the model uses GPU / TPU for computational acceleration, but similar heterogeneity only has obvious improvements in operators such as convolution / MLP (multi-layer perceptron) that are mainly multiplication and addition operations. Operators such as element-wise (element-by-element operation) / reduce (gathering operation) are not well supported by GPU / TPU (tensor processing unit). The general reasoning framework here has two options. The overall solution falls back to the CPU, or the unsupported operator rolls back to the CPU calculation. However, there is a problem with this. The communication cost between the CPU and GPU / TPU is very high, and the time saved by the operation may be much lower than the communication time. Similar situations are encountered in the subsequent verification process. The CPU+GPU time consumption of some modules may be higher than that of the CPU solution. If it is split into several decoupled sub-modules, each can be equipped with hardware acceleration to avoid mutual influence. b. Reduce the peak performance of the running process. The problem with edge devices is that very little computing power is allocated to the application, and multiple programs may be running at the same time in the background and foreground. The operating system monitors resources, and if they are too large, the resources may be withdrawn, causing a crash. Moreover, the heat dissipation of edge devices cannot be compared with that of computer rooms. High-intensity computing equipment will quickly heat up and reduce the frequency. Therefore, ensuring that the overhead of CPU / memory is stable when the program is running is an important challenge in edge device development. For ease of understanding, please refer to Figure 5 , Figure 5 A schematic diagram of merging vocal parts determined in an embodiment of the present invention. Figure 5 The human voice data described in the description is data for determining the segmentation points of the human voice data, and the data format may be JSON (JavaScript Object Notation, a lightweight data exchange format). Figure 5The non-dotted frame part in is the silent part obtained by splitting the unsplit template audio (i.e., the audio for singing synthesis), and the dotted frame part is the vocal part. The entire vocal data can be obtained by merging the various vocal parts.
[0091] S402, converting the vocal data into FP16 format to obtain the target format vocal data.
[0092] This embodiment converts the vocal data into FP16 format. It can be understood that the resource files that each song depends on for vocal synthesis are two files, ppg / pitch. Among them, the pitch file is a float32 (4 bytes) every 10ms, and the ppg file is 1024 float32 (4096 bytes) every 20ms. For a simple reference, a 10s singing synthesis requires a 2kb pitch file and a 2M ppg. The resource size of the song "C" is a total of 40.1M. If the synthesis is performed in the background, the m4a file sent is about 5M. There are two big problems here: a. The bandwidth cost introduced by the resource file is too large (mainly ppg) is very high b. The current average download speed is about 2M / s. If the resource file is too large, it means that the download time will be an additional 15-20s compared to the background synthesis. In order to compress the size of resource files, we mainly start from two aspects: there is a certain proportion of silence in the whole song. The data of this part can actually be directly filled with silence data, without going through model synthesis. We can directly use the existing pitch resource file to do boundary detection and set the threshold thresholdDb to segment the pitch sequence. The segmented fragments are very fragmented, basically the area where the vowels are located. On this basis, the small fragments are merged. The resources of the vocal part will be merged, marked and sent down, and the edge device will expand on this basis.
[0093] S403, splitting the target format human voice data according to a fixed frame length to obtain subtasks; wherein the fixed frame length is greater than 300 frames.
[0094] S404, using a circular double linked list to connect each subtask to obtain a subtask linked list.
[0095] S405, perform reasoning based on the subtask linked list and the target audio, and after the current task reasoning is completed, traverse backward in sequence to find the next reasoning task to be executed.
[0096] The reasoning in this embodiment refers to reasoning about the subtasks corresponding to the target timbre and the resource file.
[0097] S406: When it is determined that the number of executed tasks is equal to the total number of tasks, the task ends.
[0098] S407, after each inference task is completed, the sum of squares of the corresponding range of the corresponding rms (root mean square) template is obtained according to the timestamp information, and a second sum is performed and the square root is taken to obtain the target root mean square, and the inference root mean square is calculated at the same time.
[0099] S408, determining a proportional factor according to the target RMS and the inferred RMS, and performing loudness equalization adjustment on the target RMS according to the proportional factor to obtain loudness-adjusted singing voice data.
[0100] After synthesizing the vocals, the background of this embodiment will refer to the original singer's voice-to-accompaniment ratio before remixing (remixing or editing) with the accompaniment, and adjust the loudness of the synthesized vocals and accompaniment. The calculation of the loudness requires the overall audio data. However, the end side performs streaming processing, and plays the first segment directly after synthesizing it, while the background performs inference calculations for the next subtask. In other words, there is no audio data for the entire song during playback, and it is impossible to align the background logic to perform loudness balance. At the same time, because the segment size of the edge device is smaller, there is also the problem of inconsistent loudness between segments. In order to solve this problem, the lead singer of the PPG is used as a reference, and the RMS calculation method is used to align the RMS in segments. , where N represents the number, Xi represents the i-th number, i is greater than or equal to 0 and less than or equal to N. Combining the calculation formula of rms, we can see that the calculation of rms is to square each element, sum it up, and then take the square root. In order to make it more flexible and convenient for edge devices to make adjustments, the square sum of every 10ms is calculated in advance with the previous pitch granularity of 10ms as a reference. At the same time, using the existing sharding logic on the end side, the square sum is obtained according to the offset of the pitch equivalent each time the shards are synthesized, and the target rms can be obtained by accumulating and taking the square root again. ; Scale is the final scaling coefficient, rms_infer is the rms of each piece of inferred data, rms_global is the rms of the template, and Rate is the ratio of rms_infer. The value range here is [0,1]. Rate is user-defined and is generally set to 1 by default, representing the distribution of the original dry sound and the target dry sound. Scale is the coefficient applied to each sampling point, which is equivalent to multiplying each point by scale. It is understandable that the client side performs inference and playback at the same time, so there is no complete vocal data at the beginning of playback, and the loudness of all vocals cannot be calculated. Since it cannot be calculated, it is impossible to make corresponding adjustments like the backend to pull the vocals to the target loudness. Therefore, we try to extract this rms template and adjust the rms of each vocal segment, just to ensure that the final synthesized vocal is the expected loudness, rather than an uncontrollable state. For example, if the reference vocal for extracting the rms template is -17dB, then each segment is aligned with this rms when synthesizing the vocal, and the loudness of the final synthesized vocal work will also be around -17dB. If it is not aligned, the loudness will have a large fluctuation range.
[0101] S409, performing fade-in and fade-out processing on adjacent subtasks in the loudness-adjusted singing voice data, inserting a silent task between each segment of the vocal data, and performing compression processing.
[0102] In this embodiment, the loudness singing voice data may be compressed and then the mute task may be performed, and the mute task may be inserted as long as it can be ensured. In order to avoid the overlaw (overload) that may be introduced by the gain, a limiter (limiter) post-processing is required.
[0103] S410, re-sample and output the data after the compression processing, and write it into the dry sound memory according to the corresponding offset.
[0104] The beneficial effects of the technical solution of the embodiment of the present invention include: 1. Improving the start speed, reducing the waiting time from at least 8 seconds to about 1-3 seconds; 2. Saving the background GPU inference cost.
[0105] The singing voice synthesis device provided by an embodiment of the present invention is introduced below. The singing voice synthesis device described below and the singing voice synthesis method described above can be referred to each other.
[0106] Please refer to Figure 6 , Figure 6 A structural schematic diagram of a singing voice synthesis device provided by an embodiment of the present invention may include:
[0107] The vocal data splitting module 100 is used to split the vocal data according to a fixed frame length to obtain vocal subtasks; wherein the fixed frame length is less than a preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song;
[0108] A successive reasoning and synthesis module 200 is used to perform reasoning and synthesis on the target timbre and the vocal subtasks in sequence to obtain the target timbre vocal data;
[0109] The silence task inserting module 300 is used to insert silence tasks between each segment of the human voice data in the target timbre human voice data to obtain the target synthesized singing voice.
[0110] Further, based on the above embodiment, the above singing voice synthesis device may further include:
[0111] The cropping module is used to crop the resource file according to the audio characteristics to obtain the vocal part;
[0112] The merging module is used to merge the vocal parts to obtain the vocal data.
[0113] Further, based on any of the above embodiments, the above singing voice synthesis device may further include:
[0114] A vocal subtask merging module, used for merging the vocal subtasks to obtain merged vocal data;
[0115] A fade-in and fade-out processing module, used for performing fade-in and fade-out processing on the spliced parts in the combined vocal data to obtain a target vocal subtask;
[0116] Accordingly, the successive reasoning synthesis module 200 includes:
[0117] A successive reasoning and synthesis unit is used to perform reasoning and synthesis on the target timbre and the target vocal subtask in sequence to obtain the target timbre vocal data
[0118] Further, based on any of the above embodiments, the above successive reasoning synthesis module 200 may include:
[0119] An initial segmented vocal accompaniment data determination unit, used for inferring and synthesizing the target timbre with each of the vocal subtasks in sequence to obtain a plurality of initial segmented timbre data;
[0120] A root mean square determination unit, used for performing root mean square processing on the initial segmented timbre data according to the timestamp information of the vocal subtask, until the root mean square processing of all the vocal subtasks is completed, and an inferred root mean square is obtained; wherein the root mean square processing is to square the vocal subtasks element by element according to a preset duration, then sum them, and then perform a square root processing;
[0121] The vocal accompaniment data adjustment unit is used to adjust the multiple initial segmented timbre data according to the comparison result of the target root mean square and the inferred root mean square to obtain the target timbre vocal data; wherein the target root mean square is the root mean square corresponding to the original singing data.
[0122] Further, based on the above embodiment, the vocal accompaniment data adjustment unit may include:
[0123] A proportional factor determination subunit, configured to determine a proportional factor according to the target RMS and the inferred RMS;
[0124] The loudness equalization adjustment unit is used to perform loudness equalization adjustment on the plurality of initial segmented timbre data according to the proportional factor to obtain the target timbre vocal data.
[0125] Further, based on any of the above embodiments, the above singing voice synthesis device may further include:
[0126] The human voice subtask double linked list determination module is used to connect each of the human voice subtasks using a circular double linked list to obtain a human voice subtask double linked list.
[0127] Further, based on the above embodiment, the above singing voice synthesis device may further include:
[0128] The target vocal subtask search module is used to move the file pointer to the corresponding target vocal subtask in the vocal subtask double linked list according to the timestamp when a file pointer movement operation occurs; wherein the file pointer is a variable identifying the timestamp position, each subtask in the vocal subtask double linked list corresponds to a timestamp, and the timestamp is a number marking the time node at which the vocal subtask is located.
[0129] Further, based on any of the above embodiments, the above voice data splitting module 100 may include:
[0130] The human voice data splitting unit is used to split the human voice data in the half-precision floating point data format according to the fixed frame length to obtain the human voice subtask.
[0131] Further, based on any of the above embodiments, the fixed frame is greater than or equal to 300 frames.
[0132] It should be noted that the order of the modules and units in the above-mentioned singing synthesis device can be changed without affecting the logic.
[0133] The singing voice synthesis device provided by an embodiment of the present invention may include: a vocal data splitting module 100, which is used to split the vocal data according to a fixed frame length to obtain vocal subtasks; wherein the fixed frame length is less than a preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song; a successive inference synthesis module 200, which is used to infer and synthesize the target timbre with the vocal subtasks in sequence to obtain the target timbre vocal data; a silent task insertion module 300, which is used to insert silent tasks between each segment of vocal data in the target timbre vocal data to obtain the target synthesized singing voice. This application divides the vocal data into multiple vocal subtasks according to a fixed frame length. Since the fixed frame length is greater than the preset frame length threshold, the vocal synthesis can be kept stable, and the memory peak is lower than the threshold of the highest memory peak of the edge device, so that the reasoning of the entire vocal synthesis can be moved to the edge device for implementation, and since the target timbre is sequentially reasoned and synthesized with the vocal subtasks, it can be played after the reasoning and synthesis of a subtask is completed, that is, it can be played while synthesizing, effectively reducing the waiting time for starting the broadcast, increasing the starting speed, and saving the background GPU reasoning cost. In addition, this embodiment splits the vocal data by a fixed frame length of not less than 300 frames, so that singing synthesis can be performed on the edge device; and, the vocal data is converted into a half-precision floating point data format, reducing the resource volume; and, the vocal part is extracted by pitch, thereby improving the effect of vocal data determination; and, this embodiment improves the naturalness of the transition by fading in and out the spliced part, thereby improving the effect of singing synthesis; and, the file pointer can be moved to the corresponding target vocal subtask in the double linked list of the vocal subtask according to the timestamp, thereby improving the target vocal subtask The efficiency of positioning; and, according to the comparison result of the target root mean square and the inference root mean square, multiple initial segmented timbre data are adjusted to improve the effect of singing synthesis; and, through loudness equalization adjustment, this embodiment allows users to obtain a consistent listening experience on different devices, whether in headphones, car audio or home theater systems, they can enjoy similar sound intensity and quality, so the user experience can be improved; and, this implementation will perform compression processing to prevent the audio signal from exceeding the maximum volume that the system can withstand at the peak, thereby preventing distortion and improving the quality of vocal accompaniment data.
[0134] The following is an introduction to a singing voice synthesis device provided by an embodiment of the present invention. The singing voice synthesis device described below and the singing voice synthesis method described above can be referenced to each other.
[0135] Please refer to Figure 7, Figure 7 A structural schematic diagram of a singing voice synthesis device provided by an embodiment of the present invention may include:
[0136] A memory 10, used for storing computer programs;
[0137] The processor 20 is used to execute the computer program to implement the above-mentioned singing synthesis method.
[0138] The memory 10 , the processor 20 , and the communication interface 30 all communicate with each other via a communication bus 40 .
[0139] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:
[0140] The vocal data is split according to a fixed frame length to obtain a vocal subtask; wherein the fixed frame length is less than a preset maximum frame length threshold, and the preset maximum frame length threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is the data of the vocal part in the song;
[0141] The target timbre is sequentially inferred and synthesized with the vocal subtasks to obtain the target timbre vocal data;
[0142] A silence task is inserted between each segment of the target timbre vocal data to obtain the target synthesized singing voice.
[0143] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.
[0144] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.
[0145] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .
[0146] The communication interface 30 may be an interface of a communication module, and is used to connect to other devices or systems.
[0147] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the singing voice synthesis device in the embodiment of the present invention. In actual applications, the singing voice synthesis device may include Figure 7 More or fewer components than shown, or combinations of certain components.
[0148] The computer-readable storage medium provided in an embodiment of the present invention is introduced below. The computer-readable storage medium described below and the singing synthesis method described above can be referenced to each other.
[0149] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned singing synthesis method are implemented.
[0150] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0151] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned singing synthesis method when executed by a processor.
[0152] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0153] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0154] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0155] The above is a detailed introduction to a singing synthesis method, device and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A singing voice synthesis method, characterized in that: Applied to edge devices, including: The vocal data is split according to a fixed frame length to obtain a vocal subtask; wherein the fixed frame length is less than a preset frame length maximum threshold, and the preset frame length maximum threshold is the longest singing voice inference synthesis time supported by the edge device; the vocal data is data of the vocal part of the song; Inferring and synthesizing the target timbre with the vocal subtasks in sequence to obtain the target timbre vocal data; A silence task is inserted between each segment of the human voice data in the target timbre human voice data to obtain a target synthesized singing voice.
2. The singing voice synthesis method according to claim 1, characterized in that: Before splitting the human voice data according to the fixed frame length to obtain the human voice subtask, the method further includes: The resource file is trimmed according to the audio features to obtain the vocal part; The vocal parts are merged to obtain the vocal data.
3. The singing voice synthesis method according to claim 1, characterized in that: After splitting the human voice data according to the fixed frame length to obtain the human voice subtask, the method further includes: Merging the vocal subtasks to obtain merged vocal data; Performing fade-in and fade-out processing on the spliced parts in the combined vocal data to obtain a target vocal subtask; Accordingly, the target timbre is sequentially inferred and synthesized with the vocal subtasks to obtain the target timbre vocal data, including: The target timbre is sequentially inferred and synthesized with the target vocal subtask to obtain the target timbre vocal data.
4. The singing voice synthesis method according to any one of claims 1 to 3, characterized in that: The target timbre is sequentially inferred and synthesized with the vocal subtasks to obtain the target timbre vocal data, including: Inferring and synthesizing the target timbre with each of the vocal subtasks in sequence to obtain a plurality of initial segmented timbre data; According to the timestamp information of the vocal subtask, the initial segmented timbre data is subjected to RMS processing until the RMS processing of all the vocal subtasks is completed to obtain an inferred RMS; wherein the RMS processing is to square the vocal subtasks element by element according to a preset duration, then sum them, and then take the square root; According to the comparison result between the target root mean square and the inferred root mean square, the multiple initial segmented timbre data are adjusted to obtain the target timbre vocal data; wherein the target root mean square is the root mean square corresponding to the original singing voice data.
5. The singing voice synthesis method according to claim 4, characterized in that: The step of adjusting the plurality of initial segmented timbre data according to the comparison result between the target root mean square and the inferred root mean square to obtain the target timbre vocal data comprises: Determine a scaling factor according to the target root mean square and the inferred root mean square; The loudness equalization adjustment is performed on the plurality of initial segmented timbre data according to the proportional factor to obtain the target timbre vocal data.
6. The singing voice synthesis method according to any one of claims 1 to 3, characterized in that: After splitting the human voice data according to the fixed frame length to obtain the human voice subtask, the method further includes: Each of the human voice subtasks is connected using a circular double linked list to obtain a human voice subtask double linked list.
7. The singing voice synthesis method according to claim 6, characterized in that: The method further comprises: When a file pointer movement operation occurs, the file pointer is moved to the corresponding target vocal subtask in the double linked list of vocal subtasks according to the timestamp; wherein the file pointer is a variable identifying the timestamp position, each subtask in the double linked list of vocal subtasks corresponds to a timestamp, and the timestamp is a number marking the time node at which the vocal subtask is located.
8. The singing voice synthesis method according to claim 1, characterized in that: The human voice data is split according to the fixed frame length to obtain the human voice subtask, including: The human voice data in the half-precision floating point data format is split according to the fixed frame length to obtain the human voice subtask.
9. A singing voice synthesis device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the singing synthesis method as described in any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the singing synthesis method according to any one of claims 1 to 8.
Citation Information
Cited By
Multifunctional integrated song ordering machine
CN119851639A
Voice interaction method and device, medium, equipment and computer program product
CN120977282A
