Singing beautifying method and device and medium

Abstract

The invention discloses a song beautification method and device and a medium, and relates to the technical field of song conversion, and the method comprises the steps: determining a to-be-reasoned region matched with a beautification service instruction based on the singing score of a target recorded song when the beautification service instruction for the target recorded song is received; determining an octave offset tone based on the range distribution of the target recorded song, and calculating a target reasoning tone according to the octave offset tone and the singing tone of the target user; cutting the singing guide feature file based on the to-be-reasoned area to obtain a target feature file; the singing guide feature file is a feature file of an original song; inputting the target inference tone, the target feature file and the timbre vector of the target user into a target timbre inference model to output a target beautified audio; and issuing the target beautified audio to the user side. The singing reasoning effect does not depend on the accuracy of a traditional sound correction template, and the singing beautifying accuracy is guaranteed on the premise that the characteristics of part of users are reserved.

Description

Technical Field

[0001] The present invention relates to the technical field of singing voice conversion, and in particular to a singing voice beautification method, equipment and medium. Background Art

[0002] At present, most of the singing voice beautification schemes are based on signal processing, and the singing voice beautification effect based on signal processing depends on the accuracy of the template. When the accuracy of the template is low, the signal processing process will have problems such as correction deviation and misjudgment of timbre spectrum characteristics, resulting in the singing voice beautification effect cannot be guaranteed. Therefore, the above problems urgently need to be solved by those skilled in the art. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a singing voice beautification method, device and medium, so that the singing voice reasoning effect does not rely on the accuracy of the traditional voice modification template, and the accuracy of the singing voice beautification is guaranteed under the premise of retaining some of the user's own characteristics. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a method for beautifying singing voice, comprising:

[0005] When receiving a beautification service instruction for a target recorded song, determining a region to be inferred in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song; the target recorded song is obtained by a target user singing a target song;

[0006] Determine the octave offset pitch of the area to be inferred based on the pitch distribution of the target recorded song, and calculate the target inferred pitch based on the octave offset pitch and the singing pitch of the target user;

[0007] The lead-in feature file is cut based on the area to be inferred to obtain a target feature file; wherein the lead-in feature file is a feature file of the original song of the target track;

[0008] Inputting the target inference tone, the target feature file and the timbre vector of the target user into a target timbre inference model, so as to output a target beautified audio through the target timbre inference model;

[0009] The target beautified audio is sent to the user end.

[0010] Optionally, the singing voice beautification method further includes:

[0011] Obtaining a user dry voice file of the target user;

[0012] Performing noise reduction processing on the user dry voice file to obtain a noise-reduced user dry voice file;

[0013] Filtering a plurality of songs that meet preset conditions from the user dry sound file after noise reduction;

[0014] A sample training set is constructed according to the plurality of songs, and the timbre model to be trained is trained according to the sample training set to obtain a target timbre inference model whose training loss satisfies a loss condition.

[0015] Optionally, the beautification service instruction is a trial listening segment beautification service instruction, and the step of determining the area to be inferred in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song includes:

[0016] If the singing scores of the singing sentences of the target recorded song are not zero, the target song segment is obtained according to the singing sentence with the lowest singing score;

[0017] The target song segment is cropped according to a first cropping strategy to obtain a first audition segment, and the time region corresponding to the first audition segment is determined as the area to be inferred; the first cropping strategy is obtained based on lyrics timestamp information, a reserved duration on the left side of the audition segment, and an expected duration of the audition segment, and the reserved duration on the left side of the audition segment is a duration reserved for preventing cropping deviation.

[0018] Optionally, the beautification service instruction is a trial listening segment beautification service instruction, and the step of determining the area to be inferred in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song includes:

[0019] If the singing scores of the various singing sentences in the target recorded song are all zero or the number of singing scores of the target recorded song is different from the length of the sentence list, then a preset time-length segment of the target recorded song is determined as the target song segment; the preset time-length segment includes the start timestamp of the target recorded song to the target timestamp, and the length of the sentence list is the number of the singing sentences;

[0020] The target song segment is cropped according to a second cropping strategy to obtain a second audition segment, and a time region corresponding to the second audition segment is determined as the region to be inferred; the second cropping strategy is obtained based on lyrics timestamp information and an expected duration of the audition segment.

[0021] Optionally, the beautification service instruction is a complete song beautification service instruction, and the determining, based on the performance score of the target recorded song, the area to be inferred that matches the beautification service instruction in the target recorded song includes:

[0022] The area to be inferred is determined based on whether all the singing sentences in the target recorded song fail to meet the standards; any dimension score in the singing score of the singing sentences that fail to meet the standards is lower than the corresponding score threshold, and the singing score includes scores in multiple dimensions.

[0023] Optionally, determining the area to be inferred according to whether all the singing sentences in the target recorded song fail to meet the standard includes:

[0024] If all the singing sentences in the target recorded song do not meet the standards, the time region corresponding to the target recorded song is determined as the region to be inferred;

[0025] If each singing sentence part in the target recorded song does not meet the standard, the time area corresponding to the lead singing feature file is determined as the area to be inferred;

[0026] If all the singing sentences in the target recorded song meet the standards, no processing will be performed.

[0027] Optionally, the area to be inferred is a time area corresponding to the lead-in feature file, and the target inference pitch, the target feature file, and the timbre vector of the target user are input into a target timbre inference model to output a target beautified audio through the target timbre inference model, including:

[0028] Input the target inference tone, the lead singing feature file and the timbre vector of the target user into a target timbre inference model to obtain the audio after the first inference;

[0029] Determine the singing sentence that does not meet the standard in the target recorded song as the first singing sentence, and determine the singing sentence that meets the standard in the target recorded song as the second singing sentence;

[0030] Filtering out a third singing sentence corresponding to the first singing sentence from the audio after the first inference, and splicing the second singing sentence with the third singing sentence to obtain a spliced ​​result;

[0031] The spliced ​​result is cropped according to the third cropping strategy and the target recorded song to obtain a cropped result, and the cropped result is directly determined as the target beautified audio.

[0032] Optionally, after the spliced ​​result is trimmed according to the third trimming strategy and the target recorded song to obtain the trimmed result, the method further includes:

[0033] According to the clipped result, an updated lead vocal feature file is obtained;

[0034] The target inference pitch, the updated lead singing feature file and the timbre vector of the target user are input into a target timbre inference model to obtain the target beautified audio.

[0035] Optionally, sending the target beautified audio to the user terminal includes:

[0036] If the beautification service instruction is the audition clip beautification service instruction, a singing clip corresponding to the target beautified audio is determined from the target recorded song, and the original singing clip and the target beautified audio are sent to the user end; wherein the singing clip is a clip corresponding to the area to be inferred;

[0037] If the beautification service instruction is the complete song beautification service instruction, the target beautified audio is sent to the user end.

[0038] In a second aspect, the present application discloses a singing beautification device, comprising:

[0039] A module for determining an area to be inferred, for determining an area to be inferred in the target recorded song that matches the beautification service instruction based on a performance score of the target recorded song when a beautification service instruction for the target recorded song is received; the target recorded song is obtained by a target user singing a target song;

[0040] An inference pitch determination module, used to determine the octave offset pitch of the area to be inferred based on the pitch distribution of the target recorded song, and calculate the target inference pitch according to the octave offset pitch and the singing pitch of the target user;

[0041] A target feature file determination module is used to cut the lead-in feature file based on the area to be inferred to obtain a target feature file; wherein the lead-in feature file is a feature file of the original song of the target track;

[0042] An audio beautification module, used for inputting the target inference tone, the target feature file and the timbre vector of the target user into a target timbre inference model, so as to output a target beautified audio through the target timbre inference model;

[0043] The beautified audio sending module is used to send the target beautified audio to the user end.

[0044] In a third aspect, the present application discloses an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] The processor is used to execute the computer program to implement the aforementioned disclosed singing beautification method.

[0047] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method for beautifying singing voice is implemented.

[0048] In a fifth aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the aforementioned singing beautification method when executed by a processor.

[0049] It can be seen that the present application proposes a singing beautification method, comprising: when a beautification service instruction for a target recorded song is received, the area to be inferred in the target recorded song that matches the beautification service instruction is determined based on the singing score of the target recorded song; the target recorded song is obtained by the target user singing the target song; the octave offset pitch of the area to be inferred is determined based on the range distribution of the target recorded song, and the target inference pitch is calculated based on the octave offset pitch and the singing pitch of the target user; the lead singing feature file is trimmed based on the area to be inferred to obtain a target feature file; wherein the lead singing feature file is a feature file of the original song of the target song; the target inference pitch, the target feature file and the timbre vector of the target user are input into the target timbre inference model to output the target beautified audio through the target timbre inference model; and the target beautified audio is sent down to the user end.

[0050] Beneficial effects: For different beautification service instructions, this application will determine different areas to be inferred according to the singing scores of the target recorded songs, and cut the feature files of the original songs of the target tracks according to the areas to be inferred to obtain the target feature files, so as to input the target inference pitch, the target feature files and the timbre vector of the target user into the pre-trained target timbre inference model, and output the target beautified audio through the target timbre inference model. In this way, the fusion of timbre, pitch and original singing characteristics is achieved, so that the beautified audio finally output can achieve better results in terms of timbre performance and pitch accuracy. Furthermore, for the target inference pitch used in reasoning, this application calculates the octave difference between the target recorded song and the original song, and performs corresponding offset synthesis, so that the target beautified audio is more in line with the user's original singing habits, so that the final beautified audio can not only retain the charm of the original song, but also highlight the singing characteristics of the target user and perform targeted beautification. In summary, the audio beautification of the present application does not depend on the accuracy of the template, and solves the problems of correction deviation and misjudgment of timbre spectrum characteristics in the signal processing process due to inaccurate templates in traditional technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A flow chart of a singing voice beautification method disclosed in this application;

[0053] Figure 2 A flow chart for beautifying an audition clip disclosed in this application;

[0054] Figure 3 A complete song beautification flow chart disclosed in this application;

[0055] Figure 4 A specific singing beautification flow chart disclosed in this application;

[0056] Figure 5 An interactive schematic flow chart disclosed in this application;

[0057] Figure 6 A specific flow chart of a singing voice beautification method disclosed in this application;

[0058] Figure 7 A flow chart of sentence replacement strategy in a complete song beautification disclosed in this application;

[0059] Figure 8 This is a structural schematic diagram of a singing voice beautification device disclosed in this application;

[0060] Fig. 9 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] At present, most of the singing voice beautification solutions are based on signal processing, and the singing voice beautification effect based on signal processing depends on the accuracy of the template. When the accuracy of the template is low, the signal processing process will have problems such as correction deviation and misjudgment of timbre spectrum characteristics, resulting in the beautification effect cannot be guaranteed.

[0063] To this end, the embodiment of the present application proposes a singing beautification solution, which makes the singing beautification effect independent of the accuracy of the traditional tuning template and ensures the accuracy of the singing beautification.

[0064] The present application embodiment discloses a method for beautifying singing voice, see Figure 1 As shown, the method includes:

[0065] Step S11: When a beautification service instruction for a target recorded song is received, a region to be inferred in the target recorded song that matches the beautification service instruction is determined based on the performance score of the target recorded song; the target recorded song is obtained by a target user singing a target song.

[0066] In this embodiment, the beautification service instruction selected by the user is obtained through the singing beautification service setting interface on the interactive interface, and the area to be inferred in the target recorded song that matches the beautification service instruction is determined according to the beautification service instruction selected by the user and the singing score of the target recorded song. It should be pointed out that the singing score includes multiple dimensional scores, and the multiple dimensional scores constitute a dimensional score array, which includes but is not limited to breath score, pitch score, rhythm score, etc.

[0067] The following describes how to obtain the singing score of the target recorded song:

[0068] First, parse the timestamps of the song sentences to get ;in, Indicates the line number, The timestamp of the beginning of the lyrics. The timestamp of the end of the lyrics. Further, check whether the number of singing scores (the length of the dimension score array) is consistent with the length of the sentence list (the number of song sentences). For example, if the length of the sentence list is 20, check whether the number of singing scores is 20. If not, select the audition clip bottom determination strategy, which will be described later. If yes, store the dimension scores corresponding to the sentences. The storage format is as follows: ;in, , , Dimensions, express Dimension scoring. It is understandable that since the target user may only sing a few lines of lyrics, this embodiment needs to retain the sentence scores of the target user's singing part and filter out the invalid sentence scores of other parts. Since the target user may start singing from the middle of a certain line of lyrics, it is necessary to handle the abnormal situation of the left and right boundaries. First, the left boundary is processed. If the whole line of lyrics is on the left side of the start time of the target user's singing segment, the sentence is discarded. If the whole line of lyrics is on the right side of the start time of the singing segment, the sentence is retained and its right boundary is determined in the subsequent steps. If the sentence just crosses the left boundary, the start timestamp of the sentence is truncated according to the start time of the singing segment. Further, the right boundary is processed. If the whole line of lyrics is on the right side of the end time of the target user's singing segment, the sentence is discarded. If the whole line of lyrics is on the left side of the end time of the singing segment, it is directly retained. If the sentence just crosses the right boundary, it is necessary to determine whether the effective singing rate threshold of the singing sentence reaches the threshold. The threshold can be preset to A. If it does not reach A, it is directly discarded. Otherwise, it is necessary to determine whether the sentence score is non-zero. If the sentence score is non-zero, it is retained, otherwise it is discarded. Among them, the effective singing rate of a sentence is the ratio of the time area of ​​the user singing the part of a certain sentence of lyrics to the complete time area of ​​the sentence of lyrics.

[0069] The following describes how to determine the area to be inferred:

[0070] In the first embodiment, the user needs to beautify the audition clip service. On this basis, if the singing score of each singing sentence of the target recorded song is not zero, the target song clip is obtained according to the singing sentence with the lowest singing score, and the target song clip is clipped according to the first clipping strategy to obtain the first audition clip, and then the time area corresponding to the first audition clip is determined as the area to be inferred, wherein the first clipping strategy is obtained based on the lyrics timestamp information, the reserved time on the left side of the audition clip and the expected time of the audition clip, and the reserved time on the left side of the audition clip is the time reserved to prevent clipping deviation. Exemplarily, if the target recorded song contains three singing sentences, it is determined whether the singing scores of these three singing sentences are not zero. It should be pointed out that the singing score is not zero, which means that the scores of multiple dimensions are not all zero. For example, in the first singing sentence, the breath dimension score is not zero, in the second singing sentence, the pitch dimension score is not zero, and in the third singing sentence, the rhythm dimension score is not zero, which can be called the singing score of the three singing sentences is not zero. Furthermore, if there is any singing sentence whose singing score is not zero and the score of each dimension in the singing score is lower than the corresponding dimension threshold, the singing sentence is considered to be a singing sentence that does not meet the standard. The singing sentence with the lowest singing score is selected from the singing sentences that do not meet the standard, and the selected singing sentence is determined as the target song segment, and then the target song segment is cropped according to the first cropping strategy to obtain a first audition segment, and the time area corresponding to the first audition segment is determined as the area to be inferred.

[0071] The first clipping strategy is described in detail below:

[0072] ;

[0073] ;

[0074] in, is the start timestamp of the first audition segment, is the start timestamp of the target song segment, Reserve time for the left side of the audition clip. The timestamp of the start of the part sung by the user. is the end timestamp of the first audition segment, The expected duration of the audition clip. The timestamp of the end of the part sung by the user. For example, It can be 80 milliseconds, It can be 10 seconds.

[0075] In the second implementation, the user needs to beautify the audition clip service. On this basis, if the singing scores of all the singing sentences in the target recorded song are all zero or the number of singing scores of the target recorded song is different from the length of the sentence list, the preset time segment of the target recorded song is determined as the target song segment, and the target song segment is clipped according to the second clipping strategy to obtain the second audition segment, and then the time area corresponding to the second audition segment is determined as the area to be inferred, and the preset time segment includes the start timestamp of the target recorded song to the target timestamp, and the second clipping strategy is obtained based on the lyrics timestamp information and the expected duration of the audition segment. Among them, the preset time segment can be 10 seconds, for example, the first 10 seconds of the target recorded song are directly determined as the target song segment, and the target song segment is clipped to obtain the second audition segment. In this embodiment, the target song segment is clipped to avoid the last word in the segment being truncated. It should be pointed out that this implementation is a bottom-up determination strategy for the audition segment when the singing scores of each singing sentence sung by the user are all zero.

[0076] The second clipping strategy is described in detail below:

[0077] ;

[0078] ;

[0079] The meanings of the various parts in the second clipping strategy are as described above and will not be repeated here.

[0080] In the third implementation, the user needs a complete song beautification service. On this basis, the area to be inferred is determined according to whether all the singing sentences in the target recorded song do not meet the standard. If all the singing sentences in the target recorded song do not meet the standard, the time area corresponding to the target recorded song is determined as the area to be inferred. If some of the singing sentences in the target recorded song do not meet the standard, the time area corresponding to the lead singing feature file is determined as the area to be inferred; if all the singing sentences in the target recorded song meet the standard, no processing is performed. That is to say, if the user needs a complete song beautification service, then when all the singing sentences do not meet the standards, each singing sentence is inferred. At this time, the time area corresponding to the target recorded song is determined as the area to be inferred. When several singing sentences do not meet the standards, the time area corresponding to the lead singing feature file is determined as the area to be inferred, and the sentences that do not meet the standards are replaced according to the reasoning results. The details of the replacement will be explained later. It should also be pointed out that the lead singing feature file is the feature file of the original song of the target song, which is obtained specifically in the following way: download the original song and separate the original dry voice, remove the reverberation / harmony / other noise in the original dry voice, and extract PPG (phonetic posterior grams, voice posterior probability) & F0 (fundamental frequency of the speech signal) to obtain the lead singing feature file, and store the lead singing feature file to the CFS (Cloud File Storage, shared file system) shared disk so that it can be shared among various services.

[0081] Step S12: Determine the octave offset pitch of the area to be inferred based on the range distribution of the target recorded song, and calculate the target inferred pitch based on the octave offset pitch and the singing pitch of the target user.

[0082] Considering that boys often sing in a lower octave when singing a girl's song with accompaniment, and girls may sing in a higher octave when singing a boy's song with accompaniment, in order to make the beautified singing voice close to the user's original singing habits, this embodiment calculates the octave difference between the target recorded song and the original song, and performs corresponding offset synthesis. Specifically, the vocal audio file of the target recorded song is subjected to noise reduction processing to obtain the sung song after noise reduction, and the range distribution of the sung song after noise reduction is calculated to obtain the median of the range, and the octave offset pitch is calculated according to the median of the range, and the target inference pitch is calculated according to the addition result of the octave offset pitch and the user's singing pitch, and the user's singing pitch can be obtained from the relevant recording information of the target recorded song.

[0083] ;

[0084] in, Represents the median of the vocal range.

[0085] Step S13: cutting the lead-in feature file based on the area to be inferred to obtain a target feature file; wherein the lead-in feature file is a feature file of the original song of the target track.

[0086] In this embodiment, the lead-in feature file is cropped based on the area to be inferred. If the area to be inferred is 2s to 5s of the target recorded song, 2s to 5s of the lead-in feature file are determined as the target feature file.

[0087] Step S14: inputting the target inference tone, the target feature file and the timbre vector of the target user into a target timbre inference model, so as to output a target beautified audio through the target timbre inference model.

[0088] In this embodiment, a user dry voice file is downloaded from a streaming media system to a file storage shared network disk, and a check is performed to see whether the effective vocal duration of the user dry voice file reaches a first threshold value; wherein, the user dry voice file is a vocal recording file of a target user, and if the effective vocal duration of the user dry voice file reaches the first threshold value, the user dry voice file is subjected to noise reduction processing to obtain a noise-reduced user dry voice file, and a plurality of singing songs are screened out from the noise-reduced user dry voice file according to the headphone wearing condition of the user during singing, the Musical Instrument Digital Interface score, and the effective vocal duration, and then a sample training set is constructed based on the plurality of singing songs, and the timbre model to be trained is trained based on the sample training set to obtain a target timbre inference model whose training loss satisfies the loss condition.

[0089] Specifically, this embodiment first collects the PCM (Pulse Code Modulation) of the dry voice recorded by the user through a mobile client or a WEB page, with a sampling rate of 44100 or 48000, and a mono or dual channel, which is not limited here, and encodes it into AAC (Advanced Audio Coding, a file compression format designed for sound data) format and uploads it to the streaming media system for storage. The streaming media system includes but is not limited to COS (Cloud Object Storage, object storage) or other cloud storage, and records the corresponding recording information, such as whether only part of the segment is sung, the start timestamp of the singing segment (relative to the accompaniment), the end timestamp of the singing segment (relative to the accompaniment), the headphone plug-in information, the lyrics version information, multiple dimension scores, MIDI (Musical Instrument Digital Interface) scores, etc. Further, check whether the duration of the encoded file (that is, the user dry voice file) reaches the second threshold, and if so, download it to the CFS shared network disk to avoid repeated downloads and reduce development complexity. Further, check whether the effective vocal duration of the user's dry voice file reaches the first threshold, which can be 30s. If it does not reach the first threshold, it is directly discarded, and the dry voice in the user's dry voice file that reaches the first threshold is subjected to noise reduction processing to minimize the interference of noise caused by the environment, equipment, abnormal operations during recording, etc. on subsequent calculations. According to the headphone wearing situation of the user during singing, the musical instrument digital interface score, and the effective vocal duration, multiple dry voices are selected from the user's dry voice file after noise reduction. For example, the dry voice of the user wearing headphones is given priority, which can effectively avoid environmental noise, accompaniment back-stepping, etc., and the sound reception effect will be significantly better than external playback. Priority is given to songs with higher musical instrument digital interface scores and dry voices with longer effective vocal duration. Finally, a sample training set is constructed based on the multiple dry voices selected, and the timbre model to be trained is trained based on the sample training set to obtain a target timbre inference model whose training loss meets the loss condition. It should be noted that in order to avoid a large number of dry sounds being screened, which would result in a large amount of calculation and a long time consumption, this embodiment may preset a screening threshold, for example, taking the top 5 dry sounds in the comprehensive ranking as part of the training set and discarding other dry sounds.

[0090] The following is an explanation of how to train the target timbre inference model: (1) Divide the training set into a training subset and a validation subset according to a certain ratio. The training subset is used for the model training process, and the validation subset is used to evaluate the model performance during the training process so as to adjust the model parameters and training strategy in a timely manner. (2) Select the timbre model architecture to be trained, such as convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), etc. (3) Define the loss function. Commonly used loss functions include cross entropy loss function. (4) Set the training parameters. First, select an initial learning rate, such as 0.001. If it is found that the loss value decreases too slowly or oscillates during model training, the learning rate can be appropriately reduced or increased. (5) Set the training rounds and decide whether to increase or decrease the rounds based on the performance evaluation results on the validation subset. (6) Determine the batch size. Furthermore, the data of the training subset is sent to the timbre model to be trained in batches according to the set batch size. For each batch of data, the output value of the model for the batch of data is calculated, and the loss value of the batch of data is calculated by the defined loss function. The back propagation algorithm is used to derive the various parameters of the model according to the loss value to obtain the gradient of each parameter. Finally, the various parameters of the model are updated according to the obtained gradient and the set learning rate to reduce the loss value of the next training until the training of all batches of data in the entire training subset is completed. After each round of training, the performance of the model is evaluated using the validation subset. When the performance evaluation result of the model on the validation subset meets the set loss condition, the model at this time is the target timbre inference model whose training loss meets the loss condition.

[0091] In this embodiment, one user corresponds to one timbre inference model. The voices of different users have unique physiological characteristics. Training different timbre inference models for different users can better adapt to these physiological characteristics and achieve accurate beautification of each user's voice.

[0092] After obtaining the target timbre inference model, the target inference pitch, target feature file and the timbre vector of the target user are input into the target timbre inference model to output the target beautified audio through the target timbre inference model. The user's timbre vector is the average value of the user's multiple dry voice timbre vectors in each dimension.

[0093] Step S14: Send the target beautified audio to the user end.

[0094] In this embodiment, different contents are sent to the user terminal according to the service instruction input by the user.

[0095] See also Figure 2As shown, if the user inputs an audition clip beautification service instruction, the original singing clip corresponding to the target beautified audio is determined from the target recorded song, and the original singing clip and the target beautified audio (here refers to the beautified audio obtained after reasoning the area to be inferred based on the audition clip) are sent to the user end, so that the user can intuitively compare the singing clip with the beautified clip.

[0096] See also Figure 3 As shown, if the user inputs a paid complete song beautification service instruction, the target beautified audio (here refers to the beautified audio of the complete song) is sent to the user end, so that the user can enjoy a more professional and comprehensive song beautification service. It should be pointed out that Figure 3 The merged sentences in refers to merging adjacent sentences that do not meet the standards to reduce the damage to the original audio.

[0097] See also Figure 4 As shown, after singing a piece of work, the user enters the mixer to open the super tuning order panel, and chooses to initiate a trial listening (i.e., beautify the trial listening segment) or pay to start super tuning (i.e., beautify the complete work). If the user chooses to audition, the trial listening segment beautification task is initiated, and the processing flow of the trial listening segment beautification operation generates the original sound segment with the worst quality and the effect of the segment after tuning for the user, and finally the user can hear two audios, namely the original sound segment with the worst quality and the effect of the segment after tuning. If the user chooses to pay to start super tuning, the complete work beautification task is initiated, and the processing flow of the complete work beautification operation generates the complete beautified singing product, and finally the user can hear one audio, i.e., the complete singing product. See. Figure 5 As shown, Figure 5 The left part is the audition interface, and the right part is the complete beautification interface. The user can click the button before processing to play the user's singing segment, and click the button after processing to play the beautified audition segment. If you are willing, you can click the bottom button to pay and then start the beautification of the complete work. Furthermore, this embodiment also supports the visual display of the process of beautification of singing voices through technical discourse. The user can view the processing progress and processing method in real time, and wait for the task to be completed to play the complete beautified work.

[0098] In addition, the beautified audio can be encrypted and uploaded to the streaming media system so that it can be sent to the user end.

[0099] It can be seen that the present application proposes a singing beautification method, comprising: when a beautification service instruction for a target recorded song is received, the area to be inferred in the target recorded song that matches the beautification service instruction is determined based on the singing score of the target recorded song; the target recorded song is obtained by the target user singing the target song; the octave offset pitch of the area to be inferred is determined based on the range distribution of the target recorded song, and the target inference pitch is calculated based on the octave offset pitch and the singing pitch of the target user; the lead singing feature file is trimmed based on the area to be inferred to obtain a target feature file; wherein the lead singing feature file is a feature file of the original song of the target song; the target inference pitch, the target feature file and the timbre vector of the target user are input into the target timbre inference model to output the target beautified audio through the target timbre inference model; and the target beautified audio is sent down to the user end.

[0100] Beneficial effects: For different beautification service instructions, this application will determine different areas to be inferred according to the singing scores of the target recorded songs, and cut the feature files of the original songs of the target tracks according to the areas to be inferred to obtain the target feature files, so as to input the target inference pitch, the target feature files and the timbre vector of the target user into the pre-trained target timbre inference model, and output the target beautified audio through the target timbre inference model. In this way, the fusion of timbre, pitch and original singing characteristics is achieved, so that the beautified audio finally output can achieve better results in terms of timbre performance and pitch accuracy. Furthermore, for the target inference pitch used in reasoning, this application calculates the octave difference between the target recorded song and the original song, and performs corresponding offset synthesis, so that the target beautified audio is more in line with the user's original singing habits, so that the final beautified audio can not only retain the charm of the original song, but also highlight the singing characteristics of the target user and perform targeted beautification. In summary, the audio beautification of the present application does not depend on the accuracy of the template, and solves the problems of correction deviation and misjudgment of timbre spectrum characteristics in the signal processing process due to inaccurate templates in traditional technologies.

[0101] Next, the scenario where the user needs a complete song beautification service and only a part of the singing sentences in the target recorded song do not meet the standards and therefore needs to replace the singing sentences that do not meet the standards is explained:

[0102] The embodiment of the present application discloses a specific method for beautifying singing voice. Compared with the previous embodiment, the present embodiment further explains and optimizes the technical solution. Figure 6 As shown, specifically including:

[0103] Step S21: input the target inference tone, the lead singing feature file and the timbre vector of the target user into a target timbre inference model to obtain the audio after the first inference.

[0104] When some singing sentences do not meet the standard, the time area corresponding to the lead singing feature file is determined as the area to be inferred, and the sentences that do not meet the standard are replaced according to the inference results, which is also the singing sentence replacement strategy. Specifically, the target inference pitch, the lead singing feature file and the timbre vector of the target user are first input into the target timbre inference model to obtain the audio after the first inference.

[0105] Step S22: determining the singing sentences in the target recorded song that do not meet the standards as the first singing sentences, and determining the singing sentences in the target recorded song that meet the standards as the second singing sentences.

[0106] See also Figure 7 As shown, line1, line3, and line4 in the user's singing work are qualified segments, that is, the second singing sentence in this embodiment (the sentence after merging adjacent and unqualified singing sentences). Line2 in the user's singing work is a non-qualified segment, that is, the first singing sentence in this embodiment.

[0107] Step S23: Filter out a third singing sentence corresponding to the first singing sentence from the audio after the first inference, and splice the second singing sentence with the third singing sentence to obtain a spliced ​​result.

[0108] See also Figure 7 As shown, line0, line2, and line5 in the audio after the first reasoning correspond to the non-compliant segments in the target recorded song, that is, the third singing sentence corresponding to the first singing sentence. It can be understood that line0, line2, and line5 in the audio after the first reasoning are the compliant segments after reasoning. In this embodiment, the compliant segment in the target recorded song, that is, the second singing sentence, is spliced ​​with the compliant segment, the third singing sentence, after reasoning, to obtain the spliced ​​result, and the spliced ​​result is the complete song length.

[0109] Step S24: cropping the spliced ​​result according to the third cropping strategy and the target recorded song to obtain a cropped result, and directly determining the cropped result as the target beautified audio.

[0110] In this embodiment, since the target user does not necessarily sing the entire song, it is necessary to crop the spliced ​​result according to the area of ​​the complete song where the target user's current singing segment is located to obtain the cropped result, which is the third cropping strategy. Further, the cropped result is the target beautified audio.

[0111] It should be pointed out that in order to reduce the sense of connection caused by replacement splicing, this embodiment introduces secondary reasoning, that is, the updated lead singing feature file is obtained according to the cropped result, and the target reasoning pitch, the updated lead singing feature file and the timbre vector of the target user are input into the target timbre reasoning model to obtain the target beautified audio. In this embodiment, the pitch used in the secondary reasoning is 0, which is consistent with the pitch of the updated lead singing feature file. Furthermore, the beautified audio is encrypted and uploaded to the streaming media system so that it can be subsequently sent to the user end.

[0112] In summary, this embodiment beautifies and repairs sentences that do not meet the singing standards, while retaining sentences that meet the standards. It can not only retain some of the user's excellent singing skills, but also beautify sentences of slightly lower quality to be comparable to the original singing. It can also make the connection more natural and smooth through secondary reasoning, which greatly improves the reasoning accuracy and the user experience.

[0113] It can be seen that this application proposes a singing analysis technology based on multi-dimensional scoring and a singing beautification technology based on singing conversion, which can be applied in multiple singing beautification scenarios. The user asynchronous recording scene can obtain the singing beautification effect of the complete singing song. Furthermore, this embodiment covers the secondary creation scene of the work. For the works that have been deposited in history, by beautifying them, cooperating with professional record-level mixing, video cropping, special effects rendering and other technologies to create high-quality works, and distributing them to users in consumption scenarios or launching customized package services.

[0114] Next, the scenario where the user needs a complete song inference service and all the singing sentences in the target recorded song do not meet the standard, so it is necessary to infer all the singing sentences that do not meet the standard is explained:

[0115] When all the singing sentences do not meet the standard, it is necessary to infer all the singing sentences. At this time, the time area corresponding to the target recorded song is determined as the area to be inferred. Further, considering that boys often sing girls' songs with accompaniment in a low octave, girls may sing boys' accompaniment songs in a high octave. Therefore, in order to better make the beautified singing close to the user's original singing habits, this embodiment needs to calculate the octave difference between the target recorded song and the original song, and perform corresponding offset synthesis. Specifically, the human voice audio file of the target recorded song is subjected to noise reduction processing to obtain the denoised singing song, and the range distribution of the denoised singing song is calculated to obtain the range median, and then the octave offset pitch is calculated according to the range median, and the target inference pitch is calculated according to the octave offset pitch and the user's singing pitch. The specific calculation formula of the target inference pitch is shown in the aforementioned disclosed embodiment, and will not be described in detail here. Finally, the target inference pitch, the target feature file and the timbre vector of the target user are input into the target timbre inference model to output the target beautified audio through the target timbre inference model.

[0116] Correspondingly, the present application also discloses a singing beautification device, see Figure 8 As shown, the device comprises:

[0117] The module 11 for determining the area to be inferred is used to determine the area to be inferred in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song when receiving the beautification service instruction for the target recorded song; the target recorded song is obtained by the target user singing the target song;

[0118] An inference pitch determination module 12 is used to determine the octave offset pitch of the area to be inferred based on the range distribution of the target recorded song, and calculate the target inference pitch according to the octave offset pitch and the singing pitch of the target user;

[0119] The target feature file determination module 13 is used to cut the lead-in feature file based on the area to be inferred to obtain a target feature file; wherein the lead-in feature file is a feature file of the original song of the target track;

[0120] The audio beautification module 14 is used to input the target inference tone, the target feature file and the timbre vector of the target user into a target timbre inference model, so as to output a target beautified audio through the target timbre inference model;

[0121] The beautified audio sending module 15 is used to send the target beautified audio to the user end.

[0122] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0123] It can be seen that the present application proposes a singing beautification method, comprising: when a beautification service instruction for a target recorded song is received, the area to be inferred in the target recorded song that matches the beautification service instruction is determined based on the singing score of the target recorded song; the target recorded song is obtained by the target user singing the target song; the octave offset pitch of the area to be inferred is determined based on the range distribution of the target recorded song, and the target inference pitch is calculated based on the octave offset pitch and the singing pitch of the target user; the lead singing feature file is trimmed based on the area to be inferred to obtain a target feature file; wherein the lead singing feature file is a feature file of the original song of the target song; the target inference pitch, the target feature file and the timbre vector of the target user are input into the target timbre inference model to output the target beautified audio through the target timbre inference model; and the target beautified audio is sent down to the user end.

[0124] Beneficial effects: For different beautification service instructions, this application will determine different areas to be inferred according to the singing scores of the target recorded songs, and cut the feature files of the original songs of the target tracks according to the areas to be inferred to obtain the target feature files, so as to input the target inference pitch, the target feature files and the timbre vector of the target user into the pre-trained target timbre inference model, and output the target beautified audio through the target timbre inference model. In this way, the fusion of timbre, pitch and original singing characteristics is achieved, so that the beautified audio finally output can achieve better results in terms of timbre performance and pitch accuracy. Furthermore, for the target inference pitch used in reasoning, this application calculates the octave difference between the target recorded song and the original song, and performs corresponding offset synthesis, so that the target beautified audio is more in line with the user's original singing habits, so that the final beautified audio can not only retain the charm of the original song, but also highlight the singing characteristics of the target user and perform targeted beautification. In summary, the audio beautification of the present application does not depend on the accuracy of the template, and solves the problems of correction deviation and misjudgment of timbre spectrum characteristics in the signal processing process due to inaccurate templates in traditional technologies.

[0125] Furthermore, an embodiment of the present application also provides an electronic device. Fig. 9 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0126] Fig. 9The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the singing beautification method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0127] In this embodiment, the power supply 26 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 24 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0128] In addition, the memory 22, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc., and the resources stored thereon may include a computer program 221, and the storage method may be a temporary storage or a permanent storage. In addition to including a computer program that can be used to complete the singing voice beautification method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 221 may further include a computer program that can be used to complete other specific tasks.

[0129] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method for beautifying singing voice is implemented.

[0130] For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0131] The various embodiments in this application are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0132] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0134] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0135] The above is a detailed introduction to a singing beautification method, device, and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A singing beautification method, characterized in that: include: When receiving a beautification service instruction for a target recorded song, determining a region to be inferred in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song; The target recorded song is obtained by the target user singing the target song; Determine the octave offset pitch of the area to be inferred based on the pitch distribution of the target recorded song, and calculate the target inferred pitch based on the octave offset pitch and the singing pitch of the target user; The lead-in feature file is cut based on the area to be inferred to obtain a target feature file; wherein the lead-in feature file is a feature file of the original song of the target track; Inputting the target inference tone, the target feature file and the timbre vector of the target user into a target timbre inference model, so as to output a target beautified audio through the target timbre inference model; The target beautified audio is sent to the user end.

2. The singing voice beautification method according to claim 1, characterized in that: Also includes: Obtaining a user dry voice file of the target user; Performing noise reduction processing on the user dry voice file to obtain a noise-reduced user dry voice file; Filtering a plurality of songs that meet preset conditions from the user dry sound file after noise reduction; A sample training set is constructed according to the plurality of songs, and the timbre model to be trained is trained according to the sample training set to obtain a target timbre inference model whose training loss satisfies a loss condition.

3. The singing beautification method according to claim 1, characterized in that: The beautification service instruction is a trial listening segment beautification service instruction, and the determining of the to-be-inferred region in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song includes: If the singing scores of the singing sentences of the target recorded song are not zero, the target song segment is obtained according to the singing sentence with the lowest singing score; The target song segment is cropped according to a first cropping strategy to obtain a first audition segment, and the time region corresponding to the first audition segment is determined as the area to be inferred; the first cropping strategy is obtained based on lyrics timestamp information, a reserved duration on the left side of the audition segment, and an expected duration of the audition segment, and the reserved duration on the left side of the audition segment is a duration reserved for preventing cropping deviation.

4. The singing voice beautification method according to claim 1, characterized in that: The beautification service instruction is a trial listening segment beautification service instruction, and the determining of the to-be-inferred region in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song includes: If the singing scores of the various singing sentences in the target recorded song are all zero or the number of singing scores of the target recorded song is different from the length of the sentence list, then a preset time-length segment of the target recorded song is determined as the target song segment; the preset time-length segment includes the start timestamp of the target recorded song to the target timestamp, and the length of the sentence list is the number of the singing sentences; The target song segment is cropped according to a second cropping strategy to obtain a second audition segment, and a time region corresponding to the second audition segment is determined as the region to be inferred; the second cropping strategy is obtained based on lyrics timestamp information and an expected duration of the audition segment.

5. The singing voice beautification method according to claim 1, characterized in that: If the beautification service instruction is a complete song beautification service instruction, then determining the area to be inferred in the target recorded song that matches the beautification service instruction based on the performance score of the target recorded song includes: The area to be inferred is determined based on whether all the singing sentences in the target recorded song fail to meet the standards; any dimension score in the singing score of the singing sentences that fail to meet the standards is lower than the corresponding score threshold, and the singing score includes scores in multiple dimensions.

6. The singing voice beautification method according to claim 5, characterized in that: The determining the area to be inferred according to whether all the singing sentences in the target recorded song fail to meet the standard includes: If all the singing sentences in the target recorded song do not meet the standards, the time region corresponding to the target recorded song is determined as the region to be inferred; If each singing sentence part in the target recorded song does not meet the standard, the time area corresponding to the lead singing feature file is determined as the area to be inferred; If all the singing sentences in the target recorded song meet the standards, no processing will be performed.

7. The singing voice beautification method according to claim 6, characterized in that: The area to be inferred is a time area corresponding to the lead-in feature file, and the target inference pitch, the target feature file, and the timbre vector of the target user are input into a target timbre inference model to output a target beautified audio through the target timbre inference model, including: Input the target inference tone, the lead singing feature file and the timbre vector of the target user into a target timbre inference model to obtain the audio after the first inference; Determine the singing sentence that does not meet the standard in the target recorded song as the first singing sentence, and determine the singing sentence that meets the standard in the target recorded song as the second singing sentence; Filtering out a third singing sentence corresponding to the first singing sentence from the audio after the first inference, and splicing the second singing sentence with the third singing sentence to obtain a spliced ​​result; The spliced ​​result is cropped according to the third cropping strategy and the target recorded song to obtain a cropped result, and the cropped result is directly determined as the target beautified audio.

8. The singing voice beautification method according to claim 7, characterized in that: After the spliced ​​result is trimmed according to the third trimming strategy and the target recorded song to obtain the trimmed result, the method further includes: According to the clipped result, an updated lead vocal feature file is obtained; The target inference pitch, the updated lead singing feature file and the timbre vector of the target user are input into a target timbre inference model to obtain the target beautified audio.

9. The singing voice beautification method according to any one of claims 1 to 8, characterized in that: The step of sending the target beautified audio to the user terminal includes: If the beautification service instruction is the audition clip beautification service instruction, a singing clip corresponding to the target beautified audio is determined from the target recorded song, and the original singing clip and the target beautified audio are sent to the user end; wherein the singing clip is a clip corresponding to the area to be inferred; If the beautification service instruction is the complete song beautification service instruction, the target beautified audio is sent to the user end.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the singing beautification method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the singing beautification method as described in any one of claims 1 to 9 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the singing voice beautification method described in any one of claims 1 to 9 is implemented.

Images