Model training, song composition, audio tagging, and song scoring methods and components

Abstract

The application discloses a model training, song synthesis, audio marking and song scoring method and components thereof. The model training method comprises the following steps: obtaining a song audio and a song template corresponding to the song audio; determining a breathing position in the song audio by using the song template, and determining an interval duration of the breathing position from a previous singing section based on the breathing position; performing energy extraction at the breathing position to obtain an energy peak value corresponding to the breathing position and a breathing duration; marking the breathing position, the interval duration, the energy peak value and the breathing duration in the song audio to obtain a training sample; and training a regression model by using the training sample to obtain a feature extraction model. The application can train a feature extraction model for extracting breathing features of multiple different dimensions, and the feature extraction model can meet more application requirements, such as song synthesis, audio marking and song scoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a model training method, a song synthesis method, an audio marking method and a song scoring method and components thereof. BACKGROUND

[0002] Singing breathing is a singing expression form with music theory. Breathing sound can reflect the singing level of a singer, and appropriate breathing sound can also make the synthesized music more natural. However, the current breathing position detection only detects the time of the breathing point, which is relatively single and cannot meet the actual application requirements.

[0003] In summary, how to effectively solve the problems such as breathing detection of song audio is a technical problem that needs to be solved by the technical personnel in the field at present. SUMMARY

[0004] The purpose of the present application is to provide a model training method, a song synthesis method, an audio marking method, a song scoring method, equipment and a readable storage medium, which can train a feature extraction model that detects more breathing attributes, and can mark audio and score songs based on the breathing attributes extracted by the feature extraction model.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] A model training method comprises:

[0007] Obtaining a song audio and a song template corresponding to the song audio;

[0008] Using the song template to determine the breathing position in the song audio, and based on the breathing position, determining the interval duration of the breathing position from the previous verse;

[0009] Performing energy extraction at the breathing position to obtain an energy peak value and a breathing duration corresponding to the breathing position;

[0010] Annotating the breathing position, the interval duration, the energy peak value and the breathing duration in the song audio to obtain a training sample;

[0011] Using the training sample to train a regression model to obtain a feature extraction model.

[0012] Preferably, using the song template to determine the breathing position in the song audio comprises:

[0013] Determine a breathing point in the song audio by using the time interval information in the song template; wherein the song template is a melody template and / or a lyric template; in the lyric template, the time interval information is the interval duration between each word; in the melody template, the time interval information is the interval duration between each note;

[0014] Perform breathing detection on the breathing point to obtain the breathing position.

[0015] Preferably, the method for obtaining the breathing position by performing breathing detection on the breathing point comprises:

[0016] If the breathing positions corresponding to the melody template and the lyric template are different, perform breathing detection on the breathing point determined by the lyric template to obtain the breathing position.

[0017] Preferably, the method for obtaining the energy peak value and the breathing duration corresponding to the breathing position by performing energy extraction at the breathing position comprises:

[0018] Sample the breathing position in the song audio to obtain a sampling point;

[0019] Extract the energy of the sampling point;

[0020] Smooth the energy by using a filter function to obtain an energy envelope line;

[0021] Determine the energy peak value and the breathing duration by using the energy envelope line.

[0022] A song synthesis method further comprises:

[0023] Obtain a synthesized song audio that is artificially synthesized and has no breathing feature;

[0024] From a reference song library, obtain a reference audio corresponding to the synthesized song audio;

[0025] Perform breathing detection on the reference audio by using the feature extraction model to obtain a reference breathing feature corresponding to the reference audio;

[0026] Correspondingly add a breathing audio to the synthesized song audio by referring to the breathing reference feature.

[0027] An audio labeling method further comprises:

[0028] Obtain a to-be-labeled audio uploaded by a teaching client;

[0029] performing breath detection on the to-be-labeled audio by using the feature extraction model to obtain a breath position corresponding to the reference audio, the breath position having an interval duration from a previous singing section, and the breath position corresponding to an energy peak value, a breath duration, and the interval duration;

[0030] labeling teaching singing information on the to-be-labeled audio by using the breath position, the interval duration, the energy peak value, and the breath duration to obtain a labeled audio;

[0031] feeding back the labeled audio to the teaching singing client.

[0032] A song scoring method, comprising:

[0033] obtaining target singing audio uploaded by a user end, and obtaining reference breath features corresponding to the target singing audio from a reference song feature library;

[0034] performing breath detection on the target singing audio by using the feature extraction model to obtain target breath features corresponding to the target singing audio;

[0035] comparing breath positions, interval durations, energy peak values, and breath durations in the reference breath features and the target breath features to obtain a singing score of the target singing audio.

[0036] Preferably, comparing the reference breath features and the target breath features to obtain the singing score of the target singing audio comprises:

[0037] comparing the reference breath features and the target breath features to obtain feature differences;

[0038] weighting and mapping the feature differences according to a category of the target song to obtain the singing score.

[0039] An electronic device, comprising:

[0040] a memory configured to store a computer program;

[0041] a processor configured to execute the computer program to implement steps of the model training method, steps of the song synthesis method, steps of the audio labeling method, or steps of the song scoring method.

[0042] A readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement steps of the model training method, steps of the song synthesis method, steps of the audio labeling method, or steps of the song scoring method.

[0043] The model training method provided in the embodiment of the application is applied to obtain a song audio and a song template corresponding to the song audio; the song template is used to determine a breathing position in the song audio, and based on the breathing position, an interval duration of the breathing position from a previous singing section is determined; energy extraction is performed at the breathing position to obtain an energy peak value corresponding to the breathing position and a breathing duration; the breathing position, the interval duration, the energy peak value and the breathing duration are labeled on the song audio to obtain a training sample; and the training sample is used to train a regression model to obtain a feature extraction model.

[0044] In the application, the breathing position in the song audio is first determined based on the song template, and the interval duration of the breathing position from the previous singing section is determined, and then energy extraction is performed on the breathing position to obtain the energy peak value and the breathing duration. The breathing position, the interval duration, the energy peak value and the breathing duration are labeled on the song audio. The breathing position indicates the specific position of the breathing, the interval duration indicates the corresponding time relationship between the breathing and the previous singing section, the energy peak value corresponds to the strength of the breathing sound, and the breathing duration corresponds to the duration of the breathing sound. After labeling, the training sample is obtained, the regression model is trained based on the training sample, and the feature extraction model is obtained. The feature extraction model can extract the breathing position, the interval duration, the energy peak value and the breathing duration from the input audio. That is, the feature extraction model that extracts breathing features of multiple different dimensions can be trained and obtained, and the feature extraction model can meet more application requirements.

[0045] Correspondingly, the application also provides a song synthesis method corresponding to the above-mentioned model training method. Based on the feature extraction model, the breathing position, the interval duration, the energy peak value and the breathing duration can be extracted, and the synthesized song audio synthesized by the artificial can be added with the breathing audio corresponding to the reference breathing feature of the reference audio. The final synthesized song audio has breathing information and is more consistent with the real human singing audio, and the audio is more natural.

[0046] Correspondingly, the application also provides an audio labeling method corresponding to the above-mentioned model training method. Based on the feature extraction model, the breathing position, the interval duration, the energy peak value and the breathing duration can be extracted, so that more rich teaching singing information can be labeled on the audio to be labeled. The labeled audio is fed back to the teaching singing client, so that the user can obtain more teaching singing information.

[0047] Correspondingly, the application also provides a song scoring method corresponding to the above-mentioned model training method. In the method, the breathing position, the interval duration, the energy peak value and the breathing duration in the reference breathing feature and the target breathing feature are used as the basis for scoring, and finally the singing score is obtained. That is, the scoring reference dimension is multiple, which can be considered from multiple angles, and finally a more humanized singing score is obtained.

[0048] Correspondingly, the embodiments of the present application also provide an electronic device and a readable storage medium corresponding to the above method, which have the above technical effects, and details are not described here. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 The embodiment of the present application is a model training method flowchart, which comprises the following steps:

[0051] Figure 2 The embodiment of the present application is an energy extraction schematic diagram, which comprises the following steps:

[0052] Figure 3 The embodiment of the present application is a peak energy and breathing time schematic diagram, which comprises the following steps:

[0053] Figure 4 The embodiment of the present application is a song synthesis processing method flowchart, which comprises the following steps:

[0054] Figure 5 The embodiment of the present application is an audio marking method flowchart, which comprises the following steps:

[0055] Figure 6 The embodiment of the present application is a song scoring method flowchart, which comprises the following steps:

[0056] Figure 7 The embodiment of the present application is a structure schematic diagram of an electronic device, which comprises the following steps:

[0057] Figure 8 The embodiment of the present application is a specific structure schematic diagram of an electronic device, which comprises the following steps. DETAILED DESCRIPTION

[0058] In order to make the person skilled in the art better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] Please refer to Figure 1 , Figure 1 The embodiment of the present application is a model training method flowchart, which comprises the following steps:

[0060] Step S001, obtaining a song audio and a song template corresponding to the song audio.

[0061] The song template has lyrics information or melody information of the song, that is, the song template can be specifically a lyrics template or a melody template. Specifically, the lyrics template corresponds to lyrics and interval time between each word or character; correspondingly, the melody template corresponds to melody and interval time corresponding to each note. The lyrics or melody of the song can be sorted in advance, and the song template is prepared in advance.

[0062] In the present application, the song audio can be all audio corresponding to a song or can be a segment audio. The song audio can correspond to the original singer of the song, or can correspond to other performers, or a synthesized audio of the song.

[0063] The song audio and the corresponding lyrics template can be obtained by reading a storage space, online receiving, or near field transmission, etc.

[0064] Step S002, determining a breathing position in the song audio by using the song template, and determining an interval duration of the breathing position from a previous singing section based on the breathing position.

[0065] Considering that a singer needs to breathe after singing for a period of time, the breathing position in the song audio and the interval duration of the breathing position from a previous singing section can be determined by using the song template. Specifically, for the part to be sung in the song template, the breathing position is determined according to the general breathing interval rule and the time interval in the template. For example, according to the lyrics template of the song, when the interval between words or characters (or letter groups) is greater than a specified duration (such as 250 milliseconds, which can be adjusted in practice), it is recorded as a breathing point, that is, the breathing position; according to the melody template of the song, when the interval between notes is greater than a specified duration (such as 250 milliseconds, which can be adjusted in practice), it is recorded as a breathing point, that is, the breathing position.

[0066] Considering that the breathing in actual performance may not strictly follow a specific breathing frequency, the song audio can be further detected for breathing while referring to the song template. That is, in actual application, the step S002 of determining the breathing position in the song audio by using the song template includes:

[0067] Step one, determining a breathing point in the song audio by using time interval information in the song template; wherein the song template is a melody template and / or a lyrics template; in the lyrics template, the time interval information is the interval duration between each word or character; in the melody template, the time interval information is the interval duration between each note;

[0068] Step two, breath detection is performed on the breath point to obtain a breath position.

[0069] For the convenience of description, the above two steps are described below.

[0070] The lyrics template and / or the melody template refer to that when determining the breath point in the song audio, only the lyrics template can be used, only the melody template can be used, or both can be used. When both are used, the corresponding breath points can be obtained based on the lyrics template and the melody template respectively, and then the two sets of breath points are merged, which can be performed in a union manner. Preferably, if the breath positions corresponding to the melody template and the lyrics template are different, the breath point determined by the lyrics template is subjected to breath detection to obtain a breath position. That is, for the melody template and the lyrics template, when the breath positions are different, the lyrics template is used as the reference.

[0071] After the breath point is determined, breath detection can be performed on the breath point position, so as to determine the breath position according to the breath detection result. That is, when the breath sound is detected, the breath point corresponds to the breath position; when the breath sound is not detected, the breath point does not correspond to the breath position.

[0072] Further, since a person cannot breathe for a long time, the position without obvious breath, such as the interval between the current and the next breath point > a specific time length (such as 7 seconds, and other time lengths can also be set in practice), or the silence time length of the sentence > a specific time length (such as 3 seconds, and other time lengths can also be set in practice), can be marked as a breath position.

[0073] After the breath position is obtained, the breath position distance from the previous singing section can be obtained directly by using the song template or by accumulating the interval distance between notes.

[0074] Step S003, energy extraction is performed at the breath position to obtain an energy peak value and a breath duration corresponding to the breath position.

[0075] After the breath position is determined, energy extraction can be performed at the breath position to obtain an energy peak value and a breath duration corresponding to the breath position.

[0076] Specifically, in order to obtain different dimensional breath features, after the breath position is determined, energy extraction can be performed on the breath position to obtain an energy peak value and a breath duration corresponding to the breath position. The energy peak value is the maximum energy value. In actual application, the audio corresponding to the breath position can also be processed to obtain an energy shape curve, and the energy peak value and the breath duration are determined based on the energy shape curve.

[0077] Specifically, the breathing duration is the duration corresponding to the breathing energy greater than a threshold (which can be set as the energy corresponding to no breathing). In actual applications, the breathing duration can be specifically determined by the distance between the concave points (corresponding to the energy peak and valley) on the left and right sides of the energy peak.

[0078] Specifically, in actual applications, step S003 extracts energy at the breathing position to obtain the energy peak value and the breathing duration corresponding to the breathing position, including:

[0079] Step one, sampling the breathing position in the song audio to obtain a sampling point;

[0080] Step two, extracting the energy of the sampling point;

[0081] Step three, smoothing the energy using a filter function to obtain an energy envelope;

[0082] Step four, determining the energy peak value and the breathing duration using the energy envelope.

[0083] For ease of description, the above four steps will be combined and described with reference to Figure 2

[0084] Sampling the audio waveform at the breathing position (such as Figure 2 ) and extracting the energy (RMS Energy) of the sampling point as Figure 2 , smoothing the energy using a filter function to obtain an energy envelope, such as Figure 2 .

[0085] RMS Energy = mean(abs(x)**2) x, which is the amplitude of the sampling point.

[0086] The filter function can be Savitzky-Golay filter; the Savitzky-Golay filter fitting method is to determine appropriate filter parameters according to the average trend of the NDVI time series curve, and to realize least square fitting in the sliding window using a polynomial. As shown in Figure 3 , the smoothed curve can use the maximum value point E (smoothed breathing energy peak value) and the distance between the concave points on the left and right sides of the peak value (the length of the breathing L) as the breathing characteristic value.

[0087] Step S004, labeling the breathing position, interval duration, energy peak value, and breathing duration in the song audio to obtain a training sample.

[0088] Specifically, the song audio can be established with a corresponding breathing label, and the breathing position, interval duration, energy peak value, and breathing duration can be noted in the breathing label, thereby obtaining a labeled training sample. ​

[0089] That is, the breathing label of the training sample, that is, the breathing feature, includes the breathing position, the interval duration, the energy peak value, and the breathing duration. The breathing position indicates the specific position of the breathing, the interval duration with the previous verse can indicate the corresponding time relationship of the breathing with the previous verse, the energy peak value corresponds to the strength of the breathing sound, and the breathing duration corresponds to the duration of the breathing sound.

[0090] In particular, in actual application, when labeling the breathing feature, the song BPM (Beat Per Minute, the unit of the number of beats per minute), the breathing position, the energy of the paragraphs before and after the breathing, and the paragraph breathing feature can be specifically labeled. Among them, the song BPM can be calculated using an open source tool (for example, madmom, a music analysis tool that can be used to extract BPM); the energy of the paragraphs before and after the breathing can be calculated as the root mean square error (RMSE, Root Mean Square Error) or Ebur128 loudness (wherein Ebu, European Broadcasting Union, Ebur128 loudness normalization standard); the duration of the previous sentence is in milliseconds; the duration of the next sentence is in milliseconds (ms); the breathing position is the duration T from the previous sentence, and the unit is millisecond.

[0091] Step S005, training the regression model using the training sample to obtain a feature extraction model.

[0092] The regression model is a predictive modeling technique that studies the relationship between the dependent variable (target) and the independent variable (predictor). This technique can be used for predictive analysis, time series modeling, and discovery of causal relationships between variables. In this embodiment, the specific architecture of the regression model and the specific regression method are not limited.

[0093] The trained regression model can be determined as a detection model after the model converges or the training round reaches a predetermined round. After training, the regression model can perform breathing detection on the input song audio and obtain breathing features, including breathing position, interval duration, energy peak value, and breathing duration.

[0094] By applying the model training method provided in the embodiments of the present application, the song audio and the song template corresponding to the song audio are obtained; the breathing position in the song audio is determined using the song template, and the interval duration of the breathing position from the previous verse is determined based on the breathing position; energy extraction is performed at the breathing position to obtain the energy peak value and the breathing duration corresponding to the breathing position; the breathing position, the interval duration, the energy peak value, and the breathing duration are labeled in the song audio to obtain a training sample; and the regression model is trained using the training sample to obtain a feature extraction model.

[0095] In the present application, first, the breathing position in the song audio is determined based on the song template, and the interval duration of the breathing position from the previous verse is determined, and then the energy extraction is performed on the breathing position, so that the energy peak and the breathing duration can be obtained. Then, the breathing position, interval duration, energy peak and breathing duration are labeled on the song audio. Among them, the breathing position indicates the specific position of the breathing, and the interval duration from the previous verse can indicate the corresponding time relationship of the breathing and the previous verse, the energy peak corresponds to the strength of the breathing sound, and the breathing duration corresponds to the duration of the breathing sound. After labeling, the training sample can be obtained, and the regression model is trained based on the training sample, so that the feature extraction model can be obtained, which can extract the breathing position, interval duration, energy peak and breathing duration from the input audio. That is, the feature extraction model of extracting multiple different dimensions of breathing features can be trained and obtained, which can meet more application requirements.

[0096] It should be noted that based on the above embodiments, the present application also provides corresponding improvement schemes. The steps involved in the preferred / improved embodiments can be mutually referenced between the above embodiments or corresponding steps, and the corresponding beneficial effects can also be mutually referenced. In the preferred / improved embodiments of the present application, they will not be described one by one.

[0097] Please refer to Figure 4 , Figure 4 A flowchart of a song synthesis processing method in an embodiment of the present application, the method comprising the following steps:

[0098] S101, obtaining a synthetic song audio artificially synthesized and without breathing features.

[0099] The synthetic song audio is an audio artificially synthesized and without breathing-related content.

[0100] S102, obtaining a reference audio corresponding to the synthetic song audio from a reference song library.

[0101] The reference audio is an audio belonging to the same song as the synthetic song audio, but the reference audio has breathing features.

[0102] S103, performing breathing detection on the reference audio by using a feature extraction model to obtain reference breathing features corresponding to the reference audio.

[0103] The feature extraction model is specifically a model trained by using the model training method described in the above embodiments, and the model can extract at least four dimensions of breathing features, i.e., including the breathing position, the interval duration of the breathing position from the previous verse, the energy peak corresponding to the breathing position, and the breathing duration and the interval duration.

[0104] S104, refer to the breathing reference feature, and add breathing audio corresponding to the breathing reference feature in the synthesized song audio.

[0105] After obtaining the breathing reference feature, the breathing audio corresponding to the breathing reference feature can be added in the synthesized song audio with reference to the breathing reference feature. Finally, the breathing audio in the synthesized audio can correspond to the breathing reference feature. The synthesized audio also has natural and harmonious breathing sound as the reference singing audio, that is, the synthesized audio is more natural and smooth.

[0106] By applying the song synthesis method provided in the embodiments of the present application, the breathing position, interval duration, energy peak value, and breathing duration can be extracted based on the feature extraction model, and the breathing audio corresponding to the reference audio can be added to the artificially synthesized synthesized song audio. The synthesized song audio finally obtained has breathing information, is more consistent with real human singing audio, and is more natural.

[0107] Please refer to Figure 5 , Figure 5 The flowchart of an audio marking method in the embodiments of the present application includes the following steps:

[0108] S201, obtaining the to-be-marked audio uploaded by a teaching singing client.

[0109] S202, using the feature extraction model trained in the above embodiments to perform breathing detection on the to-be-marked audio, to obtain the breathing position corresponding to the reference audio, the interval duration of the breathing position from the previous singing section, the energy peak value corresponding to the breathing position, and the breathing duration and interval duration.

[0110] S203, using the breathing position, interval duration, energy peak value, and breathing duration to mark the teaching information on the to-be-marked audio, to obtain a marked audio.

[0111] S204, feeding back the marked audio to the teaching singing client.

[0112] Specifically, it can be directly indicated in the song template corresponding to the target song where to breathe, the strength (depth) of the breathing, the duration of the breathing, and other information, so that the user can see the teaching information corresponding to the breathing, quickly learn the breathing rhythm corresponding to the target song, and quickly learn to sing the target song.

[0113] By applying the audio marking method provided in the embodiments of the present application, the breathing position, interval duration, energy peak value, and breathing duration can be extracted based on the feature extraction model, so that more rich teaching information can be marked on the to-be-marked audio. The marked audio fed back to the teaching singing client can enable the user to obtain more teaching information.

[0114] Please refer to Figure 6 , Figure 6A flowchart of a song scoring method in an embodiment of the present application, the method comprising the following steps:

[0115] S301, obtaining a target singing audio uploaded by a user terminal, and obtaining a reference breathing feature corresponding to the target song audio from a reference song feature library.

[0116] The user terminal obtains the target singing audio, which can be obtained by microphone collection, reading a storage space, or online receiving / downloading.

[0117] The target song audio can correspond to any song audio that needs to be scored, such as a user singing recording. The reference breathing feature can be a reference singing audio of the original song or a more standard (e.g., popular) singing, i.e., the reference singing audio has a relatively standard or aesthetically pleasing breathing sound.

[0118] Specifically, the reference singing audio can be input into the feature extraction model trained by the above-mentioned model training method to detect the breathing of the reference singing audio, thereby obtaining the reference breathing feature corresponding to the target song. The reference breathing feature can be pre-stored in the reference song feature library.

[0119] The reference breathing feature includes breathing position, interval duration, energy peak value, and breathing duration.

[0120] S302, using the feature extraction model trained in the above-mentioned embodiment to detect the breathing of the target singing audio, thereby obtaining a target breathing feature corresponding to the target singing audio.

[0121] As known from the above, the feature extraction model can extract at least four dimensions of breathing features, i.e., breathing position, interval duration, energy peak value, and breathing duration. Thus, the breathing feature includes breathing position, interval duration, energy peak value, and breathing duration.

[0122] That is, inputting the target singing audio into the feature extraction model can detect the breathing of the target singing audio, thereby obtaining a target breathing feature corresponding to the target singing audio. The target breathing feature has at least four different breathing attributes, i.e., breathing position, interval duration, energy peak value, and breathing duration.

[0123] S303, comparing the breathing position, interval duration, energy peak value, and breathing duration in the reference breathing feature and the target breathing feature, thereby obtaining a singing score of the target singing audio.

[0124] By comparing the breathing reference breathing feature and the singing breathing feature, a corresponding feature difference can be obtained. Specifically, at least one of the breathing position, interval duration, energy peak value and breathing duration can be compared to obtain the difference of various attributes, and the difference is determined as the feature difference.

[0125] Mapping the feature difference can map the corresponding singing score of the singing audio.

[0126] For example, only compare the peak energy, breathing duration and breathing position to illustrate the singing score. Among them, the breathing energy proportion score is:

[0127]

[0128] Breathing time accuracy score:

[0129] Breathing duration score:

[0130] Final singing score: Breathy score = αE score + βT score + γL score ; wherein α, β, γ are weight coefficients, which can be adjusted according to requirements.

[0131] Further, step S303 compares the breathing position, interval duration, energy peak value and breathing duration in the reference breathing feature and the target breathing feature to obtain the singing score of the target singing audio, including:

[0132] Step one, compare the reference breathing feature and the target breathing feature to obtain the feature difference;

[0133] Step two, according to the classification of the target song, the feature difference is weighted and mapped to obtain the singing score.

[0134] For ease of description, the above two steps will be combined for description.

[0135] Generally, different categories of songs correspond to different characteristics of breathing sounds. For example, the breathing sound of a ballad song is long, the breathing sound in the main song part is weak, the breathing sound in the chorus part is strong, the characteristics of breathing are closely related to the emotions expressed by singing; the breathing of a rock song is short, the breathing sound in the chorus part has a rhythmic nature, and the beat points are interwoven and echoed. In practical applications, the feature differences can also be weighted and mapped according to the specific classification of the song, so that the singing score corresponds to the corresponding breathing characteristics of the song classification. For example, if the target song is a ballad song, a higher weight can be given to the breathing time in the breathing characteristics corresponding to the main song part, and a lower weight can be given to other attributes of the breathing characteristics. A higher weight can be given to the energy peak value of the chorus part, and a lower weight can be given to other attributes of the breathing characteristics.

[0136] The song scoring method provided by the embodiment of the present application is applied in the method, and the scoring is based on the breathing position, interval time, energy peak value and breathing time in the reference breathing characteristics and the target breathing characteristics, and finally the singing score is obtained. That is, the scoring reference dimension is multiple, which can consider from multiple angles, and finally obtain a more humanized singing score.

[0137] Corresponding to the above method embodiment, the embodiment of the present application also provides an electronic device. The electronic device described below can be mutually corresponding with the model training method, song synthesis method, audio marking method and song scoring method described above.

[0138] Referring to Figure 7 As shown in the figure, the electronic device includes:

[0139] The memory 332 is configured to store a computer program.

[0140] The processor 322 is configured to execute the computer program to implement the steps of the model training method, song synthesis method, audio marking method or song scoring method of the above method embodiments.

[0141] Specifically, please refer to Figure 8 , Figure 8A specific structural schematic diagram of an electronic device is provided for the embodiment. The electronic device can have great differences due to different configurations or performances, and can include one or more processors (central processing units, CPUs) 322 (for example, one or more processors) and a memory 332 storing one or more computer applications 342 or data 344. The memory 332 can be temporary storage or persistent storage. The program stored in the memory 332 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the data processing device. Further, the central processor 322 can be configured to communicate with the memory 332 and execute the series of instruction operations in the memory 332 on the electronic device 301.

[0142] The electronic device 301 can further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0143] The steps in the model training method, the song synthesis method, the audio marking method, or the song scoring method described above can be implemented by the structure of the electronic device.

[0144] Corresponding to the method embodiments above, the embodiments of the present application also provide a readable storage medium. The readable storage medium described below can be mutually corresponding with the model training method, the song synthesis method, the audio marking method, and the song scoring method described above.

[0145] A readable storage medium, the readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the model training method, the song synthesis method, the audio marking method, or the song scoring method of the method embodiments.

[0146] The readable storage medium can be a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various readable storage media that can store program codes.

[0147] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both, and the general description of the components and steps of each example has been described above in order to clarify the interchangeability of hardware and software. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

Claims

1. A model training method, characterized in that, The method comprises: acquiring song audio and a song template corresponding to the song audio; determining a breathing position in the song audio by using the song template, and determining an interval duration of the breathing position from a previous singing section based on the breathing position; performing energy extraction at the breathing position to obtain an energy peak value and a breathing duration corresponding to the breathing position; labeling the breathing position, the interval duration, the energy peak value, and the breathing duration in the song audio to obtain a training sample; training a regression model by using the training sample to obtain a feature extraction model; wherein the breathing position in the song audio is determined by using the song template, comprising: determining a breathing exchange point in the song audio by using time interval information in the song template; wherein the song template is a melody template and / or a lyrics template; in the lyrics template, the time interval information is an interval duration between each word of lyrics; in the melody template, the time interval information is an interval duration between each note; performing breathing detection on the breathing exchange point to obtain the breathing position.

2. The model training method of claim 1, wherein, The breathing detection on the breathing exchange point to obtain the breathing position comprises: if the breathing positions corresponding to the melody template and the lyrics template are different, performing breathing detection on the breathing exchange point determined by the lyrics template to obtain the breathing position.

3. The model training method of claim 1, wherein, The energy extraction at the breathing position to obtain the energy peak value and the breathing duration corresponding to the breathing position comprises: sampling at the breathing position in the song audio to obtain a sampling point; extracting energy of the sampling point; performing smoothing processing on the energy by using a filter function to obtain an energy envelope line; determining the energy peak value and the breathing duration by using the energy envelope line.

4. A song synthesis method characterized by comprising: The method further comprises: acquiring a synthetic song audio that is artificially synthesized and does not have a breathing feature; acquiring a reference audio corresponding to the synthetic song audio from a reference song library; performing breathing detection on the reference audio by using the feature extraction model to obtain a reference breathing feature corresponding to the reference audio; correspondingly adding breathing audio in the synthetic song audio with reference to the reference breathing feature.

5. An audio marking method, characterized by, The method further comprises: acquiring a to-be-labeled audio uploaded by a teaching client; performing breathing detection on the to-be-labeled audio by using the feature extraction model to obtain a breathing position corresponding to the to-be-labeled audio, an interval duration of the breathing position from a previous singing section, an energy peak value and a breathing duration corresponding to the breathing position, and the interval duration; labeling teaching information on the to-be-labeled audio by using the breathing position, the interval duration, the energy peak value, and the breathing duration to obtain a labeled audio; feeding back the labeled audio to the teaching client.

6. A song rating method, characterized by, The method comprises: acquiring target singing audio uploaded by a user terminal, and acquiring a reference breathing feature corresponding to the target singing audio from a reference song feature library; performing breathing detection on the target singing audio by using the feature extraction model to obtain a target breathing feature corresponding to the target singing audio; Comparing the reference breathing feature and the target breathing feature in terms of breathing position, interval duration, energy peak and breathing duration, to obtain a singing score of the target singing audio.

7. The song rating method of claim 6, wherein, Comparing the reference breathing feature and the target breathing feature, to obtain a singing score of the target singing audio, includes: Comparing the reference breathing feature and the target breathing feature, to obtain a feature difference; According to the classification of the target song, the feature difference is weighted and mapped to obtain the singing score.

8. An electronic device, comprising: It includes: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the model training method according to any one of claims 1 to 3, or the steps of the song synthesis method according to claim 4, or the steps of the audio labeling method according to claim 5, or the steps of the song scoring method according to claim 6 or 7.

9. A readable storage medium, characterized by, The computer program is stored on the readable storage medium, and when executed by the processor, the computer program implements the steps of the model training method according to any one of claims 1 to 3, or the steps of the song synthesis method according to claim 4, or the steps of the audio labeling method according to claim 5, or the steps of the song scoring method according to claim 6 or 7.

Images