Audio processing methods, devices, electronic equipment and storage media

Abstract

This application relates to the field of computer technology and discloses an audio processing method, apparatus, electronic device, and storage medium, comprising: acquiring a target original vocal audio segment corresponding to a target song segment in a target song; extracting pitch feature information of the target original vocal audio segment; predicting the target performance level of the target object for the target song segment based on the pitch feature information of the target original vocal audio segment and using a target performance rating model applicable to the target object; determining a target volume ratio suitable for the target object when singing the target song segment based on the target performance level of the target object for the target song segment; and adjusting the volume ratio to the target volume ratio in response to the target object starting to sing the target song segment. The method of this application can guarantee the singing playback effect and can solve the problem of unnaturalness caused by audio editing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an audio processing method, apparatus, electronic device, and storage medium. Background Art

[0002] In related technologies, in karaoke applications, to ensure the singing playback effect, if the user's singing effect is not ideal, the karaoke application will modify the user's singing audio. However, although this method improves the singing playback effect to a certain extent, the audio after modification is significantly different from the user's real voice, and the sound after modification may become unnatural or abrupt. Summary of the Invention

[0003] In view of the above problems, embodiments of this application propose an audio processing method, apparatus, electronic device, and storage medium to improve the above problems.

[0004] According to one aspect of the embodiments of this application, an audio processing method is provided, comprising: obtaining a target original vocal audio segment corresponding to a target song segment in a target song; extracting pitch feature information of the target original vocal audio segment; predicting a target singing performance level of a target object for the target song segment based on the pitch feature information of the target original vocal audio segment and using a target singing rating model applicable to the target object; determining a target volume ratio applicable to the target object when singing the target song segment based on the target singing performance level of the target object for the target song segment; the target volume ratio referring to the ratio between the recording volume, the original vocal volume, and the accompaniment volume; and adjusting the volume ratio to the target volume ratio in response to the target object starting to sing the target song segment.

[0005] According to one aspect of the embodiments of this application, an audio processing apparatus is provided, comprising: an acquisition module, configured to acquire a target original vocal audio segment corresponding to a target song segment in a target song; an extraction module, configured to extract pitch feature information of the target original vocal audio segment; a prediction module, configured to predict a target singing performance level of a target object for the target song segment based on the pitch feature information of the target original vocal audio segment and through a target singing rating model applicable to the target object; a target volume ratio determination module, configured to determine a target volume ratio applicable to the target object when singing the target song segment based on the target singing performance level of the target object for the target song segment; wherein the target volume ratio refers to the ratio between the singing recording volume, the original vocal volume, and the accompaniment volume; and a ratio adjustment module, configured to adjust the volume ratio to the target volume ratio in response to the target object starting to sing the target song segment.

[0006] In some embodiments, the audio processing apparatus further includes: a first acquisition module, configured to acquire sample singing audio of multiple sample songs sung by the target object and sample original audio of each sample song; a segmentation module, configured to segment the sample singing audio and sample original audio of the sample songs according to the multiple sample song segments, to obtain multiple first audio segments in the sample singing audio and multiple second audio segments in the sample original audio; a pitch feature extraction module, configured to extract first pitch feature information of each first audio segment and second pitch feature information of each second audio segment; a singing performance level determination module, configured to determine the sample singing performance level of the target object for each of the sample song segments based on the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment corresponding to the same sample song segment; and a training module, configured to train the target singing rating model based on the second pitch feature information of the multiple second audio segments and the sample singing performance level of the target object for each of the sample song segments.

[0007] In some embodiments, the second pitch feature information includes the feature values ​​of the corresponding second audio segment under multiple pitch features; the training module includes: a first statistics unit, configured to calculate the prior probability of the target object for each performance level based on the sample performance levels of the target object for multiple sample song segments; a second statistics unit, configured to calculate the conditional probability of each performance level for the target object under the feature values ​​of each pitch feature based on the feature values ​​of the multiple second audio segments under multiple pitch features and the sample performance levels of the target object for multiple sample song segments; and a model determination unit, configured to determine the target singing rating model based on the prior probability of the target object for each performance level and the conditional probability of each performance level under the feature values ​​of each pitch feature.

[0008] In some embodiments, the pitch feature information of the target original audio segment includes the feature values ​​of the target original audio segment under multiple pitch features; the prediction module is used to: the target singing rating model processes the target object for the target song segment according to the following process to predict the target singing performance level of the target object for the target song segment: based on the prior probability of the target object for each singing performance level and the conditional probability of the feature values ​​of each singing performance level under multiple pitch features corresponding to the target original audio segment, perform posterior probability calculation to determine that the target singing performance level of the target object for the target song segment is the posterior probability of each singing performance level; and take the singing performance level with the highest posterior probability as the target singing performance level of the target object for the target song segment.

[0009] In other embodiments, the training module includes: a prediction unit, configured to predict the singing performance rating of the target singing rating model based on the second pitch feature information of each of the second audio segments, and output the predicted singing performance level of the target object for each sample song segment corresponding to each of the second audio segments; a prediction loss calculation unit, configured to calculate the prediction loss based on the predicted singing performance level of the target object for each sample song segment corresponding to each of the second audio segments, and the sample singing performance level of the target object for each of the sample song segments; and a parameter adjustment unit, configured to adjust the parameters of the target singing rating model based on the prediction loss until the training termination condition is met.

[0010] In some embodiments, the first pitch feature information includes feature values ​​of the first audio segment under multiple pitch features; the second pitch feature information includes feature values ​​of the second audio segment under multiple pitch features; the singing performance level determination module includes: a feature value deviation calculation unit, configured to determine the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under each of the pitch features based on the feature values ​​of the first audio segment and the second audio segment corresponding to the same sample song segment under multiple pitch features; and a sample singing performance level determination unit, configured to determine the sample singing performance level of the target object for each of the sample song segments based on the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under multiple pitch features.

[0011] In some embodiments, the target volume ratio determination module includes: a default volume ratio acquisition unit, used to acquire a default volume ratio; an adjustment strategy determination unit, used to determine a target volume ratio adjustment strategy adapted to the target singing performance level of the target object for the target song segment; and a volume ratio adjustment unit, used to adjust the default volume ratio according to the target volume ratio adjustment strategy to obtain a target volume ratio suitable for the target object to sing the target song segment.

[0012] In some embodiments, the target volume ratio adjustment strategy is used to adjust the proportion of one of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio, or to adjust the proportion of at least two of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio.

[0013] In some embodiments, the adjustment strategy determining unit is configured to: if the target performance level of the target object for the target song segment is higher than the first performance level, the target volume ratio adjustment strategy adapted to the target performance level is: maintaining the default volume ratio unchanged; if the target performance level of the target object for the target song segment is not higher than the first performance level, but higher than the second performance level, the target volume ratio adjustment strategy adapted to the target performance level is: based on the default volume ratio, maintaining the proportion of the original vocal volume unchanged, and increasing the proportion of the accompaniment volume while decreasing the proportion of the vocal recording volume; if the target performance level of the target object for the target song segment is not higher than the first performance level, but higher than the second performance level, the target volume ratio adjustment strategy adapted to the second performance level is: based on the default volume ratio, maintaining the proportion of the original vocal volume unchanged, and increasing the proportion of the accompaniment volume while decreasing the proportion of the vocal recording volume; if the target performance level of the target object for the target song segment is not higher than the first performance level, but higher than the second performance level, the target volume ratio adjustment strategy ... the target volume ratio adjustment strategy is: based on the default volume ratio, maintaining the proportion of the original vocal volume unchanged, and increasing the proportion of If the target singing performance level is higher than the third singing performance level, the target volume ratio adjustment strategy adapted to the target singing performance level is: based on the default volume ratio, increase the proportion of the original vocal volume and decrease the proportion of the accompaniment volume and the proportion of the vocal recording volume; if the target singing performance level of the target object for the target song segment is not higher than the third singing performance level, the target volume ratio adjustment strategy adapted to the target singing performance level is: based on the default volume ratio, maintain the proportion of the vocal recording volume unchanged, decrease the proportion of the accompaniment volume and increase the proportion of the original vocal volume; wherein, the first singing performance level > the second singing performance level > the third singing performance level, and the singing performance level is positively correlated with the singing effect.

[0014] In other embodiments, the target volume ratio determination module is used to: determine, based on the correspondence between singing performance level and volume ratio, the volume ratio corresponding to the target singing performance level of the target object for the target song segment as the target volume ratio applicable to the target object singing the target song segment; wherein, in the correspondence between singing performance level and volume ratio, the higher the singing performance level, the lower the proportion of the original vocal volume in the corresponding volume ratio; the singing performance level is positively correlated with the singing effect.

[0015] In some embodiments, the pitch feature information includes feature values ​​of the target original audio segment under multiple pitch features; the target original audio segment includes multiple target audio frames; the extraction module includes: a fundamental frequency detection unit, used to perform fundamental frequency detection on each target audio frame in the target original audio segment and determine the fundamental frequency of each target audio frame; a pitch conversion unit, used to convert the fundamental frequency of each target audio frame into pitch to obtain the pitch value of each target audio frame; and a feature value determination unit, used to determine the feature value of the target original audio segment under multiple pitch features based on the pitch values ​​of the multiple target audio frames in the target original audio segment.

[0016] In some embodiments, the plurality of pitch features include at least two of the following: highest pitch value, lowest pitch value, maximum stable duration of pitch, pitch variance value, and duration of the preceding silence pause.

[0017] In some embodiments, the audio processing device further includes: an audio synthesis module, configured to synthesize the original vocal audio segment corresponding to the target song segment, the corresponding accompaniment audio segment, and the vocal audio segment collected during the process of the target object singing the target song segment, according to the target volume ratio, to obtain a synthesized audio segment corresponding to the target song segment; and a playback module, configured to play the synthesized audio segment.

[0018] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the audio processing method described above.

[0019] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the audio processing method described above.

[0020] According to one aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the audio processing method described above.

[0021] In this application, a target singing rating model applicable to the target subject is used to predict the target singing performance level of the target subject for a specific song segment based on the pitch characteristic information of the original audio segment. Based on the target singing performance level of the target subject for that segment, a target volume ratio suitable for the target subject singing that segment is determined. When the target subject begins singing the segment, the volume ratio is adjusted to the target volume ratio. This solution predicts the target singing performance level of the target subject in advance and automatically adjusts the volume ratio to match the target performance level. The target volume ratio refers to the ratio between the recorded volume, the original vocal volume, and the accompaniment volume. By adjusting this ratio, deficiencies in the target subject's singing are masked. This method eliminates the need for audio editing of the target subject's audio, avoiding unnatural singing effects caused by excessive or inappropriate editing. Furthermore, adjusting the volume ratio ensures the playback quality of subsequent audio recordings and guarantees a superior auditory experience. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1A This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application.

[0024] Figure 1B This is an interactive schematic diagram illustrating singing in a client according to an embodiment of this application.

[0025] Figure 2 This is a flowchart illustrating an audio processing method according to an embodiment of this application.

[0026] Figure 3 This is a flowchart illustrating step 240 according to an embodiment of this application.

[0027] Figure 4 This is a schematic diagram illustrating an automatic percentage adjustment setting interface in a client according to an embodiment of this application.

[0028] Figure 5 This is a flowchart illustrating a training target singing rating model according to an embodiment of this application.

[0029] Figure 6 This is a schematic diagram illustrating step 540 according to an embodiment of this application.

[0030] Figure 7 This is a flowchart illustrating step 550 according to an embodiment of this application.

[0031] Figure 8 This is a flowchart illustrating step 230 according to an embodiment of this application.

[0032] Figure 9 This is a flowchart illustrating step 550 according to another embodiment of this application.

[0033] Figure 10 This is a flowchart illustrating an audio processing method according to another embodiment of this application.

[0034] Figure 11 This is a block diagram of an audio processing apparatus according to an embodiment of this application.

[0035] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0038] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0040] Figure 1A Figure 1 is a schematic diagram illustrating an application scenario of this application according to an embodiment. As shown in Figure 1, the application scenario includes a terminal 110 and a music database 120. The music database 120 can store the audio data of songs. The audio data of a song can include the original vocals and the accompaniment audio. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart TV, in-vehicle terminal, smart speaker, extended display device, wearable device, etc.

[0041] like Figure 1AAs shown, terminal 110 can run an application client. This application can be a karaoke application, or other entertainment applications that integrate karaoke functionality (such as music applications, live streaming applications, video applications), or other social applications that integrate karaoke functionality (such as instant messaging applications, content sharing applications). In other words, the application can be any application that integrates karaoke functionality. Karaoke functionality can be the main function of the application, or it can be an additional function besides the main function. The client can be a desktop client, a mobile client, or a mini-program client. The client can obtain the audio data of the target song from the music database 120. Then, from the audio data of the target song, it can obtain the target original audio segment corresponding to the target song segment. Subsequently, it can extract the pitch features of the target original audio segment to obtain the pitch feature information of the target original audio segment. Then, based on the pitch feature information of the target original audio segment, it can predict the singing performance level through a target singing rating model suitable for the target object to obtain the target singing performance level of the target object for the target song segment. Based on the target singing performance level of the target object for the target song segment, it can determine the target volume ratio suitable for the target object to sing the target song segment. The target volume ratio refers to the ratio between the singing recording volume, the original volume, and the accompaniment volume. In response to the target object starting to sing the target song segment, the volume ratio is adjusted to the target volume ratio.

[0042] Figure 1B This is an interactive diagram illustrating singing in a client according to an embodiment of this application. Figure 1B As shown, you can click Figure 1B The "Start a Live Stream" control on the public account page shown in Figure (1) displays... Figure 1B The live content selection window shown in (2) shows that if you click... Figure 1B The "Live" option in the live content selection window shown in (2) can display... Figure 1B The karaoke page shown in (3) is accessed; if you click... Figure 1B The “Voice Karaoke Room” control shown in Figure (3) can display the karaoke entry page. Figure 1B The karaoke mode selection window shown in (4) is displayed; if you click the "Karaoke Room" option in the karaoke mode selection window, you can display... Figure 1B The karaoke page shown in (5) allows users to sing karaoke live with one or more other users.

[0043] When multiple users participate in a live karaoke session, and the different users are on different clients, the method described in this application can be used to determine the target volume ratio applicable to each client when the user on each client sings the corresponding song segment. When the user on the client side starts singing the corresponding song segment, the corresponding volume ratio is adjusted to the determined target volume ratio.

[0044] The method described in this application is not limited to execution by the terminal; it can also be executed by the server corresponding to the application, or implemented by the interaction between the client's terminal and the server. No specific limitations are made here.

[0045] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0046] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0047] Figure 2 This is a flowchart illustrating an audio processing method according to an embodiment of this application. This method can be executed by an electronic device with processing capabilities, such as a terminal or server, and is not specifically limited thereto. (Refer to...) Figure 2 As shown, the method includes at least steps 210 to 250, which are described in detail below:

[0048] Step 210: Obtain the target original audio segment corresponding to the target song segment in the target song.

[0049] The target song can be a song that the target person is currently singing or a song that they are about to sing. A target song segment refers to any segment within the target song, such as the segment where the target person is about to sing. In some embodiments, the target song can be divided into multiple segments of equal length, with each segment having the same duration. For example, 0 seconds to 5 seconds of the target song could be considered the first segment, 5 seconds to 10 seconds the second segment, and so on.

[0050] In other embodiments, song data for the target song can be obtained from a song database. This song data includes basic information about the target song (e.g., lyrics, composer, singer, arrangement, song title, lyrics, etc.), as well as the target original vocal audio and the target accompaniment audio. Based on the target original vocal audio and the pitch values ​​of each original vocal audio frame, the target original vocal audio can be segmented into multiple segments. The time period within a single original vocal audio segment in the target song is considered a song segment. Each original vocal audio segment comprises multiple consecutive original vocal audio frames, and the pitch difference (i.e., the absolute value of the pitch difference) between different original vocal audio frames within a single original vocal audio segment does not exceed a first threshold. For example, if the original audio frames in the target original audio include audio frames 1 to 100 from first to last, and the pitch difference between any two audio frames 1 to 50 does not exceed a first threshold, and the pitch difference between audio frame 50 and audio frame 51 exceeds the first threshold, then audio frames 1 to 50 are divided into one original audio segment. The timestamp of audio frame 1 in the target original audio is time point A, and the timestamp of audio frame 50 in the target original audio is time point B. Therefore, a segment from time point A to time point B in the target song is taken as a song segment.

[0051] Step 220: Extract pitch feature information of the target original audio segment.

[0052] In some embodiments, the pitch feature information of the target original audio segment can be a sequence of pitch values ​​formed by the pitch values ​​of all original audio frames in the target original audio segment.

[0053] In other embodiments, pitch feature information may include feature values ​​of the target original audio segment under one or more pitch features. Pitch features may include the highest pitch value, the lowest pitch value, the maximum stable duration of the pitch, the pitch variance value, and the duration of the preceding silence pause.

[0054] Understandably, if the pitch feature information of the target original audio segment includes feature values ​​of the target original audio segment under multiple pitch features, then the multiple pitch features include at least two of the following: highest pitch value, lowest pitch value, maximum stable duration of pitch, pitch variance, and duration of the preceding silence. It is also understandable that the more pitch features involved in the pitch feature information, the more comprehensive the pitch characteristics of the target original audio segment reflected by the pitch feature information, ensuring a more accurate prediction of the subsequent performance.

[0055] A characteristic value of an original audio segment at its highest pitch can be either the numerical value of the highest pitch in that segment, or the first pitch level mapped to the highest pitch in that segment, determined based on a mapping relationship between the highest pitch value and a first pitch level. The mapping relationship between the highest pitch value and the second pitch level can be: the larger the highest pitch value, the higher the first pitch level. For example, four first pitch levels can be set: First Pitch Level I, First Pitch Level II, First Pitch Level III, and First Pitch Level IV. When the value of the highest pitch is less than 65, the first pitch level mapped to that value is First Pitch Level I; when the value of the highest pitch is ∈ [65, 70], the first pitch level mapped to that value is First Pitch Level II; when the value of the highest pitch is ∈ [71, 75], the first pitch level mapped to that value is First Pitch Level III; and when the value of the highest pitch is greater than 75, the first pitch level mapped to that value is First Pitch Level IV.

[0056] Similarly, the characteristic value of an original audio segment at its lowest pitch can be either the lowest pitch value in the original audio segment, or the second pitch level determined by the mapping relationship between the lowest pitch value and the second pitch level. The mapping relationship between the lowest pitch value and the second pitch level can be: the larger the lowest pitch value, the higher the second pitch level. For example, four second pitch levels can be set: second pitch level I, second pitch level II, second pitch level III, and second pitch level IV. When the value of the lowest pitch is less than 50, the second pitch level mapped to that value is second pitch level I; when the value of the lowest pitch is in the range [50, 55], the second pitch level mapped to that value is second pitch level II; when the value of the lowest pitch is in the range [56, 60], the second pitch level mapped to that value is second pitch level III; and when the value of the lowest pitch is greater than 60, the second pitch level mapped to that value is second pitch level IV.

[0057] The maximum stable duration of pitch for an original audio segment refers to the maximum number of consecutive original audio frames in the segment whose pitch differences do not exceed a second threshold. The second threshold is less than the first threshold; for example, the second threshold is 1. For instance, assuming the second threshold is 1, if an original audio segment contains a total of 50 original audio frames, and the absolute value of the pitch difference between any two adjacent original audio frames (1-10) does not exceed 1, the absolute value of the pitch difference between any two adjacent original audio frames (10-15) exceeds 1, the absolute value of the pitch difference between any two adjacent original audio frames (15-35) does not exceed 1, and the absolute value of the pitch difference between any two adjacent original audio frames (35-50) exceeds 1, then the maximum stable duration of pitch for this original audio segment is 21.

[0058] Similarly, the feature value of an original audio segment at its maximum stable duration can be either the numerical value of the maximum stable duration corresponding to that original audio segment, or the stability level mapped to the maximum stable duration value within the original audio segment, determined according to a set mapping relationship between the maximum stable duration value and the stability level. The mapping relationship between the maximum stable duration value and the stability level can be: the larger the maximum stable duration value, the higher the stability level. For example, four stability levels are defined as stability level 1, stability level 2, stability level 3, and stability level 4. If the maximum stable duration of a pitch in a given original audio segment is less than two original audio frames, the stability level corresponding to that pitch is stability level 1. If the maximum stable duration of a pitch in a given original audio segment is between 3 and 5 original audio frames, the stability level corresponding to that pitch is stability level 2. If the maximum stable duration of a pitch in a given original audio segment is between 6 and 10 original audio frames, the stability level corresponding to that pitch is stability level 3. If the maximum stable duration of a pitch in a given original audio segment is greater than 10 original audio frames, the stability level corresponding to that pitch is stability level 4.

[0059] The pitch variance of an original audio segment refers to the variance of the pitch values ​​of all original audio frames within that segment. Similarly, the characteristic value of an original audio segment under the pitch variance value can be either the numerical value of the pitch variance corresponding to that segment, or it can be the variance level mapped to the numerical pitch variance values ​​within the original audio segment, determined according to a predefined mapping relationship between pitch variance values ​​and variance levels. The mapping relationship between the numerical pitch variance values ​​and variance levels can be: the larger the numerical pitch variance value, the higher the variance level. For example, four variance levels are defined as variance level 1, variance level 2, variance level 3, and variance level 4. If the pitch variance value in a segment of original vocals does not exceed 3, the variance level mapped to that pitch variance value is variance level 1; if the pitch variance value in a segment of original vocals is ∈ (3, 5], the variance level mapped to that pitch variance value is variance level 2; if the pitch variance value in a segment of original vocals is ∈ (5, 10], the variance level mapped to that pitch variance value is variance level 3; and if the pitch variance value in a segment of original vocals is greater than 10, the variance level mapped to that pitch variance value is variance level 4.

[0060] The duration of the preceding silence in an original vocal audio segment refers to the number of consecutive silent original vocal audio frames preceding the first original vocal audio frame in that segment. Whether an original vocal audio frame is a silent original vocal audio frame can be determined based on its amplitude. If the amplitude of an original vocal audio frame is less than an amplitude threshold (which is a non-negative number), it can be determined that the original vocal audio frame is a silent original vocal audio frame; conversely, if the amplitude of an original vocal audio frame is not less than the amplitude threshold, it can be determined that the original vocal audio frame is not a silent original vocal audio frame.

[0061] Similarly, the feature value of the preceding silence duration for a segment of original vocal audio can be either the numerical value of the preceding silence duration corresponding to that segment, or it can be the pause level mapped to by the numerical value of the preceding silence duration, determined according to a set mapping relationship between the preceding silence duration value and the pause level. The mapping relationship between the preceding silence duration value and the pause level can be: the larger the preceding silence duration value, the higher the pause level.

[0062] For example, four pause levels can be set: pause level 1, pause level 2, pause level 3, and pause level 4. In a segment of original vocal audio, if the value of the preceding silence pause is less than 2, the pause level mapped to that pitch is pause level 1; if the value of the preceding silence pause in a segment of original vocal audio is ∈ [3, 8], the pause level mapped to that pitch is pause level 2; if the value of the preceding silence pause in a segment of original vocal audio is ∈ [9, 15], the pause level mapped to that pitch is pause level 3; and if the value of the preceding silence pause in a segment of original vocal audio is greater than 15, the pause level mapped to that pitch is pause level 4.

[0063] It is worth mentioning that the mapping relationships between the highest pitch value and the first pitch level, the lowest pitch value and the second pitch level, the maximum stable duration and the stability level, the pitch variance value and the variance level, and the duration of the preceding silence and the pause level are merely illustrative examples and should not be considered as limitations on the scope of this application.

[0064] In some embodiments, the pitch feature information of the target original audio segment includes the feature values ​​of the target original audio segment under multiple pitch features. Step 220 includes the following ①-③:

[0065] ① Perform fundamental frequency detection on each target audio frame in the target original audio segment to determine the fundamental frequency of each target audio frame.

[0066] In this application, the original audio frame in the target original audio segment is referred to as the target audio frame. The target original audio segment can be divided into multiple consecutive target audio frames according to equal duration. For example, the duration of each target audio frame can be 100ms.

[0067] The fundamental frequency of a target audio frame is equal to the reciprocal of its fundamental period. The fundamental period of a target audio frame refers to the longest repetition period in the target audio frame (equivalent to a short segment of sound signal). In some embodiments, the fundamental period of the target audio frame can be calculated using autocorrelation or Fourier transform methods.

[0068] In a specific embodiment, considering the various noise interferences present in the audio signal, the Fourier transform method may be significantly affected by noise in low signal-to-noise ratio environments. Noise can blur the spectral peaks in the spectrum, thus affecting the accurate estimation of the pitch period. Therefore, to reduce the impact of noise on the accuracy of pitch period calculation, the autocorrelation method can be used to calculate the pitch period of the target audio frame.

[0069] The autocorrelation method finds the pitch period by calculating the similarity between a sound signal and its delayed version (i.e., the short-time autocorrelation function). For a discrete sound signal x[n], its autocorrelation function Rxx[k] is defined as:

[0070] Rxx[k] = Σx[n] * x[nk]; (Formula 1)

[0071] Here, k is the delay, which can also be understood as the number of sampling points between x[nk] and x[n].

[0072] Then, the points where k=0 are removed from the autocorrelation function, and the delay k corresponding to the maximum value of the autocorrelation function is found. The determined k is the pitch period.

[0073] Furthermore, considering that the target original audio segment (or the target original audio) is sampled at a preset sampling rate, that is, the target audio frame includes multiple sampling points sampled at this sampling rate, assuming the sampling rate of the target audio frame is fs, the fundamental frequency F of the target audio frame can be determined according to the fundamental period k determined by Formula 1 above, as follows:

[0074] F = fs / k; (Formula 2)

[0075] ② Convert the fundamental frequency of each target audio frame to obtain the pitch value of each target audio frame.

[0076] The pitch value T of the target audio frame can be determined according to the fundamental frequency of the target audio frame using the following formula 3:

[0077] T = 12 × log2(F / F1) + 69; (Formula 3)

[0078] Where F1 is the reference base frequency, for example, F1 can be 440.

[0079] ③ Based on the pitch values ​​of multiple target audio frames in the target original audio segment, determine the feature values ​​of the target original audio segment under multiple pitch features.

[0080] After determining the pitch values ​​of each target audio frame in the target original audio segment, the highest pitch value, lowest pitch value, maximum stable duration of pitch, pitch variance value, and pitch features in the preceding silence pause duration can be determined accordingly, thereby determining the feature values ​​of the target original audio segment under each pitch feature.

[0081] Step 230: Based on the pitch feature information of the target original audio segment, predict the target singing performance level of the target object for the target song segment using a target singing rating model applicable to the target object.

[0082] A performance rating model suitable for evaluating the performance of a target subject is called a target performance rating model. This model can be trained using sample performance audio of multiple sample songs sung by the target subject and the original audio samples of each song. The sample performance audio refers to the audio recorded by the target subject during their performance of the sample song. The performance rating model can be a Bayesian classification model, an SVM (Support Vector Machine) model, or a deep learning network model; no specific limitations are specified here.

[0083] Since the target singing rating model can be trained using sample audio recordings of multiple sample songs sung by the target subject and sample original audio recordings of each song, during training, the model can learn the target subject's singing performance for different song segments with different pitch characteristics. The target singing performance level for a specific segment of a target song is the predicted performance level of the target subject when singing that segment; in other words, the target singing performance level reflects the performance effect of the target subject when singing that segment.

[0084] Step 240: Based on the target performance level of the target subject for the segment of the target song, determine the target volume ratio applicable to the target subject when singing the segment of the target song; the target volume ratio refers to the ratio between the recording volume, the original vocal volume, and the accompaniment volume.

[0085] Vocal recording volume refers to the volume of the audio recorded while the singer is performing. Original vocal volume refers to the playback volume set for the original vocal audio. Accompaniment volume refers to the playback volume set for the accompaniment audio.

[0086] Understandably, if the target performance level for a specific segment of a target song indicates that the target's performance is poor, then playing only the audio recorded during that segment will result in a poor auditory experience for the listener. Conversely, if the target performance level indicates that the target's performance is good, then playing only the audio recorded during that segment will result in a better auditory experience and a more pleasant listening experience for the listener.

[0087] Therefore, in this application, to ensure the auditory effect during playback of the recorded vocal audio, if the vocal performance is predicted to be poor, the volume ratio can be adjusted in advance to adjust the volume proportions of the three audio components (vocal audio component, original vocal audio component, and accompaniment audio component) in the synthesized audio. This ensures that the volumes of the three audio components differ when the synthesized audio is played. For example, if the adjusted volume ratio is: vocal recording volume : original vocal volume : accompaniment volume = k1 : k2 : k3, then in the synthesized audio, the vocal recording volume : original vocal volume : accompaniment volume = k1 : k2 : k3. Assuming the playback volume is set to B, theoretically, the playback volume of the vocal audio component will be [missing value]. The playback volume of the original audio component is The playback volume of the accompaniment audio component is By adjusting the volume ratio, the listener perceives different volumes of different audio components when playing synthesized audio; in other words, the proportion of the volume of different audio components perceived by the listener in the overall playback volume is different.

[0088] Therefore, when the performance of a singer is not good, by adjusting the volume ratio in advance, the volume of the singer's recording in the synthesized audio can be lower than the volume of the audio played without adjusting the volume ratio, while the volume of the accompaniment and / or the volume of the original vocals can be higher than the volume of the audio played without adjusting the volume ratio, so as to ensure that the overall listening effect is better.

[0089] In the volume ratio, the sum of the percentages of the recorded vocal volume, the original vocal volume, and the accompaniment volume is 1. The target volume ratio refers to the volume ratio applicable to a segment of a target song performed by a target artist. Specifically, the applicable volume ratio for a segment of a target song performed by a target artist refers to the ratio between the recorded vocal volume, the original vocal volume, and the accompaniment volume when the auditory effect of the recorded vocal audio segment is optimal.

[0090] Understandably, the proportion of vocal recording volume in the target volume ratio is related to the singing effect indicated by the target singing performance level. That is, when the singing effect indicated by the target singing performance level is poor, the proportion of original vocal volume and accompaniment volume in the target volume ratio can be relatively high overall; when the singing effect indicated by the target singing performance level is good, the proportion of original vocal volume and accompaniment volume in the target volume ratio can be relatively low overall.

[0091] In some embodiments, step 240 includes: according to the correspondence between singing performance level and volume ratio, taking the volume ratio corresponding to the target singing performance level of the target object for the target song segment as the target volume ratio applicable to the target object singing the target song segment; wherein, in the correspondence between singing performance level and volume ratio, the higher the singing performance level, the lower the proportion of the original vocal volume in the corresponding volume ratio; the singing performance level is positively correlated with the singing effect.

[0092] A higher performance level indicates a better singing effect. With a better performance, the original vocal volume accounts for a lower percentage of the final audio, while the target singer's voice accounts for a higher percentage. This ensures the target singer's participation and, due to the better performance, provides a better listening experience. Conversely, with a worse performance, the original vocal volume accounts for a higher percentage of the final audio. This allows the original vocals to mask any shortcomings in the target singer's performance, thus ensuring a better overall listening experience.

[0093] In other words, a volume ratio applicable to each singing performance level can be preset. Thus, after determining the target singing performance level of the target audience for the segment of the target song, the volume ratio applicable to the target singing performance level can be used as the target volume ratio applicable to the target audience when singing the segment of the target song.

[0094] Table 1 below illustrates the correspondence between singing performance levels and volume ratios according to an embodiment of this application. In Table 1, singing performance level I is higher than singing performance level II, singing performance level II is higher than singing performance level III, singing performance level III is higher than singing performance level IV, and singing performance level IV is higher than singing performance level V. In other words, the singing effect indicated by singing performance level I is better than the singing effect indicated by singing performance level II, the singing effect indicated by singing performance level II is better than the singing effect indicated by singing performance level III, and so on. The correspondence shown in Table 1 is merely an exemplary example and should not be considered as a limitation on the scope of this application.

[0095] Singing performance level Volume ratio (vocal recording volume: accompaniment volume: original vocal volume) Singing performance levels I and II 0.5：0.5：0 Singing Performance Level III 0.4：0.6：0 Singing Performance Level IV 0.45：0.4：0.15 Singing performance level V 0.5：0.2：0.3

[0096] Table 1

[0097] For example, if there are a total of 5 levels of singing performance, namely: excellent, good, average, slightly poor, and very poor, then the singing performance level I can be excellent, singing performance level II can be good, singing performance level III can be average, singing performance level IV can be slightly poor, and singing performance level V can be very poor.

[0098] In other embodiments, such as Figure 3 As shown, step 240 includes steps 310-330:

[0099] Step 310: Obtain the default volume ratio.

[0100] The default volume ratio in the client can be either the volume ratio set by default within the client itself, or the volume ratio set by the target user in the client's interactive interface before they sing the song. A possible default volume ratio is: vocal recording volume : accompaniment volume : original vocal volume = 0.5 : 0.5 : 0, but it is not limited to this and can be other ratios.

[0101] In the karaoke function provided by the client, since the karaoke function mainly provides lyrics and accompaniment to assist users in singing, the original vocal volume can be set to 0% in the default volume ratio setting of the client.

[0102] Step 320: Based on the target performance level of the target audience for each segment of the target song, determine the target volume ratio adjustment strategy that is compatible with the target performance level.

[0103] The correspondence between singing performance level and volume ratio adjustment strategy can be preset, with one singing performance level corresponding to one volume ratio adjustment strategy. In this way, after determining the target singing performance level, the corresponding volume ratio adjustment strategy can be determined as the target volume ratio adjustment strategy that is adapted to the target singing performance level.

[0104] Each performance level corresponds to a volume ratio adjustment strategy used to adjust the default volume ratio. A volume ratio adjustment strategy can set the adjustment amount and direction for the proportion of each audio component's volume (i.e., vocal recording volume, original vocal volume, and accompaniment volume), with the adjustment direction including decreasing or increasing. Specifically, when a performance level indicates a poor performance, the corresponding volume ratio adjustment strategy for that performance level will at least indicate an increase in the proportion of the original vocal volume from the default volume ratio. Of course, the adjustment amount for the proportion of the same audio component's volume can differ across volume ratio adjustment strategies corresponding to different performance levels.

[0105] In some embodiments, step 320 includes the following 1)-4):

[0106] 1) If the target performance level of the target object for the target song segment is higher than the first performance level, the target volume ratio adjustment strategy that is adapted to the target performance level is to maintain the default volume ratio unchanged.

[0107] The first performance level indicates a good performance. If the target performance level is not lower than the first performance level, it means that the performance indicated by the target performance level is not worse than that indicated by the first performance level. This suggests that the target person's performance of the target song segment may be better. Therefore, the default volume ratio should be maintained and no adjustment is needed.

[0108] 2) If the target performance level of the target object for the target song segment is not higher than the first performance level but higher than the second performance level, the target volume ratio adjustment strategy that matches the target performance level is: on the basis of the default volume ratio, maintain the original vocal volume ratio unchanged, and increase the accompaniment volume ratio while decreasing the vocal recording volume ratio.

[0109] The performance level indicated by the first performance level is superior to that indicated by the second performance level. When the target performance level is no higher than the first performance level but higher than the second performance level, a slight intervention is made to the default volume ratio. Based on the default volume ratio, the proportion of the original vocal volume remains unchanged, while the proportion of the accompaniment volume is increased and the proportion of the vocal recording volume is decreased. This is equivalent to increasing the proportion of the accompaniment volume to mask any deficiencies in the target vocal audio and ensure the overall playback effect.

[0110] 3) If the target performance level of the target object for the segment of the target song is not higher than the second performance level but higher than the third performance level, the target volume ratio adjustment strategy that matches the target performance level is: on the basis of the default volume ratio, increase the proportion of the original vocal volume and decrease the proportion of the accompaniment volume and the proportion of the vocal recording volume.

[0111] The second performance level indicates a better performance than the third performance level. If the target performance level is no higher than the second but higher than the third, it means the target's performance of a segment of the target song is better than the third performance level, but not as good as the second. In this case, simply increasing the accompaniment volume may not be enough for a satisfactory overall playback. Therefore, a moderate intervention is made to the default volume ratio, increasing the proportion of the original vocals to introduce the original audio and ensure a better overall playback. Understandably, introducing the original vocals volume has a greater impact than introducing the accompaniment volume, as it is more effective at masking deficiencies in the vocal audio.

[0112] 4) If the target vocal performance level for a segment of the target song is not higher than the third vocal performance level, the target volume ratio adjustment strategy adapted to the target vocal performance level is: based on the default volume ratio, maintain the proportion of the vocal recording volume unchanged, reduce the proportion of the accompaniment volume, and increase the proportion of the original vocal volume. Wherein, the first vocal performance level > the second vocal performance level >

[0113] The third performance level is positively correlated with the performance effect.

[0114] If the target singing performance level is not higher than the third singing performance level, it indicates that the target's singing effect for the target song segment is not as good as the singing effect indicated by the third singing performance level. Therefore, the default volume ratio is heavily intervened. On the basis of the default volume ratio, the proportion of the singing recording volume remains unchanged, the proportion of the accompaniment volume is reduced and the proportion of the original vocal volume is increased.

[0115] In some embodiments, the adjustment amount of the original vocal volume percentage defined in the target volume ratio adjustment strategy corresponding to case 4) above can be greater than the adjustment amount of the original vocal volume percentage defined in the target volume ratio adjustment strategy corresponding to case 3) above, so that after adjusting the default volume ratio according to the target volume ratio adjustment strategy corresponding to case 4) above, the original vocal volume percentage is greater than the original vocal volume percentage after adjusting the default volume ratio according to the target volume ratio adjustment strategy corresponding to case 3) above.

[0116] Step 330: Adjust the default volume ratio according to the target volume ratio adjustment strategy to obtain the target volume ratio suitable for the target object to sing the target song segment.

[0117] The target volume ratio adjustment strategy is used to adjust the proportion of one of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio, or to adjust the proportion of at least two of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio.

[0118] In some embodiments, the sum of the proportions of the three volume components in the volume ratio can be set to not exceed 1, that is, the sum of the proportions of the three audio components in the volume ratio can be less than 1. In this case, the target volume ratio adjustment strategy can be limited to adjusting the proportion of one of the vocal recording volume, the original vocal volume, and the accompaniment volume in the volume ratio. Of course, it can also be limited to adjusting the proportion of at least two of the vocal recording volume, the original vocal volume, and the accompaniment volume in the volume ratio.

[0119] In some embodiments, the sum of the proportions of the three audio components in the volume ratio can be set to always be 1. In this way, the target volume ratio adjustment strategy is limited to adjusting the proportion of at least two of the vocal recording volume, original vocal volume and accompaniment volume in the volume ratio.

[0120] Table 2 below shows the correspondence between singing performance levels and volume ratio adjustment strategies according to an embodiment of this application, and illustrates the volume ratios before and after volume ratio adjustment according to the corresponding volume ratio adjustment strategy. The volume ratios in Table 2 are all: vocal recording volume: accompaniment volume: original vocal volume.

[0121]

[0122] Table 2

[0123] In the above embodiments, volume ratio adjustment strategies are set for the singing performance levels in four ranges. In other embodiments, the singing performance levels can be divided into more refined ranges for more precise ratio adjustments. The percentage adjustment of the volume of each audio component under each volume ratio adjustment strategy in Table 2 above is merely an example and should not be considered as a limitation on the scope of this application.

[0124] In some embodiments, if there are a total of 5 singing performance levels: Excellent, Good, Average, Slightly Poor, and Very Poor, and the first singing performance level is set to Average, the second to Slightly Poor, and the third to Very Poor, then the target singing performance level is Excellent or Good. In this case, the default volume ratio is maintained according to the volume ratio adjustment strategy of "maintaining the default volume ratio unchanged," for example, maintaining the vocal recording volume: accompaniment volume: original vocal volume = 0.5:0.5:0.

[0125] If the target singing performance level is average, the volume ratio adjustment strategy of "maintaining the original vocal volume ratio unchanged on the basis of the default volume ratio, and increasing the proportion of the accompaniment volume and decreasing the proportion of the vocal recording volume" will be followed. The default volume ratio (vocal recording volume: accompaniment volume: original vocal volume = 0.5:0.5:0) will be adjusted to vocal recording volume: accompaniment volume: original vocal volume = 0.4:0.6:0.

[0126] If the target singing performance level is slightly poor, the volume ratio adjustment strategy of "increasing the proportion of the original vocal volume and decreasing the proportion of the accompaniment volume and the proportion of the vocal recording volume" will be followed. The default volume ratio (vocal recording volume: accompaniment volume: original vocal volume = 0.5:0.5:0) will be adjusted to vocal recording volume: accompaniment volume: original vocal volume = 0.45:0.4:0.15.

[0127] If the target singing performance level is extremely poor, the volume ratio adjustment strategy of "maintaining the proportion of the vocal recording volume unchanged, reducing the proportion of the accompaniment volume and increasing the proportion of the original vocal volume" will be followed. The default volume ratio (vocal recording volume: accompaniment volume: original vocal volume = 0.5:0.5:0) will be adjusted to vocal recording volume: accompaniment volume: original vocal volume = 0.5:0.2:0.3.

[0128] By using the volume ratio adjustment strategy described above, the performance deficiencies of the user (target) when singing the target song segment can be masked by using the accompaniment and original vocal audio, ensuring the auditory effect of the subsequent synthesized audio playback. Moreover, it allows the user to adjust their state in a timely manner during the masking intervals of the accompaniment and original vocal audio, which is conducive to improving the performance of subsequent song segments.

[0129] In some embodiments, before step 330, the method further includes: obtaining adjustment instruction information from the client; if the adjustment instruction information indicates that automatic volume ratio adjustment is allowed, then adjusting the default volume ratio according to the target volume ratio adjustment strategy to obtain the target volume ratio suitable for the target object to sing the target song segment; if the adjustment instruction information indicates that automatic volume ratio adjustment is not allowed, then there is no need to adjust the default volume ratio.

[0130] The client interface provides an option to set whether to allow automatic volume adjustment. If the user sets the option to allow automatic volume adjustment, an adjustment instruction indicating that automatic volume adjustment is allowed is generated and stored; if the user does not set the option to allow automatic volume adjustment, an adjustment instruction indicating that automatic volume adjustment is not allowed is saved.

[0131] In some embodiments, the adjustment instruction information may also indicate whether the proportion of each audio component volume (vocal recording volume, accompaniment volume, and original vocal volume) is allowed to be automatically adjusted. For ease of description, the audio component volume indicated by the adjustment instruction information as allowing automatic adjustment is referred to as the first audio component volume, and the audio component volume indicated as not allowing automatic adjustment is referred to as the second audio component volume. In this case, in step 330, the proportion of the first audio component volume in the default volume ratio can be adjusted according to the target volume ratio adjustment strategy, without needing to adjust the proportion of the second audio component volume in the default volume ratio, thus maintaining the proportion of the second audio component volume in the default volume ratio.

[0132] Figure 4 This is a schematic diagram illustrating an automatic percentage adjustment settings interface in a client according to an embodiment of this application, such as... Figure 4 As shown, the "Automatic Microphone Volume Adjustment" option is used to set whether the proportion of vocal recording volume can be automatically adjusted. If the "Automatic Microphone Volume Adjustment" option is checked, it means that the proportion of vocal recording volume can be automatically adjusted. The "Automatic Backing Track Volume Adjustment" option is used to set whether the proportion of backing track volume can be automatically adjusted, and the "Automatic Vocal Volume Adjustment" option is used to set whether the proportion of vocal volume can be automatically adjusted. If all three options are checked, it means that the proportion of volume of each individual audio component can be automatically adjusted.

[0133] Understandably, in Figure 4 In a corresponding embodiment, the sum of the proportions of the three volume components in the volume ratio can be preset to not exceed 1. In this way, the proportion of the volume of a single volume component can be set to allow adjustment.

[0134] Step 250: In response to the target object starting to sing the target song segment, adjust the volume ratio to the target volume ratio.

[0135] Adjusting the volume ratio to the target volume ratio means adjusting the volume ratio in the target client to the target volume ratio. After adjustment, during the target object's performance of a segment of the target song, the recorded audio segments of the target object's performance, the corresponding original vocal audio segments, and the corresponding accompaniment audio segments are synthesized according to the target volume ratio to obtain a synthesized audio segment. In other words, in the synthesized audio segment: the volume ratio of the vocal audio segment equals the proportion of the vocal recording volume in the target volume ratio; the volume ratio of the original vocal audio segment equals the proportion of the original vocal volume in the target volume ratio; and the volume ratio of the accompaniment audio segment equals the proportion of the accompaniment volume in the target volume ratio. When the target object begins performing a segment of the target song, the volume ratio is adjusted to the target volume ratio. Thus, during the target object's performance of that segment, the recorded volume is calculated as the product of the set total volume (e.g., a set recording volume) and the proportion of the vocal recording volume in the target volume ratio. And subsequently, the accompaniment volume is determined according to the proportion of the accompaniment volume in the target volume ratio, and the accompaniment audio segment corresponding to the target song segment is played; and the original vocal volume is determined according to the proportion of the original vocal volume in the target volume ratio, and the original vocal audio segment corresponding to the target song segment is played.

[0136] In some embodiments, after step 250, the method further includes: according to the target volume ratio, synthesizing the original vocal audio segment corresponding to the target song segment, the corresponding accompaniment audio segment, and the vocal audio segment collected during the target object's performance of the target song segment to obtain the synthesized audio segment corresponding to the target song segment; and playing the synthesized audio segment.

[0137] Based on the recording volume on the client side where the target object is located (i.e., the microphone volume set on the client side) and the target volume ratio, the volumes of the original vocal audio segment, the accompaniment audio segment, and the captured vocal audio segment corresponding to the target song segment to be synthesized can be determined separately. Assuming the target volume ratio is: vocal recording volume : original vocal volume : accompaniment volume = k1 : k2 : k3; assuming the set recording volume is V, in some embodiments, the set recording volume can be used to constrain the volume of the vocal audio segment, that is, to ensure that the volume of the vocal audio segment is V; the volume of the original vocal audio segment is... The volume of the accompaniment audio segments is After combining the three, the volume of the vocal audio segment in the synthesized audio segment is set to V; the volume of the original vocal audio segment is set to V. The volume of the accompaniment audio segments is

[0138] In other embodiments, the set recording volume can be used to limit the volume of the synthesized audio segments, that is, to ensure that the volume of the synthesized audio segments is V. Correspondingly, the volume of the singing audio segments can be determined to be V. The volume of the original audio segments is The volume of the accompaniment audio segments is After combining the three, the volume of the vocal audio segment in the combined audio segment is set to [value]. The volume of the original audio segments is The volume of the accompaniment audio segments is

[0139] In this application, a target singing performance rating model applicable to the target audience is used to predict the target singing performance level of the target audience for a specific segment of the target song based on the pitch characteristics of the original audio segment. Based on this target singing performance level, a target volume ratio is determined for the target audience when singing that segment. The volume ratio is then adjusted to the target volume ratio as soon as the target audience begins singing the segment. This solution predicts the target singing performance level in advance and automatically adjusts the volume ratio to match the target performance level. The target volume ratio refers to the ratio between the recorded volume, the original vocal volume, and the accompaniment volume. By adjusting this ratio, deficiencies in the target audience's singing are masked. This method eliminates the need for audio editing of the target audience's audio, avoiding unnatural singing effects caused by excessive or inappropriate editing. Furthermore, adjusting the volume ratio ensures the playback quality of subsequent audio recordings and guarantees a superior auditory experience.

[0140] In some embodiments, such as Figure 5 As shown, prior to step 230, the method further includes:

[0141] Step 510: Obtain the sample singing audio of the target object singing multiple sample songs and the sample original audio of each sample song.

[0142] Sample songs refer to the songs used to train the target singing rating model. The original audio of the sample song is called the sample original audio. The sample singing audio of the target subject singing the sample song is obtained by recording the audio of the target subject singing the sample song. The sample original audio of the sample song can be obtained from a song database.

[0143] Step 520: According to the multiple sample song segments of the sample song, the sample singing audio and the sample original audio of the sample song are segmented to obtain multiple first audio segments in the sample singing audio and multiple second audio segments in the sample original audio.

[0144] The method for segmenting the sample song is similar to that for segmenting the target song, as described above, and will not be repeated here. The audio segment corresponding to the sample song segment in the sample vocal audio is called the first audio segment, and the audio segment corresponding to the original vocal audio in the sample original audio is called the second audio segment. It is understandable that for the same sample song segment, the number of first and second audio segments is equal to the number of sample song segments in the sample song itself. One sample song segment corresponds to one first audio segment and one second audio segment. For example, if a sample song segment is from 0s to 10s in the sample song, then the first audio segment corresponding to this sample song segment refers to the segment from 0s to 10s in the sample vocal audio, and the second audio segment corresponding to this sample song segment refers to the segment from 0s to 10s in the sample original audio.

[0145] Step 530: Extract the first pitch feature information of each first audio segment and the second pitch feature information of each second audio segment.

[0146] In some embodiments, the first pitch feature information may be a pitch value sequence formed by the pitch values ​​of all audio frames in the first audio segment, and correspondingly, the second pitch feature information may be a pitch value sequence formed by the pitch values ​​of all audio frames in the second audio segment.

[0147] In other embodiments, the first pitch feature information may include the feature values ​​of the first audio segment under one or more pitch features. Similarly, the second pitch feature information includes the feature values ​​of the second audio segment under one or more pitch features. The pitch features may be the highest pitch value, the lowest pitch value, the maximum stable duration of the pitch, the pitch variance value, and the duration of the preceding silence. The pitch features involved in the first pitch feature information are the same as those involved in the second pitch feature information. The specific methods for extracting the first pitch feature information from the first audio segment and the second pitch feature information from the second audio segment are similar to the methods for extracting pitch feature information from the target original audio segment described above, and will not be repeated here.

[0148] Step 540: Determine the sample singing performance level of the target object for each sample song segment based on the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment corresponding to the same sample song segment.

[0149] The sample performance level of a target subject for a given segment of a sample song is used to indicate the performance effect of the target subject when singing that segment. For the first and second audio segments corresponding to the same segment of a sample song, the higher the similarity between the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment, the closer the target subject's performance of the sample song segment is to the original singer, and the better the performance effect indicated by the sample performance level of the target subject for that segment. Conversely, the lower the similarity between the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment, the greater the difference between the target subject's performance of the sample song segment and the original singer, and the worse the performance effect indicated by the sample performance level of the target subject for that segment.

[0150] In some embodiments, the first pitch feature information of each first audio segment can be vectorized to obtain the first pitch feature vector of each first audio segment, and the second pitch feature information of each second audio segment can be vectorized to obtain the second pitch feature vector of each second audio segment. The first pitch feature vector and the second pitch feature vector have the same dimension, and the same dimension represents the same pitch feature. Then, the similarity between the first pitch feature vector of the first audio segment and the second pitch feature vector of the second audio segment corresponding to the same sample song segment is calculated to obtain the pitch feature similarity between the two. Then, the sample singing performance level of the target object for each sample song segment is determined based on the pitch feature similarity.

[0151] In some embodiments, if the first pitch feature information can be a pitch value sequence formed by the pitch values ​​of all audio frames in the first audio segment, and the second pitch feature information can be a pitch value sequence formed by the pitch values ​​of all audio frames in the second audio segment, then two relatively aligned audio frames in the first and second audio segments corresponding to the same sample song segment are considered as an audio frame group. One audio frame in an audio frame group comes from the first audio segment, and the other audio frame comes from the second audio segment. The t-th audio frame in the first audio segment and the t-th audio frame in the second audio segment corresponding to the same sample song segment are two relatively aligned audio frames, where t is a positive integer. Based on this, the pitch values ​​of the two relatively aligned audio frames in each audio frame group can be subtracted according to the first and second pitch feature information, and the absolute value is taken to obtain the pitch value difference corresponding to each audio frame group. Then, the average pitch value difference corresponding to all audio frame groups determined for the same sample song segment is calculated to obtain the average pitch value difference of the target object for each sample song segment. Subsequently, based on the correspondence between pitch difference and performance level, the performance level corresponding to the calculated average pitch difference is determined, which serves as the sample performance level for the corresponding sample song segment.

[0152] In other embodiments, a correspondence between pitch difference and performance score can also be established. After calculating the pitch difference for each audio frame group, the performance score of the target object for each sample song segment is determined. For example, if the pitch difference does not exceed a first pitch threshold, the corresponding performance score is increased by N1 points (N1 is a positive integer). If the pitch difference is greater than (first pitch threshold, second pitch threshold), the corresponding performance score is decreased by N2 points (N2 is a positive integer). The second pitch threshold is greater than the first pitch threshold, for example, the first pitch threshold is 2 and the second pitch threshold is 4. If the pitch difference exceeds the second pitch threshold, the corresponding performance score is decreased by N3 points (N3 is a positive integer). N1, N2, and N3 can be set according to actual needs. For example, N2 is less than N1 and less than N3, for example, N1 is 3, N2 is 1, and N3 is 3. Next, the performance scores corresponding to all audio frame groups determined for the same sample song segment are accumulated and averaged to obtain the first average performance score of the target object for each sample song segment. Subsequently, based on the correspondence between performance scores and performance levels, the performance level corresponding to the average performance score of the target object for each sample song segment is taken as the sample performance level of the target object for the corresponding sample song segment.

[0153] In some embodiments, the first pitch feature information includes feature values ​​of the first audio segment under multiple pitch features; the second pitch feature information includes feature values ​​of the second audio segment under multiple pitch features; such as Figure 6 As shown, step 540 includes:

[0154] Step 610: Based on the feature values ​​of the first audio segment and the second audio segment corresponding to the same sample song segment under multiple pitch features, determine the feature value deviation of the first audio segment and the second audio segment under multiple pitch features.

[0155] For the first and second audio segments corresponding to the same sample song segment, the feature values ​​of the first and second audio segments under the same pitch feature are subtracted, and the absolute value is taken to obtain the feature value deviation under that pitch feature.

[0156] Step 620: Determine the sample singing performance level of the target object for each sample song segment based on the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under multiple pitch features.

[0157] In some embodiments, the feature value deviations of the first audio segment and the second audio segment corresponding to the same sample song segment under multiple pitch features can be weighted and calculated to obtain the reference feature feature value deviation. Then, according to the correspondence between the feature value deviation and the performance level, the performance level corresponding to the reference feature feature value deviation is used as the sample performance level of the target object for the corresponding sample song segment.

[0158] In some embodiments, a correspondence between feature value deviation and singing score can be established. Then, based on the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under each pitch feature, a singing score corresponding to the feature value deviation under each pitch feature is determined. Then, the singing scores corresponding to the feature value deviation under multiple pitch features are accumulated and averaged to obtain a second average singing score. Then, based on the correspondence between singing score and singing performance level, the singing performance level corresponding to the second average singing score is used as the sample singing performance level of the target object for the corresponding sample song segment.

[0159] For example, if there are a total of 5 performance levels: Excellent, Good, Average, Slightly Poor, and Very Poor, the correspondence between the performance score and the performance level can be as follows: 1) If the performance score exceeds the first score threshold, the corresponding performance level is Excellent; 2) If the performance score ∈ (second score threshold, first score threshold), the corresponding performance level is Good; 3) If the performance score ∈ (third score threshold, second score threshold), the corresponding performance level is Average; 4) If the performance score ∈ (fourth score threshold, third score threshold), the corresponding performance level is Slightly Poor; 5) If the performance score does not exceed the fourth score threshold, the corresponding performance level is Very Poor. The first score threshold > second score threshold > third score threshold > fourth score threshold, and all are non-negative numbers. These can be set according to actual needs, for example, the first score threshold is 2.5, the second score threshold is 2.0, the third score threshold is 1.2, and the fourth score threshold is 0.5.

[0160] Step 550: Based on the second pitch feature information of multiple second audio segments and the sample singing performance level of the target object for each sample song segment, train the target singing rating model.

[0161] In some embodiments, the second pitch feature information includes the feature values ​​of the corresponding second audio segment under multiple pitch features; the target singing rating model can be a Bayesian classification model. For example... Figure 7 As shown, step 550 includes the following steps 710-730:

[0162] Step 710: Based on the target object's sample singing performance level for multiple sample song segments, calculate the prior probability of the target object for each singing performance level.

[0163] When the target singing performance rating model is a Bayesian classification model, the process of training the target singing performance rating model mainly involves determining the prior probability of the target object for each singing performance level, as well as the conditional probability of each singing performance level of the target object under the feature value of each pitch feature. This facilitates the target singing performance rating model, after training, to predict the probability of the target object singing each singing performance level of the song segment based on the prior probability and the determined conditional probability determined during the training process, combined with the pitch feature information of the original audio segment corresponding to the song segment to be sung by the target object, and thus determine the final singing performance level corresponding to the target object when singing the song segment.

[0164] In some embodiments, the number of sample song segments corresponding to each performance level can be counted among the sample performance levels of multiple sample song segments for the target object, and then the probability of each performance level can be counted.

[0165] For example, if a total of 5 performance levels are set, namely performance level I, performance level II, performance level III, performance level IV, and performance level V, and among the sample performance levels of 1000 sample song segments for the target object, the number of sample song segments with performance level I, performance level II, performance level III, performance level IV, and performance level V are 50, 150, 200, 350, and 250, respectively, then the prior probabilities of the target object performing at performance level I, performance level II, performance level III, performance level IV, and performance level V can be statistically determined to be 0.05, 0.15, 0.2, 0.35, and 0.25, respectively.

[0166] Step 720: Based on the feature values ​​of multiple second audio segments under multiple pitch features and the sample singing performance level of the target object for multiple sample song segments, calculate the conditional probability of each singing performance level of the target object under the feature value of each pitch feature.

[0167] For example, if there are a total of 3 feature values ​​for pitch feature 1, namely feature value A1, feature value A2 and feature value A3, and assuming that when the feature value of pitch feature 1 is feature value A1, it is represented as pitch feature 1_A1, in the above 50 sample song segments with a performance level of performance level I, there are a total of 3 sample song segments with the feature value of pitch feature 1 being feature value A1, then the conditional probability that the target object has the feature value A1 for pitch feature 1 in performance level I is P(pitch feature 1_A1|performance level I) = 3 / 50 = 0.06.

[0168] Step 730: Determine the target singing rating model based on the prior probability of the target object for each singing performance level and the conditional probability of each singing performance level under the feature value of each pitch feature.

[0169] The prior probability of the target object for each singing performance level, and the conditional probability of each singing performance level under the feature value of each pitch feature, can be used as parameters of the target singing rating model, and serve as the basis for subsequent singing performance level prediction.

[0170] Through the training process described above, a target singing rating model suitable for the target audience can be specifically determined to ensure the accuracy of the singing performance predictions made by the target singing rating model in the future.

[0171] In some embodiments, if the target vocal rating model is a Bayesian classification model, the pitch feature information of the target original vocal audio segment includes the feature values ​​of the target original vocal audio segment under multiple pitch features; such as Figure 8 As shown, in step 230, the target singing rating model processes the data according to steps 810-820 to predict the target singing performance level of the target object for each segment of the target song:

[0172] Step 810: Based on the prior probability of the target object for each singing performance level and the conditional probability of the feature value of each singing performance level under multiple pitch features corresponding to the target original audio segment, perform posterior probability calculation to determine the target singing performance level of the target object for the target song segment as the posterior probability of each singing performance level.

[0173] Step 820: Take the performance level with the highest posterior probability as the target performance level for the target song segment.

[0174] For example, if a total of 5 pitch features are involved, namely pitch feature 1, pitch feature 2, pitch feature 3, pitch feature 4, and pitch feature 5, these 5 pitch features can be the highest pitch value, the lowest pitch value, the maximum stable duration of the pitch, the pitch variance value, and the duration of the preceding silence pause mentioned above. If the pitch feature information of the target original audio segment is as follows: the feature value of pitch feature 1 is feature value A3 (represented as pitch feature 1_A3), the feature value of pitch feature 2 is feature value B1 (represented as pitch feature 2_B1), the feature value of pitch feature 3 is feature value C2 (represented as pitch feature 3_C2), the feature value of pitch feature 4 is feature value D3 (represented as pitch feature 4_D3), and the feature value of pitch feature 5 is feature value E4 (represented as pitch feature 5_E4). If there are a total of 5 performance levels, namely Performance Level I, Performance Level II, Performance Level III, Performance Level IV, and Performance Level V, then the posterior probability of the target performance level for each segment of the target song being a specific performance level can be determined using the following formula:

[0175] P(Singing Performance Level I | Pitch Feature 1_A3, Pitch Feature 2_B1, Pitch Feature 3_C2, Pitch Feature 4_D3, Pitch Feature 5_E4) = P(Singing Performance Level I) * P(Pitch Feature 1_A3 | Singing Performance Level I) * P(Pitch Feature 2_B1 | Singing Performance Level I) * P(Pitch Feature 3_C2 | Singing Performance Level I) * P(Pitch Feature 4_D3 | Singing Performance Level I) * P(Pitch Feature 5_E4 | Singing Performance Level I); (Formula 4)

[0176] P(Singing Performance Level II | Pitch Feature 1_A3, Pitch Feature 2_B1, Pitch Feature 3_C2, Pitch Feature 4_D3, Pitch Feature 5_E4) = P(Singing Performance Level II) * P(Pitch Feature 1_A3 | Singing Performance Level II) * P(Pitch Feature 2_B1 | Singing Performance Level II) * P(Pitch Feature 3_C2 | Singing Performance Level II) * P(Pitch Feature 4_D3 | Singing Performance Level II) * P(Pitch Feature 5_E4 | Singing Performance Level II); (Formula 5)

[0177] P(Singing Performance Level III | Pitch Feature 1_A3, Pitch Feature 2_B1, Pitch Feature 3_C2, Pitch Feature 4_D3, Pitch Feature 5_E4) = P(Singing Performance Level III) * P(Pitch Feature 1_A3 | Singing Performance Level III) * P(Pitch Feature 2_B1 | Singing Performance Level III) * P(Pitch Feature 3_C2 | Singing Performance Level III) * P(Pitch Feature 4_D3 | Singing Performance Level III) * P(Pitch Feature 5_E4 | Singing Performance Level III); (Formula 6)

[0178] P(Singing Performance Level IV | Pitch Feature 1_A3, Pitch Feature 2_B1, Pitch Feature 3_C2, Pitch Feature 4_D3, Pitch Feature 5_E4) = P(Singing Performance Level IV) * P(Pitch Feature 1_A3 | Singing Performance Level IV) * P(Pitch Feature 2_B1 | Singing Performance Level IV) * P(Pitch Feature 3_C2 | Singing Performance Level IV) * P(Pitch Feature 4_D3 | Singing Performance Level IV) * P(Pitch Feature 5_E4 | Singing Performance Level IV); (Formula 7)

[0179] P(Singing Performance Level V|Pitch Feature 1_A3, Pitch Feature 2_B1, Pitch Feature 3_C2, Pitch Feature 4_D3, Pitch Feature 5_E4) = P(Singing Performance Level V) * P(Pitch Feature 1_A3|Singing Performance Level V) * P(Pitch Feature 2_B1|Singing Performance Level V) * P(Pitch Feature 3_C2|Singing Performance Level V) * P(Pitch Feature 4_D3|Singing Performance Level V) * P(Pitch Feature 5_E4|Singing Performance Level V); (Formula 8)

[0180] In the above equation, P(performance level I|pitch feature 1_A3, pitch feature 2_B1, pitch feature 3_C2, pitch feature 4_D3, pitch feature 5_E4) represents the posterior probability that the target performance level of the target object for the target song segment is performance level I; P(performance level I) represents the prior probability of the target object for performance level I.

[0181] P(Pitch Feature 1_A3|Singing Performance Level I) represents the conditional probability of singing performance level I for a target subject given that the feature value of pitch feature 1 is A3. Similarly, P(Pitch Feature 2_B1|Singing Performance Level I) represents the conditional probability of singing performance level I for a target subject given that the feature value of pitch feature 2 is B1. The physical meanings of the other characters are similar and will not be explained in detail here.

[0182] After determining the posterior probabilities of the target performance level (I to V) for each segment of the target song according to Formulas 4-8 above, the performance level corresponding to the highest posterior probability is taken as the target performance level for each segment of the target song. For example, if the calculated P(performance level V|pitch feature 1_A3, pitch feature 2_B1, pitch feature 3_C2, pitch feature 4_D3, pitch feature 5_E4) is the largest, then the target performance level for each segment of the target song can be determined as performance level V.

[0183] In other embodiments, if the target singing rating model is a support vector machine model or a deep learning network model, it can be performed as follows: Figure 9 The process shown is used for training, such as Figure 9 As shown, step 550 includes:

[0184] Step 910: The target singing rating model predicts the singing performance rating based on the second pitch feature information of each second audio segment, and outputs the predicted singing performance level of the target object for the sample song segment corresponding to each second audio segment.

[0185] After inputting the second pitch feature information of the second audio segment into the target singing rating model, the target singing rating model can extract features from the second pitch feature information, and then classify them according to the extracted features. The predicted singing performance level of the target object for each sample song segment corresponding to each second audio segment is the probability of each singing performance level, and the singing performance level with the highest probability is taken as the predicted singing performance level of the target object for each sample song segment corresponding to each second audio segment.

[0186] Step 920: Calculate the prediction loss based on the predicted performance level of the target object for each sample song segment corresponding to each second audio segment, and the sample performance level of the target object for each sample song segment.

[0187] The prediction loss can be calculated by combining the predicted performance level of the target object for each second audio segment and the sample performance level of the target object for each sample song segment using a loss function. The loss function can be an absolute value loss function, a mean squared error loss function, a root mean square error loss function, a cross-entropy loss function, etc., and no specific limitation is made here.

[0188] Step 930: Adjust the parameters of the target singing rating model based on the predicted loss until the training termination condition is met.

[0189] The training termination condition can be that the number of iterations of the target singing rating model reaches a threshold, or the loss function converges; no specific limitation is made here. In the training process described above, the sample singing performance levels of the target object for each song segment are used as supervised information to supervise the training of the target singing rating model. This allows the model to learn the pitch features of the original audio segments (second audio segments) for different singing performance levels, enabling the model to accurately predict the target object's singing performance level for the input audio segment based on its pitch features.

[0190] Figure 10 This is a flowchart illustrating an audio processing method according to an embodiment of this application, such as... Figure 10 As shown, it includes:

[0191] Step 1010: Record audio while the target person is singing the sample song to obtain the sample singing audio of the sample song.

[0192] Step 1020: Obtain the original audio sample of the sample song.

[0193] Step 1030: Calculate the pitch feature deviation of the sample singing audio relative to the original sample audio.

[0194] Specifically, the sample song audio and the original sample audio can be segmented according to multiple sample song segments. The sample song audio is divided into multiple first audio segments, and the original sample audio is divided into multiple second audio segments. Then, the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment are extracted respectively. For relatively aligned first and second audio segments (i.e., first and second audio segments corresponding to the same sample song segment), the pitch feature deviation is calculated based on the corresponding first and second pitch feature information. Please refer to the description above for the specific calculation method.

[0195] Then, based on the pitch feature deviation between the relatively aligned first and second audio segments, the sample singing performance level of the target object for each sample song segment in the sample song is determined.

[0196] Step 1040: Train the target singing rating model online.

[0197] That is, the target singing rating model is trained using the target object's performance level for each segment of the sample song and the second pitch feature information of multiple second audio segments. Please refer to the description above for the specific training process.

[0198] Step 1050: Predict the target performance level of the target subject for the target song segment using the target performance rating model.

[0199] It can obtain the target original audio segment corresponding to the target song fragment, extract the pitch feature information of the target original audio segment, input the pitch feature information of the target original audio segment into the target singing rating model, and the target singing rating model predicts the singing performance level based on the pitch feature information of the target original audio segment, and outputs the target singing performance level of the target object for the target song fragment.

[0200] Step 1060: Dynamically adjust the volume ratio according to the target singing performance level.

[0201] Specifically, the target volume ratio can be determined based on the target singing performance level when the target person sings the target song segment, and the volume ratio in the client can be adjusted to the target volume ratio in response to the start of the target person singing the target song segment.

[0202] In this application, a targeted singing performance rating model is specifically trained for the target audience. Based on the pitch characteristics of the original audio segments, the model predicts the target audience's performance level for a specific song segment and adjusts the volume ratio during the performance. The target performance level reflects whether the singer (target audience) is unable to handle the song segment. If the performance level indicates poor quality, adjusting the volume ratio—specifically, the ratio between the recording volume, the original vocal volume, and the accompaniment volume—masks the target audience's shortcomings in the song segment. Furthermore, it allows the singer to adjust their vocal state in a timely manner to improve performance and ensure a better final auditory experience.

[0203] In related technologies, to improve the auditory effect of singing, singing aid applications use automatic pitch correction technology to correct the pitch of the singer's recorded audio. However, this method has two problems: 1. The pitch correction effect is strongly correlated with the pitch correction algorithm. Especially in online real-time applications, automatic pitch correction technology generally does not reach the level of perfectly matching the generated audio features with the singer's vocal features. Therefore, listeners can easily perceive that the audio has been pitch-corrected and has obvious differences from the singer's real voice; 2. After pitch correction, the sound may become unnatural or abrupt, and the effect after pitch correction does not meet expectations.

[0204] The proposed solution does not modify the singer's recorded audio, thus avoiding the negative impact of unnatural performance caused by excessive or inappropriate audio editing. Furthermore, it employs an adaptive volume adjustment mechanism throughout the entire performance, allowing the singer to fully enjoy the experience rather than simply receiving an objective score. This approach enables singers to perform more freely and enjoyably, enhancing their overall singing experience.

[0205] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0206] Figure 11 This is a block diagram of an audio processing apparatus according to an embodiment of this application, such as... Figure 11 As shown, the audio processing device includes: an acquisition module 1110, used to acquire the target original audio segment corresponding to the target song segment in the target song; an extraction module 1120, used to extract the pitch feature information of the target original audio segment; a prediction module 1130, used to predict the target performance level of the target object for the target song segment based on the pitch feature information of the target original audio segment and through a target performance rating model applicable to the target object; a target volume ratio determination module 1140, used to determine the target volume ratio applicable to the target object when singing the target song segment based on the target performance level of the target object for the target song segment; the target volume ratio refers to the ratio between the recording volume, the original vocal volume, and the accompaniment volume; and a ratio adjustment module 1150, used to adjust the volume ratio to the target volume ratio in response to the target object starting to sing the target song segment.

[0207] In some embodiments, the audio processing apparatus further includes: a first acquisition module, configured to acquire sample singing audio of a target object singing multiple sample songs and sample original audio of each sample song; a segmentation module, configured to segment the sample singing audio of the sample songs and the sample original audio of the sample songs according to the multiple sample song segments, to obtain multiple first audio segments in the sample singing audio and multiple second audio segments in the sample original audio; a pitch feature extraction module, configured to extract first pitch feature information of each first audio segment and second pitch feature information of each second audio segment; a singing performance level determination module, configured to determine the sample singing performance level of the target object for each sample song segment based on the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment corresponding to the same sample song segment; and a training module, configured to train a target singing rating model based on the second pitch feature information of multiple second audio segments and the sample singing performance level of the target object for each sample song segment.

[0208] In some embodiments, the second pitch feature information includes feature values ​​of the corresponding second audio segment under multiple pitch features; the training module includes:

[0209] The first statistical unit is used to calculate the prior probability of the target object for each performance level based on the sample singing performance levels of the target object for multiple sample songs.

[0210] The second statistical unit is used to calculate the conditional probability of each performance level of the target object under the feature value of each pitch feature based on the feature values ​​of multiple second audio segments under multiple pitch features and the sample performance level of the target object for multiple sample song segments.

[0211] The model determination unit is used to determine the target singing rating model based on the prior probability of the target object for each singing performance level, and the conditional probability of each singing performance level under the feature value of each pitch feature.

[0212] In some embodiments, the pitch feature information of the target original audio segment includes the feature values ​​of the target original audio segment under multiple pitch features; the prediction module 1130 is used to: the target singing rating model processes the target object for the target song segment according to the following process to predict the target singing performance level of the target object for the target song segment: based on the prior probability of the target object for each singing performance level and the conditional probability of the feature values ​​of each singing performance level under multiple pitch features corresponding to the target original audio segment, the posterior probability is calculated to determine the target singing performance level of the target object for the target song segment as the posterior probability of each singing performance level; the singing performance level with the highest posterior probability is taken as the target singing performance level of the target object for the target song segment.

[0213] In some embodiments, the training module includes:

[0214] The prediction unit is used by the target singing rating model to predict the singing performance rating based on the second pitch feature information of each second audio segment, and outputs the predicted singing performance level of the target object for each sample song segment corresponding to each second audio segment.

[0215] The prediction loss calculation unit is used to calculate the prediction loss based on the predicted performance level of the target object for each sample song segment corresponding to each second audio segment, and the sample performance level of the target object for each sample song segment.

[0216] The parameter adjustment unit is used to adjust the parameters of the target singing rating model based on the prediction loss until the training termination condition is met.

[0217] In some embodiments, the first pitch feature information includes the feature values ​​of the first audio segment under multiple pitch features; the second pitch feature information includes the feature values ​​of the second audio segment under multiple pitch features;

[0218] The singing performance level determination module includes:

[0219] The feature value deviation calculation unit is used to determine the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under various pitch features based on the feature values ​​of the first audio segment under multiple pitch features and the feature values ​​of the second audio segment under multiple pitch features.

[0220] The sample singing performance level determination unit is used to determine the sample singing performance level of the target object for each sample song segment based on the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under multiple pitch features.

[0221] In some embodiments, the target volume ratio determination module 1140 includes:

[0222] The default volume ratio acquisition unit is used to obtain the default volume ratio.

[0223] The adjustment strategy determination unit is used to determine the target volume ratio adjustment strategy that matches the target singing performance level based on the target performance level of the target object for the segment of the target song.

[0224] The volume ratio adjustment unit is used to adjust the default volume ratio according to the target volume ratio adjustment strategy to obtain the target volume ratio suitable for the target object to sing the target song segment.

[0225] In some embodiments, the target volume ratio adjustment strategy is used to adjust the proportion of one of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio, or to adjust the proportion of at least two of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio.

[0226] In some embodiments, the adjustment strategy determination unit is configured to:

[0227] If the target performance level of the target object for the target song segment is higher than the first performance level, the target volume ratio adjustment strategy that is adapted to the target performance level is: to maintain the default volume ratio unchanged;

[0228] If the target performance level of the target object for the segment of the target song is not higher than the first performance level but higher than the second performance level, the target volume ratio adjustment strategy that matches the target performance level is: on the basis of the default volume ratio, maintain the proportion of the original vocal volume unchanged, and increase the proportion of the accompaniment volume while decreasing the proportion of the vocal recording volume.

[0229] If the target performance level of the target object for the segment of the target song is no higher than the second performance level but higher than the third performance level, the target volume ratio adjustment strategy that matches the target performance level is: on the basis of the default volume ratio, increase the proportion of the original vocal volume and decrease the proportion of the accompaniment volume and the proportion of the vocal recording volume.

[0230] If the target performance level of the target object for the segment of the target song is not higher than the third performance level, the target volume ratio adjustment strategy that matches the target performance level is: on the basis of the default volume ratio, keep the proportion of the vocal recording volume unchanged, reduce the proportion of the accompaniment volume and increase the proportion of the original vocal volume.

[0231] Among them, the first performance level > the second performance level > the third performance level, and the performance level is positively correlated with the performance effect.

[0232] In other embodiments, the target volume ratio determination module 1140 is used for:

[0233] Based on the correspondence between performance level and volume ratio, the volume ratio corresponding to the target performance level of the target subject for the segment of the target song is used as the target volume ratio applicable to the target subject singing the segment of the target song. In the correspondence between performance level and volume ratio, the higher the performance level, the lower the proportion of the original vocal volume in the corresponding volume ratio; performance level and performance effect are positively correlated.

[0234] In some embodiments, the pitch feature information includes feature values ​​of the target original vocal audio segment under multiple pitch features; the target original vocal audio segment includes multiple target audio frames; the extraction module 1120 includes:

[0235] The fundamental frequency detection unit is used to detect the fundamental frequency of each target audio frame in the target original audio segment and determine the fundamental frequency of each target audio frame.

[0236] The pitch conversion unit is used to convert the fundamental frequency of each target audio frame into pitch to obtain the pitch value of each target audio frame;

[0237] The feature value determination unit is used to determine the feature value of the target original audio segment under multiple pitch features based on the pitch values ​​of multiple target audio frames in the target original audio segment.

[0238] In some embodiments, the plurality of pitch features include at least two of the following: highest pitch value, lowest pitch value, maximum stable duration of pitch, pitch variance value, and duration of the preceding silence pause.

[0239] In some embodiments, the audio processing device further includes: an audio synthesis module, configured to synthesize the original vocal audio segment corresponding to the target song segment, the corresponding accompaniment audio segment, and the singing audio segment collected during the process of the target object singing the target song segment, according to the target volume ratio, to obtain the synthesized audio segment corresponding to the target song segment; and a playback module, configured to play the synthesized audio segment.

[0240] Figure 12 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 12 The computer system 1200 of the illustrated electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application. This electronic device can be used to execute the audio processing method provided in the application, and the electronic device can be a smartphone, tablet computer, smart TV, or other terminal.

[0241] like Figure 12As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0242] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0243] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0244] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0245] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0246] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0247] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0248] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal. It can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0249] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods of any of the above embodiments.

[0250] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0251] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0252] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0253] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that, include: Obtain the target original audio segment corresponding to the target song segment in the target song; Extract the pitch feature information of the target original audio segment; Based on the pitch feature information of the target original audio segment, the target singing performance level of the target object for the target song segment is predicted by using a target singing rating model applicable to the target object. Based on the target performance level of the target object for the target song segment, determine the target volume ratio applicable to the target object when singing the target song segment; The target volume ratio refers to the ratio between the recording volume, the original vocal volume, and the accompaniment volume. In response to the target object starting to sing the target song segment, the volume ratio is adjusted to the target volume ratio.

2. The method according to claim 1, characterized in that, Before predicting the target singing performance level of the target object for the target song segment based on the pitch feature information of the target original audio segment and using a target singing rating model suitable for the target object, the method further includes: Obtain sample singing audio of the target object singing multiple sample songs and sample original audio of each sample song; According to the multiple sample song segments of the sample song, the sample singing audio and the sample original audio of the sample song are segmented to obtain multiple first audio segments in the sample singing audio and multiple second audio segments in the sample original audio. Extract the first pitch feature information of each first audio segment and the second pitch feature information of each second audio segment; Based on the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment corresponding to the same sample song segment, the sample singing performance level of the target object for each of the sample song segments is determined; Based on the second pitch feature information of multiple second audio segments, and the sample singing performance level of the target object for each sample song segment, the target singing rating model is trained.

3. The method according to claim 2, characterized in that, The second pitch feature information includes the feature values ​​of the corresponding second audio segment under multiple pitch features; The step of training the target singing rating model based on the second pitch feature information of multiple second audio segments and the sample singing performance level of the target object for multiple sample song segments includes: Based on the sample singing performance levels of the target object for multiple sample song segments, the prior probability of the target object for each singing performance level is calculated. Based on the feature values ​​of multiple second audio segments under multiple pitch features and the sample singing performance level of the target object for multiple sample song segments, the conditional probability of each singing performance level of the target object under the feature value of each pitch feature is calculated. The target singing rating model is determined based on the prior probability of the target object for each singing performance level, and the conditional probability of each singing performance level under the feature value of each pitch feature.

4. The method according to claim 3, characterized in that, The pitch feature information of the target original audio segment includes the feature values ​​of the target original audio segment under multiple pitch features; The step of predicting the target singing performance level of the target object for the target song segment based on the pitch feature information of the target original audio segment and using a target singing rating model suitable for the target object includes: The target singing rating model processes data according to the following process to predict the target singing performance level of the target object for the target song segment: Based on the prior probability of the target object for each singing performance level and the conditional probability of the feature value of each singing performance level under the multiple pitch features corresponding to the target original audio segment, the posterior probability is calculated to determine that the target singing performance level of the target object for the target song segment is the posterior probability of each singing performance level. The performance level with the highest posterior probability is taken as the target performance level for the target object in the segment of the target song.

5. The method according to claim 2, characterized in that, The step of training the target singing rating model based on the second pitch feature information of multiple second audio segments and the sample singing performance level of the target object for multiple sample song segments includes: Based on the second pitch feature information of each second audio segment, the target singing performance rating model predicts the singing performance rating and outputs the predicted singing performance level of the target object for each sample song segment corresponding to each second audio segment. The prediction loss is calculated based on the predicted performance level of the target object for each sample song segment corresponding to the second audio segment, and the sample performance level of the target object for each sample song segment. Based on the predicted loss, adjust the parameters of the target singing rating model until the training termination condition is met.

6. The method according to claim 2, characterized in that, The first pitch feature information includes the feature values ​​of the first audio segment under multiple pitch features; the second pitch feature information includes the feature values ​​of the second audio segment under multiple pitch features; The step of determining the target object's sample singing performance level for each of the sample song segments based on the first pitch feature information of the first audio segment and the second pitch feature information of the second audio segment corresponding to the same sample song segment includes: Based on the feature values ​​of the first audio segment corresponding to the same sample song segment under multiple pitch features and the feature values ​​of the second audio segment under multiple pitch features, the feature value deviation of the first audio segment and the second audio segment corresponding to the same sample song segment under each pitch feature is determined; Based on the feature value deviation of the first and second audio segments corresponding to the same sample song segment under multiple pitch features, the sample singing performance level of the target object for each of the sample song segments is determined.

7. The method according to claim 1, characterized in that, The step of determining the target volume ratio applicable to the target object's performance of the target song segment based on the target object's target singing performance level for the target song segment includes: Get the default volume ratio; Based on the target performance level of the target object for the segment of the target song, determine a target volume ratio adjustment strategy that is compatible with the target performance level; According to the target volume ratio adjustment strategy, the default volume ratio is adjusted to obtain a target volume ratio suitable for the target object to sing the target song segment.

8. The method according to claim 7, characterized in that, The target volume ratio adjustment strategy is used to adjust the proportion of one of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio, or to adjust the proportion of at least two of the vocal recording volume, original vocal volume, and accompaniment volume in the volume ratio.

9. The method according to claim 7, characterized in that, The step of determining a target volume ratio adjustment strategy that matches the target singing performance level of the target object for the target song segment includes: If the target performance level of the target object for the target song segment is higher than the first performance level, the target volume ratio adjustment strategy adapted to the target performance level is: to maintain the default volume ratio unchanged; If the target performance level of the target object for the target song segment is not higher than the first performance level but higher than the second performance level, the target volume ratio adjustment strategy adapted to the target performance level is: based on the default volume ratio, maintain the proportion of the original vocal volume unchanged, and increase the proportion of the accompaniment volume while decreasing the proportion of the vocal recording volume. If the target performance level of the target object for the target song segment is not higher than the second performance level but higher than the third performance level, the target volume ratio adjustment strategy adapted to the target performance level is: based on the default volume ratio, increase the proportion of the original vocal volume and decrease the proportion of the accompaniment volume and the proportion of the vocal recording volume. If the target performance level of the target object for the target song segment is not higher than the third performance level, the target volume ratio adjustment strategy adapted to the target performance level is: based on the default volume ratio, maintain the proportion of the singing recording volume unchanged, reduce the proportion of the accompaniment volume and increase the proportion of the original vocal volume. Among them, the first performance level > the second performance level > the third performance level, and the performance level is positively correlated with the performance effect.

10. The method according to claim 1, characterized in that, The step of determining the target volume ratio applicable to the target object's performance of the target song segment based on the target object's target singing performance level for the target song segment includes: Based on the correspondence between singing performance level and volume ratio, the volume ratio corresponding to the target singing performance level of the target object for the target song segment is taken as the target volume ratio applicable to the target object when singing the target song segment. Among them, in the correspondence between singing performance level and volume ratio, the higher the singing performance level, the lower the proportion of the original singer's volume in the corresponding volume ratio; the singing performance level is positively correlated with the singing effect.

11. The method according to any one of claims 1 to 10, characterized in that, The pitch feature information includes the feature values ​​of the target original vocal audio segment under multiple pitch features; the target original vocal audio segment includes multiple target audio frames; The extraction of pitch feature information from the target original audio segment includes: The fundamental frequency of each target audio frame in the target original audio segment is detected to determine the fundamental frequency of each target audio frame; The fundamental frequency of each target audio frame is converted to a pitch value to obtain the pitch value of each target audio frame. Based on the pitch values ​​of multiple target audio frames in the target original audio segment, the feature values ​​of the target original audio segment under multiple pitch features are determined.

12. The method according to claim 11, characterized in that, The plurality of pitch features include at least two of the following: highest pitch value, lowest pitch value, maximum stable duration of pitch, pitch variance value, and duration of the preceding silence pause.

13. The method according to any one of claims 1 to 10, characterized in that, After adjusting the volume ratio to the target volume ratio in response to the target object starting to sing the target song segment, the method further includes: According to the target volume ratio, the original vocal audio segment corresponding to the target song segment, the corresponding accompaniment audio segment, and the vocal audio segment collected during the target object's singing of the target song segment are synthesized to obtain the synthesized audio segment corresponding to the target song segment. Play the synthesized audio segment.

14. An audio processing apparatus, characterized in that, include: The acquisition module is used to acquire the target original audio segment corresponding to the target song segment in the target song; The extraction module is used to extract the pitch feature information of the target original audio segment; The prediction module is used to predict the target singing performance level of the target object for the target song segment based on the pitch feature information of the target original audio segment and through a target singing rating model applicable to the target object. The target volume ratio determination module is used to determine the target volume ratio applicable to the target object when singing the target song segment based on the target object's target singing performance level for the target song segment; The target volume ratio refers to the ratio between the recording volume, the original vocal volume, and the accompaniment volume. The volume adjustment module is used to adjust the volume ratio to the target volume ratio in response to the target object starting to sing the target song segment.

15. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-13.

16. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, the method as described in any one of claims 1-13 is implemented.

17. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-13.

Images