Background music volume adjustment method, device, electronic device and storage medium
Through the combination of preset models and linear filters, the background music volume is separated and adjusted in real time, which solves the problem of poor separation of vocals and background music in the prior art, and achieves real-time performance of volume adjustment and user experience.
Patent Information
- Application Number
- CN202210911782.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-07-29
AI Technical Summary
When the prior art adjusts the volume of background music in real time, the separation effect of the human voice and background music is poor, resulting in the change of the human voice when adjusting the volume of background music, affecting the user experience.
By obtaining the currently played original audio, using a preset model to estimate the human voice to obtain the human voice audio, and using a linear filter to filter the original audio to separate the background music audio, and then the human voice audio and the human voice audio are based on the signal-to-noise ratio parameters and The background music audio is mixed to adjust the volume of the background music.
It improves the separation effect of vocals and background music, ensures real-time volume adjustment, and improves user experience.
Smart Images

Figure CN115277935B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio processing, and in particular to a method, device, electronic device and storage medium for adjusting the volume of background music. Background Art
[0002] With the popularity of audio applications in smart mobile terminals, users have more and more demands for audio effects. One demand is that when users listen to audio content with background music, they can adjust the volume of the background music in real time according to their own auditory perception. Therefore, how to adjust the volume in real time to improve the user experience is an issue worthy of attention in this field. Summary of the invention
[0003] In view of this, an object of the present invention is to provide a background music volume adjustment method, device, electronic device and storage medium.
[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:
[0005] In a first aspect, the present invention provides a method for adjusting the volume of background music, the method comprising:
[0006] Obtain the original audio currently being played, where the original audio includes human voice and background music;
[0007] Based on the volume adjustment operation of the user, a signal-to-noise ratio parameter is obtained, where the signal-to-noise ratio parameter represents an energy ratio between human voice and background music;
[0008] Using a preset model to estimate the human voice in the original audio to obtain an estimation result, and obtaining the human voice audio based on the estimation result;
[0009] Filtering the original audio based on the human voice audio using a linear filter to obtain background music audio;
[0010] The human voice audio and the background music audio are mixed according to the signal-to-noise ratio parameter to adjust the volume of the background music.
[0011] In an optional implementation, the step of estimating the human voice in the original audio using a preset model to obtain an estimation result, and obtaining the human voice audio based on the estimation result includes:
[0012] Dividing the original audio into a plurality of audio segments of the same length in chronological order;
[0013] Performing feature extraction on each of the audio clips to obtain a feature vector of each of the audio clips;
[0014] Inputting the feature vector of each audio segment into the preset model to obtain a target ideal amplitude masking parameter representing the estimation result; wherein the target ideal amplitude masking parameter represents a ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio;
[0015] The human voice audio is obtained based on the original audio and the target ideal amplitude masking parameter.
[0016] In an optional implementation, the step of extracting features from each of the audio segments to obtain a feature vector for each of the audio segments includes:
[0017] Using any audio segment from all audio segments as the audio segment to be processed;
[0018] Performing Fourier transform on the audio clip to be processed to obtain a full-band energy spectrum;
[0019] Dividing the full-band energy spectrum into sub-band energy spectra according to an equivalent rectangular bandwidth scale using a triangular filter bank;
[0020] Calculating cepstral coefficients based on the sub-band energy spectrum;
[0021] Calculating first-order differences and second-order differences of the cepstral coefficients;
[0022] Obtaining a feature vector of the audio segment to be processed based on the first-order difference, the second-order difference and the sub-band energy spectrum;
[0023] Each of the audio segments is traversed to obtain a feature vector of each of the audio segments.
[0024] In an optional implementation, the step of obtaining the human voice audio based on the original audio and the target ideal amplitude masking parameter includes:
[0025] Based on the original audio, obtaining an amplitude spectrum and a phase spectrum of the original audio;
[0026] Multiplying the target ideal amplitude masking parameter by the amplitude spectrum of the original audio to obtain the amplitude spectrum of the human voice audio;
[0027] The human voice audio is obtained by performing inverse Fourier transform based on the amplitude spectrum of the human voice audio and the phase spectrum of the original audio.
[0028] In an optional implementation manner, the preset model is obtained in the following manner:
[0029] According to a preset signal-to-noise ratio range, multiple human voice audio samples and multiple music audio samples are mixed to obtain multiple mixed audio samples;
[0030] Dividing each of the mixed audio samples into a plurality of audio segment samples of the same duration in time order to obtain all audio segment samples;
[0031] The basic model is trained based on all audio clip samples to obtain the preset model.
[0032] In an optional implementation, the step of training the basic model based on all audio clip samples to obtain the preset model includes:
[0033] Based on each of the audio segment samples, obtaining a real human voice amplitude spectrum of each of the audio segment samples;
[0034] Performing feature extraction on each of the audio clip samples to obtain a feature vector of each of the audio clip samples;
[0035] Inputting the feature vector of each audio clip sample into the basic model to obtain a predicted ideal amplitude mask corresponding to each audio clip sample;
[0036] Based on the predicted ideal amplitude mask corresponding to each of the audio segment samples, a predicted human voice amplitude spectrum of each of the audio segment samples is obtained;
[0037] Based on a preset loss function, a real human voice amplitude spectrum and a predicted human voice amplitude spectrum of each of the audio clip samples, the basic model is trained to obtain the preset model.
[0038] In a second aspect, the present invention provides a background music volume adjustment device, the device comprising:
[0039] An acquisition module is used to acquire the original audio currently being played, wherein the original audio includes human voice and background music;
[0040] Based on the volume adjustment operation of the user, a signal-to-noise ratio parameter is obtained, where the signal-to-noise ratio parameter represents an energy ratio between human voice and background music;
[0041] A processing module, configured to estimate the human voice in the original audio using a preset model to obtain an estimation result, and obtain the human voice audio based on the estimation result;
[0042] Filtering the original audio based on the human voice audio using a linear filter to obtain background music audio;
[0043] The adjustment module is used to mix the human voice audio and the background music audio according to the signal-to-noise ratio parameter to adjust the volume of the background music.
[0044] In an optional implementation manner, the processing module is specifically used to:
[0045] Dividing the original audio into a plurality of audio segments of the same length in chronological order;
[0046] Performing feature extraction on each of the audio clips to obtain a feature vector of each of the audio clips;
[0047] Inputting the feature vector of each audio segment into the preset model to obtain a target ideal amplitude masking parameter representing the estimation result; wherein the target ideal amplitude masking parameter represents a ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio;
[0048] The human voice audio is obtained based on the original audio and the target ideal amplitude masking parameter.
[0049] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in any one of the aforementioned embodiments is implemented.
[0050] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in any one of the aforementioned embodiments is implemented.
[0051] The background music volume adjustment method, device, electronic device and storage medium provided by the embodiment of the present invention obtain the original audio currently being played, which includes human voice and background music; and based on the volume adjustment operation of the user, obtain the signal-to-noise ratio parameter, which represents the energy ratio of human voice and background music; then use the preset model to estimate the human voice in the original audio to obtain the estimation result, and obtain the human voice audio based on the estimation result; then use the linear filter to filter the original audio based on the human voice audio to obtain the background music audio; finally, mix the human voice audio and the background music audio according to the signal-to-noise ratio parameter to adjust the volume of the background music. By adopting the model and the filter, the effect of separating the human voice and the background music is improved, while ensuring the real-time performance of the volume adjustment, and improving the user experience.
[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention is shown;
[0055] Figure 2 A schematic diagram showing a flow chart of a method for adjusting the volume of background music provided by an embodiment of the present invention;
[0056] Figure 3 Another schematic diagram of a process of adjusting the volume of background music provided by an embodiment of the present invention is shown;
[0057] Figure 4 An example diagram showing a method for adjusting the volume of background music provided by an embodiment of the present invention is shown;
[0058] Figure 5 Another example diagram of the background music volume adjustment method provided by an embodiment of the present invention is shown;
[0059] Figure 6 A functional module diagram of a background music volume adjustment device provided by an embodiment of the present invention is shown.
[0060] Icon: 110 - bus; 120 - processor; 130 - memory; 150 - I / O module; 170 - communication interface; 300 - background music volume adjustment device; 310 - acquisition module; 330 - processing module; 350 - adjustment module; 370 - training module. DETAILED DESCRIPTION
[0061] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.
[0063] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0064] With the popularity of audio applications in smart mobile terminals, users have more and more demands for audio effects. One demand is that when users listen to audio content with background music, they can adjust the volume of the background music in real time according to their own auditory perception. The prior art usually uses signal processing technology to achieve real-time volume adjustment. Although it meets the real-time requirement, the separation effect of human voice and background music is poor, and adjusting the volume of background music will cause the volume of human voice to change. In addition, an embodiment of the present invention provides a background music volume adjustment method to solve the above problem.
[0065] Please refer to Figure 1 , is a block diagram of an electronic device provided by an embodiment of the present invention. The electronic device includes a bus 110 , a processor 120 , a memory 130 , an I / O module 150 , and a communication interface 170 .
[0066] The bus 110 may be a circuit that connects the above elements to each other and transfers communications (eg, control messages) between the above elements.
[0067] The processor 120 may receive commands from the other elements described above (eg, the memory 130 , the I / O module 150 , the communication interface 170 , etc.) through the bus 110 , may interpret the received commands, and may perform calculations or data processing according to the interpreted commands.
[0068] The processor 120 may be an integrated circuit chip with signal processing capabilities. The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0069] The memory 130 may store commands or data received from the processor 120 or other elements (eg, the I / O module 150 , the communication interface 170 , etc.) or commands or data generated by the processor 120 or other elements.
[0070] The memory 130 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM).
[0071] The I / O module 150 may receive commands or data input from a user via input-output means (e.g., a sensor, a keyboard, a touch screen, etc.), and may transmit the received commands or data to the processor 120 or the memory 130 via the bus 110. It is also used to display various information (e.g., multimedia data, text data) received, stored, and processed from the above elements, and may display videos, images, data, etc. to the user.
[0072] The communication interface 170 may be used to communicate signaling or data with other node devices.
[0073] Understandably, Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device. The electronic device may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0074] The electronic device provided in the embodiment of the present invention may be a smart phone, a personal computer, a tablet computer, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present invention does not impose any limitation on this.
[0075] It is understandable that an audio application is installed in the electronic device, and during the operation of the audio application, the background music volume adjustment method provided by the embodiment of the present invention is executed.
[0076] The above-mentioned electronic device will be used as an execution subject to execute each step in each method provided in the embodiment of the present invention and achieve corresponding technical effects.
[0077] See also Figure 2 , Figure 2 The present invention is a flowchart of a method for adjusting the volume of background music provided by an embodiment of the present invention.
[0078] Step S202, obtaining the original audio currently being played, the original audio including human voice and background music;
[0079] In this embodiment, the user can listen to audio including human voice and background music through the audio application in the electronic device. While the user is listening to the audio, the audio currently played by the user can be obtained in real time, that is, the original audio.
[0080] Step S204, based on the volume adjustment operation of the user, a signal-to-noise ratio parameter is obtained, where the signal-to-noise ratio parameter represents the energy ratio between human voice and background music;
[0081] In this embodiment, when the user wants to adjust the volume of the background music in the original audio, the volume adjustment operation can be performed. For example, the volume adjustment operation can be performed by interactive keys, physical keys, external devices, etc.
[0082] Based on the user's volume adjustment operation, an expected volume value, i.e., the volume value that the user expects to achieve, is obtained. Based on the correspondence between the preset volume value and the signal-to-noise ratio, a signal-to-noise ratio parameter corresponding to the expected volume value can be obtained. The signal-to-noise ratio parameter represents the energy ratio of the human voice and the background music in the original audio.
[0083] Step S206, estimating the human voice in the original audio using a preset model to obtain an estimation result, and obtaining human voice audio based on the estimation result;
[0084] Among them, the preset model uses a lightweight RNN model, which is trained based on a large number of mixed audio samples and is used to evaluate the human voice in the mixed audio. The mixed audio can be understood as audio containing human voice and background music.
[0085] In this embodiment, the preset model is used to estimate the human voice in the original audio to obtain an estimation result, and the human voice audio is obtained based on the estimation result. It can be understood that the preset model obtains information of the human voice signal from the original audio, and then obtains the human voice audio based on the information. From the effect, the preset model can eliminate the information of the background music signal in the original audio, and then the preset model can also be understood as a background music elimination model.
[0086] Step S208, filtering the original audio based on the human voice audio using a linear filter to obtain background music audio;
[0087] In this embodiment, the linear filter can be an FIR filter. The original audio can be understood as the superimposed audio of the human voice audio and the background music audio. Based on the obtained human voice audio, the human voice audio is filtered by a linear filter to obtain the filtered human voice audio, and then the background music audio is obtained according to the filtered human voice audio and the original audio.
[0088] Optionally, a normalized least mean square algorithm may be used to adaptively update the linear filter to improve the separation effect of human voice and background music.
[0089] Step S210, mixing the human voice audio and the background music audio according to the signal-to-noise ratio parameter to adjust the volume of the background music;
[0090] In this embodiment, the human voice audio and the background music audio are mixed according to the signal-to-noise ratio parameter, and the mixed audio is the audio after the volume of the background music is adjusted. That is, the human voice audio is regarded as a signal and the background music is regarded as noise. The embodiment of the present invention adjusts the volume of the background music by adjusting the ratio of the human voice to the background music.
[0091] It can be understood that the embodiment of the present invention adopts a lightweight model and filter to extract the human voice audio and background music audio respectively, thereby improving the separation effect of the human voice and background music. At the same time, due to the small amount of calculation, the real-time requirement of volume adjustment can be met.
[0092] It can be seen that based on the above steps, by obtaining the original audio currently being played, the original audio includes human voice and background music; and based on the user's volume adjustment operation, the signal-to-noise ratio parameter is obtained, and the signal-to-noise ratio parameter represents the energy ratio of human voice and background music; then the human voice in the original audio is estimated using the preset model to obtain an estimation result, and the human voice audio is obtained based on the estimation result; then the original audio is filtered based on the human voice audio using a linear filter to obtain background music audio; finally, the human voice audio and background music audio are mixed according to the signal-to-noise ratio parameter to adjust the volume of the background music. The use of models and filters improves the effect of separating human voice and background music, while ensuring the real-time performance of volume adjustment.
[0093] Optionally, for the above step S206, the embodiment of the present invention provides a possible implementation method, please refer to Figure 3 , wherein step S206 includes the following steps:
[0094] Step S206-1, dividing the original audio into a plurality of audio segments of the same length in chronological order;
[0095] In this embodiment, the original audio is framed according to the set frame length parameter and frame shift parameter, that is, the original audio is divided into a plurality of continuous audio segments of the same length.
[0096] For example, the original audio can be divided into frames according to the frame length parameter of 20ms and the frame shift parameter of 10ms to obtain audio segments of 20ms each. The frame shift parameter can be understood as the overlap amount between the tail of the previous audio segment and the head of the next audio segment in two adjacent audio segments.
[0097] It should be understood that the frame length parameter and the frame shift parameter may be set according to actual application, and the embodiment of the present invention does not limit this.
[0098] Step S206-3, extracting features from each audio clip to obtain a feature vector for each audio clip;
[0099] In this embodiment, based on each obtained audio segment, feature extraction is performed on each audio segment to obtain a feature vector of each audio segment, so that a subsequent model can extract information of a human voice signal through the feature vector.
[0100] Step S206-5, inputting the feature vector of each audio clip into a preset model to obtain a target ideal amplitude masking parameter representing the estimation result; wherein the target ideal amplitude masking parameter represents the ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio;
[0101] Among them, the ideal amplitude mask, namely Ideal Amplitude Mask, can be referred to as IAM, which is used to characterize the amplitude ratio of the pure vocal signal and the mixed audio signal.
[0102] In this embodiment, based on the feature vectors of all the audio clips obtained, the feature vector of each audio clip is input into a preset model, and the preset model is used to estimate the human voice in the original audio to obtain a target ideal amplitude masking parameter, which refers to the ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio. It can be understood that the preset model estimates the amplitude information of the human voice signal in the original audio.
[0103] Step S206-7, obtaining human voice audio based on the original audio and the target ideal amplitude masking parameter;
[0104] In this embodiment, based on the obtained target ideal amplitude masking parameter and combined with the original audio, the audio of the human voice in the original audio can be obtained.
[0105] It can be seen that the original audio is divided into multiple continuous audio segments and feature vectors are extracted, and the feature vectors of each audio segment are input into the preset model to obtain the ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio. The amplitude information of the human voice signal is extracted from the original audio through the model and a pure human voice is obtained based on the amplitude information, thereby improving the separation effect and facilitating the subsequent acquisition of background music audio based on the human voice audio.
[0106] Optionally, for the above step S206-3, an embodiment of the present invention provides a possible implementation method.
[0107] Step S206-3-1, taking any audio segment from all audio segments as the audio segment to be processed;
[0108] It can be understood that, for each audio segment, the feature vector of each audio segment is obtained according to step S206-3-3 to step S206-3-11. The following steps will be described by taking any audio segment from all the audio segments as the audio segment to be processed.
[0109] Step S206-3-3, performing Fourier transform on the audio clip to be processed to obtain a full-band energy spectrum;
[0110] Step S206-3-5, using a triangular filter bank to divide the full-band energy spectrum into sub-band energy spectra according to an equivalent rectangular bandwidth scale;
[0111] Among them, Equivalent Rectangular Bandwidth, or ERB for short, can be understood as a measurement method that simulates human hearing. Through this measurement method, the linear frequency band is divided into several ERB frequency bands, i.e., sub-bands; the number of ERB frequency bands is a configurable parameter. For example, for 48kHz audio, it can be set to 32. The triangular filter bank is dense at low frequencies and sparse at high frequencies, which just conforms to the rule that the higher the frequency, the less sensitive the human ear is.
[0112] In this embodiment, the audio clip to be processed is subjected to Fourier transform to obtain a full-band energy spectrum; then a triangular filter bank is used to divide the full-band energy spectrum into sub-band energy spectra according to an equivalent rectangular bandwidth scale.
[0113] It can be understood that by using equivalent rectangular bandwidth and triangular filter banks, the auditory perception of the human ear can be better matched, thereby optimizing the volume adjustment effect and improving the user's auditory experience. In addition, the number of ERB bands is smaller, which can effectively reduce the computational complexity of the model and ensure the real-time performance of volume adjustment.
[0114] Step S206-3-7, calculating cepstral coefficients based on the sub-band energy spectrum;
[0115] Step S206-3-9, calculating the first-order difference and the second-order difference of the cepstral coefficients;
[0116] Step S206-3-11, obtaining a feature vector of the audio segment to be processed based on the first-order difference, the second-order difference and the sub-band energy spectrum;
[0117] In this embodiment, for the obtained sub-band energy spectrum, the base-10 logarithm of the sub-band energy spectrum is calculated, and the cepstral coefficients corresponding to the sub-band energy spectrum are obtained after discrete cosine transform; then the first-order difference and second-order difference of the cepstral coefficients are calculated, and combined with the sub-band energy spectrum, the feature vector of the audio clip to be processed is obtained.
[0118] Step S206-3-13, traverse each audio clip to obtain a feature vector of each audio clip;
[0119] It can be understood that, for each audio segment, the feature vector of each audio segment can be obtained by following the above steps S206-3-3 to S206-3-11.
[0120] Optionally, for the above step S206-7, an embodiment of the present invention provides a possible implementation method.
[0121] Step S206-7-1, based on the original audio, obtaining the amplitude spectrum and phase spectrum of the original audio;
[0122] Step S206-7-3, multiplying the target ideal amplitude masking parameter by the amplitude spectrum of the original audio to obtain the amplitude spectrum of the human voice audio;
[0123] Step S206-7-5, obtaining the human voice audio through inverse Fourier transform based on the amplitude spectrum of the human voice audio and the phase spectrum of the original audio.
[0124] In this embodiment, the original audio signal may be processed to obtain the amplitude spectrum and phase spectrum of the original audio signal; then the target ideal amplitude masking parameter is multiplied by the amplitude spectrum of the original audio signal to obtain the amplitude spectrum of the human voice signal.
[0125] The phase spectrum of the original audio signal can be regarded as the phase spectrum of the human voice signal, that is, the human voice audio can be obtained by inverse Fourier transform based on the amplitude spectrum of the human voice signal and the phase spectrum of the original audio signal.
[0126] Optionally, for the preset model in the above embodiment, the embodiment of the present invention provides a possible implementation method for obtaining the preset model, please refer to Figure 4 .
[0127] Step S212, mixing the plurality of human voice audio samples and the plurality of music audio samples according to a preset signal-to-noise ratio interval to obtain a plurality of mixed audio samples;
[0128] In this embodiment, the preset signal-to-noise ratio interval can be 5dB to 25dB, or can be set according to actual applications, which is not limited in the embodiment of the present invention. According to the signal-to-noise ratio parameter within the interval, multiple pure human voice audio samples and multiple music audio samples are mixed to obtain multiple mixed audio samples.
[0129] Step S214, dividing each mixed audio sample into a plurality of audio segment samples of the same duration in time order, to obtain all audio segment samples;
[0130] In this embodiment, for each mixed audio sample, the mixed audio sample is framed, that is, it is divided into multiple continuous audio segment samples of the same length, and the audio segment samples divided by each mixed audio sample are obtained, that is, all audio segment samples are obtained.
[0131] Step S216, training the basic model based on all audio clip samples to obtain a preset model;
[0132] In this embodiment, the basic model is trained based on all audio clip samples, and the preset model is obtained when the training iteration reaches a preset number of times or the value of the loss function reaches a preset value.
[0133] Optionally, for the above step S216, the embodiment of the present invention provides a possible implementation method, please refer to Figure 5 , wherein step S216 includes the following steps:
[0134] Step S216-1, based on each audio segment sample, obtaining a real human voice amplitude spectrum of each audio segment sample;
[0135] In this embodiment, for each audio segment sample, there is a corresponding human voice audio sample, and the corresponding human voice segment can be obtained from the corresponding human voice audio sample, and the real human voice amplitude spectrum of the audio segment sample can be obtained based on the human voice segment. Based on each audio segment sample, the real human voice amplitude spectrum of each audio segment sample is obtained. The real human voice amplitude spectrum can be used Represented by , where u represents the index of the audio slice sample.
[0136] Step S216-3, extracting features from each audio clip sample to obtain a feature vector for each audio clip sample;
[0137] It is understandable that the manner of extracting the feature vector for each audio segment sample is similar. For the sake of simplicity, the following is an explanation of the implementation manner of extracting the feature vector of an audio segment sample.
[0138] In this embodiment, the audio clip sample is subjected to Fourier transform to obtain a full-band energy spectrum such as Y(t,f), where t represents the time index and f represents the frequency index; then, after being processed by a triangular filter bank according to an equivalent rectangular bandwidth scale, a sub-band energy spectrum such as Y(t,b) is obtained, where b represents the frequency band index; then, the logarithm of the sub-band energy spectrum with ten as the lower number is calculated, and after discrete cosine transform, the cepstrum coefficients corresponding to the sub-band energy spectrum are obtained; finally, the first-order difference and the second-order difference of the cepstrum coefficients are calculated, and the feature vector of the audio clip sample such as V is obtained in combination with the sub-band energy spectrum. u (t).
[0139] For each audio segment sample, a feature vector of each audio segment sample is obtained in the above-mentioned manner.
[0140] Step S216-5, inputting the feature vector of each audio clip sample into the basic model to obtain the predicted ideal amplitude mask corresponding to each audio clip sample;
[0141] Step S216-7, obtaining a predicted human voice amplitude spectrum of each audio segment sample based on the predicted ideal amplitude mask corresponding to each audio segment sample;
[0142] In this embodiment, the basic model includes three layers of GRU (Gated Recurrent Unit) and one fully connected layer.
[0143] The feature vector of each audio clip sample is input into the basic model, and the basic model is used to predict the ideal amplitude mask of each audio clip sample, that is, the predicted ideal amplitude mask corresponding to each audio clip sample is obtained. The predicted ideal amplitude mask can be used u (t,f) represents.
[0144] For each audio clip sample, obtain the amplitude spectrum of the audio clip sample and multiply it with the corresponding predicted ideal amplitude mask to obtain the predicted human voice amplitude spectrum of the audio clip sample. Based on each audio clip sample, obtain the predicted human voice amplitude spectrum of each audio clip sample. The predicted human voice amplitude spectrum can be used S u (t,f) indicates that S u (t,f)=O u (t,f)|Y u (t,f)|.
[0145] Step S216-9, based on the preset loss function, the real human voice amplitude spectrum and the predicted human voice amplitude spectrum of each audio clip sample, the basic model is trained to obtain the preset model.
[0146] In this embodiment, the minimum mean square error can be used as the loss function, which can be expressed as Where α is a parameter and α∈(0,1], optionally α=0.5.
[0147] Based on the preset loss function, the real vocal amplitude spectrum and the predicted vocal amplitude spectrum of each audio clip sample, the basic model is trained. When the training iteration reaches a preset number of times or the value of the loss function reaches a preset value, the preset model is obtained.
[0148] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a background music volume adjustment device is provided below. Figure 6 , Figure 6 This is a functional module diagram of a background music volume adjustment device 300 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the background music volume adjustment device 300 provided in this embodiment are the same as those of the above-mentioned embodiments. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned embodiments. The background music volume adjustment device 300 includes:
[0149] The acquisition module 310 is used to acquire the original audio currently being played, the original audio including human voice and background music; based on the volume adjustment operation of the user, obtain a signal-to-noise ratio parameter, the signal-to-noise ratio parameter represents the energy ratio of the human voice and the background music;
[0150] The processing module 330 is used to estimate the human voice in the original audio using a preset model to obtain an estimation result, and obtain the human voice audio based on the estimation result; and filter the original audio based on the human voice audio using a linear filter to obtain background music audio;
[0151] The adjustment module 350 is used to mix the human voice audio and the background music audio according to the signal-to-noise ratio parameter to adjust the volume of the background music.
[0152] Optionally, the processing module 330 is specifically used to: divide the original audio into multiple audio segments of the same length in chronological order; perform feature extraction on each audio segment to obtain a feature vector for each audio segment; input the feature vector of each audio segment into a preset model to obtain a target ideal amplitude masking parameter that characterizes the estimation result; wherein the target ideal amplitude masking parameter represents the ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio; and obtain the human voice audio based on the original audio and the target ideal amplitude masking parameter.
[0153] Optionally, the processing module 330 is specifically used to: take any one audio segment among all the audio segments as the audio segment to be processed; perform Fourier transform on the audio segment to be processed to obtain a full-band energy spectrum; use a triangular filter group to divide the full-band energy spectrum into sub-band energy spectra according to an equivalent rectangular bandwidth scale; calculate cepstral coefficients based on the sub-band energy spectrum; calculate the first-order difference and second-order difference of the cepstral coefficients; obtain a feature vector of the audio segment to be processed based on the first-order difference, the second-order difference and the sub-band energy spectrum; traverse each audio segment to obtain a feature vector of each audio segment.
[0154] Optionally, the processing module 330 is specifically used to: obtain the amplitude spectrum and phase spectrum of the original audio based on the original audio; multiply the target ideal amplitude masking parameter by the amplitude spectrum of the original audio to obtain the amplitude spectrum of the human voice audio; and obtain the human voice audio based on the amplitude spectrum of the human voice audio and the phase spectrum of the original audio through inverse Fourier transform.
[0155] Optionally, the background music volume adjustment device 300 also includes a training module 370, which is used to: mix multiple human voice audio samples and multiple music audio samples according to a preset signal-to-noise ratio range to obtain multiple mixed audio samples; divide each mixed audio sample into multiple audio segment samples of the same length in chronological order to obtain all audio segment samples; train the basic model based on all audio segment samples to obtain a preset model.
[0156] Optionally, the training module 370 is specifically used to: obtain the real human voice amplitude spectrum of each audio clip sample based on each audio clip sample; perform feature extraction on each audio clip sample to obtain the feature vector of each audio clip sample; input the feature vector of each audio clip sample into the basic model to obtain the predicted ideal amplitude mask corresponding to each audio clip sample; obtain the predicted human voice amplitude spectrum of each audio clip sample based on the predicted ideal amplitude mask corresponding to each audio clip sample; and train the basic model based on the preset loss function, the real human voice amplitude spectrum of each audio clip sample and the predicted human voice amplitude spectrum to obtain the preset model.
[0157] An embodiment of the present invention further provides an electronic device, including a processor 120 and a memory 130. The memory 130 stores a computer program. When the processor executes the computer program, the background music volume adjustment method disclosed in the above embodiment is implemented.
[0158] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 120, the background music volume adjustment method disclosed in the embodiment of the present invention is implemented.
[0159] In summary, the background music volume adjustment method, device, electronic device and storage medium provided by the embodiments of the present invention. The original audio currently being played is obtained, and the original audio includes human voice and background music; and based on the user's volume adjustment operation, a signal-to-noise ratio parameter is obtained, and the signal-to-noise ratio parameter represents the energy ratio of human voice and background music; then the human voice in the original audio is estimated using a preset model to obtain an estimation result, and human voice audio is obtained based on the estimation result; then a linear filter is used to filter the original audio based on the human voice audio to obtain background music audio; finally, the human voice audio and background music audio are mixed according to the signal-to-noise ratio parameter to adjust the volume of the background music. The use of models and filters improves the effect of separating human voice and background music, while ensuring the real-time performance of volume adjustment and improving the user experience.
[0160] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0161] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0162] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0163] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for adjusting the volume of background music, characterized in that: The method comprises: Obtain the original audio currently being played, where the original audio includes human voice and background music; Based on the volume adjustment operation of the user, a signal-to-noise ratio parameter is obtained, where the signal-to-noise ratio parameter represents an energy ratio between human voice and background music; Using a preset model to estimate the human voice in the original audio to obtain an estimation result, and obtaining the human voice audio based on the estimation result; Filtering the original audio based on the human voice audio using a linear filter to obtain background music audio; Mixing the human voice audio and the background music audio according to the signal-to-noise ratio parameter to adjust the volume of the background music; The step of estimating the human voice in the original audio using a preset model to obtain an estimation result, and obtaining the human voice audio based on the estimation result includes: Dividing the original audio into a plurality of audio segments of the same length in chronological order; Performing feature extraction on each of the audio clips to obtain a feature vector of each of the audio clips; Inputting the feature vector of each audio segment into the preset model to obtain a target ideal amplitude masking parameter representing the estimation result; wherein the target ideal amplitude masking parameter represents a ratio of the amplitude spectrum of the human voice audio to the amplitude spectrum of the original audio; Based on the original audio and the target ideal amplitude masking parameter, obtaining the human voice audio; The step of extracting features from each of the audio clips to obtain a feature vector for each of the audio clips comprises: Using any audio segment from all audio segments as the audio segment to be processed; Performing Fourier transform on the audio clip to be processed to obtain a full-band energy spectrum; Dividing the full-band energy spectrum into sub-band energy spectra according to an equivalent rectangular bandwidth scale using a triangular filter bank; Calculating cepstral coefficients based on the sub-band energy spectrum; Calculating first-order differences and second-order differences of the cepstral coefficients; Obtaining a feature vector of the audio segment to be processed based on the first-order difference, the second-order difference and the sub-band energy spectrum; Each of the audio segments is traversed to obtain a feature vector of each of the audio segments.
2. The method according to claim 1, characterized in that The step of obtaining the human voice audio based on the original audio and the target ideal amplitude masking parameter comprises: Based on the original audio, obtaining an amplitude spectrum and a phase spectrum of the original audio; Multiplying the target ideal amplitude masking parameter by the amplitude spectrum of the original audio to obtain the amplitude spectrum of the human voice audio; The human voice audio is obtained by performing inverse Fourier transform based on the amplitude spectrum of the human voice audio and the phase spectrum of the original audio.
3. The method according to claim 1, characterized in that: The preset model is obtained in the following manner: According to a preset signal-to-noise ratio range, multiple human voice audio samples and multiple music audio samples are mixed to obtain multiple mixed audio samples; Dividing each of the mixed audio samples into a plurality of audio segment samples of the same duration in time order to obtain all audio segment samples; The basic model is trained based on all audio clip samples to obtain the preset model.
4. The method according to claim 3, characterized in that The step of training the basic model based on all audio clip samples to obtain the preset model includes: Based on each of the audio segment samples, obtaining a real human voice amplitude spectrum of each of the audio segment samples; Performing feature extraction on each of the audio clip samples to obtain a feature vector of each of the audio clip samples; Inputting the feature vector of each audio clip sample into the basic model to obtain a predicted ideal amplitude mask corresponding to each audio clip sample; Based on the predicted ideal amplitude mask corresponding to each of the audio segment samples, a predicted human voice amplitude spectrum of each of the audio segment samples is obtained; Based on a preset loss function, a real human voice amplitude spectrum and a predicted human voice amplitude spectrum of each of the audio clip samples, the basic model is trained to obtain the preset model.
5. A background music volume adjustment device, characterized in that: The device comprises: An acquisition module is used to acquire the original audio currently being played, wherein the original audio includes human voice and background music; Based on the volume adjustment operation of the user, a signal-to-noise ratio parameter is obtained, where the signal-to-noise ratio parameter represents an energy ratio between human voice and background music; A processing module, configured to estimate the human voice in the original audio using a preset model to obtain an estimation result, and obtain the human voice audio based on the estimation result; Filtering the original audio based on the human voice audio using a linear filter to obtain background music audio; An adjustment module, used for mixing the human voice audio and the background music audio according to the signal-to-noise ratio parameter to adjust the volume of the background music; The processing module is specifically used to: divide the original audio into multiple audio segments of the same length in time sequence; perform feature extraction on each of the audio segments to obtain a feature vector of each of the audio segments; input the feature vector of each of the audio segments into the preset model to obtain a target ideal amplitude masking parameter that characterizes the estimation result; wherein the target ideal amplitude masking parameter represents a ratio of an amplitude spectrum of the human voice audio to an amplitude spectrum of the original audio; and obtain the human voice audio based on the original audio and the target ideal amplitude masking parameter; The processing module is specifically used to: take any one audio segment from all audio segments as the audio segment to be processed; perform Fourier transform on the audio segment to be processed to obtain a full-band energy spectrum; use a triangular filter group to divide the full-band energy spectrum into sub-band energy spectra according to an equivalent rectangular bandwidth scale; calculate cepstral coefficients based on the sub-band energy spectra; calculate the first-order difference and second-order difference of the cepstral coefficients; obtain a feature vector of the audio segment to be processed based on the first-order difference, the second-order difference and the sub-band energy spectrum; traverse each of the audio segments to obtain the feature vector of each of the audio segments.
6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.
7. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.
