Adaptive scene speech enhancement method and device, equipment and storage medium
By extracting audio features and using the scene recognition model to adjust the audio signal output ratio, the shortcomings of automatic voice gain control in different scenarios in remote teaching are solved, adaptive voice enhancement is achieved, and teaching experience is improved.
Patent Information
- Application Number
- CN202411288148.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-03
AI Technical Summary
In existing remote teaching, the voice automatic gain control technology is difficult to adapt to the needs of different teaching scenarios, especially in classroom and intercourse scenarios, where the volume cannot be effectively adjusted, affecting the teaching experience.
By obtaining the audio signal, extracting audio features and inputting it to the pre-trained scene recognition model, determining the current scene, adjusting the output ratio of the audio signal according to the scene confidence, and achieving adaptive voice enhancement.
It realizes adaptive voice enhancement in different teaching scenarios, adapts to the needs of the actual teaching environment, and improves the flexibility of teaching experience and volume adjustment.
Smart Images

Figure CN119068892A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of artificial intelligence, and more specifically, to a method, device, equipment and storage medium for adaptive scene speech enhancement. Background Art
[0002] With the continuous advancement of science and technology, more and more teaching is conducted through distance learning. Distance learning has the advantages of not being restricted by geographical location and having high flexibility in participation.
[0003] In the scenario of remote teaching, in order to ensure that the teacher's lectures or students' speeches can be clearly collected and the volume is kept within a reasonable range, the voice automatic gain control (AGC) technology is generally used to adjust the voice audio data to improve the user experience and reduce the user's manual volume adjustment.
[0004] However, in the actual teaching environment, there are various different links and scenarios, and different scenarios have different requirements for automatic voice gain. For example, during the break, when students are noisy, there is no need to turn on automatic voice gain control.
[0005] Therefore, there is an urgent need for a speech enhancement method that can adapt to different scenarios to meet the requirements of actual teaching scenarios. Summary of the invention
[0006] In order to overcome at least one of the defects of the prior art described above, the present invention provides a method, device, equipment and storage medium for adaptive scene speech enhancement. The technical solution adopted by the present invention is as follows.
[0007] In a first aspect, the present invention provides a method for adaptive scene speech enhancement, comprising:
[0008] Get audio signal;
[0009] Processing the audio signal to extract audio features of each audio frame after the audio signal is framed;
[0010] Inputting the audio features of the audio frames into a pre-trained scene recognition model to obtain the scene to which each audio frame belongs; wherein the scene recognition model is a scene recognition model that is obtained by training a pre-established machine learning model based on training data and is capable of obtaining a corresponding scene based on the audio features;
[0011] A first audio signal and a second audio signal are received, and a corresponding method is selected to control an output ratio of the first audio signal and the second audio signal according to scenes to which each audio frame belongs, wherein the second audio signal is an audio signal after the first audio signal is gain-processed.
[0012] In one implementation, the audio features include: time domain features and frequency domain features.
[0013] In one implementation, the audio features include: Mel-frequency cepstral coefficients, Mel-feature differential coefficients, short-time energy, zero-crossing rate, short-time energy mean, and pitch period.
[0014] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0015] The process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs includes:
[0016] Obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model;
[0017] According to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained;
[0018] In the time corresponding to each audio frame, the output size of the second audio signal is adjusted according to the confidence of the classroom scene of each audio frame, and the output size of the first audio signal is adjusted according to the confidence of the break scene of each audio frame.
[0019] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0020] The process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs includes:
[0021] Obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model;
[0022] According to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained;
[0023] reducing the output volume of the first audio signal to form a new first audio signal;
[0024] In the time corresponding to each audio frame, the output size of the second audio signal is adjusted according to the confidence of the classroom scene of each audio frame, and the output size of the new first audio signal is adjusted according to the confidence of the break scene of each audio frame.
[0025] In one implementation, the process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs includes:
[0026] According to the scene to which each audio frame belongs, the current scene is obtained;
[0027] The corresponding speech enhancement mode is selected according to the current scene to control the output ratio of the first audio signal and the second audio signal.
[0028] In one implementation, the process of obtaining the current scene according to the scene to which each audio frame belongs includes:
[0029] According to the scene to which each audio frame belongs, the number of audio frames determined to be in each scene within a period of N consecutive frames is counted;
[0030] When the number of audio frames of a scene is greater than a preset number threshold, the scene with the number of audio frames greater than the preset number threshold is determined as the current scene.
[0031] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0032] The process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal includes:
[0033] When the current scene is a classroom scene, outputting a second audio signal;
[0034] When the current scene is a class break scene, a first audio signal is output.
[0035] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0036] The process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal includes:
[0037] When the current scene is a classroom scene, outputting a second audio signal;
[0038] When the current scene is a recess scene, the output volume of the first audio signal is reduced to form a new first audio signal, and the new first audio signal is output.
[0039] In one implementation, the process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal further includes:
[0040] When switching from the break scene to the classroom scene, gradually reducing the weight of the first audio signal and gradually increasing the weight of the second audio signal;
[0041] When switching from the classroom scene to the break scene, the proportion of the second audio signal is gradually reduced, and the proportion of the first audio signal is gradually increased.
[0042] In a second aspect, the present invention provides a device for adaptive scene speech enhancement, comprising:
[0043] An acquisition module, used for acquiring an audio signal;
[0044] An extraction module, used to process the audio signal and extract audio features of each audio frame after the audio signal is framed;
[0045] A recognition module, used to input the audio features of the audio frame into a pre-trained scene recognition model to obtain the scene to which each audio frame belongs; wherein the scene recognition model is a scene recognition model that is obtained by training a pre-established machine learning model based on training data and is capable of obtaining a corresponding scene based on the audio features;
[0046] The output module is used to receive a first audio signal and a second audio signal, and to select a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs, wherein the second audio signal is an audio signal after the first audio signal is gain-processed.
[0047] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods when executing the program.
[0048] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method of any of the above-mentioned embodiments.
[0049] In the present invention, the audio features in the audio signal are extracted, and then the scene of the audio signal is determined based on the audio features using a pre-trained machine learning model, and finally the output ratio between the original input audio and the audio after gain processing by AGC is adjusted according to the scene. The present invention fully considers the needs of speech enhancement in different scenes in the actual teaching environment, can adapt to speech enhancement in different scenes, and adapts to the requirements of actual teaching scenes for automatic gain control. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a schematic diagram of the overall process of an implementation method of Example 1 of the present invention.
[0051] Figure 2 It is a schematic diagram of an audio feature vector of an implementation method of the first embodiment of the present invention.
[0052] Figure 3 It is a schematic diagram of the overall structure of an implementation method of the second embodiment of the present invention. DETAILED DESCRIPTION
[0053] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0054] It should be noted that the terms "first\second\..." involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\..." can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\..." can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0055] Embodiment 1
[0056] See also Figure 1 , Figure 1 1 is a flow chart of a method for adaptive scene speech enhancement provided in Embodiment 1 of the present invention, the method comprising step S110, step S120, step S130 and step S140. It should be noted that step S110, step S120, step S130 and step S140 are merely reference numerals for the purpose of clearly explaining the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.
[0057] Step S110, obtaining an audio signal;
[0058] Step S120, processing the audio signal to extract audio features of each audio frame after the audio signal is framed;
[0059] Step S130, inputting the audio features of the audio frames into a pre-trained scene recognition model to obtain the scene to which each audio frame belongs; wherein the scene recognition model is a scene recognition model that is obtained by training a pre-established machine learning model based on training data and is capable of obtaining a corresponding scene based on the audio features;
[0060] Step S140, receiving a first audio signal and a second audio signal, and selecting a corresponding method to control an output ratio of the first audio signal and the second audio signal according to scenes to which each audio frame belongs, wherein the second audio signal is an audio signal after the first audio signal is gain-processed.
[0061] The method is generally used in a classroom, so the method is described by taking a classroom scene as an example. In the classroom, audio signal data of the usage scene is obtained through a digital microphone, and in step S110, the audio signal is obtained.
[0062] It should be noted here that the audio signal acquired in step S110 may be an audio signal that has been preprocessed such as noise reduction, normalization, pre-emphasis, etc., or may be an audio signal that has not been preprocessed, which is not limited in this method.
[0063] The process of step S120 is mainly to obtain the audio features of the audio signal. Generally speaking, the first step is to preprocess the audio signal (if the audio signal obtained in step S110 is not preprocessed), which may include steps such as denoising, normalization, and pre-emphasis. The second step is framing, which divides the continuous audio signal into shorter frames. This is because the audio signal is usually non-stationary, that is, its characteristics may remain relatively stable in a short time, but will change in a longer time. Framing allows us to analyze the local characteristics of the signal. The frame length of the framing is usually between 20ms and 100ms, and the frame shift (or frame step length) is usually half of the frame length or less. For example, if the frame length is 25ms (about 400 samples at a sampling rate of 16kHz), the frame shift may be 10ms or 200 samples. In this method, preferably, the frame length is 40ms and the frame shift is 20ms. The third step is windowing. To reduce discontinuities at frame boundaries, each frame is usually multiplied by a window function, such as a Hamming window or a Hanning window. The window function helps smooth the start and end of the frame, reduce boundary effects, and reduce spectral leakage. The fourth step is feature extraction. Various algorithms are applied to each frame to extract audio features. Common audio features include: Mel frequency cepstral coefficients (MFCCs), spectral features obtained by short-time Fourier transform (STFT), short-time energy, zero crossing rate, short-time energy mean, pitch period, etc.
[0064] It should be noted that there are many audio features, and only those audio features needed are extracted in step S120. As for which audio features are needed, those skilled in the art can reasonably select corresponding audio features according to actual needs.
[0065] In one implementation, the audio features include: time domain features and frequency domain features.
[0066] Time domain features are features extracted directly from the original audio signal on the time axis without involving frequency conversion. The following are some common time domain audio features:
[0067] Zero Crossing Rate (ZCR): The number of times a signal crosses zero, reflecting the degree of signal fluctuation.
[0068] Energy: The total energy of a signal, usually used to measure the strength of the signal.
[0069] Amplitude Envelope: The change in signal amplitude can be used to analyze the dynamic range of the signal.
[0070] Peak Detection: The local maximum value of a signal can be used to analyze the instantaneous characteristics of the signal.
[0071] Zero Crossing Duration: The time that a signal lasts near the zero point, which can be used to analyze the stability of the signal.
[0072] Time Difference: The time difference between consecutive samples reflects the speed of change of the signal.
[0073] Frequency domain features are extracted by converting time domain signals to frequency domain, usually using Fourier transform (such as fast Fourier transform FFT). The following are some common frequency domain audio features:
[0074] Frequency distribution: The distribution of a signal at different frequencies can be used to analyze the spectral characteristics of the signal.
[0075] Frequency Center (Spectral Centroid): The center frequency of the signal spectrum, reflecting the main frequency components of the signal.
[0076] Band Energy: The energy of a signal within a specific frequency band, which can be used to analyze the frequency band characteristics of the signal.
[0077] Spectral Flatness: The flatness of the signal spectrum, which can be used to distinguish different types of audio signals.
[0078] Spectral Contrast: The difference in frequency distribution in different frequency ranges can be used to analyze the richness of the signal.
[0079] Mel-Frequency Cepstral Coefficients (MFCCs): Frequency domain features based on the auditory characteristics of the human ear, widely used in speech recognition.
[0080] The combination of these features in this embodiment provides a comprehensive description, including both time domain features and frequency domain features, and can more accurately reflect the multi-dimensional characteristics of the speech signal.
[0081] In one implementation, the audio features include: Mel-frequency cepstral coefficients, Mel-feature differential coefficients, short-time energy, zero-crossing rate, short-time energy mean, and pitch period.
[0082] This embodiment is preferably used which audio features. Figure 2 As shown, Figure 2 It is a schematic diagram of an audio feature vector of an implementation method, which includes five audio features: Mel-frequency cepstral coefficients (MFCCs), short-time energy, zero-crossing rate, short-time energy mean and fundamental frequency period.
[0083] Mel Frequency Cepstral Coefficients (MFCCs) are an acoustic feature widely used in speech processing and audio analysis. MFCCs can provide a simulation of human auditory perception and are very effective for feature extraction of speech and music signals.
[0084] The extraction process of MFCCs usually includes the following steps:
[0085] 1. Fast Fourier Transform (FFT): Perform Fast Fourier Transform (FFT) on each frame to convert the time domain signal into a frequency domain signal and calculate the power spectrum of the frequency domain signal;
[0086] 2. Mel filter bank: The spectrum obtained by FFT is passed through a series of Mel filters to obtain the eigenvalues of each filter. The design of Mel filter simulates the nonlinear frequency perception characteristics of the human ear. The output of Mel filter is usually expressed as Mel-scaled spectrum.
[0087] 3. Logarithmic operation: Perform logarithmic operation on the output of the Mel filter to further simulate the logarithmic perception characteristics of the human ear to sound intensity.
[0088] 4. Discrete Cosine Transform (DCT): Discrete cosine transform is performed on the logarithmic Mel spectrum to obtain MFCCs. The result of DET is a series of coefficients that capture the cepstrum characteristics of the signal.
[0089] 5. Coefficient selection: Usually only the first few DCT coefficients are kept because they contain most of the useful information, while higher coefficients may contain noise.
[0090] In one embodiment, the Mel frequency cepstral coefficients are the first 6 coefficients. In the process of coefficient selection, the first 6 coefficients are extracted from the MFCC of each frame.
[0091] Short-time energy and short-time energy mean describe the energy level of the audio signal within a certain time window, providing important information about signal strength and dynamic changes, especially suitable for scenarios that need to distinguish different sound types or detect sound changes. For real-time updating of the short-time energy mean, the sliding window method or the cumulative average method can be used to update the short-time energy mean.
[0092] The zero-crossing rate describes the number of zero-point crossings of a signal from positive to negative or from negative to positive per unit time. This feature is very useful for analyzing the periodicity and frequency characteristics of audio signals.
[0093] Pitch period is a key audio feature that describes periodic sound signals, such as human speech. It refers to the time required for a sound wave to complete a full cycle in one vibration. The method for calculating pitch period is: calculate the autocorrelation function for each frame, find the first significant peak other than zero delay in the autocorrelation function, calculate the delay time corresponding to the peak position, and the time corresponding to the delay T is the pitch period T0.
[0094] The combination of these features in this embodiment provides a comprehensive description, including both spectrum information (MFCCs) and time domain information (short-time energy, zero crossing rate, pitch period), which can more accurately reflect the multi-dimensional characteristics of the speech signal.
[0095] Short-term energy and zero-crossing rate can effectively distinguish speech from background noise, improving the effect of speech detection and speech enhancement. The short-term energy mean reflects the average energy level of the signal over a longer period of time, provides information about the overall strength of the signal, helps to identify background noise and steady-state noise, and smoothes out instantaneous fluctuations.
[0096] MFCC is based on the Mel frequency scale, which simulates the human ear's perception of different frequencies and is more in line with the characteristics of the human auditory system. For lightweight design, we only use the first 6 feature coefficients. The reason for choosing the first 6 MFCC coefficients is that they can effectively capture and represent the main spectral features of the speech signal while maintaining computational efficiency. Under the Mel frequency scale, these coefficients cover the low-frequency to mid-frequency information of the speech signal, especially the fundamental frequency and formant structure, which are the most important feature parts of the speech signal.
[0097] Mel Frequency Cepstral Coefficients Derivatives (MFCC Deltas) are features extracted from Mel Frequency Cepstral Coefficients (MFCCs) to capture the dynamic changes of audio signals. MFCCs describe the spectral features of audio signals on the Mel frequency scale, while MFCC Deltas describe the rate at which these features change over time.
[0098] In this implementation, the first 6 MFCC coefficients are further processed. By combining static MFCC features, first-order differences, and second-order differences, a more comprehensive and richer description of speech features can be provided. This combination of features can improve the robustness of the system in different environments, especially in noisy environments, and can help the system better distinguish between speech and noise.
[0099] First-order difference: captures the rate of change of MFCC features over time, which can reflect dynamic changes in speech signals, such as the beginning and end of speech, changes in syllables, and the pronunciation of consonants.
[0100] Second-order difference: captures the acceleration of MFCC features over time, which further captures more complex dynamic changes in speech signals, such as rapid ups and downs and turning points in speech.
[0101] First-order difference calculation: ΔM i =M i+1 -M i
[0102] Second-order difference calculation: Δ 2 Mi=M i+1 -ΔM i
[0103] Take the first two first-order differences and second-order differences as the MFCC feature mixture features.
[0104] The pitch period is insensitive to background noise and helps to extract speech features in noisy environments.
[0105] In step S130, the audio features of each audio frame are input into a pre-trained scene recognition model, and the model outputs the confidence Conf of the scene to which the frame of audio belongs. Based on the confidence Conf of the scene, the scene to which each audio frame belongs can be determined.
[0106] Among them, the scene recognition model is to train a pre-established machine learning model based on training data, and the scene recognition model obtained after training is based on the audio features to derive the corresponding scene. Taking the two scenes of classroom and break as an example, the audio of the two scenes of classroom / break is collected, and the corresponding audio features are extracted using step S120, and the audio features and the corresponding scenes are combined into data pairs as training data. The scene recognition model is a neural network model, which is composed of a recurrent neural network (including but not limited to RNN, LSTM, GRU). The input and output layers can be combined with CNN convolutional layers or FC fully connected layers to construct higher-dimensional audio features.
[0107] In step S140, two audio signals are received, the original input audio, i.e., the first audio signal, is IN, and the original input audio, i.e., the audio after the gain processed by AGC, i.e., the second audio signal, is OUT_agc. In step S130, the scene to which each audio frame belongs is obtained, and step S140 selects a paired output mode according to the specific scene to which it belongs, and different output modes represent outputting their mixed audio signals in different output ratios of the first audio signal and the second audio signal.
[0108] In one embodiment, the scenes to which the audio frames belong include: classroom scenes and break scenes; the process of step S140 of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scenes to which each audio frame belongs includes: step S210, step S220 and step S230.
[0109] Step S210, obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model;
[0110] Step S220, according to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained;
[0111] Step S230, within the time corresponding to each audio frame, the output size of the second audio signal is adjusted according to the confidence of the classroom scene of each audio frame, and the output size of the first audio signal is adjusted according to the confidence of the break scene of each audio frame.
[0112] This implementation is the first processing mode, the output mode of steady-state superposition. Step S210 obtains the confidence of each scene obtained in step S130. In this implementation, there are only two scenes, the classroom scene and the break scene, so if the confidence of the classroom scene is Conf, then the confidence of the break scene is naturally (1_Conf).
[0113] In this implementation, the confidence of the scene is used as the output weight of the classroom audio.
[0114] Generally speaking, in a classroom scenario, in order to highlight the speaker's voice, it is more suitable to output the second audio signal 0UT_agc. In a break scenario, the sound is more chaotic and does not need to be processed by AGC, so it is more suitable to output the first audio signal IN.
[0115] The new system output audio is 0new.
[0116] 0new=k*(1-Conf)*IN+Conf*0UTagc
[0117] Among them, k is the input audio control parameter, which is used to adjust the actual output on site.
[0118] In one embodiment, the scenes to which the audio frames belong include: classroom scenes and break scenes; the process of step S140 of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scenes to which each audio frame belongs includes: step S210, step S220 and step S230.
[0119] Step S210, obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model;
[0120] Step S220, according to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained;
[0121] Step S240, reducing the output volume of the first audio signal to form a new first audio signal;
[0122] Step S250, within the time corresponding to each audio frame, adjust the output size of the second audio signal according to the confidence of the classroom scene of each audio frame, and adjust the output size of the new first audio signal according to the confidence of the break scene of each audio frame.
[0123] The idea of this embodiment is the same as that of the previous embodiment, the only difference is that the sound during class breaks is noisy and not the focus of audio. We do not need it, so its sound output can be reduced by amplitude suppression. Therefore, in step S240, the output volume of the first audio signal is reduced to form a new first audio signal, that is, the new first audio signal IN_sup = W_sup*IN, where W_sup is the input audio volume control parameter. When W_sup is set to less than 1, the output volume value can be suppressed. Step S250 adjusts the new first audio signal IN_sup, that is:
[0124] O_new=(1-Conf)*IN_sup+Conf*OUT_agc.
[0125] In one implementation, the process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs in step S140 includes: step S310 and step S320.
[0126] Step S310, deriving the current scene according to the scenes to which each audio frame belongs;
[0127] Step S320: Select a corresponding speech enhancement method according to the current scene to control the output ratio of the first audio signal and the second audio signal.
[0128] In one implementation, the process of step S310 includes: step S311 and step S312.
[0129] Step S311, according to the scene to which each audio frame belongs, counting the number of audio frames determined to be in each scene within a period of N consecutive frames;
[0130] Step S312: When the number of audio frames of a scene is greater than a preset number threshold, the scene with the number of audio frames greater than the preset number threshold is determined as the current scene.
[0131] This is the second processing mode, where multiple frames of audio jointly determine the scene switching state. In step S130, the scene to which each audio frame belongs is obtained, and a statistical array is created to count the number of audio frames determined to be in each scene in N consecutive time frames. In step S312, the current scene is determined based on the number of audio frames in each scene. If the number of audio frames in a certain scene is greater than a preset threshold number, then the current scene is determined to be that scene.
[0132] Compared with the method of directly deriving the current scene based on the scene to which a single audio frame belongs, this implementation can determine the current scene only when the state is relatively stable. In addition, in this application, the current scene is finally determined only when the number is greater than the preset number threshold, which can also avoid jumping between scenes and make the judgment result more stable.
[0133] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0134] The process of step S320 includes:
[0135] When the current scene is a classroom scene, outputting a second audio signal;
[0136] When the current scene is a class break scene, a first audio signal is output.
[0137] As mentioned above, in the classroom scene, in order to highlight the speaker's voice, it is more suitable to output the second audio signal OUT_agc. In the break scene, the sound is relatively chaotic and does not need to be processed by AGC, so it is more suitable to output the first audio signal IN.
[0138] In one implementation, the process of step S320 further includes:
[0139] When switching from the break scene to the classroom scene, gradually reducing the weight of the first audio signal and gradually increasing the weight of the second audio signal;
[0140] When switching from the classroom scene to the break scene, the proportion of the second audio signal is gradually reduced, and the proportion of the first audio signal is gradually increased.
[0141] In this implementation, a gentle slope switching method is used when switching scenes, a gentle slope interval is set, the length of which is t, the interval [0, 1] is evenly divided into t values, and the size of the i-th value in the interval is w_i=i / t;
[0142] OUT_final=(1-w_i)*IN+w_i*OUT_agc, the break scene switches to the classroom scene;
[0143] OUT_final=(1-w_i)*OUT_agc+w_i*IN, the classroom scene switches to the break scene.
[0144] In one implementation, the process of step S320 includes:
[0145] When the current scene is a classroom scene, outputting a second audio signal;
[0146] When the current scene is a recess scene, the output volume of the first audio signal is reduced to form a new first audio signal, and the new first audio signal is output.
[0147] As mentioned above, in the classroom scene, in order to highlight the voice of the speaker, it is more suitable to output the second audio signal 0UT_agc. In the recess scene, the sound is relatively chaotic and does not need to be processed by AGC. Moreover, the sound during the recess is noisy and not the focus of the audio. We do not need it, so its sound output can be reduced by amplitude suppression, that is, the output volume of the first audio signal is reduced to form a new first audio signal, that is, the new first audio signal IN_sup=W_sup*IN.
[0148] Correspondingly, a gentle slope switching method is used when switching scenes, and a gentle slope interval is set with a length of t. The interval [0, 1] is evenly divided into t values, and the size of the i-th value in the interval is w_i=i / t;
[0149] OUT_final=(1-w_i)*IN_sup+w_i*OUT_agc, the break scene switches to the classroom scene;
[0150] OUT_final=(1-w_i)*OUT_agc+w_i*IN_sup, the classroom scene switches to the break scene.
[0151] In this method, the audio features in the audio signal are extracted, and then the scene of the audio signal is determined based on the audio features using a pre-trained machine learning model, and finally the output ratio between the original input audio and the audio after AGC processing is adjusted according to the scene. This method fully considers the needs of speech enhancement in different scenes in the actual teaching environment, can adapt to speech enhancement in different scenes, and adapt to the requirements of automatic gain control in actual teaching scenes.
[0152] Embodiment 2
[0153] Corresponding to the method for adaptive scene speech enhancement in the first embodiment, as shown in FIG. Figure 3 As shown, the present invention also provides a device 3 for adaptive scene speech enhancement, including: an acquisition module 310, an extraction module 320, a recognition module 330 and an output module 340.
[0154] An acquisition module 310, configured to acquire an audio signal;
[0155] An extraction module 320, configured to process the audio signal and extract audio features of each audio frame after the audio signal is framed;
[0156] The recognition module 330 is used to input the audio features of the audio frame into a pre-trained scene recognition model to obtain the scene to which each audio frame belongs; wherein the scene recognition model is a scene recognition model that can obtain the corresponding scene based on the audio features by training a pre-established machine learning model based on the training data;
[0157] The output module 340 is used to receive the first audio signal and the second audio signal, and select a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs, wherein the second audio signal is an audio signal after the first audio signal is gain-processed.
[0158] In one implementation, the audio features include: time domain features and frequency domain features.
[0159] In one implementation, the audio features include: Mel-frequency cepstral coefficients, Mel-feature differential coefficients, short-time energy, zero-crossing rate, short-time energy mean, and pitch period.
[0160] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0161] The output module executes the process of selecting a corresponding method according to the scene to which each audio frame belongs to control the output ratio of the first audio signal and the second audio signal, including:
[0162] Obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model;
[0163] According to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained;
[0164] In the time corresponding to each audio frame, the output size of the second audio signal is adjusted according to the confidence of the classroom scene of each audio frame, and the output size of the first audio signal is adjusted according to the confidence of the break scene of each audio frame.
[0165] In one implementation, the output module executes the process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs, including:
[0166] According to the scene to which each audio frame belongs, the current scene is obtained;
[0167] The corresponding speech enhancement mode is selected according to the current scene to control the output ratio of the first audio signal and the second audio signal.
[0168] In one implementation, the process of obtaining the current scene according to the scene to which each audio frame belongs includes:
[0169] According to the scene to which each audio frame belongs, the number of audio frames determined to be in each scene within a period of N consecutive frames is counted;
[0170] When the number of audio frames of a scene is greater than a preset number threshold, the scene with the number of audio frames greater than the preset number threshold is determined as the current scene.
[0171] In one implementation, the scenes to which the audio frames belong include: classroom scenes and break scenes;
[0172] The process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal includes:
[0173] When the current scene is a classroom scene, outputting a second audio signal;
[0174] When the current scene is a class break scene, a first audio signal is output.
[0175] In one implementation, the process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal further includes:
[0176] When switching from the break scene to the classroom scene, gradually reducing the weight of the first audio signal and gradually increasing the weight of the second audio signal;
[0177] When switching from the classroom scene to the break scene, the proportion of the second audio signal is gradually reduced, and the proportion of the first audio signal is gradually increased.
[0178] In this device, the audio features in the audio signal are extracted, and then the scene of the audio signal is determined based on the audio features using the pre-trained machine learning model, and finally the output ratio between the original input audio and the audio after AGC processing is adjusted according to the scene. This device fully considers the needs of speech enhancement in different scenes in the actual teaching environment, can adapt to speech enhancement in different scenes, and adapt to the requirements of actual teaching scenes for automatic gain control.
[0179] Embodiment 3
[0180] An embodiment of the present invention further provides a storage medium on which computer instructions are stored. When the instructions are executed by a processor, the method for adaptive scene speech enhancement of any of the above embodiments is implemented.
[0181] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, random access memories (RAM), read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0182] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.
[0183] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein when the encoder executes the program, a method for adaptive scene speech enhancement as described in any one of the above-mentioned embodiments is implemented.
[0184] The above-mentioned computer device extracts audio features from the audio signal, and then determines the scene of the audio signal based on the audio features using a pre-trained machine learning model, and finally adjusts the output ratio between the original input audio and the audio after gain processing by AGC according to the scene. The above-mentioned computer device fully considers the needs of speech enhancement in different scenes in the actual teaching environment, can adapt to speech enhancement in different scenes, and adapt to the requirements of actual teaching scenes for automatic gain control.
[0185] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0186] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A method for adaptive scene speech enhancement, characterized in that: include: Get audio signal; Processing the audio signal to extract audio features of each audio frame after the audio signal is framed; Inputting the audio features of the audio frames into a pre-trained scene recognition model to obtain the scene to which each audio frame belongs; wherein the scene recognition model is a scene recognition model that is obtained by training a pre-established machine learning model based on training data and is capable of obtaining a corresponding scene based on the audio features; A first audio signal and a second audio signal are received, and a corresponding method is selected to control an output ratio of the first audio signal and the second audio signal according to scenes to which each audio frame belongs, wherein the second audio signal is an audio signal after the first audio signal is gain-processed.
2. The method for scene-adaptive speech enhancement according to claim 1, characterized in that: The audio features include: time domain features and frequency domain features.
3. The method for scene-adaptive speech enhancement according to claim 2, characterized in that: The audio features include: Mel frequency cepstral coefficients, Mel feature difference coefficients, short-time energy, zero crossing rate, short-time energy mean and pitch period.
4. The method for scene-adaptive speech enhancement according to claim 1, characterized in that: The scenes to which the audio frames belong include: classroom scenes and break scenes; The process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs includes: Obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model; According to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained; In the time corresponding to each audio frame, the output size of the second audio signal is adjusted according to the confidence of the classroom scene of each audio frame, and the output size of the first audio signal is adjusted according to the confidence of the break scene of each audio frame.
5. The method for scene-adaptive speech enhancement according to claim 1, characterized in that: The scenes to which the audio frames belong include: classroom scenes and break scenes; The process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs includes: Obtaining the confidence of the scene to which each audio frame belongs after inputting the audio features of the audio frame into a pre-trained scene recognition model; According to the confidence of the scene to which each audio frame belongs, the confidence of the classroom scene and the confidence of the break scene of each audio frame are obtained; reducing the output volume of the first audio signal to form a new first audio signal; In the time corresponding to each audio frame, the output size of the second audio signal is adjusted according to the confidence of the classroom scene of each audio frame, and the output size of the new first audio signal is adjusted according to the confidence of the break scene of each audio frame.
6. The method for scene-adaptive speech enhancement according to claim 1, characterized in that: The process of selecting a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs includes: According to the scene to which each audio frame belongs, the current scene is obtained; The corresponding speech enhancement mode is selected according to the current scene to control the output ratio of the first audio signal and the second audio signal.
7. The method for scene-adaptive speech enhancement according to claim 6, characterized in that: The process of obtaining the current scene according to the scene to which each audio frame belongs includes: According to the scene to which each audio frame belongs, the number of audio frames determined to be in each scene within a period of N consecutive frames is counted; When the number of audio frames of a scene is greater than a preset number threshold, the scene with the number of audio frames greater than the preset number threshold is determined as the current scene.
8. The method for scene-adaptive speech enhancement according to claim 7, characterized in that: The scenes to which the audio frames belong include: classroom scenes and break scenes; The process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal includes: When the current scene is a classroom scene, outputting a second audio signal; When the current scene is a class break scene, a first audio signal is output.
9. The method for scene-adaptive speech enhancement according to claim 7, characterized in that: The scenes to which the audio frames belong include: classroom scenes and break scenes; The process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal includes: When the current scene is a classroom scene, outputting a second audio signal; When the current scene is a recess scene, the output volume of the first audio signal is reduced to form a new first audio signal, and the new first audio signal is output.
10. The method for scene-adaptive speech enhancement according to claim 8 or 9, characterized in that: The process of selecting a corresponding speech enhancement mode according to the current scene to control the output ratio of the first audio signal and the second audio signal also includes: When switching from the break scene to the classroom scene, gradually reducing the weight of the first audio signal and gradually increasing the weight of the second audio signal; When switching from the classroom scene to the break scene, the proportion of the second audio signal is gradually reduced, and the proportion of the first audio signal is gradually increased.
11. A device for scene-adaptive speech enhancement, characterized in that: include: An acquisition module, used for acquiring an audio signal; An extraction module, used to process the audio signal and extract audio features of each audio frame after the audio signal is framed; A recognition module, used to input the audio features of the audio frame into a pre-trained scene recognition model to obtain the scene to which each audio frame belongs; wherein the scene recognition model is a scene recognition model that is obtained by training a pre-established machine learning model based on training data and is capable of obtaining a corresponding scene based on the audio features; The output module is used to receive a first audio signal and a second audio signal, and to select a corresponding method to control the output ratio of the first audio signal and the second audio signal according to the scene to which each audio frame belongs, wherein the second audio signal is an audio signal after the first audio signal is gain-processed.
12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the method described in any one of claims 1 to 10.
