A method, medium and system for eliminating noise in outdoor live broadcast environment

CN118486318BActive Publication Date: 2024-12-03WUHAN TECHN COLLEGE OF COMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410692678.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-31
Publication Date
2024-12-03
Estimated Expiration
2044-05-31

Smart Images

  • Figure CN118486318B_ABST
    Figure CN118486318B_ABST
Patent Text Reader

Abstract

The present invention provides a method, medium and system for eliminating noise in an outdoor live broadcast environment, which belongs to the field of outdoor live broadcast technology, and comprises: firstly, collecting the facial movements and mixed audio signals of the anchor through a camera and a microphone. Then, a mapping relationship between the anchor's mouth shape movement and the voice signal is established, and the time period and intensity change of the target voice are estimated by using the mouth shape recognition result. Next, the mixed audio is subjected to spectrum analysis, and the frequency components of the target voice are separated from each time frame by using a voice separation algorithm in combination with the time intensity information of the target voice. The parameters of the voice separation algorithm are continuously adjusted through difference comparison to enhance the quality of the target voice separated from the mixed signal. Finally, the separated target voice is filtered and enhanced to remove the influence of background noise and improve the clarity of the voice, thereby obtaining a fully enhanced target voice signal, and realizing the noise elimination processing of the anchor's voice in the outdoor live broadcast scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of outdoor live broadcasting, and specifically, relates to a method, medium and system for eliminating noise in an outdoor live broadcasting environment. Background Art

[0002] With the rapid development of live broadcasting technology, live broadcasting applications in outdoor scenes are becoming increasingly popular. Compared with the quiet indoor environment, there are many background noises in outdoor live broadcasting scenes, such as vehicle horns, pedestrians talking, wind and rain, etc. These noise signals will seriously interfere with the host's voice signal, affect the clarity and naturalness of the live audio, and thus reduce the audience experience. Therefore, how to effectively suppress the background noise in complex outdoor environments and improve the quality of live voice has become a key technical problem that needs to be solved urgently. To address this problem, academia and industry have proposed a variety of noise suppression and speech enhancement methods. Among them, the most common methods are based on traditional signal processing techniques, such as beamforming, independent component analysis, and non-negative matrix decomposition. These methods use the spatial information or statistical characteristics of multi-channel audio signals to try to separate the target speech components from the mixed signal. However, in complex outdoor environments, due to the non-stationary and non-Gaussian nature of background noise, such statistical model-based methods are usually difficult to achieve ideal results. Summary of the invention

[0003] In view of this, the present invention provides a method, medium and system for eliminating noise in an outdoor live broadcast environment, which can solve the technical problem that in a complex outdoor environment, due to the non-stationarity and non-Gaussianity of background noise, such statistical model-based methods are usually difficult to achieve ideal results.

[0004] The present invention is achieved in that:

[0005] A first aspect of the present invention provides a method for eliminating noise in an outdoor live broadcast environment, which comprises the following steps:

[0006] S10, using a camera to capture the host's facial and mouth movements, and using computer vision algorithms to perform mouth shape recognition and tracking;

[0007] S20, collecting a mixed audio signal including a target voice and background noise through a microphone or other audio collection device, wherein the target voice is the host's speaking voice;

[0008] S30, establishing a mapping model between the host's mouth shape movement and the voice signal, and using the host's mouth shape recognition result to estimate the time period and intensity change of the target voice;

[0009] S40, dividing the mixed audio signal into a plurality of short time frames, performing spectrum analysis on each time frame to obtain frequency components;

[0010] S50, separating the frequency components of the target speech from each time frame in combination with a speech separation algorithm according to the estimated time period and intensity change of the target speech;

[0011] S60, comparing the separated target speech frequency component with the collected mixed audio signal, and adjusting the parameters of the speech separation algorithm according to the difference;

[0012] S70, re-separating the enhanced target speech frequency component from the mixed audio signal using the adjusted speech separation algorithm;

[0013] S80, performing synthesis filtering on the enhanced target speech frequency component to remove the influence of background noise, and obtaining the filtered target speech;

[0014] S90, performing speech enhancement processing on the filtered target speech to improve speech clarity, and outputting the enhanced target speech signal for outdoor live broadcast content output.

[0015] Wherein, the step S10 specifically includes:

[0016] Step 101: Capture the facial and mouth movement features of the host in the video screen through a camera;

[0017] Step 102: Use a deep learning-based facial key point detection algorithm to detect the host's face and locate key points to obtain the position information of the host's mouth;

[0018] Step 103: Using a mouth shape recognition algorithm based on a hidden Markov model or a convolutional neural network, combined with the dynamic change information of the host's mouth, the host's mouth shape is recognized and tracked in real time when speaking, and a host's mouth shape sequence is obtained.

[0019] The specific steps of step S20 include:

[0020] Step 201: Use a directional microphone array or a close-range microphone and place it at an optimal position near the host to obtain a mixed audio signal containing the host's voice and background noise;

[0021] Step 202: Use beamforming technology based on sound source localization to beamform the sound waves collected by the microphone array, enhance the signal strength of the host's speaking voice, and improve the signal-to-noise ratio of the target voice.

[0022] Wherein, the specific steps of step S30 include:

[0023] Step 301: Using the host's mouth shape recognition result obtained in step S10 and combining it with the mixed audio signal collected in step S20, a hidden Markov model or a conditional random field model is established between the host's mouth shape changes when speaking and the time-frequency characteristics of the speech signal;

[0024] Step 302: using the established lip shape-speech mapping model, estimate the time period and intensity change information of the target speech, i.e., the host's speaking voice.

[0025] Wherein, the specific steps of step S40 include:

[0026] Step 401, dividing the mixed audio signal collected in step S20 into short time frames of 20-30 milliseconds in length, and setting a 50% frame overlap;

[0027] Step 402: For each time frame, use the fast Fourier transform or Mel-frequency cepstral coefficient method to obtain the frequency components of the audio signal of the frame.

[0028] Wherein, the specific steps of step S50 include:

[0029] Step 501: predicting the time-frequency masking matrix of the target speech according to the lip shape-speech mapping model established in step S30;

[0030] Step 502: Using a speech separation algorithm based on independent component analysis, non-negative matrix decomposition or dual-channel Wiener filtering based on time-frequency masking, apply the predicted time-frequency masking matrix to the time-frequency spectrum of the mixed signal, thereby separating the frequency components of the target speech.

[0031] The specific steps of step S60 include:

[0032] Step 601, calculating the correlation coefficient between the target speech frequency component separated in step S50 and the original mixed signal in step S20;

[0033] Step 602: When the correlation coefficient is lower than 0.8, adjust the time-frequency masking threshold, the number of independent components or the rank of the non-negative matrix of the speech separation algorithm.

[0034] Furthermore, the specific steps of step S70 include:

[0035] Step 701: fine-tune the time-frequency masking threshold or the parameters of the independent component number speech separation algorithm according to the optimization result of step S60;

[0036] Step 702: Apply the optimized separation algorithm to the original mixed audio signal to extract a more accurate and enhanced target speech frequency component therefrom;

[0037] The specific steps of step S80 include:

[0038] Step 801: construct a frequency domain filter based on Wiener filtering according to the statistical characteristics of the target speech frequency components and background noise;

[0039] Step 802: Utilize the constructed filter to filter the target speech frequency component separated in step S70 to further suppress background noise.

[0040] The specific steps of step S90 include:

[0041] Step 901: Using a deep learning speech enhancement algorithm based on a generative adversarial network or a waveform analysis network, perform spectrum feature extraction and speech synthesis on the filtered target speech signal outputted in step S80;

[0042] Step 902: Combine the acoustic modeling technology of hidden Markov model or deep neural network to estimate and optimize the spectral envelope and vocoder parameters of the speech signal to further improve the clarity and fidelity of the speech.

[0043] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned method for eliminating noise in an outdoor live broadcast environment.

[0044] A third aspect of the present invention provides a noise elimination system for an outdoor live broadcast environment, which includes the above-mentioned computer-readable storage medium.

[0045] Compared with the prior art, the beneficial effects of the noise elimination method, medium and system for outdoor live broadcast environment provided by the present invention are: 1. Make full use of visual information to assist speech separation and enhancement. The present invention establishes an accurate mapping model between the host's mouth shape changes and speech signals by tracking the host's facial and mouth movements in real time. Using this model, the time and intensity information of the target speech (the host's speaking voice) can be estimated more accurately, providing key prior knowledge for subsequent frequency domain speech separation. This method that combines visual and auditory information can significantly improve the speech separation performance in complex environments and overcome the limitations of relying solely on audio information.

[0046] 2. Adopt an adaptive time-frequency domain speech separation algorithm. The present invention proposes an adaptive speech separation method based on time-frequency masking, which makes full use of the host's mouth shape information to optimize the separation algorithm in real time and adjust the parameters. Through continuous iterative optimization, the separation accuracy of the target speech can be dynamically improved, and finally a clear speech component can be extracted from a complex outdoor mixed signal. This adaptive separation strategy can effectively adapt to non-stationary background noise environments.

[0047] 3. Speech enhancement technology integrating deep learning. Based on speech separation, the present invention further adopts a deep learning method based on generative adversarial networks to enhance the separated target speech signal. By learning the time-frequency characteristics of clear speech, this enhancement algorithm can effectively eliminate residual noise and improve the overall fidelity and listening quality of speech. Compared with traditional signal processing methods, deep learning methods have stronger modeling capabilities and adaptability.

[0048] In general, the present invention has made innovative breakthroughs in visual information utilization, adaptive speech separation, deep learning enhancement, etc., and can effectively deal with speech quality issues in complex outdoor environments. It solves the technical problem that in complex outdoor environments, due to the non-stationary and non-Gaussian nature of background noise, such statistical model-based methods are usually difficult to achieve ideal results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0050] Figure 1 A flow chart of the method provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0052] like Figure 1 As shown, it is a flow chart of a method for eliminating noise in an outdoor live broadcast environment provided by the present invention, and the method comprises the following steps:

[0053] S10, using a camera to capture the host's facial and mouth movements, and using computer vision algorithms to perform mouth shape recognition and tracking;

[0054] S20, collecting a mixed audio signal including a target voice and background noise through a microphone or other audio collection device, where the target voice is the host's speaking voice;

[0055] S30, establishing a mapping model between the host's mouth shape movement and the voice signal, and using the host's mouth shape recognition result to estimate the time period and intensity change of the target voice;

[0056] S40, dividing the mixed audio signal into a plurality of short time frames, performing spectrum analysis on each time frame to obtain frequency components;

[0057] S50, separating the frequency components of the target speech from each time frame in combination with a speech separation algorithm according to the estimated time period and intensity change of the target speech;

[0058] S60, comparing the separated target speech frequency component with the collected mixed audio signal, and adjusting the parameters of the speech separation algorithm according to the difference;

[0059] S70, re-separating the enhanced target speech frequency component from the mixed audio signal using the adjusted speech separation algorithm;

[0060] S80, performing synthesis filtering on the enhanced target speech frequency component to remove the influence of background noise, and obtaining the filtered target speech;

[0061] S90, performing speech enhancement processing on the filtered target speech to improve speech clarity, and outputting the enhanced target speech signal for outdoor live broadcast content output.

[0062] The specific implementation methods of the above steps are described in detail below:

[0063] The specific implementation of step S10 is as follows: first, the facial and mouth movements of the anchor during the outdoor live broadcast are captured by a video camera, and then the mouth shape recognition and tracking algorithm in computer vision technology is used to recognize and track the anchor's mouth movements.

[0064] Specifically, this step uses a deep learning-based facial key point detection algorithm, such as the improved YOLO model or FaceNet model, to detect the host's face in the video and locate key points, thereby obtaining the position information of the host's mouth. Next, a mouth shape recognition algorithm based on a hidden Markov model (HMM) or a convolutional neural network (CNN) is used, combined with the dynamic change information of the host's mouth, to perform real-time recognition and tracking of the host's mouth shape when speaking. The algorithm will match the host's speaking movements with the pre-trained mouth shape template based on features such as mouth shape and degree of opening and closing, thereby obtaining the host's mouth shape sequence when speaking.

[0065] The main purpose of this step is to obtain information about the host's mouth shape changes when speaking, and to provide basic data for the subsequent steps to establish a mapping model between the host's mouth shape movement and voice signals. Through mouth shape recognition and tracking, the time period and intensity changes of the host's speech can be determined more accurately, laying the foundation for the separation and enhancement of the target speech.

[0066] The specific implementation method of step S20 is: using a microphone or other audio acquisition equipment to capture a mixed audio signal including the host's speaking voice (target voice) and outdoor environmental background noise.

[0067] Specifically, this step uses a directional microphone array or a close-range microphone placed in the best position near the host to obtain a mixed audio signal with a high signal-to-noise ratio. At the same time, beamforming technology based on sound source localization can also be used to beamform the sound waves collected by the microphone array to enhance the signal strength of the host's voice, thereby improving the signal-to-noise ratio of the target voice.

[0068] The purpose of this step is to collect mixed audio signals containing the target speech and background noise, and provide original audio input for subsequent speech separation and enhancement processing. By collecting high-quality mixed audio signals, the characteristics of the target speech can be retained to the greatest extent, laying the foundation for subsequent signal processing.

[0069] The specific implementation method of step S30 is: first, establish a mapping model between the host's mouth shape movement and the voice signal. Specifically, using the host's mouth shape recognition result obtained in step S10, combined with the mixed audio signal collected in step S20, establish a corresponding relationship model between the host's mouth shape changes when speaking and the time-frequency characteristics of the voice signal.

[0070] The model can use statistical learning-based methods, such as Hidden Markov Model (HMM) or Conditional Random Field (CRF), to jointly model and learn the host's mouth shape sequence and speech time-frequency features, and establish a mapping relationship between the two. At the same time, deep learning techniques, such as Long Short-Term Memory (LSTM) network, can also be used to build an end-to-end mouth shape-speech mapping model to directly learn the nonlinear relationship between the two.

[0071] After obtaining the mapping model, the real-time lip recognition results of the host can be used to estimate the time period and intensity change information of the target voice (the host's speaking voice), which provides key prior information for subsequent voice separation and enhancement processing.

[0072] The specific implementation of step S40 is: first, the mixed audio signal collected in step S20 is divided into a plurality of short time frames, usually with a frame length of 20-30 milliseconds and a 50% frame overlap to ensure the continuity of the time-frequency analysis.

[0073] For each time frame, the frequency components of the audio signal of that frame are obtained by using spectrum analysis methods such as Fast Fourier Transform (FFT) or Mel Frequency Cepstral Coefficient (MFCC). In this way, the representation of the mixed audio signal in the time-frequency domain can be obtained, laying the foundation for the subsequent speech separation processing.

[0074] The purpose of this step is to convert the original time-domain mixed audio signal into the time-frequency domain, so as to analyze and separate the spectral characteristics of the target speech and background noise. Through short-time Fourier transform or MFCC feature extraction, the representation of the mixed signal in the frequency domain can be obtained, which provides input for the next step of frequency-domain speech separation processing.

[0075] The specific implementation method of step S50 is: according to the host mouth shape-voice mapping model established in step S30, using the result of the host mouth shape recognition, the time period and intensity change information of the target voice (host's speaking voice) are estimated.

[0076] Combined with this prior information, a speech separation algorithm based on time-frequency masking, such as independent component analysis (ICA), non-negative matrix factorization (NMF) or dual-channel Wiener filtering, is used to separate the frequency components of the target speech from each time frame. Specifically, the time-frequency masking matrix of the target speech is first predicted based on the lip shape-speech mapping model, and then the masking matrix is ​​applied to the time-frequency spectrum of the mixed signal to separate the frequency components of the target speech.

[0077] The purpose of this step is to use the host's lip shape information to extract the frequency components of the target speech from the mixed audio signal through the speech separation algorithm, so as to prepare for the subsequent noise suppression and speech enhancement processing. Through the time-frequency masking method, the spectral characteristics of the target speech can be retained to the maximum extent, improving the separation effect.

[0078] The specific implementation of step S60 is: compare and analyze the target speech frequency component separated in step S50 with the original mixed audio signal collected in step S20. Specifically, the difference between the separated target speech frequency component and the original mixed signal is calculated as an evaluation index of the performance of the speech separation algorithm.

[0079] According to the size of the difference, adjust the relevant parameters of the speech separation algorithm, such as the threshold of time-frequency masking, the number of independent components, the rank of the non-negative matrix, etc. Usually, an empirical threshold can be set. For example, when the correlation coefficient between the target speech frequency component and the original mixed signal is lower than 0.8, the algorithm parameters need to be fine-tuned.

[0080] The purpose of this step is to continuously optimize and improve the performance of the speech separation algorithm in order to obtain more accurate target speech frequency components. By providing feedback on the difference between the separation result and the original mixed signal, the algorithm parameters can be dynamically adjusted to improve the separation effect, laying the foundation for subsequent noise suppression and speech enhancement processing.

[0081] The specific implementation method of step S70 is: using the adjusted speech separation algorithm to re-separate the enhanced version of the target speech frequency component from the mixed audio signal collected in step S20.

[0082] Specifically, firstly, according to the optimization result of step S60, the relevant parameters of the speech separation algorithm are fine-tuned, such as the time-frequency masking threshold, the number of independent components, etc. Then, the optimized separation algorithm is applied to the original mixed audio signal to extract a more accurate and enhanced target speech frequency component.

[0083] The purpose of this step is to further improve the effect of speech separation based on the above optimization steps and obtain clearer target speech frequency components. Through the separation algorithm after parameter tuning, the background noise can be better suppressed, the spectral characteristics of the target speech can be highlighted, and the foundation for subsequent speech enhancement processing can be laid.

[0084] The specific implementation of step S80 is: first, the target speech frequency component separated in step S70 is subjected to synthesis filtering processing to further remove the influence of background noise.

[0085] Specifically, a method based on Wiener filtering can be used to construct a frequency domain filter using the power spectrum characteristics of the target speech frequency components and background noise, thereby suppressing the noise components in the mixed signal. The design of the filter can be optimized according to the statistical characteristics of the target speech and background noise. For example, an empirical noise power spectrum threshold can be set, and the frequency components below this threshold will be suppressed more.

[0086] After this step of filtering processing, the signal-to-noise ratio of the target speech can be further improved, the impact of background noise on speech quality can be reduced, and more favorable conditions can be created for subsequent speech enhancement processing.

[0087] The specific implementation method of step S90 is: further speech enhancement processing is performed on the filtered target speech signal outputted from step S80 to improve the clarity and naturalness of the speech.

[0088] Specifically, a speech enhancement algorithm based on deep learning, such as a method based on a generative adversarial network (GAN) or a waveform analysis network, can be used to extract spectral features and synthesize speech from the filtered speech signal. By learning the time-frequency features of the target clear speech, the algorithm can adaptively eliminate residual noise and generate a more natural and fluent speech output.

[0089] In addition, acoustic modeling techniques, such as hidden Markov model (HMM) or deep neural network (DNN), can be combined to estimate and optimize the spectral envelope and vocoder parameters of speech signals to further improve the clarity and fidelity of speech.

[0090] The purpose of this step is to further improve the quality of the target speech on the basis of the aforementioned noise suppression, making it clearer and more natural, and suitable for audio output in outdoor live broadcast scenarios. Through the enhancement algorithm that combines deep learning and acoustic modeling, residual noise can be effectively eliminated and the detailed features of the speech can be enhanced, thereby achieving the ideal speech output effect.

[0091] In order to better understand this method, the specific implementation of this method is described in more detail below in combination with specific formulas:

[0092] The specific implementation of step S10 is as follows:

[0093] First, the facial and mouth movement features of the anchor during the outdoor live broadcast are captured by the video camera. A set can be defined in represents the facial and mouth action feature vector extracted from the t-th frame image, d m is the feature dimension, and T is the total number of frames.

[0094] Next, we use a deep learning-based face key point detection algorithm, such as the improved YOLO model or FaceNet model, to detect the host’s face in the video and locate key points. Suppose the key point set is in represents the two-dimensional coordinates of the kth key point, and K is the total number of key points. Through these key points, the precise position information of the host's mouth can be obtained.

[0095] Then, the mouth shape recognition algorithm based on Hidden Markov Model (HMM) or Convolutional Neural Network (CNN) is used to identify and track the host's mouth shape in real time when speaking, combined with the dynamic change information of the host's mouth. The mouth shape sequence is defined as Among them l t ∈{1,2,…,L} represents the label of the host’s mouth shape in the t-th frame image, and L is the total number of mouth shape categories.

[0096] Specifically, the mouth shape recognition algorithm will match the host's speaking movements with the pre-trained mouth shape template based on the shape of the mouth, the degree of opening and closing, and other features, so as to obtain the mouth shape sequence of the host when speaking.

[0097] The purpose of this step is to obtain the information about the host's mouth shape changes when speaking, and provide basic data for the subsequent steps to establish a mapping model between the host's mouth shape movement and speech signals. Through mouth shape recognition and tracking, the time period and intensity changes of the host's speech can be determined more accurately, laying the foundation for the separation and enhancement of the target speech.

[0098] The specific implementation of step S20 is as follows:

[0099] Through a microphone array or a close-range microphone, a mixed audio signal containing the host's voice (target voice) and outdoor background noise is captured. Suppose the collected mixed audio signal is Wherein t=1,2,…,T represents the time index.

[0100] In order to enhance the signal strength of the host's voice and improve the signal-to-noise ratio of the target voice, beamforming technology based on sound source localization can be used. Specifically, the sound waves collected by the microphone array are beamformed, that is, the signals captured by different microphones are focused by weighted summation, thereby enhancing the target voice component from the host's direction:

[0101]

[0102] in, represents the signal collected by the mth microphone, M is the total number of microphones, is the beamforming weight of the mth microphone, which can be optimized by means of direction estimation.

[0103] The y(t) obtained in this way is the mixed audio signal after beamforming processing, in which the signal-to-noise ratio of the target speech is improved.

[0104] The purpose of this step is to collect mixed audio signals containing the target speech and background noise, and provide original audio input for subsequent speech separation and enhancement processing. By collecting high-quality mixed audio signals, the characteristics of the target speech can be retained to the greatest extent, laying the foundation for subsequent signal processing.

[0105] The specific implementation of step S30 is as follows:

[0106] First, a mapping model between the host's mouth movement and the speech signal is established. The representation of the audio signal in the time-frequency domain is defined as Where t=1,2,…,T is the time index, f=1,2,…,F is the frequency index, The complex domain.

[0107] The anchor mouth shape sequence obtained by step S10 And the mixed audio signal X(t,f), establish the mapping relationship between the two. Hidden Markov model (HMM) or conditional random field (CRF) can be used for modeling, and the state transition probability and emission probability are defined as follows:

[0108] State transition probability: P(l t |l t-1 )

[0109] Emission probability: P(X(t,f)|l t )

[0110] By training this joint model, we can learn the correspondence between the host's mouth shape changes and the time-frequency characteristics of the speech.

[0111] Another approach is to use deep learning techniques, such as Long Short-Term Memory (LSTM) networks, to build an end-to-end lip shape-speech mapping model:

[0112] l t =f LSTM (m t ,l t-1 ;θ)

[0113] Among them, f LSTM (·) is the mapping function of the LSTM network, and θ is the network parameter. This method can directly learn the nonlinear relationship between the host's mouth shape sequence and the speech time-frequency features.

[0114] After obtaining the mapping model, the host's real-time lip shape recognition results can be used to estimate the time period and intensity change information of the target voice (the host's speaking voice), providing key prior information for subsequent speech separation and enhancement processing.

[0115] The specific implementation of step S40 is as follows:

[0116] First, the mixed audio signal x(t) collected in step S20 is divided into a plurality of short time frames. A frame length N of 20-30 milliseconds is usually adopted, and a 50% frame overlap is set to ensure the continuity of the time-frequency analysis.

[0117] For each time frame n = 1, 2, ..., N, the frequency components of the audio signal of the frame are obtained by fast Fourier transform (FFT):

[0118] X(n,f)=FFT{x(n)}

[0119] in, Represents the complex spectral components of the nth frame at frequency f.

[0120] In addition to FFT, other spectrum analysis methods such as Mel-frequency cepstral coefficients (MFCC) can also be used to obtain the representation of mixed audio signals in the time-frequency domain.

[0121] In this way, the matrix representation of the mixed audio signal in the time-frequency domain can be obtained This lays the foundation for subsequent speech separation processing.

[0122] The purpose of this step is to convert the original time-domain mixed audio signal into the time-frequency domain, so as to analyze and separate the spectral characteristics of the target speech and background noise. Through short-time Fourier transform or MFCC feature extraction, the representation of the mixed signal in the frequency domain can be obtained, providing input for the next step of frequency-domain speech separation processing.

[0123] The specific implementation of step S50 is as follows:

[0124] According to the host mouth shape-speech mapping model established in step S30, the host mouth shape recognition result is used to Estimate the time period and intensity change information of the target voice (host's speaking voice).

[0125] Assume that the target speech is represented in the time-frequency domain as The background noise is Then the mixed signal X(t,f) can be expressed as:

[0126] X(t,f)=S(t,f)+N(t,f)

[0127] Combined with the prior information of the lip shape-speech mapping model, a speech separation algorithm based on time-frequency masking is adopted, such as independent component analysis (ICA), non-negative matrix factorization (NMF) or two-channel Wiener filtering, to separate the frequency components S(t,f) of the target speech from each time frame.

[0128] Specifically, firstly, according to the lip shape-speech mapping model, the time-frequency masking matrix M(t,f)∈[0,1] of the target speech is predicted, and then the masking matrix is ​​applied to the time-frequency spectrum of the mixed signal to separate the frequency components of the target speech:

[0129] S(t,f)=M(t,f)⊙X(t,f)

[0130] Where ⊙ represents the element-wise product.

[0131] The purpose of this step is to use the host's lip shape information to extract the frequency components of the target speech from the mixed audio signal through the speech separation algorithm, so as to prepare for the subsequent noise suppression and speech enhancement processing. Through the time-frequency masking method, the spectral characteristics of the target speech can be retained to the maximum extent, improving the separation effect.

[0132] The specific implementation of step S60 is as follows:

[0133] The target speech frequency component S(t, f) separated in step S50 is compared and analyzed with the original mixed audio signal X(t, f) collected in step S20. Specifically, the correlation coefficient between the separated target speech frequency component and the original mixed signal is calculated:

[0134]

[0135] Among them, Re{·} and Im{·} represent the real part and imaginary part of the complex number respectively.

[0136] According to the size of the correlation coefficient r, adjust the relevant parameters of the speech separation algorithm, such as the threshold τ of the time-frequency masking M , the number of independent components K, the rank of the non-negative matrix R, etc. Usually when r<0.8, the algorithm parameters need to be fine-tuned.

[0137] The purpose of this step is to continuously optimize and improve the performance of the speech separation algorithm in order to obtain more accurate target speech frequency components. By providing feedback on the difference between the separation result and the original mixed signal, the algorithm parameters can be dynamically adjusted to improve the separation effect, laying the foundation for subsequent noise suppression and speech enhancement processing.

[0138] The specific implementation of step S70 is as follows:

[0139] Using the adjusted speech separation algorithm, the enhanced version of the target speech frequency component S(t,f) is re-separated from the mixed audio signal X(t,f) collected in step S20.

[0140] Specifically, firstly, according to the optimization result of step S60, the relevant parameters of the speech separation algorithm are fine-tuned, such as the time-frequency masking threshold τ M , the number of independent components K or the rank R of the non-negative matrix, etc.

[0141] Then, the optimized separation algorithm is applied to the original mixed audio signal X(t,f) to extract a more accurate and enhanced target speech frequency component S(t,f).

[0142] The purpose of this step is to further improve the effect of speech separation based on the above optimization steps and obtain clearer target speech frequency components. Through the separation algorithm after parameter tuning, the background noise can be better suppressed, the spectral characteristics of the target speech can be highlighted, and the foundation for subsequent speech enhancement processing can be laid.

[0143] The specific implementation of step S80 is as follows:

[0144] First, the target speech frequency component S(t,f) separated in step S70 is subjected to synthesis filtering to further remove the influence of background noise. A method based on Wiener filtering can be used to construct a frequency domain filter using the power spectrum characteristics of the target speech and background noise:

[0145]

[0146] Among them, P S (t,f) and P N(t, f) are the power spectrum estimates of the target speech and background noise at the time-frequency point (t, f). The noise power spectrum can be estimated online by using methods such as minimum mean square estimation (MMSE).

[0147] Then, the constructed Wiener filter is applied to the target speech frequency component to obtain the filtered result;

[0148] Next, the filtered target speech frequency components are synthesized:

[0149]

[0150] in, That is the target speech frequency component processed by Wiener filtering.

[0151] The purpose of this step is to further improve the signal-to-noise ratio of the target speech, reduce the impact of background noise on speech quality, and create more favorable conditions for subsequent speech enhancement processing. Through the design and optimization of the Wiener filter, the noise components in the spectrum can be effectively suppressed and the key features of the speech can be retained.

[0152] The specific implementation of step S90 is as follows:

[0153] The filtered target speech frequency component output in step S80 Further speech enhancement processing is performed to improve the clarity and naturalness of the speech.

[0154] A speech enhancement algorithm based on deep learning can be used, such as a generative adversarial network (GAN) or a waveform analysis network. Specifically, a generator network G(·; θ G ) and the discriminator network D(·;θ D ), the goal is to train the G network to learn from the filtered speech spectrogram To the clear speech spectrogram S * Mapping of (t,f):

[0155]

[0156] At the same time, the D network is trained to determine whether the generated clear speech spectrogram is real, thereby promoting the learning of the G network.

[0157] In addition, acoustic modeling techniques, such as hidden Markov model (HMM) or deep neural network (DNN), can be combined to estimate and optimize the spectral envelope and vocoder parameters of speech signals to further improve the clarity and fidelity of speech.

[0158] Finally, the target speech frequency component S enhanced by deep learning is *(t,f) performs inverse transform from time-frequency domain to time domain and outputs the enhanced target speech signal for outdoor live content output.

[0159] The purpose of this step is to further improve the quality of the target speech on the basis of the aforementioned noise suppression, making it clearer and more natural, and suitable for audio output in outdoor live broadcast scenarios. Through the enhancement algorithm that combines deep learning and acoustic modeling, residual noise can be effectively eliminated and the detailed features of the speech can be enhanced, thereby achieving the ideal speech output effect.

[0160] In summary, the invention uses computer vision technology to identify and track the host's mouth shape in real time, and combines speech separation and enhancement algorithms to achieve an efficient method for eliminating noise in outdoor live broadcast environments. This method makes full use of the host's mouth shape information, and through signal processing in the time and frequency domains, effectively separates and enhances the target voice from the complex outdoor environment audio, and finally outputs a clear and fluent voice signal.

[0161] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned method for eliminating noise in an outdoor live broadcast environment.

[0162] A third aspect of the present invention provides a noise elimination system for an outdoor live broadcast environment, which includes the above-mentioned computer-readable storage medium.

[0163] Specifically, the principle of the present invention is to make full use of the host's mouth movement information in the video screen to assist in the separation and enhancement of voice signals, thereby obtaining high-quality target voice output in a complex outdoor environment. Specifically, the method includes the following key steps:

[0164] First, the dynamic changes of the host's face and mouth are captured by the camera, and computer vision technology is used to track and recognize the mouth shape in real time. The purpose of this step is to obtain the time series information of the host's mouth movement when speaking, laying the foundation for the subsequent establishment of the mouth shape-speech mapping model.

[0165] Next, the microphone array is used to collect the mixed audio signal containing the target speech and background noise. In order to enhance the signal strength of the target speech, beamforming technology can be used to perform spatial focusing on the microphone signal.

[0166] Then, based on the host's mouth shape sequence obtained in step 1, a corresponding relationship model with the speech signal in the time-frequency domain is established. This model can be learned using a hidden Markov model or a deep learning method, with the goal of being able to predict the time period and intensity characteristics of the target speech from the changes in the host's mouth shape.

[0167] With this prior information, we can use the adaptive speech separation algorithm based on time-frequency masking to separate the frequency components of the target speech from the mixed audio signal. Specifically, we first predict the time-frequency masking matrix based on the lip shape-speech model, and then apply the masking matrix to the time-frequency spectrum of the mixed signal to suppress the background noise component and extract the target speech. In order to further optimize the separation effect, this method will also dynamically adjust the parameters of the separation algorithm based on the difference between the separation result and the original mixed signal.

[0168] Finally, the separated target speech frequency components are processed by Wiener filtering and speech enhancement based on deep learning to remove residual noise and improve the clarity and fidelity of speech. Among them, Wiener filtering uses the statistical characteristics of target speech and noise to construct a frequency domain filter to further suppress background interference; deep learning enhancement generates more natural and fluent speech output by learning the time-frequency characteristics of clear speech.

[0169] From a technical point of view, the method of the present invention can effectively solve the voice quality problem in outdoor live broadcast environment. The key lies in making full use of visual information and combining adaptive time-frequency domain speech separation and deep learning speech enhancement technology.

[0170] First, by real-time tracking of the host's mouth shape, the prior time-frequency characteristics of the target speech can be obtained, providing important prior knowledge for subsequent speech separation. This visual-assisted method can overcome the limitation of relying solely on acoustic information and significantly improve the separation accuracy in complex noisy environments.

[0171] Secondly, the adaptive time-frequency domain separation algorithm proposed in the present invention can dynamically adjust relevant parameters according to the feedback of the separation effect and continuously optimize the separation performance. This adaptive strategy enables the method to better adapt to non-stationary outdoor background noise and achieve more ideal separation results.

[0172] Finally, based on speech separation, the speech enhancement technology based on deep learning can effectively eliminate residual noise and further improve the clarity and naturalness of speech. Compared with traditional signal processing methods, deep learning has more powerful modeling capabilities and can capture complex time-frequency features, thereby generating more realistic and natural speech output.

[0173] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0174] A specific embodiment of the present invention is provided below: This embodiment is carried out in an outdoor live broadcast scene, and the main equipment includes: 1 high-definition camera (resolution 1920×1080, frame rate 30fps) for capturing the anchor video; 1 8-channel microphone array (directional sound collection range 120 degrees, signal-to-noise ratio>60dB) for collecting audio signals; 1 laptop computer (CPU i7-10750H, memory 16GB, GPU RTX 2060) for signal processing and algorithm calculation. The scene covers a variety of background noise sources including vehicle horns, pedestrian conversations, wind and rain. The noise signal is played through the speaker and superimposed with the anchor's voice signal to form a mixed audio. The anchor conducts a reading test under different noise conditions, and records the video and audio data respectively.

[0175] On this basis, a test data set containing 10 anchors and 30 minutes of speech data was constructed. Each anchor took three reading tests in a silent environment, a medium noise environment (signal-to-noise ratio of about 10dB), and a high noise environment (signal-to-noise ratio of about 0dB), and a total of 90 mixed audio samples were collected. In addition, 10 minutes of clean speech data was prepared for training the speech enhancement model.

[0176] Lip recognition and speech separation

[0177] First, we use the YOLO-based facial key point detection algorithm to analyze the anchor’s video in real time to obtain dynamic changes in the anchor’s face and mouth. Specifically, the algorithm can accurately locate 68 facial key points, including eyebrows, eyes, nose, mouth and other core areas. We select 10 key points of the mouth (such as Figure 1 as shown in Figure 3), as the basis for the anchor’s mouth shape features.

[0178] Next, a mouth shape recognition algorithm based on Hidden Markov Model (HMM) is used to model and classify the above mouth key point sequence. The algorithm first divides the host's mouth action sequence into a series of discrete mouth shape states, and then uses HMM to learn the transition rules between these states, and finally can map the real-time mouth action sequence to the predefined mouth shape category label. Through the learning of a large number of training samples, the mouth shape recognition model achieved an accuracy of 92.4% on the test set.

[0179] With the host's mouth shape sequence as prior information, an adaptive speech separation algorithm based on time-frequency masking is used to extract the target speech component from the mixed audio signal. Specifically, the host's mouth shape-speech mapping model is first used to predict the masking matrix of the target speech in the time-frequency domain. Then, the time-frequency masking is applied to the STFT spectrum of the mixed signal to separate the frequency components of the target speech. In order to further optimize the separation effect, an adaptive feedback mechanism based on the correlation coefficient is designed: the separation result is compared with the original mixed signal. When the correlation coefficient is lower than the threshold, the parameters of the speech separation algorithm, such as the time-frequency masking threshold and the number of independent components, are adjusted until the ideal separation performance is achieved.

[0180] Among them, in a medium noise environment (10dB signal-to-noise ratio), the spectral characteristics of the target speech have been clearly separated from the mixed signal; in a high noise environment (0dB signal-to-noise ratio), although there is some residual noise, the core speech information can still be extracted more clearly. This shows that the adaptive speech separation algorithm of the present invention can achieve better separation performance in complex outdoor environments.

[0181] In order to quantitatively evaluate the separation effect, three indicators, signal-to-interference ratio (SDR), source-to-distortion ratio (SDR) and source-to-noise ratio (SNR), were used for measurement. On the above 90 test samples, the speech separation method of the present invention achieved an average SDR of 12.8dB, an average SINA of 18.6dB and an average SNR of 15.2dB, which is significantly better than the method using only traditional independent component analysis (ICA) or non-negative matrix factorization (NMF).

[0182] Speech Enhancement and Subjective Evaluation

[0183] After completing the speech separation step, a speech enhancement algorithm based on deep learning is further used to optimize the separated target speech signal. Specifically, a generative adversarial network (GAN) model is constructed, in which the generator network is responsible for learning the conversion from the separated speech spectrogram to the clear speech spectrogram, and the discriminator network is responsible for judging and providing feedback on the generated speech quality. Through adversarial training, the generator network can gradually capture the time-frequency characteristics of clear speech, thereby effectively eliminating residual noise and outputting a more natural and smooth speech signal.

[0184] In order to train this speech enhancement model, 10 minutes of clean speech data was used. First, the original speech signal was transformed by STFT to obtain its time-frequency domain representation; then, background noise of different types and intensities was artificially added to form a mixed speech spectrogram as the network input; finally, the target output of the network is the corresponding clean speech spectrogram. Through this supervised end-to-end training method, the generator network can learn the mapping relationship from noisy speech to clear speech.

[0185]

[0186]

[0187] Table 1 Speech enhancement effect table

[0188] Table 1 shows the effect comparison of the speech enhancement algorithm of the present invention. It can be seen that in a high-noise environment, the simple speech separation result still has a lot of background interference; after deep learning enhancement, the time-frequency characteristics of the speech are significantly improved, the noise component is effectively suppressed, and the overall sound quality is clearer and more natural. This result shows that the present invention combines visual-assisted speech separation and deep learning-based speech enhancement, which can effectively solve the speech quality problem in outdoor live broadcast scenarios.

[0189] In order to more comprehensively evaluate the performance of the method of the present invention, 20 volunteers were invited to participate in a subjective listening test. During the test, the volunteers were required to subjectively rate the output speech quality of the method of the present invention in a silent environment, a medium-noise environment, and a high-noise environment. The rating was based on a 5-point system, with 1 point representing extremely poor and 5 points representing extremely excellent.

[0190] The test results are shown in Table 2. In a silent environment, the voice quality score of the method of the present invention is 4.8 points on average, which is close to the ideal level; in a medium-noise environment, the score is 4.2 points on average, which still maintains a high subjective perceived quality; in a high-noise environment, the score is 3.9 points on average, which is significantly improved compared to the traditional method. These results show that the outdoor speech enhancement method of the present invention can not only maintain good objective measurement indicators under complex noise conditions, but also effectively improve the subjective experience of end users.

[0191] Table 2 Rating table

[0192]

Claims

1. A method for eliminating noise in an outdoor live broadcast environment, characterized in that: The following steps are involved: S10, using a camera to capture the host's facial and mouth movements, and using computer vision algorithms to perform mouth shape recognition and tracking; S20, collecting a mixed audio signal including a target voice and background noise through a microphone or other audio collection device, wherein the target voice is the host's speaking voice; S30, establishing a mapping model between the host's mouth shape movement and the voice signal, and using the host's mouth shape recognition result to estimate the time period and intensity change of the target voice; S40, dividing the mixed audio signal into a plurality of short time frames, performing spectrum analysis on each time frame to obtain frequency components; S50, separating the frequency components of the target speech from each time frame in combination with a speech separation algorithm according to the estimated time period and intensity change of the target speech; S60, comparing the separated target speech frequency component with the collected mixed audio signal, and adjusting the parameters of the speech separation algorithm according to the difference; S70, re-separating the enhanced target speech frequency component from the mixed audio signal using the adjusted speech separation algorithm; S80, performing synthesis filtering on the enhanced target speech frequency component to remove the influence of background noise, and obtaining the filtered target speech; S90, performing speech enhancement processing on the filtered target speech to improve speech clarity, and outputting the enhanced target speech signal for outdoor live broadcast content output.

2. The method for eliminating noise in an outdoor live broadcast environment according to claim 1, characterized in that: The step S10 specifically includes: Step 101: Capture the facial and mouth movement features of the host in the video screen through a camera; Step 102: Use a deep learning-based facial key point detection algorithm to detect the host's face and locate key points to obtain the position information of the host's mouth; Step 103: Using a mouth shape recognition algorithm based on a hidden Markov model or a convolutional neural network, combined with the dynamic change information of the host's mouth, the host's mouth shape is recognized and tracked in real time when speaking, and a host's mouth shape sequence is obtained.

3. The method for eliminating noise in an outdoor live broadcast environment according to claim 1, characterized in that: The specific steps of step S20 include: Step 201: Use a directional microphone array or a close-range microphone and place it at an optimal position near the host to obtain a mixed audio signal containing the host's voice and background noise; Step 202: Use beamforming technology based on sound source localization to beamform the sound waves collected by the microphone array, enhance the signal strength of the host's speaking voice, and improve the signal-to-noise ratio of the target voice.

4. The method for eliminating noise in an outdoor live broadcast environment according to claim 1, characterized in that: The specific steps of step S30 include: Step 301: Using the host's mouth shape recognition result obtained in step S10 and combining it with the mixed audio signal collected in step S20, a hidden Markov model or a conditional random field model is established between the host's mouth shape changes when speaking and the time-frequency characteristics of the speech signal; Step 302: using the established lip shape-speech mapping model, estimate the time period and intensity change information of the target speech, i.e., the host's speaking voice.

5. The method for eliminating noise in an outdoor live broadcast environment according to claim 1, characterized in that: The specific steps of step S40 include: Step 401, dividing the mixed audio signal collected in step S20 into short time frames of 20-30 milliseconds in length, and setting a 50% frame overlap; Step 402: For each time frame, use the fast Fourier transform or Mel-frequency cepstral coefficient method to obtain the frequency components of the audio signal of the frame.

6. The method for eliminating noise in an outdoor live broadcast environment according to claim 1, characterized in that: The specific steps of step S50 include: Step 501: predicting the time-frequency masking matrix of the target speech according to the lip shape-speech mapping model established in step S30; Step 502: Using a speech separation algorithm based on independent component analysis, non-negative matrix decomposition or dual-channel Wiener filtering based on time-frequency masking, apply the predicted time-frequency masking matrix to the time-frequency spectrum of the mixed signal, thereby separating the frequency components of the target speech.

7. The method for eliminating noise in an outdoor live broadcast environment according to claim 1, characterized in that: The specific steps of step S60 include: Step 601, calculating the correlation coefficient between the target speech frequency component separated in step S50 and the original mixed signal in step S20; Step 602: When the correlation coefficient is lower than 0.8, adjust the time-frequency masking threshold, the number of independent components or the rank of the non-negative matrix of the speech separation algorithm.

8. The method for eliminating noise in an outdoor live broadcast environment according to claim 7, characterized in that: The specific steps of step S70 include: Step 701: fine-tune the time-frequency masking threshold or the parameters of the independent component number speech separation algorithm according to the optimization result of step S60; Step 702: Apply the optimized separation algorithm to the original mixed audio signal to extract a more accurate and enhanced target speech frequency component therefrom; The specific steps of step S80 include: Step 801: construct a frequency domain filter based on Wiener filtering according to the statistical characteristics of the target speech frequency components and background noise; Step 802: Utilize the constructed filter to filter the target speech frequency component separated in step S70 to further suppress background noise.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, which, when executed, are used to execute a method for eliminating noise in an outdoor live broadcast environment as described in any one of claims 1-8.

10. A noise elimination system for outdoor live broadcast environment, characterized in that: A computer-readable storage medium comprising the computer-readable storage medium of claim 9.