Noise filtering method, device and medium based on deep learning to build noise model

Abstract

The present application provides a noise filtering method, device and medium for constructing a noise model based on deep learning. The method includes: preprocessing the speech signal to be processed, calculating the energy value corresponding to each speech frame of the preprocessed speech signal, and determining the suspected noise frame based on the energy value and a first threshold; using training data containing a variety of background noises and clean speech to train the deep learning noise model; inputting the speech signal corresponding to the suspected noise frame into the trained deep learning noise model, and outputting the noise probability value, and performing noise reduction processing on the speech frame determined to be noise based on the comparison result of the noise probability value and the second threshold; splicing and smoothing the speech frame after noise reduction processing, and fine-tuning the reconstructed speech signal using gain control or filtering to generate the final speech signal. The present application improves the accuracy of noise detection, improves the noise filtering effect, avoids speech signal distortion, and improves the accuracy of model noise recognition.

Description

Technical Field

[0001] The present application relates to the technical field of speech noise reduction, and in particular to a noise filtering method, device, and medium for constructing a noise model based on deep learning. Background Art

[0002] With the rapid development of online games, voice communication and collaboration between players has become a common requirement. However, due to the diverse environments players operate in, background noise such as keyboard sounds, mouse clicks, fan noise, and surrounding voices often mixes in with voice quality, affecting the gaming experience.

[0003] To ensure the clarity of game team voice, the following methods are usually used in the existing technology to suppress noise:

[0004] The noise detection method based on fixed threshold determines whether there is noise by energy value or decibel value. However, when the ambient noise level changes dynamically, the fixed threshold is prone to missed detection or false detection.

[0005] Traditional filtering algorithms (such as spectral subtraction and Wiener filtering) are effective for noise within a certain range, but when faced with diverse and complex background noise, the noise suppression effect is limited and it is easy to cause distortion in the speech signal.

[0006] Simple statistical models (such as Gaussian mixture models) lack the ability to fully express various noise types when modeling noise characteristics. For new noise types or usage environments, the recognition rate is often low.

[0007] Therefore, existing technologies find it difficult to balance real-time performance and accuracy, and it is difficult to accurately filter complex and changeable noise environments. There is an urgent need for a processing method that can maintain high recognition in diverse noise scenarios to improve the communication quality of game team voice. Summary of the Invention

[0008] In view of this, the embodiments of the present application provide a noise filtering method, device and medium for constructing a noise model based on deep learning to solve the problems of poor noise detection accuracy, poor noise filtering effect, easy distortion of voice signals and low model recognition accuracy in the prior art.

[0009] In a first aspect of an embodiment of the present application, a noise filtering method for constructing a noise model based on deep learning is provided, comprising: obtaining a speech signal to be processed, and preprocessing the speech signal to be processed to obtain a preprocessed speech signal; calculating the energy value corresponding to each speech frame of the preprocessed speech signal, and marking the speech frame with an energy value lower than a first threshold as a suspected noise frame, wherein the first threshold is dynamically adjusted through environmental parameters; based on a structure combining a convolutional neural network and a recurrent neural network, the deep learning noise model is trained using training data containing a variety of background noises and clean speech; the speech signal corresponding to the suspected noise frame is input into the trained deep learning noise model, and a noise probability value is output, and noise reduction processing is performed on the speech frame determined to be noise based on a comparison result between the noise probability value and the second threshold; splicing and smoothing operations are performed on the speech frames after noise reduction processing to obtain a reconstructed speech signal, and the reconstructed speech signal is fine-tuned using gain control or filtering to generate a final speech signal.

[0010] According to a second aspect of an embodiment of the present application, a noise filtering device for constructing a noise model based on deep learning is provided, comprising: an acquisition module for acquiring a speech signal to be processed and preprocessing the speech signal to obtain a preprocessed speech signal; a calculation module for calculating the energy value corresponding to each speech frame of the preprocessed speech signal, and marking the speech frame with an energy value lower than a first threshold as a suspected noise frame, wherein the first threshold is dynamically adjusted by environmental parameters; a training module for training the deep learning noise model based on a structure combining a convolutional neural network and a recurrent neural network, using training data containing a variety of background noises and clean speech; a noise reduction module for inputting the speech signal corresponding to the suspected noise frame into the trained deep learning noise model, and outputting a noise probability value, and performing noise reduction processing on the speech frame determined to be noise based on the comparison result of the noise probability value and the second threshold; a generation module for performing splicing and smoothing operations on the speech frames after the noise reduction processing to obtain a reconstructed speech signal, and fine-tuning the reconstructed speech signal using gain control or filtering to generate a final speech signal.

[0011] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0012] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0013] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0014] The preprocessed speech signal is obtained by obtaining a speech signal to be processed and preprocessing the speech signal to be processed; the energy value corresponding to each speech frame of the preprocessed speech signal is calculated, and the speech frame with an energy value lower than a first threshold is marked as a suspected noise frame, wherein the first threshold is dynamically adjusted by environmental parameters; based on a structure combining a convolutional neural network and a recurrent neural network, a deep learning noise model is trained using training data containing a variety of background noises and clean speech; the speech signal corresponding to the suspected noise frame is input into the trained deep learning noise model, and a noise probability value is output, and according to the comparison result between the noise probability value and the second threshold, noise reduction processing is performed on the speech frame determined to be noise; the speech frame after noise reduction processing is spliced ​​and smoothed to obtain a reconstructed speech signal, and the reconstructed speech signal is fine-tuned using gain control or filtering to generate a final speech signal. This application improves the accuracy of noise detection, improves the noise filtering effect, avoids speech signal distortion, and improves the accuracy of model noise recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 1 is a flow chart of a method for reducing game team voice noise based on a dynamic threshold mechanism provided by an embodiment of the present application;

[0017] Figure 2 This is a schematic diagram of the structure of a game team voice noise reduction device based on a dynamic threshold mechanism provided by an embodiment of the present application;

[0018] Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0020] With the popularity of online games and the growing demand for teamwork, in-game voice chat has become a crucial means of communication and collaboration. However, due to the complex and diverse environments players operate in, background noise (such as keyboard sounds, mouse clicks, fan noise, and surrounding human voices) is inevitably present when using microphones to capture voice. Excessive noise can significantly impact voice communication between team members, and in severe cases, even lead to communication interruption, impacting the player's gaming experience.

[0021] In this case, how to efficiently identify and filter noise in game team voice becomes a key technical challenge to improve the quality of game voice communication.

[0022] In the prior art, methods for voice noise filtering generally include the following categories:

[0023] Fixed-threshold noise detection determines whether the energy or decibel level of a speech signal exceeds a fixed threshold and treats any signal below that threshold as noise. However, due to the wide variation in noise levels in gaming scenarios, fixed thresholds are difficult to dynamically adjust to the actual environment, leading to missed or false detections.

[0024] General noise suppression is performed based on traditional filtering algorithms (such as spectral subtraction and Wiener filtering). Although these methods can reduce noise to a certain extent, their filtering effects in complex noise scenarios are limited, and they may cause sound quality distortion if used improperly.

[0025] Simple statistical models (such as Gaussian mixture models) are used for noise identification. Such models often lack sufficient expression capabilities for the diversity of noise types. When the noise type or environment changes, the accuracy of the model will drop significantly.

[0026] Therefore, existing technologies often cannot achieve ideal real-time filtering effects when dealing with diverse and complex noises in gaming environments.

[0027] In light of the existing challenges, this application provides a method for filtering voice noise in gaming groups. Based on energy threshold filtering, this application introduces a noise model based on deep learning (combining CNN and RNN) to more fully capture the characteristics of different types of noise. Furthermore, through dynamic thresholding and real-time feedback mechanisms, the model is continuously optimized, enabling the entire system to more accurately and quickly identify and filter noise in a variety of noise scenarios.

[0028] The contents of the technical solution of this application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 This is a flow chart of the method for reducing the noise of game team voice based on the dynamic threshold mechanism provided by the embodiment of the present application. Figure 1 As shown, the game team voice noise reduction method based on the dynamic threshold mechanism may specifically include:

[0030] S101, obtaining a speech signal to be processed, and preprocessing the speech signal to be processed to obtain a preprocessed speech signal;

[0031] S102, calculating an energy value corresponding to each speech frame of the preprocessed speech signal, and marking a speech frame having an energy value lower than a first threshold as a suspected noise frame, wherein the first threshold is dynamically adjusted according to an environmental parameter;

[0032] S103, based on a structure combining convolutional neural networks and recurrent neural networks, uses training data containing various background noises and clean speech to train a deep learning noise model;

[0033] S104: Inputting the speech signal corresponding to the suspected noise frame into the trained deep learning noise model and outputting a noise probability value. Based on a comparison result between the noise probability value and a second threshold, performing noise reduction processing on the speech frame determined to be noise.

[0034] S105 , performing splicing and smoothing operations on the speech frames after the noise reduction processing to obtain a reconstructed speech signal, and fine-tuning the reconstructed speech signal by using gain control or filtering to generate a final speech signal.

[0035] In some embodiments, obtaining a speech signal to be processed and preprocessing the speech signal to be processed to obtain a preprocessed speech signal includes:

[0036] It helps the audio acquisition device to collect the voice signal to be processed and convert the voice signal to be processed into a digital signal;

[0037] Perform DC removal and pre-emphasis processing on the digitized signal;

[0038] Window function processing and frame processing are performed on the digitized signal after DC removal and pre-emphasis processing to obtain a preprocessed speech signal suitable for subsequent noise filtering.

[0039] Specifically, in order to effectively improve the noise filtering effect in the game team voice, it is necessary to perform voice signal acquisition and preprocessing operations on the voice signal to be processed to obtain high-quality voice signals suitable for subsequent noise feature extraction and filtering processing. The specific implementation method is as follows:

[0040] During real-time communication within a gaming team, the client collects the user's raw analog voice signal in real time through their audio acquisition device (e.g., headset, external microphone, etc.). This analog voice signal is then fed into an analog-to-digital converter (ADC) module for digitization, resulting in digitized raw voice data. This digitized voice data is then further processed and analyzed by a computer.

[0041] After obtaining the digitized raw voice data, DC offset processing is further performed on it. This method includes analyzing the digitized raw voice signal to determine the DC offset present in the signal, and then removing the DC offset from the raw digitized voice data. This DC offset-free voice data effectively reduces the impact of DC offset on subsequent spectrum analysis and feature extraction.

[0042] To highlight the high-frequency components of the speech signal and improve the clarity of high-frequency speech features in subsequent analysis, pre-emphasis is performed on the digitized speech data after DC removal. Pre-emphasis is implemented using a digital filter, enhancing the high-frequency portion of the speech signal. This makes the extraction of speech features, especially high-frequency noise features, more distinct and accurate in subsequent steps.

[0043] After completing the aforementioned pre-emphasis processing, the pre-emphasized speech data needs to be windowed and framed to further facilitate subsequent noise detection and feature extraction steps. Framing involves dividing the speech signal into a number of time-continuous, fixed-length speech frames according to a preset length, with a certain percentage of overlap between each frame to preserve the continuity of the speech signal between frames. Windowing, based on the framing process, applies a specific window function (such as a Hamming window or Hanning window) to weightedly adjust the signal amplitude of each speech frame, thereby reducing spectral leakage caused by segmentation at the speech frame boundaries.

[0044] After the above series of preprocessing steps, a preprocessed speech signal suitable for subsequent feature extraction, noise recognition, and filtering is finally obtained. The speech acquisition and preprocessing method in the above embodiment can significantly improve the quality of the speech signal, laying an important foundation for the subsequent accurate recognition and effective filtering of the noise filtering model based on deep learning.

[0045] In some embodiments, calculating the energy value corresponding to each speech frame of the preprocessed speech signal and marking the speech frame having an energy value lower than a first threshold as a suspected noise frame includes:

[0046] Divide the preprocessed speech signal into frames according to a fixed or dynamically adjustable frame length so that each speech frame corresponds to an independent time segment;

[0047] Performing amplitude or power calculation on the digitized signal of each speech frame to obtain an energy value used to characterize the signal strength;

[0048] determining a first threshold in real time or periodically according to environmental parameters, the environmental parameters including an environmental noise reference level;

[0049] When the energy value is lower than the first threshold, the corresponding speech frame is marked as a suspected noise frame.

[0050] Specifically, in this embodiment, in order to achieve efficient preliminary filtering of speech noise, it is necessary to perform frame energy calculation and threshold determination on the pre-processed speech signal, which specifically includes the following steps:

[0051] First, the preprocessed digitized voice signal is divided into multiple continuous, partially overlapping voice frames. In specific implementations, the voice signal can be framed according to a preset fixed frame length, such as 20 to 30 milliseconds (typically 25 milliseconds), based on subsequent processing requirements. The frame length can be dynamically adjusted based on actual application requirements or changes in noise type to ensure flexibility and adaptability in subsequent processing. After the framing process is complete, each voice frame corresponds to an independent time segment of the voice signal and can be independently processed.

[0052] Next, amplitude or power calculations are performed on the digitized signal of each speech frame to obtain an energy value that represents the signal strength of the speech frame. For example, power or amplitude analysis can be performed on the sampled data of each speech frame, such as calculating the sum of the squares of the amplitudes at each sampling point, to effectively reflect the signal energy level corresponding to the speech frame. This energy calculation method facilitates the rapid and accurate identification of speech frames with significantly low energy.

[0053] Subsequently, based on the noise level in the user's current environment, a first energy threshold is determined as a criterion for determining suspected noise frames. In specific implementations, the first threshold may be dynamically or periodically updated based on environmental parameters, such as an ambient noise reference level (i.e., the system monitors the background noise energy level in the user's current environment in real time without voice input). The first threshold is then adaptively adjusted based on the monitored background noise energy level to ensure the accuracy and reliability of suspected noise frame determination.

[0054] Finally, based on the energy values ​​of each speech frame calculated in the previous steps, the energy values ​​are compared frame by frame with a dynamically adjusted first threshold. If the energy value of a speech frame falls below the first threshold, the frame is marked as a suspected noise frame for further analysis and filtering. This method can quickly and accurately remove low-energy speech frames that are clearly noise or background interference, thereby reducing the burden on subsequent deep learning model recognition and processing, and effectively improving the real-time performance and accuracy of the overall noise filtering process.

[0055] This embodiment makes full use of the synergistic effect of the first threshold of the dynamic adjustment of the speech frame energy and the environmental parameters to effectively achieve preliminary noise filtering, laying a solid foundation for the subsequent deep learning model to perform complex noise recognition and precise filtering.

[0056] In some embodiments, based on a structure combining a convolutional neural network and a recurrent neural network, a deep learning noise model is trained using training data containing a variety of background noises and clean speech, including:

[0057] Build a model architecture that combines convolutional neural networks with recurrent neural networks. The convolutional neural network is used to extract the characteristics of speech signals in the spectral domain, and the recurrent neural network is used to analyze the changes in speech signals over time.

[0058] Collect and annotate training data containing various types of background noise and clean speech, and divide the training data into training and validation sets;

[0059] The training data is fed into the model architecture for iterative training, and the network parameters of the convolutional and recurrent layers are gradually updated using the noise annotations of the training data as a guide.

[0060] Environmental parameters and user device information are used as auxiliary input features to enhance the model's generalization ability in diverse noisy environments;

[0061] The network parameters of the model are tuned according to the recognition accuracy, speech clarity or noise detection accuracy indicators on the validation set to obtain the trained deep learning noise model.

[0062] Specifically, in this embodiment, in order to effectively realize the accurate recognition of complex and diverse noises in the game team voice environment, a deep learning noise model based on the combination of convolutional neural network (CNN) and recurrent neural network (RNN) was established and trained.

[0063] First, a composite neural network structure for the deep learning noise model is constructed. This composite network structure mainly consists of two parts: the CNN module and the RNN module.

[0064] The CNN module is mainly used to extract the features of the speech signal in the spectral domain. It captures the characteristic distribution patterns of different noise types in the frequency dimension through convolution operations, including spectral features such as spectral peaks or spectral energy distribution unique to different noise signals.

[0065] The RNN module is used to further analyze the spectral feature sequence output by the CNN module, extract the dynamic characteristics of the speech signal changing over time, and capture the noise correlation between different speech frames, so that the model can learn the changing patterns of the speech signal in the time series and effectively distinguish between noise and valid speech.

[0066] Secondly, in order to train the above-mentioned deep learning noise model, a large amount of voice data generated by users in real-life scenarios when forming teams in games was specially collected and annotated to form the training data set and verification data set required for training. In the specific implementation, the annotated data covers a variety of typical game scene background noises, such as keyboard tapping, mouse clicking, fan running, air conditioning, indoor and outdoor human voices, and other background noises, while annotating clean voice signals that do not contain obvious noise interference. The annotation work is completed by trained annotators or annotation systems to ensure the accuracy and reliability of the training data. The training data is divided into a training set and a verification set according to a certain ratio, such as 8:2 or 9:1, which are used for model parameter training and model effect verification respectively.

[0067] Furthermore, to enhance the model's generalization capabilities across diverse user environments and device conditions, this embodiment also collects and records user environment parameters and device information as auxiliary input features for model training. These include, but are not limited to, the decibel level measured during voice signal acquisition, the spectral characteristics of ambient noise, the user's microphone model, sound card type, and other hardware parameters that may affect voice characteristics. These auxiliary input features, along with the voice signal characteristics, are fed into the model, effectively enhancing the model's adaptive recognition capabilities across diverse user environments and device conditions.

[0068] The labeled training data is then fed into the deep learning noise model for iterative training. During training, the noise and clean speech categories labeled in the training data serve as supervisory signals, and the backpropagation algorithm gradually optimizes the CNN and RNN network parameters. Training utilizes a batch processing approach, with a certain size of data batches fed each time. A loss function is used to evaluate the noise recognition performance after each iteration, thereby gradually updating the network parameters.

[0069] While the model is training, the recognition accuracy, speech clarity indicators (such as subjective clarity MOS or objective signal-to-noise ratio SNR), and noise detection accuracy of the model are periodically evaluated on the validation dataset to provide real-time feedback on the model training effect. Based on this feedback, the network hyperparameters, such as the number of network structure layers, number of nodes, and learning rate, are dynamically adjusted to further optimize model performance.

[0070] Through the above-mentioned composite CNN and RNN deep learning model structure, and the use of a large amount of labeled data and auxiliary environmental parameters for training, the deep learning noise model obtained in this embodiment can efficiently and accurately identify and filter the complex and diverse noises in game team voice scenarios, significantly improving the generalization performance of the model under different user environments, device conditions and new noise types.

[0071] In some embodiments, performing noise reduction processing on the speech frame determined to be noise based on the comparison result of the noise probability value and the second threshold value includes:

[0072] comparing the noise probability value with a second threshold, wherein the second threshold is dynamically adjusted based on real-time feedback or environmental parameters;

[0073] If the noise probability value is higher than the second threshold, the corresponding speech frame is determined to be a noise frame;

[0074] A noise reduction process is performed on the speech data determined to be a noise frame, wherein the noise reduction process includes an operation of attenuating the noise component and retaining the main speech signal.

[0075] Specifically, in this embodiment, in order to further accurately filter the complex noise in the game team voice, after the aforementioned preliminary filtering through the energy threshold, noise probability assessment and noise reduction processing are performed on the suspected noise frames based on the deep learning model.

[0076] First, speech frames marked as suspected noise after the initial energy threshold filtering step are fed into a trained deep learning noise model in real time. This model, based on a composite structure combining the aforementioned convolutional neural network (CNN) and recurrent neural network (RNN), can analyze and identify the noise characteristics of each input speech frame in real time.

[0077] The deep learning noise model then analyzes each input speech frame and outputs a probability value, or "noise probability value," indicating the probability that the current speech frame is noise. The noise probability value represents the degree of match or similarity between the speech signal in that frame and known noise patterns in the training data. A higher probability value indicates a greater likelihood that the speech frame is noise.

[0078] After obtaining the noise probability value, the system further sets a second threshold for judgment and adjusts it in real time or periodically to adapt to the voice filtering needs of different environments. This second threshold is dynamically optimized based on real-time environmental parameters and user feedback on filtering effectiveness, including the current background noise level and the user's subjective rating of voice clarity. This real-time feedback mechanism allows the second threshold to adapt dynamically and adaptively, ensuring the accuracy of subsequent noise frame determinations.

[0079] In specific implementation, the noise probability value of each speech frame output by the deep learning model is compared frame by frame with the above-mentioned dynamically adjusted second threshold. If the noise probability value is higher than the second threshold, the corresponding speech frame is judged as a noise frame; if the noise probability value is lower than the second threshold, it is judged as a valid speech frame and no further noise reduction processing is required.

[0080] For the speech data determined as noise frames, further noise reduction processing is performed to suppress the noise components as much as possible and retain the effective speech information. The specific noise reduction processing method can adopt conventional and effective speech enhancement technology, such as spectral subtraction or Wiener filtering technology. Taking spectral subtraction as an example, in the specific implementation, the noise spectrum components in the noise frame are first estimated, and then the spectral components of the speech frame are correspondingly subtracted in the spectral domain, thereby significantly attenuating the noise signal; taking Wiener filtering as an example, in the specific implementation, the power spectral density of the signal and noise is estimated, and the speech signal is optimized accordingly to retain the main speech signal to the greatest extent, while effectively reducing the noise signal strength.

[0081] After the above-mentioned noise probability determination and noise reduction processing, the complex noise in the voice signal can be significantly weakened or even eliminated, making subsequent voice reconstruction, smooth splicing and fine-tuning operations more effectively implemented, thereby significantly improving the overall quality and clarity of the game team voice signal.

[0082] In some embodiments, the speech frames after noise reduction are spliced ​​and smoothed to obtain a reconstructed speech signal, and the reconstructed speech signal is fine-tuned using gain control or filtering to generate a final speech signal, including:

[0083] Arrange the noise-reduced speech frames according to a preset time sequence;

[0084] In the joint area of ​​adjacent speech frames, a weighted or window function method is used to smooth the boundaries of adjacent speech frames;

[0085] Merging the smoothed speech frames into a reconstructed speech signal;

[0086] Performing a gain control or filtering operation on the reconstructed speech signal, wherein the gain control includes adjusting the amplitude of the speech signal, and the filtering operation includes performing frequency domain or time domain processing on the speech signal;

[0087] The speech signal after gain control or filtering is output as the final speech signal.

[0088] Specifically, in this embodiment, in order to ensure that the game team voice signal still has good continuity and naturalness after noise reduction processing, it is necessary to splice and smooth each voice frame after noise reduction processing, and further perform gain control or filtering processing on the voice signal after splicing and reconstruction.

[0089] First, the system arranges and organizes the speech frames that have undergone noise reduction processing in the previous step according to the time sequence when the original speech signal was collected to ensure that each speech frame can accurately correspond to its position in the original speech signal, thereby providing an accurate timing basis for subsequent splicing processing.

[0090] Subsequently, to avoid noticeable seams or sudden changes in sound during the splicing process between speech frames, a specific smoothing method is applied to the joint area between adjacent speech frames to achieve a smooth transition between frames. Specifically, this can be achieved using a weighted superposition or window function approach. For example, within the overlapping area of ​​adjacent speech frames, a window function such as a Hamming window or a Hanning window is applied, and the data of two adjacent speech frames are appropriately weighted and superimposed to achieve a continuous transition of the speech signal amplitude between frame boundaries, significantly reducing the abruptness that may occur at the speech frame boundaries.

[0091] After completing the smooth splicing process, the system merges all smoothed speech frames to form a complete and continuous reconstructed speech signal. The reconstructed speech signal now has good continuity, but there may still be some local amplitude or spectral imbalances that require further fine-tuning.

[0092] Furthermore, this embodiment implements a post-processing algorithm for the reconstructed speech signal, including but not limited to gain control or filtering processing:

[0093] Gain control processing: By calculating the overall or local amplitude level of the voice signal, the amplitude (volume) of the voice signal is adjusted dynamically or statically to unify the overall volume level of the voice signal and avoid the discomfort caused by sudden volume changes;

[0094] Filtering: Frequency domain or time domain filtering techniques can be used to further fine-tune the speech signal. For example, bandpass filtering, high-pass filtering, or low-pass filtering can be used to remove any residual low-frequency background noise or high-frequency noise components, making the final output speech signal cleaner and clearer.

[0095] Finally, the voice signal that has undergone the aforementioned gain control and filtering becomes the final output. This final voice signal maintains the continuity and naturalness of the voice content while effectively reducing the distortion or signal imbalance caused by the splicing and reconstruction process, ensuring a good voice communication experience for users in team gaming scenarios.

[0096] In some embodiments, the method further comprises:

[0097] Collecting quality indicators corresponding to the noise-filtered speech signal;

[0098] Receive user feedback on speech signals and dynamically adjust the parameters of the deep learning noise model based on quality indicators and feedback information;

[0099] Establish a model update mechanism to periodically obtain new noise samples from user feedback or environmental data.

[0100] Incremental learning is used to combine new noise samples with existing training data to update the deep learning noise model.

[0101] Specifically, in this embodiment, to ensure that the deep learning noise model can effectively adapt to different user environments, devices, and new noise types in the long term, a quality feedback and continuous model optimization mechanism is set up. The specific implementation process is as follows:

[0102] First, after the system applies the aforementioned noise filtering to the speech signal and outputs the final speech signal, it monitors the output signal's quality in real time to objectively evaluate the effectiveness of the noise filtering. In practice, objective metrics such as the signal-to-noise ratio (SNR) or subjective speech clarity (MOS) can be calculated in real time or collected periodically to obtain specific quality data for the current speech processing result, serving as a basis for subsequent dynamic model optimization.

[0103] Secondly, the system implements a feedback mechanism on the client side to collect subjective feedback from users on the noise-filtered speech signal in real time or periodically. This feedback can be collected through, but is not limited to, user evaluations via the client software interface, such as rating speech clarity or listening satisfaction, selecting evaluation tags, or directly entering feedback. This feedback can intuitively reflect the user's satisfaction with the actual speech noise reduction effect, forming subjective evaluation data.

[0104] Next, the system combines and analyzes the above objective quality indicator data with the subjective feedback information from the user end, and dynamically adjusts the relevant parameters of the deep learning noise model based on the comprehensive analysis results, such as network layer weights, judgment thresholds or other hyperparameters, so as to continuously optimize the actual performance of the noise model and further improve the noise filtering effect.

[0105] Furthermore, to enable the deep learning noise model to continue to effectively respond to the challenges of new environments, new devices, and new types of noise, this embodiment also establishes a model update mechanism. For example, the system regularly or periodically collects and records typical noise samples involved in user-side feedback, or actively acquires new noise samples in the current environment through environmental noise collection. Such new samples include, but are not limited to, microphone pickup noise from new devices, new types of background noise in gaming environments, and typical noise samples with poor recognition performance in user-side feedback.

[0106] Finally, the newly collected noise samples are combined with the existing training data using incremental learning techniques to update the deep learning noise model. For example, by adding new noise samples to the original model training results and performing incremental training, the model can continuously improve its adaptability to new noise types and changing environments while retaining its original recognition capabilities.

[0107] The comprehensive application of real-time quality monitoring, user-side feedback mechanism, dynamic parameter adjustment and incremental learning continuous optimization mechanism in the above embodiments enables the deep learning noise model to always maintain efficient and accurate noise filtering performance, meeting the complex and changeable actual needs in the real-time voice communication scenario of game teams.

[0108] According to the technical solution of the above-mentioned embodiment of the present application, this application proposes a hybrid noise filtering method that combines "sound energy threshold preliminary filtering" and "CNN-RNN deep learning model recognition" to address the complex noise problem in the game team environment, and through real-time feedback, dynamic threshold, continuous incremental learning and other strategies, it achieves both accuracy and adaptability in noise recognition and filtering.

[0109] This application first uses the sound energy threshold to quickly eliminate obvious low-energy noise; then uses a deep learning model to capture the various characteristics of noise in the spectral domain and time series dimension; finally, combines post-processing with incremental learning mechanisms to enable the entire system to dynamically adapt to different user devices, noise environments and complex application scenarios, and continuously improve the quality of voice communication and user experience.

[0110] Through the above technical solutions of this embodiment, the noise interference in the game team voice is significantly reduced, the communication efficiency between players is improved, and a clearer and more natural voice environment is provided for game interaction and team collaboration.

[0111] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0112] Figure 2 Schematic diagram of the structure of the noise filtering device based on deep learning to build a noise model provided by the embodiment of the present application. Figure 2 As shown, the noise filtering device based on deep learning to build a noise model includes:

[0113] The acquisition module 201 is used to acquire a speech signal to be processed and preprocess the speech signal to be processed to obtain a preprocessed speech signal;

[0114] a calculation module 202 for calculating an energy value corresponding to each speech frame of the preprocessed speech signal and marking a speech frame having an energy value lower than a first threshold as a suspected noise frame, wherein the first threshold is dynamically adjusted according to an environmental parameter;

[0115] A training module 203 is configured to train a deep learning noise model based on a structure combining a convolutional neural network and a recurrent neural network using training data containing a variety of background noises and clean speech;

[0116] A noise reduction module 204 is configured to input the speech signal corresponding to the suspected noise frame into the trained deep learning noise model, output a noise probability value, and perform noise reduction processing on the speech frame determined to be noise based on a comparison result of the noise probability value with a second threshold;

[0117] The generation module 205 is used to perform splicing and smoothing operations on the speech frames after noise reduction processing to obtain a reconstructed speech signal, and fine-tune the reconstructed speech signal by using gain control or filtering to generate a final speech signal.

[0118] In some embodiments, Figure 2 The acquisition module 201 facilitates the audio acquisition device to acquire the speech signal to be processed and convert the speech signal to be processed into a digitized signal; performs DC removal and pre-emphasis processing on the digitized signal; performs window function processing and frame processing on the digitized signal after DC removal and pre-emphasis processing to obtain a pre-processed speech signal suitable for subsequent noise filtering.

[0119] In some embodiments, Figure 2The calculation module 202 divides the preprocessed speech signal into frames according to a fixed or dynamically adjustable frame length so that each speech frame corresponds to an independent time segment; performs amplitude or power calculation on the digitized signal of each speech frame to obtain an energy value used to characterize the signal strength; determines a first threshold in real time or periodically based on environmental parameters, the environmental parameters including an environmental noise reference level; and marks the corresponding speech frame as a suspected noise frame when the energy value is lower than the first threshold.

[0120] In some embodiments, Figure 2 The training module 203 constructs a model architecture that combines a convolutional neural network and a recurrent neural network, wherein the convolutional neural network is used to extract the characteristics of the speech signal in the spectral domain, and the recurrent neural network is used to analyze the changes in the speech signal in the time series; collects and labels training data containing various types of background noise and clean speech, and divides the training data into a training set and a validation set; inputs the training data into the model architecture for iterative training, and gradually updates the network parameters of the convolution layer and the recurrent layer based on the noise labeling of the training data; uses environmental parameters and user device information as auxiliary input features to enhance the generalization ability of the model in diverse noise environments; tunes the network parameters of the model according to the recognition accuracy, speech clarity or noise detection accuracy indicators on the validation set to obtain a trained deep learning noise model.

[0121] In some embodiments, Figure 2 The noise reduction module 204 compares the noise probability value with a second threshold value, wherein the second threshold value is dynamically adjusted according to real-time feedback or environmental parameters; if the noise probability value is higher than the second threshold value, the corresponding speech frame is determined to be a noise frame; and noise reduction processing is performed on the speech data determined to be a noise frame, wherein the noise reduction processing includes the operation of attenuating the noise component and retaining the main speech signal.

[0122] In some embodiments, Figure 2 The generation module 205 arranges the noise-reduced speech frames in a preset time sequence; smoothes the boundaries of adjacent speech frames in the joint area of ​​adjacent speech frames using a weighted or window function method; merges the smoothed speech frames into a reconstructed speech signal; performs gain control or filtering operations on the reconstructed speech signal, wherein the gain control includes adjusting the amplitude of the speech signal and the filtering operation includes performing frequency domain or time domain processing on the speech signal; and outputs the gain-controlled or filtered speech signal as the final speech signal.

[0123] In some embodiments, Figure 2The training module 203 is also used to collect quality indicators corresponding to the voice signal after noise filtering; receive feedback information on the voice signal from the user end, and dynamically adjust the parameters of the deep learning noise model based on the quality indicators and feedback information; establish a model update mechanism to periodically obtain new noise samples from user feedback information or environmental collection; use incremental learning to combine the new noise samples with existing training data to update the deep learning noise model.

[0124] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0125] Figure 3 Schematic diagram of the structure of the electronic device 3 provided in the embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps of the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0126] For example, computer program 303 may be divided into one or more modules / units, which are stored in memory 302 and executed by processor 301 to implement the present application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of computer program 303 in electronic device 3.

[0127] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server or other electronic device. The electronic device 3 may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will understand that Figure 3 It is only an example of electronic device 3 and does not constitute a limitation of electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0128] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0129] Memory 302 can be an internal storage unit of electronic device 3, such as a hard drive or memory of electronic device 3. Memory 302 can also be an external storage device of electronic device 3, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, memory 302 can include both an internal storage unit of electronic device 3 and an external storage device. Memory 302 is used to store computer programs and other programs and data required by the electronic device. Memory 302 can also be used to temporarily store data that has been output or is about to be output.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0131] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0132] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which may be electrical, mechanical or other forms.

[0134] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0136] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0137] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A noise filtering method based on deep learning to build a noise model, characterized in that: include: Acquiring a speech signal to be processed, and preprocessing the speech signal to be processed to obtain a preprocessed speech signal; Calculating an energy value corresponding to each speech frame of the preprocessed speech signal, and marking a speech frame having an energy value lower than a first threshold as a suspected noise frame, wherein the first threshold is dynamically adjusted according to an environmental parameter; Based on a structure that combines convolutional neural networks and recurrent neural networks, a deep learning noise recognition model is trained using training data containing a variety of background noises and clean speech. Inputting the speech signal corresponding to the suspected noise frame into a trained deep learning noise recognition model and outputting a noise probability value, and performing noise reduction processing on the speech frame determined to be noise based on a comparison result of the noise probability value and a second threshold; The denoised speech frames are spliced ​​and smoothed to obtain a reconstructed speech signal, which is then fine-tuned using gain control or filtering to generate the final speech signal. The structure based on the combination of convolutional neural network and recurrent neural network uses training data containing a variety of background noises and clean speech to train the deep learning noise recognition model, including: Constructing a composite neural network structure that combines a convolutional neural network with a recurrent neural network, wherein the convolutional neural network is used to extract the characteristics of the speech signal in the spectral domain, and the recurrent neural network is used to analyze the changes in the speech signal in the time series; Collecting and labeling training data containing various types of background noise and clean speech, and dividing the training data into a training set and a validation set; Inputting the training data into the composite neural network structure for iterative training, and gradually updating the network parameters of the convolutional layer and the recurrent layer based on the noise annotation of the training data; Environmental parameters and user device information are used as auxiliary input features to enhance the model's generalization ability in diverse noisy environments; The network parameters of the model are tuned according to the recognition accuracy, speech clarity or noise detection accuracy indicators on the validation set to obtain the trained deep learning noise recognition model.

2. The method according to claim 1, characterized in that The step of obtaining a speech signal to be processed and preprocessing the speech signal to be processed to obtain a preprocessed speech signal includes: Facilitating the audio acquisition device to acquire the voice signal to be processed and converting the voice signal to be processed into a digital signal; performing DC removal and pre-emphasis processing on the digitized signal; Window function processing and frame processing are performed on the digitized signal after DC removal and pre-emphasis processing to obtain a preprocessed speech signal suitable for subsequent noise filtering.

3. The method according to claim 1, characterized in that The calculating the energy value corresponding to each speech frame of the preprocessed speech signal and marking the speech frame with an energy value lower than a first threshold as a suspected noise frame includes: Dividing the preprocessed speech signal into frames according to a fixed or dynamically adjustable frame length so that each speech frame corresponds to an independent time segment; Performing amplitude or power calculation on the digitized signal of each speech frame to obtain an energy value used to characterize the signal strength; determining the first threshold in real time or periodically according to environmental parameters, wherein the environmental parameters include an environmental noise reference level; When the energy value is lower than the first threshold, the corresponding speech frame is marked as a suspected noise frame.

4. The method according to claim 1, wherein The performing noise reduction processing on the speech frame determined to be noise according to the comparison result of the noise probability value and the second threshold value includes: comparing the noise probability value with a second threshold, wherein the second threshold is dynamically adjusted according to real-time feedback or environmental parameters; If the noise probability value is higher than the second threshold, the corresponding speech frame is determined to be a noise frame; A noise reduction process is performed on the speech data determined to be a noise frame, wherein the noise reduction process includes an operation of attenuating the noise component and retaining the main speech signal.

5. The method according to claim 1, wherein The method comprises: performing splicing and smoothing operations on the speech frames after noise reduction processing to obtain a reconstructed speech signal, and fine-tuning the reconstructed speech signal by using gain control or filtering to generate a final speech signal, including: Arranging the noise-reduced speech frames according to a preset time sequence; In the joint area of ​​adjacent speech frames, a weighted or window function method is used to smooth the boundaries of adjacent speech frames; Merging the smoothed speech frames into a reconstructed speech signal; performing a gain control or filtering operation on the reconstructed speech signal, wherein the gain control includes adjusting the amplitude of the speech signal, and the filtering operation includes performing frequency domain or time domain processing on the speech signal; The speech signal after gain control or filtering is output as the final speech signal.

6. The method according to claim 1, characterized in that The method further comprises: Collecting quality indicators corresponding to the noise-filtered speech signal; receiving feedback information of the voice signal from the user terminal, and dynamically adjusting parameters of the deep learning noise recognition model based on the quality indicator and the feedback information; Establishing a model update mechanism to periodically obtain new noise samples from the user feedback information or environment collection; The new noise samples are combined with the existing training data by using incremental learning to update the deep learning noise recognition model.

7. A noise filtering device based on deep learning to build a noise model, characterized in that: include: An acquisition module is used to acquire a speech signal to be processed and preprocess the speech signal to be processed to obtain a preprocessed speech signal; a calculation module, configured to calculate an energy value corresponding to each speech frame of the preprocessed speech signal, and mark a speech frame having an energy value lower than a first threshold as a suspected noise frame, wherein the first threshold is dynamically adjusted according to an environmental parameter; A training module, which is used to train a deep learning noise recognition model based on a structure that combines convolutional neural networks and recurrent neural networks, using training data containing a variety of background noises and clean speech; a noise reduction module, configured to input the speech signal corresponding to the suspected noise frame into a trained deep learning noise recognition model, output a noise probability value, and perform noise reduction processing on the speech frame determined to be noise based on a comparison result of the noise probability value with a second threshold; The generation module is used to perform splicing and smoothing operations on the speech frames after noise reduction processing to obtain a reconstructed speech signal, and fine-tune the reconstructed speech signal using gain control or filtering to generate the final speech signal; Among them, the training module is used to construct a composite neural network structure that combines a convolutional neural network and a recurrent neural network, wherein the convolutional neural network is used to extract the characteristics of the speech signal in the spectral domain, and the recurrent neural network is used to analyze the changes in the speech signal in the time series; collect and label training data containing various types of background noise and clean speech, and divide the training data into a training set and a validation set; input the training data into the composite neural network structure for iterative training, and gradually update the network parameters of the convolution layer and the recurrent layer based on the noise labeling of the training data; use environmental parameters and user device information as auxiliary input features to enhance the generalization ability of the model in diverse noise environments; tune the network parameters of the model according to the recognition accuracy, speech clarity or noise detection accuracy indicators on the validation set to obtain a trained deep learning noise recognition model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images