Earphone noise reduction method and system based on multivariate noise

Abstract

The invention provides an earphone noise reduction method and system based on multivariate noise. The method comprises the steps that a multi-array microphone collects effective sound signals of a user and signals of different noise sources; the noise reduction mode control system realizes state switching among a motion mode, a video watching mode, a call mode, an audio loading mode, a standard noise reduction mode, a comfortable noise reduction mode and an enhanced noise reduction mode through a user terminal interface; the wireless earphone noise reduction system, the user terminal earphone noise reduction system and the data computing power cloud service center adopt a'cloud side end 'three-level architecture, noise signals in mixed voice signals are eliminated through a built-in intelligent voice noise reduction method, and the user experience is improved; the cloud transmission architecture is designed into a wired and wireless combined composite information transmission system, so that long-distance, high-capacity and efficient information transmission is realized; and the voice enhancement system performs final processing on the noise-reduced sound signal through a built-in intelligent voice enhancement method, eliminates mixed sound and echo interference, and outputs a high-fidelity sound quality sound signal.

Description

Technical Field

[0001] The present invention belongs to the technical field of headphone noise reduction, and in particular relates to a headphone noise reduction method and system based on multi-noise. Background Art

[0002] Noise pollution has become a global problem, seriously affecting people's health and quality of life. Urban transportation, industrial production and entertainment activities are the main sources of noise. Long-term exposure to noise may lead to hearing loss, cardiovascular disease and other health problems. As noise pollution becomes increasingly serious, people are beginning to seek ways to solve noise interference. As an effective solution, the market demand for noise-canceling headphones continues to grow. Noise-canceling headphones can not only isolate external noise and improve listening quality, but also play an irreplaceable role in specific industries and scenarios (such as aviation, railways, construction, etc.).

[0003] At the same time, as people pursue sound quality and user experience, the noise-canceling headphone market is gradually rising. The global noise-canceling headphone market is expected to grow at a compound annual growth rate of 10% in the next few years. At present, the market is mainly dominated by several key players such as Sony, BOSE, Beats, JBL and Sennheiser. There is fierce competition in the noise-canceling headphone market. Major brands have launched noise-canceling headphone products with innovative technologies and excellent performance to compete for market share. At the same time, traditional brands and Internet brands are also paying attention to the noise-canceling headphone market and competing to enter this field.

[0004] As the core technology of noise-cancelling headphones, sound signal noise reduction technology has gone through multiple stages of development from physical noise reduction to active noise reduction, hybrid noise reduction and intelligent noise reduction technology. Among them, active noise reduction technology is currently the mainstream technology in the field of noise-cancelling headphones. External noise is sensed through built-in sensors, and then sounds with opposite waveforms are quickly generated to offset the effect of noise. This technology can better reduce noise interference and provide clearer sound quality. In addition to active noise reduction technology, hybrid noise reduction technology has also gradually attracted attention. Hybrid noise reduction technology combines the advantages of active noise reduction and passive noise reduction, and reduces noise through physical isolation and active noise reduction to achieve better noise reduction effects. In recent years, AI noise reduction technology has begun to be applied to the field of noise-cancelling headphones. Through machine learning algorithms, AI noise-cancelling headphones can more accurately identify and eliminate noise, providing a more personalized noise reduction experience. For example, the "semantic hearing" system developed by the research team at the University of Washington can connect existing noise-cancelling headphones to smartphone applications to enhance or suppress certain sounds in real time according to the preferences selected by the user.

[0005] Based on the above technological developments and the beneficial effects they bring, noise-canceling headphones are gradually beginning to play an increasingly important role in people's daily lives, which can be specifically described as follows: 1. Improve work and study efficiency: By eliminating noise interference, noise-canceling headphones can allow people to focus more on work and study, improving efficiency and results.

[0006] 2. Reduce the damage of noise to hearing: Long-term exposure to noise environment will cause permanent damage to hearing. Noise-canceling headphones can effectively reduce the noise level and protect hearing health.

[0007] 3. Improve the quality of life: Noise-canceling headphones can create a relatively quiet environment, allowing people to better enjoy music, movies and other entertainment activities and improve the quality of life.

[0008] 4. Promote the progress of noise reduction technology: With the continuous development of noise reduction technology, the performance and effect of noise reduction headphones are also constantly improving. The research on noise reduction technology of multi-noise headphones will further promote the progress and innovation of noise reduction technology.

[0009] In the prior art CN111586522A, a method for reducing noise of headphones, a device for reducing noise of headphones, headphones and a storage medium are disclosed, including obtaining an acceleration signal collected by an acceleration sensor, and judging whether the wearer is in motion according to the acceleration signal; if so, performing a filtering and adjusting operation on the acceleration signal to obtain a footstep vibration noise signal; performing a superposition operation on the footstep vibration noise signal and the environmental noise signal to obtain a signal to be reduced noise, and generating an anti-noise signal of the signal to be reduced noise, but this method can only eliminate the noise interference caused by accelerated motion and footstep vibration during motion, and cannot eliminate the interference caused by external noise such as voice, equipment, etc. under normal use conditions. In the prior art CN113411711A, a method and system for reducing noise of smart headphones are disclosed, including the following steps: the charging compartment inputs the current noise situation into the first noise reduction model. If the current noise situation is inconsistent with the pre-stored noise, the charging compartment transmits the current noise situation to the cloud server. The cloud server inputs the current noise situation into the trained second noise reduction model, and the second noise reduction model compares the current noise situation with the noise situation in the second noise reduction model. If the current noise situation is inconsistent with the noise situation, the current noise situation is subjected to noise reduction processing, and the current noise situation and the noise reduction processing are saved as the corresponding noise reduction algorithm in the second noise reduction model. The charging bin performs noise reduction processing on the current noise situation through the corresponding noise reduction algorithm. For different noise situations, the purpose of calling the noise reduction algorithm that matches the noise situation to perform noise reduction processing on the noise situation is achieved, thereby achieving a better noise reduction effect. However, this noise reduction method is overly dependent on the charging bin and the cloud server, and the headset itself does not have direct noise recognition and filtering and removal capabilities. The noise removal and synchronization speeds are slow, and the user experience cannot be effectively improved. To this end, the present invention proposes a headphone noise reduction method and system based on multivariate noise. Summary of the invention

[0010] In order to solve the above problems, the present invention proposes a headphone noise reduction method and system based on multivariate noise to solve the problems existing in the above-mentioned prior art, improve the effectiveness and efficiency of headphone noise reduction, and improve user experience. Since the signal transmission link of wired headphones is relatively stable and not easily interfered by external noise, the purpose of noise reduction can be achieved by simply modifying the headphone structure design. Therefore, this invention is mainly aimed at the wireless Bluetooth headphones currently popular on the market.

[0011] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a headphone noise reduction method based on multivariate noise, applied to a headphone noise reduction system based on multivariate noise, including a hybrid multivariate noise recognition method, an intelligent speech noise reduction method and an intelligent speech enhancement method.

[0012] Furthermore, in the headphone noise reduction method based on multivariate noise, the hybrid multivariate noise recognition method adopts a hybrid multivariate noise recognition method that combines traditional acoustic signal analysis and deep learning.

[0013] Furthermore, the headphone noise reduction method based on multivariate noise, the hybrid multivariate noise recognition method combining traditional sound signal analysis and deep learning decomposes the mixed multivariate noise into multiple single noise signals, and performs feature analysis on the decomposed single noise signals, and finally identifies the category of each single noise through a trained classifier model, specifically including the following steps: Step 1: Collect mixed noise signals from the outside world containing multiple noise sources through a microphone receiving system.

[0014] Step 2: Preprocess the signal. First, perform preliminary filtering on the collected multivariate mixed noise signal, and use a high-pass filter, low-pass filter or band-pass filter to remove signals that are obviously not conventional noise components, such as DC components and high-frequency components. After that, divide the collected continuous multivariate mixed noise signal into shorter frames for subsequent video analysis.

[0015] Step 3: Extract features of the preprocessed noise signal, mainly including time domain features, frequency domain features and other features. Extract the time domain features of each frame signal, including short-time energy, short-time zero-crossing rate, and autocorrelation function. These features can reflect the temporal structure of the noise signal; perform Fourier transform on each frame signal to extract frequency domain features, including spectrum and power spectrum features. These features can reflect the frequency distribution of the noise signal; according to different actual application scenarios, other acoustic signal features including cepstrum features and MFCC (Mel frequency cepstrum coefficients) can be extracted.

[0016] Step 4: Perform noise source separation on the multivariate mixed noise signal. First, perform blind source separation operation and use blind source separation technology to decompose the mixed noise signal into multiple independent components. These independent components correspond to different noise sources. Then, use the spectrum analysis method to perform spectrum analysis on the separated independent components and observe the performance of each component in the frequency domain. According to the spectrum characteristics, it can be preliminarily determined whether each component corresponds to a different noise source. Finally, according to the spectrum analysis results, the parameters of blind source separation are adjusted through iterative optimization to make the separated independent components closer to the real noise source.

[0017] Step 5: Single noise signal extraction. According to the spectral characteristics or other characteristics, set a suitable threshold to extract the independent component close to a certain noise source as a single noise signal. Then, the extracted single noise signal is post-processed, including filtering and smoothing operations, to further remove other residual noise components.

[0018] Step 6: Use the signal-to-noise ratio and distortion index to objectively evaluate the separated single noise signal. If the evaluation index is qualified, proceed to the next step. If not, repeat steps 1 to 5.

[0019] Step 7: Construct a noise signal recognition database. Build a dedicated database for various common external noise interferences and sources. Collect noise signals through dedicated radio equipment. First, pre-process the signals through filtering and framing operations. Then, extract the time domain, frequency domain and other features of each noise signal through signal feature extraction methods, and give corresponding noise types and source labels.

[0020] Step 8: Divide the noise signal recognition database constructed in step 7 into a training data set and a test data set in a ratio of 8:2.

[0021] Step 9: Design and build a corresponding neural network model for noise signal type recognition, complete model training on the training data set constructed in step 8, load the trained model weights and verify the performance of the neural network model on the test data set.

[0022] Step 10: Send each single noise signal separated by noise in step 6 to the neural network model in step 9 to obtain noise information including noise type, source, time domain and frequency domain characteristics to assist in the subsequent noise removal and sound signal enhancement.

[0023] Furthermore, the headphone noise reduction method based on multivariate noise, the intelligent voice noise reduction method is an intelligent noise reduction method based on a multi-stage adaptive adjustment neural network model, and the specific steps include: Step 1: Construct a multi-stage adaptive adjustment neural network model, which includes three parts: prior information control subnetwork, noise intensity estimation subnetwork and denoising network.

[0024] Step 2: Construct a special dataset for intelligent noise reduction. Use special sound signal acquisition equipment to widely collect different noise signals from various common noise sources in a single sound signal environment, and mix them with different effective sound signals to form the input signal of the neural network. The single effective signal without noise is used as the label of the mixed input signal, and constitutes the input of the neural network model in the form of data pairs.

[0025] Step 3: Divide the dedicated dataset constructed in step 2 into training dataset and test dataset in a ratio of 8:2.

[0026] Step 4: Complete model training on the neural network model constructed in step 2 on the training data set, and obtain a highly targeted neural network model for different mixed noise scenarios through model testing on the test data set.

[0027] Step 5: Input the mixed sound signal of mixed external noise and effective sound signal into the system, first use the mixed multivariate noise recognition method to separate and identify the noise type, and give the noise type, spectral characteristics and other information into the neural network model, the neural network model's prior information control subnetwork matches the neural network model weights obtained in the most similar training scenario according to the above prior information; the noise intensity estimation subnetwork estimates the noise amplitude and phase information, and guides the denoising subnetwork to remove noise interference while retaining as much effective information as possible; the denoising subnetwork is responsible for removing noise interference and obtaining pure effective information.

[0028] Furthermore, the headphone noise reduction method based on multivariate noise and the intelligent speech enhancement method are supplementary to the intelligent speech noise reduction method, and adopt the intelligent speech enhancement method assisted by deep learning, which is mainly used to remove mixed sound and echo and improve the clarity or loudness of effective sound signals.

[0029] Furthermore, the headphone noise reduction method based on multivariate noise and the deep learning-assisted intelligent speech enhancement method are intelligent speech enhancement methods that combine fixed beamforming and generative adversarial network models.

[0030] On the other hand, a headphone noise reduction system based on multivariate noise is provided and applied to any one of the headphone noise reduction methods based on multivariate noise. The headphone noise reduction system based on multivariate noise adopts a three-level "cloud-edge-end" architecture, specifically including: Multi-array microphone: used to collect the user's effective voice signals and signals from different noise sources.

[0031] Noise reduction mode control system: The noise reduction mode control system is built into the user terminal, and function access and control can be achieved through the user terminal interface. Users can use the noise reduction mode control system to select the state of the headset or the functional tasks they want the Bluetooth headset to perform, including sports mode, video watching mode, call mode and audio loading mode. Different neural network model weights are used in different modes to perform noise reduction tasks. You can also select standard noise reduction mode, comfortable noise reduction mode and enhanced noise reduction mode according to the current noise reduction effect of the headset.

[0032] Wireless headset noise reduction system: It belongs to the end architecture in the "cloud-edge-end" three-level architecture, is closest to the user side, corresponds to the standard noise reduction mode, and is built into the intelligent noise reduction module of wireless Bluetooth headsets. The AI ​​chip provides the smaller computing power support required for the algorithm to run. The internal model weights are pre-trained in some scenarios and can be used to achieve dynamic and precise intelligent noise reduction based on real-time noise changes under normal conditions.

[0033] User terminal headphone noise reduction system: It belongs to the edge architecture in the end architecture in the "cloud-edge-end" three-level architecture, corresponding to the comfortable noise reduction mode, and is built into the user terminal intelligent noise reduction module to which the headphone is connected. It is equipped with a computing power support module with higher computing power than the wireless headphone noise reduction system. The same pre-trained model weights as the wireless headphone noise reduction system are set inside. When the user's environment is noisy and the standard noise reduction mode cannot effectively reduce noise, the comfortable noise reduction mode can be activated through the user terminal headphone noise reduction system. The user terminal headphone noise reduction system will compile the real-time noise data and sound signal data collected into a data set, and fine-tune the built-in neural network model on the actual data to better adapt to the current noise environment and achieve more thorough noise removal.

[0034] Data Computing Power Cloud Service Center: It belongs to the edge architecture in the end architecture in the "cloud-edge-end" three-level architecture. It corresponds to the enhanced noise reduction mode, is equipped with high-computing power computing equipment, is built into the operator's data operation and processing center, and realizes information interaction with user terminals and headphones through the cloud transmission architecture; when the user is in a more complex noise situation, high noise intensity or a completely new noise environment, and neither the standard mode nor the comfort mode can achieve effective noise reduction, the user terminal will send the collected mixed sound signals and effective sound signals to the data computing power cloud service center through the cloud transmission architecture for processing; with the support of the high-computing power equipment of the data computing power cloud service center, the model will be adjusted or retrained, so that the model can effectively process complex multivariate mixed noise signals, and the model weights saved after training will be transmitted back to the user terminal through the cloud transmission architecture to achieve efficient and reliable noise reduction in complex noise environments.

[0035] Cloud transmission architecture: Design a composite information transmission system that combines wired and wireless. The wired transmission part uses optical fiber as the main signal transmission carrier to achieve high-speed information transmission over short distances; the wireless transmission part uses millimeter waves and relay base stations as signal transmission carriers. Relay base stations are equipped with large-scale antenna arrays to achieve long-distance, large-capacity and efficient information transmission.

[0036] Speech enhancement system: Located between the headphone noise reduction module and the output module, it performs final processing on the noise-reduced sound signal, eliminates mixed sound and echo interference, and outputs high-fidelity sound quality sound signals.

[0037] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention decomposes multivariate mixed noise into a single interference noise signal through a hybrid multivariate noise recognition method, and combines intelligent algorithms to accurately identify the source and type of interference noise, as well as time domain characteristics and frequency domain characteristics, thereby greatly improving the denoising performance.

[0038] 2. The present invention realizes adaptive high-precision noise reduction in different scenarios and modes through an intelligent voice noise reduction method based on a multi-stage adaptive adjustment neural network model, thereby improving the noise reduction performance of headphones and user experience.

[0039] 3. The present invention uses an intelligent speech enhancement method as a supplement to the intelligent speech noise reduction method, effectively eliminating mixed sound and echo interference, and further improving the noise reduction performance of headphones.

[0040] 4. The present invention constructs a three-level "cloud-edge-end" architecture and designs three noise reduction modes that users can choose independently to meet the headphone noise reduction needs of different users in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a headphone noise reduction method based on multivariate noise of the present invention; Figure 2 This is a structural diagram of a headphone noise reduction system based on multi-noise according to the present invention; DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0043] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0044] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0045] In an embodiment of the present invention, a headphone noise reduction method based on multivariate noise is provided. Figure 1 , applied to a headphone noise reduction system based on multivariate noise, including a hybrid multivariate noise recognition method, an intelligent speech noise reduction method and an intelligent speech enhancement method.

[0046] The hybrid multivariate noise recognition method adopts a hybrid multivariate noise recognition method that combines traditional acoustic signal analysis and deep learning. It decomposes the mixed multivariate noise into multiple single noise signals, performs feature analysis on the decomposed single noise signals, and finally identifies the category of each single noise through a trained classifier model.

[0047] The intelligent speech noise reduction method is an intelligent noise reduction method based on a multi-stage adaptive adjustment neural network model, and is the core technical part of the headphone noise reduction method based on multivariate noise.

[0048] The intelligent speech enhancement method adopts an intelligent speech enhancement method that combines fixed beamforming and generative adversarial network models, which is mainly used to remove mixed sound and echo and improve the clarity or loudness of effective sound signals.

[0049] See attached Figure 2 The present invention also provides a headphone noise reduction system based on multivariate noise, which adopts a three-level "cloud-edge-end" architecture, including a multi-array microphone, a noise reduction mode control system, a wireless headphone noise reduction system, a user terminal headphone noise reduction system, a data computing power cloud service center, a cloud transmission architecture and a voice enhancement system.

[0050] The multi-array microphone is used to collect the user's effective voice signal and signals from different noise sources.

[0051] The noise reduction mode control system is built into the user terminal, and functional access and control can be achieved through the user terminal interface. The user can use the noise reduction mode control system to select the state of the headset or the functional task the user wants the Bluetooth headset to perform, including sports mode, video watching mode, call mode and audio loading mode. Different neural network model weights are used to perform noise reduction tasks in different modes. The standard noise reduction mode, comfortable noise reduction mode and enhanced noise reduction mode can also be selected according to the current noise reduction effect of the headset.

[0052] The wireless headset noise reduction system belongs to the end architecture in the "cloud-edge-end" three-level architecture. It is closest to the user side and corresponds to the standard noise reduction mode. It is built into the intelligent noise reduction module of the wireless Bluetooth headset. The AI ​​chip provides the smaller computing power support required for the algorithm to run. The internal model weights are pre-trained in some scenarios and can be used to achieve dynamic and precise intelligent noise reduction based on real-time noise changes in normal environments.

[0053] The user terminal headphone noise reduction system belongs to the edge architecture in the end architecture in the "cloud-edge-end" three-level architecture, corresponding to the comfortable noise reduction mode. It is built into the user terminal intelligent noise reduction module to which the headphone is connected, and is equipped with a computing power support module with higher computing power than the wireless headphone noise reduction system. The internal system is equipped with the same pre-trained model weights as the wireless headphone noise reduction system. When the user's environment is noisy and the standard noise reduction mode cannot effectively reduce noise, the comfortable noise reduction mode can be activated through the user terminal headphone noise reduction system. The user terminal headphone noise reduction system will compile the real-time noise data and sound signal data collected into a data set, and fine-tune the built-in neural network model on the actual data to better adapt to the current noise environment and achieve more thorough noise removal.

[0054] The data computing power cloud service center belongs to the edge architecture of the end architecture in the "cloud-edge-end" three-level architecture. It corresponds to the enhanced noise reduction mode, is equipped with high-computing power computing equipment, and is built into the operator's data operation and processing center. It realizes information interaction with user terminals and headphones through the cloud transmission architecture; when the user is in a more complex noise situation, high noise intensity or a completely new noise environment, and neither the standard mode nor the comfort mode can achieve effective noise reduction, the user terminal will send the collected mixed sound signals and effective sound signals to the data computing power cloud service center through the cloud transmission architecture for processing. With the support of the high-computing power equipment of the data computing power cloud service center, the model will be adjusted or retrained, so that the model can effectively process complex and multivariate mixed noise signals, and the model weights saved after training will be transmitted back to the user terminal through the cloud transmission architecture to achieve efficient and reliable noise reduction in complex noise environments.

[0055] The cloud transmission architecture is designed as a composite information transmission system that combines wired and wireless. The wired transmission part uses optical fiber as the main signal transmission carrier to achieve high-speed information transmission over short distances. The wireless transmission part uses millimeter waves and relay base stations as signal transmission carriers. Relay base stations are equipped with large-scale antenna arrays to achieve long-distance, large-capacity and efficient information transmission.

[0056] The speech enhancement system is located between the headphone noise reduction module and the output module. It performs final processing on the noise-reduced sound signal, eliminates mixed sound and echo interference, and outputs high-fidelity sound quality sound signals.

[0057] In a specific embodiment, the headphone user selects the standard noise reduction mode to watch a movie through the personal terminal interface connected to the headphone in a public environment. When the number of people around gradually increases, or the background noise gradually increases, the user can adjust the headphone noise reduction mode to the comfortable noise reduction mode through the personal terminal. The headphone microphone collects the noise signal in the surrounding environment, and sends the data to the user terminal headphone noise reduction system after preprocessing by the noise reduction system. The user terminal headphone noise reduction system calls the built-in computing power module to fine-tune the neural network model to improve the model's noise recognition and removal capabilities. When the ambient noise intensity is further increased, or the user is in a new noise environment, and the comfortable noise reduction mode cannot effectively remove the noise, the user can further adjust the noise reduction mode to the enhanced mode. The noise data collected by the microphone is transmitted to the operator's data computing power cloud service center through the cloud transmission architecture, and more complex data calculation processing and model training are performed with the support of large computing power equipment. After the training is completed, the model weights are sent to the headphones and user terminals to achieve more powerful noise reduction processing. When the user is in a closed environment and the mixed sound and echo interference are strong, the voice enhancement system filters the mixed sound and echo through the intelligent voice enhancement algorithm and outputs a high-fidelity sound quality sound signal. Example 1

[0058] In one embodiment, the headphone user decomposes the mixed multivariate noise into multiple single noise signals by using a mixed multivariate noise recognition method, performs feature analysis on the decomposed single noise signals, and finally recognizes the category of each single noise by using a trained classifier model, which specifically includes the following steps: Step 1: Collect mixed noise signals from the outside world containing multiple noise sources through a microphone receiving system.

[0059] Step 2: Preprocess the signal. First, perform preliminary filtering on the collected multivariate mixed noise signal, and use a high-pass filter, low-pass filter or band-pass filter to remove signals that are obviously not conventional noise components, such as DC components and high-frequency components. After that, divide the collected continuous multivariate mixed noise signal into shorter frames for subsequent video analysis.

[0060] Step 3: Extract features of the preprocessed noise signal, mainly including time domain features, frequency domain features and other features. Extract the time domain features of each frame signal, including short-time energy, short-time zero-crossing rate, and autocorrelation function. These features can reflect the temporal structure of the noise signal; perform Fourier transform on each frame signal to extract frequency domain features, including spectrum and power spectrum features. These features can reflect the frequency distribution of the noise signal; according to different actual application scenarios, other acoustic signal features including cepstrum features and MFCC (Mel frequency cepstrum coefficients) can be extracted.

[0061] Step 4: Perform noise source separation on the multivariate mixed noise signal. First, perform blind source separation. Use blind source separation technology to decompose the mixed noise signal into multiple independent components. These independent components correspond to different noise sources. The specific steps include: Centralization: For each segment of multivariate mixed noise signal Calculate its mean , and then all mixed sound signals are decentralized, that is , through the centering operation, the mean of the mixed sound signal is ensured to be zero, preparing the data for subsequent processing operations.

[0062] Whitening: The input data is converted into a new data set by eliminating the covariance of the data. The variables in the new data set are independent of each other and have the same variance, which simplifies the subsequent processing operations. First, calculate the covariance matrix of the decentralized acoustic signal data:

[0063] Then perform eigenvalue decomposition on the covariance matrix:

[0064] in D is a diagonal matrix of eigenvalues, E is the corresponding eigenvector matrix. Finally, the data is transformed using the eigenvalues ​​and eigenvectors to obtain the whitened data.

[0065] Find the maximum non-Gaussian direction: The core of this step is to use the principle of maximizing non-Gaussianity to extract independent components. First, a random unit vector w is selected as the initial value of the weight vector, and then the weight vector w is updated according to the following rules until convergence:

[0066] in, is a nonlinear function used to capture non-Gaussianity, is its derivative. After updating, we need to Normalization is performed. If multiple independent components are extracted, the new weight vectors need to be orthogonalized to ensure that they are independent of each other.

[0067] Calculate independent components: Use the final weights The vector projects the whitened data onto the independent components, namely:

[0068] Repeat the steps of finding the maximum non-Gaussian direction and calculating independent components to extract more independent components, and each time ensure that the new weight vector is orthogonal (or independent) to the previously extracted independent components.

[0069] Then, the spectrum analysis method is used to perform spectrum analysis on the separated independent components and observe the performance of each component in the frequency domain. According to the spectrum characteristics, it can be preliminarily determined whether each component corresponds to a different noise source. Finally, according to the spectrum analysis results, the parameters of blind source separation are adjusted through iterative optimization to make the separated independent components closer to the real noise source.

[0070] Step 5: Single noise signal extraction. According to the spectral characteristics or other characteristics, set a suitable threshold to extract the independent component close to a certain noise source as a single noise signal. Then, the extracted single noise signal is post-processed, including filtering and smoothing operations, to further remove other residual noise components.

[0071] Step 6: Use the signal-to-noise ratio and distortion index to objectively evaluate the separated single noise signal. If the evaluation index is qualified, proceed to the next step. If not, repeat steps 1 to 5.

[0072] Step 7: Construct a noise signal recognition database. Build a dedicated database for various common external noise interferences and sources. Collect noise signals through dedicated radio equipment. First, pre-process the signals through filtering and framing operations. Then, extract the time domain, frequency domain and other features of each noise signal through signal feature extraction methods, and give corresponding noise types and source labels.

[0073] Step 8: Divide the noise signal recognition database constructed in step 7 into a training data set and a test data set in a ratio of 8:2.

[0074] Step nine: Design and build a corresponding noise signal type recognition neural network model. The present invention completes model training on the training data set constructed in step eight, and loads the trained model weights to verify the performance of the neural network model on the test data set.

[0075] Step 10: Send each single noise signal separated by noise in step 6 to the neural network model in step 9 to obtain noise information including noise type, source, time domain and frequency domain characteristics to assist in the subsequent noise removal and sound signal enhancement. Example 2

[0076] In one embodiment, when the headset user is in a noisy environment, the noise reduction mode can be turned on through the user terminal noise reduction mode control system. At this time, the intelligent voice noise reduction method built into the headset, the user terminal and the operator cloud data center starts the working mode. The intelligent voice noise reduction method is an intelligent noise reduction method based on a multi-stage adaptive adjustment neural network model. The specific steps include: Step 1: Construct a multi-stage adaptive adjustment neural network model, which includes three parts: prior information control subnetwork, noise intensity estimation subnetwork and denoising network. Prior information control subnetwork Step 2: Construct a special dataset for intelligent noise reduction. Use special sound signal acquisition equipment to widely collect different noise signals from various common noise sources in a single sound signal environment, and mix them with different effective sound signals to form the input signal of the neural network. The single effective signal without noise is used as the label of the mixed input signal, and constitutes the input of the neural network model in the form of data pairs.

[0077] Step 3: Divide the dedicated dataset constructed in step 2 into training dataset and test dataset in a ratio of 8:2.

[0078] Step 4: Complete model training on the neural network model constructed in step 2 on the training data set, and obtain a highly targeted neural network model for different mixed noise scenarios through model testing on the test data set.

[0079] Step 5: Input the mixed sound signal of mixed external noise and effective sound signal into the system, first use the mixed multivariate noise recognition method to separate and identify the noise type, and give the noise type, spectral characteristics and other information into the neural network model, the neural network model's prior information control subnetwork matches the neural network model weights obtained in the most similar training scenario according to the above prior information; the noise intensity estimation subnetwork estimates the noise amplitude and phase information, and guides the denoising subnetwork to remove noise interference while retaining as much effective information as possible; the denoising subnetwork is responsible for removing noise interference and obtaining pure effective information. Example 3

[0080] The intelligent speech noise reduction method can effectively remove background and environmental noise interference, but it cannot remove mixed sound and echo interference when the user is indoors or in other relatively closed environments. Therefore, the user can use the built-in intelligent speech enhancement method of the speech enhancement system to remove mixed sound and echo interference and obtain pure sound output. The intelligent speech enhancement method adopts a deep learning-assisted intelligent speech enhancement method, specifically, an intelligent speech enhancement method that combines fixed beamforming and generative adversarial network models. The specific steps include: Step 1: Modeling the speech enhancement problem, assuming that mixed sound and echo interference are additive interference, and the mixed interference speech signal Clean speech signal and interfering sound signals The specific time domain expression is:

[0081] By performing discrete Fourier transform on the time domain signal, we can get the frequency domain expression of the mixed interference sound signal model:

[0082] in , , , respectively, are the time domain signals , , The complex expression of the frequency domain signal obtained by discrete Fourier transform, t is the time index, f is the frequency band index. From the above formula, we can get the power spectrum representation of the noisy speech signal:

[0083] in Represents the phase angle between two signals. In general, speech enhancement algorithms assume that speech and noise are independent of each other. Based on this assumption, we have , then the power spectrum of the noisy speech signal is expressed as:

[0084] Step 2: Speech enhancement technology based on fixed beamforming algorithm. This technology is based on microphone array. The purpose of beamforming is achieved by fixing the weight of the filter. The properties of the formed beam are related to the shape and direction of the microphone array. The specific operations are as follows: Assume that is the output signal of the microphone in the microphone array, is the corresponding filter weight, and by summing the microphone signals after filter processing, the output of the system can be obtained as:

[0085] in, F represents the number of microphones, Representative i The delay caused by the microphone. Set to constant When , the system output can be obtained as:

[0086] In addition, when Set to 1 / F When , the system output can be obtained as:

[0087]

[0088] From the above formula, we can see that i The output signal of the microphone Can be split into target speech signal and interference signals , which shows that the fixed beamformer can separate incoherent noise, thereby achieving the effect of noise suppression.

[0089] Step 3: Speech enhancement technology based on generative adversarial networks. This technology is a refinement and supplement of the speech enhancement technology based on the fixed beamforming algorithm. The specific steps are as follows: Construct training and test databases: The database contains pure sound signals and mixed signals with superimposed mixed sounds and noise interference. The pure signals are used as labels and the mixed signals are used as neural network inputs. The data set is divided into training set and test set in an 8:2 ratio.

[0090] Training the neural network: Construct a generative adversarial network model and train the model on the training data set; the generative adversarial network consists of a generator network and a discriminator network. The generator network generates an enhanced spectrogram by enhancing the input mixed sound signal spectrogram. The discriminator network attempts to classify the spectrogram as a clean spectrogram or an enhanced spectrogram under the condition of the corresponding noise spectrum. The training process uses the adversarial loss function as the objective function, and the expression function is:

[0091] in, is the discriminator function, is the generator function. According to the result of the discriminator, the generator G Try to adjust the distribution to produce better outputs until the discriminator cannot distinguish whether the output is real or fake.

[0092] Test the neural network: Perform a performance test on the trained neural network model on the test set.

[0093] Deploy neural network: The voice signal enhanced by the speech enhancement technology based on the fixed beamforming algorithm is input into the tested neural network model for further speech enhancement, eliminating mixed sound and echo interference, and improving the quality and purity of the output signal.

[0094] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0095] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A headphone noise reduction method based on multivariate noise, comprising a hybrid multivariate noise recognition method, an intelligent speech noise reduction method and an intelligent speech enhancement method, characterized in that The mixed multivariate noise recognition method adopts a mixed multivariate noise recognition method that combines traditional sound signal analysis and deep learning, decomposes the mixed multivariate noise into multiple single noise signals, and performs feature analysis on the decomposed single noise signals, and finally identifies the category of each single noise through a trained classifier model; the intelligent speech noise reduction method is an intelligent noise reduction method based on a multi-stage adaptive adjustment neural network model, which is the core technical part of the headphone noise reduction method based on multivariate noise; the intelligent speech enhancement method adopts an intelligent speech enhancement method that combines fixed beamforming and generative adversarial network models, which is mainly used to remove mixed sound and echo and improve the clarity or loudness of effective sound signals.

2. The headphone noise reduction method based on multivariate noise according to claim 1, characterized in that The hybrid multivariate noise recognition method specifically comprises the following steps: Step 1: Collect mixed noise signals from the outside world including multiple noise sources through a microphone sound receiving system; Step 2: Preprocess the signal. First, perform preliminary filtering on the collected multivariate mixed noise signal, and use a high-pass filter, a low-pass filter or a band-pass filter to remove signals that are obviously not conventional noise components, such as DC components and high-frequency components; then, divide the collected continuous multivariate mixed noise signal into shorter frames for subsequent video analysis; Step 3: Extract features of the pre-processed noise signal, mainly including time domain features, frequency domain features and other features. Extract the time domain features of each frame signal, including short-time energy, short-time zero-crossing rate, and autocorrelation function. These features can reflect the time structure of the noise signal; perform Fourier transform on each frame signal to extract frequency domain features, including spectrum and power spectrum features. These features can reflect the frequency distribution of the noise signal; according to different actual application scenarios, other acoustic signal features including cepstrum features and MFCC (Mel frequency cepstrum coefficients) can be extracted; Step 4: Perform noise source separation on the multivariate mixed noise signal. First, perform blind source separation. Use blind source separation technology to decompose the mixed noise signal into multiple independent components. These independent components correspond to different noise sources. The specific steps include: Centralization: For each segment of multivariate mixed noise signal Calculate its mean , and then all mixed sound signals are decentralized, that is , through the centering operation, ensure that the mean of the mixed sound signal is zero, and prepare data for subsequent processing operations; Whitening: The input data is converted into a new data set by eliminating the covariance of the data. The variables in the new data set are independent of each other and have the same variance, which simplifies the subsequent processing operations. First, calculate the covariance matrix of the decentralized acoustic signal data: ;Then perform eigenvalue decomposition on the covariance matrix: ;in D is a diagonal matrix of eigenvalues, E is the corresponding eigenvector matrix. Finally, the data is transformed using the eigenvalues ​​and eigenvectors to obtain the whitened data; Find the maximum non-Gaussian direction: The core of this step is to use the principle of maximizing non-Gaussianity to extract independent components. First, a random unit vector w is selected as the initial value of the weight vector, and then the weight vector w is updated according to the following rules until convergence: ;in, is a nonlinear function used to capture non-Gaussianity, is its derivative. After updating, we need to Normalize. If multiple independent components are extracted, the new weight vectors need to be orthogonalized to ensure that they are independent of each other. Calculate independent components: Use the final weights The vector projects the whitened data onto the independent components, namely: ; Repeat the steps of finding the maximum non-Gaussian direction and calculating independent components to extract more independent components. Each time you extract, make sure that the new weight vector is orthogonal (or independent) to the previously extracted independent components. Then, the spectrum analysis method is used to perform spectrum analysis on the separated independent components to observe the performance of each component in the frequency domain. According to the spectrum characteristics, it can be preliminarily judged whether each component corresponds to a different noise source. Finally, according to the spectrum analysis results, the parameters of blind source separation are adjusted through iterative optimization to make the separated independent components closer to the real noise source. Step 5: Extraction of single noise signal: according to the spectrum characteristics or other characteristics, set a suitable threshold value to extract the independent component close to a certain noise source as a single noise signal, and then post-process the extracted single noise signal, including filtering and smoothing operations, to further remove other residual noise components; Step 6: Use the signal-to-noise ratio and distortion index to objectively evaluate the separated single noise signal. If the evaluation index is qualified, proceed to the next step. If it is not qualified, repeat steps 1 to 5. Step 7: Build a noise signal recognition database. Build a dedicated database for various common external noise interferences and sources. Collect noise signals through dedicated radio equipment. First, pre-process the signals through filtering and framing operations. Then, extract the time domain, frequency domain and other features of each noise signal through signal feature extraction methods, and give the corresponding noise type and source labels. Step 8: Divide the noise signal recognition database constructed in step 7 into a training data set and a test data set in a ratio of 8:2; Step nine: design and build a corresponding noise signal type recognition neural network model. The present invention uses the neural network model to complete model training on the training data set constructed in step eight, and loads the trained model weights to perform performance verification of the neural network model on the test data set; Step 10: Send each single noise signal separated by noise in step 6 to the neural network model in step 9 to obtain noise information including noise type, source, time domain and frequency domain characteristics to assist in the subsequent noise removal and sound signal enhancement.

3. The headphone noise reduction method based on multivariate noise according to claim 1, characterized in that The specific steps of the intelligent voice noise reduction method include: Step 1: Construct a multi-stage adaptive adjustment neural network model, which includes three parts: prior information control sub-network, noise intensity estimation sub-network and denoising network; Step 2: Construct a special data set for intelligent noise reduction. Use a special sound signal acquisition device to widely collect different noise signals from various common noise sources in a single sound signal environment, and mix them with different effective sound signals to form the input signal of the neural network. The single effective signal without noise is used as the label of the mixed input signal, and constitutes the input of the neural network model in the form of data pairs. Step 3: Divide the dedicated dataset constructed in step 2 into training dataset and test dataset in a ratio of 8:2; Step 4: Complete model training of the neural network model constructed in step 2 on the training data set, and obtain a highly targeted neural network model for different mixed noise scenarios through model testing on the test data set; Step 5: Input the mixed sound signal of mixed external noise and effective sound signal into the system, first use the mixed multivariate noise recognition method to separate and identify the noise type, and give the noise type, spectral characteristics and other information into the neural network model, the neural network model's prior information control subnetwork matches the neural network model weights obtained in the most similar training scenario according to the above prior information; the noise intensity estimation subnetwork estimates the noise amplitude and phase information, and guides the denoising subnetwork to remove noise interference while retaining as much effective information as possible; the denoising subnetwork is responsible for removing noise interference and obtaining pure effective information.

4. The headphone noise reduction method based on multivariate noise according to claim 1, characterized in that The specific steps of the intelligent speech enhancement method include: Step 1: Modeling the speech enhancement problem, assuming that mixed sound and echo interference are additive interference, and the mixed interference speech signal Clean speech signal and interfering sound signals The specific time domain expression is: ; Perform discrete Fourier transform on the time domain signal to obtain the frequency domain expression of the mixed interference sound signal model: ;in , , , respectively, are the time domain signals , , The complex expression of the frequency domain signal obtained by discrete Fourier transform, t is the time index, f is the frequency band index; From the above formula, we can get the power spectrum representation of the noisy speech signal: ;in Represents the phase angle between two signals. In general, speech enhancement algorithms assume that speech and noise are independent of each other. Based on this assumption, we have , then the power spectrum of the noisy speech signal is expressed as: ; Step 2: Speech enhancement technology based on fixed beamforming algorithm. This technology is a speech enhancement technology based on microphone array. The purpose of beamforming is achieved by fixing the weight of the filter, and the properties of the formed beam are related to the shape and direction of the microphone array. The specific operations are: Assume that is the output signal of the microphone in the microphone array, is the corresponding filter weight, and by summing the microphone signals after filter processing, the output of the system can be obtained as: ;in, F represents the number of microphones, Representative i The delay caused by the microphone; when Set to constant When , the output of the system can be obtained as: In addition, when Set to 1 / F When , the output of the system can be obtained as: ； ； From the above formula, we can see that i The output signal of the microphone Can be split into target speech signal and interference signals , which shows that the fixed beamformer can separate the incoherent noise, thus achieving the effect of noise suppression; Step 3: Speech enhancement technology based on generative adversarial networks. This technology is a refinement and supplement of the speech enhancement technology based on the fixed beamforming algorithm. The specific steps are as follows: Build training and test databases: The database contains pure sound signals and mixed signals with superimposed mixed sounds and noise interference. The pure signals are used as labels and the mixed signals are used as neural network inputs. The data set is divided into training set and test set in an 8:2 ratio. Training the neural network: Construct a generative adversarial network model and train the model on the training data set; the generative adversarial network consists of a generator network and a discriminator network. The generator network generates an enhanced spectrogram by enhancing the input mixed sound signal spectrogram. The discriminator network attempts to classify the spectrogram as a clean spectrogram or an enhanced spectrogram under the condition of the corresponding noise spectrum. The training process uses the adversarial loss function as the objective function, and the expression function is: ;in, is the discriminator function, is the generator function. According to the result of the discriminator, the generator G Try to adjust the distribution to produce better outputs until the discriminator cannot distinguish whether the output is real or fake; Test the neural network: perform performance tests on the trained neural network model on the test set; Deploy neural network: The voice signal enhanced by the speech enhancement technology based on the fixed beamforming algorithm is input into the tested neural network model for further speech enhancement, eliminating mixed sound and echo interference, and improving the quality and purity of the output signal.

5. A headphone noise reduction system based on multivariate noise, characterized in that: A headphone noise reduction method based on multi-noise as described in any one of claims 1 to 4 is used to implement a headphone noise reduction system based on multi-noise, and a "cloud-edge-end" three-level architecture is adopted, including a multi-array microphone, a noise reduction mode control system, a wireless headphone noise reduction system, a user terminal headphone noise reduction system, a data computing power cloud service center, a cloud transmission architecture and a voice enhancement system.

6. The headphone noise reduction system based on multivariate noise according to claim 5, characterized in that: The noise reduction mode control system is built into the user terminal, and functional access and control can be achieved through the user terminal interface. The user can use the noise reduction mode control system to select the state of the headset or the functional task the user wants the Bluetooth headset to perform, including sports mode, video watching mode, call mode and audio loading mode. Different neural network model weights are used to perform noise reduction tasks in different modes. The standard noise reduction mode, comfortable noise reduction mode and enhanced noise reduction mode can also be selected according to the current noise reduction effect of the headset.

7. The headphone noise reduction system based on multivariate noise according to claim 5, characterized in that: The wireless headset noise reduction system belongs to the end architecture in the "cloud-edge-end" three-level architecture, which is closest to the user side and corresponds to the standard noise reduction mode. It is built into the intelligent noise reduction module of the wireless Bluetooth headset. The AI ​​chip provides the smaller computing power support required for the algorithm to run. The internal model weights are pre-trained in some scenarios and can be used to achieve dynamic and precise intelligent noise reduction according to real-time noise changes in normal environments.

8. The headphone noise reduction system based on multivariate noise according to claim 5, characterized in that: The user terminal headphone noise reduction system belongs to the edge architecture in the end architecture in the "cloud-edge-end" three-level architecture, corresponding to the comfortable noise reduction mode, and is built into the user terminal intelligent noise reduction module to which the headphone is connected. It is equipped with a computing power support module with higher computing power than the wireless headphone noise reduction system. The internal system is equipped with the same pre-trained model weights as the wireless headphone noise reduction system. When the user's environment is noisy and the standard noise reduction mode cannot effectively reduce noise, the comfortable noise reduction mode can be activated through the user terminal headphone noise reduction system. The user terminal headphone noise reduction system will make the real-time noise data and sound signal data collected into a data set, and fine-tune the built-in neural network model on the actual data to better adapt to the current noise environment and achieve more thorough noise removal.

9. The headphone noise reduction system based on multivariate noise according to claim 5, characterized in that: The data computing power cloud service center belongs to the edge architecture of the end architecture in the "cloud-edge-end" three-level architecture, corresponds to the enhanced noise reduction mode, is equipped with high-computing computing equipment, is built-in into the operator's data operation and processing center, and realizes information interaction with the user terminal and headphones through the cloud transmission architecture; when the user is in a relatively complex noise situation, a high noise intensity, or a completely new noise environment, and both the standard mode and the comfort mode cannot achieve effective noise reduction, the user terminal sends the collected mixed sound signals and effective sound signals to the data computing power cloud service center through the cloud transmission architecture for processing, and adjusts or retrains the model with the support of the high-computing power equipment of the data computing power cloud service center, so that the model can effectively process complex multivariate mixed noise signals, and transmits the model weights saved after the training back to the user terminal through the cloud transmission architecture to achieve efficient and reliable noise reduction in a complex noise environment; The cloud transmission architecture is designed as a composite information transmission system that combines wired and wireless. The wired transmission part uses optical fiber as the main signal transmission carrier to achieve high-speed information transmission over short distances. The wireless transmission part uses millimeter waves and relay base stations as signal transmission carriers. Relay base stations are equipped with large-scale antenna arrays to achieve long-distance, large-capacity and efficient information transmission.

10. The headphone noise reduction system based on multivariate noise according to claim 5, characterized in that: The speech enhancement system is located between the headphone noise reduction module and the output module, performs final processing on the noise-reduced sound signal, eliminates mixed sound and echo interference, and outputs a high-fidelity sound quality sound signal.

Images