Multi-modal human voice enhancement method for low-delay Bluetooth headset

Abstract

The invention relates to the technical field of multi-modal human voice enhancement methods, in particular to a multi-modal human voice enhancement method for a low-delay Bluetooth headset, and the method comprises the following steps: S1, collecting a bone conduction signal of the headset when a user speaks through a microphone, and recording environment sound and user voice at the same time; s2, data preprocessing; s3, a deep learning module based on end-to-end; s4, based on a signal separation module; s5, deep learning and signal separation module fusion; a thought of noise reduction and filtering is adopted, after noise components are extracted in a signal separation algorithm, the noise components in signal separation and a human voice part enhanced by a deep learning module in the previous step are subtracted from an input signal to obtain a final human voice signal, the final human voice signal is used for recovering a target signal and suppressing background environment noise, and a multi-modal signal is fused to obtain a final human voice signal. The target voice capturing capability of the system is improved, and the interference of environmental noise is effectively reduced. While the speech enhancement effect is improved, low-delay processing is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal voice enhancement methods, and particularly relates to a multi-modal voice enhancement method for a low-latency Bluetooth earphone. BACKGROUND

[0002] In recent years, with the rapid development of mobile communication technology, Bluetooth earphones have become an indispensable smart device in people's daily life. However, when using Bluetooth earphones to talk in a noisy environment, users often face problems such as unclear speech and serious background noise interference. In order to solve these problems, the industry has been actively exploring various noise reduction and speech enhancement technologies.

[0003] Traditional Bluetooth earphone noise reduction technologies mainly rely on single-channel signal processing methods such as spectral subtraction and Wiener filtering. Although these methods can suppress background noise to some extent, they often introduce speech distortion, resulting in unnatural sound heard by the user. In addition, these methods are not effective in dealing with non-stationary noise (such as burst noise), making it difficult to meet the needs of complex real-world environments.

[0004] With the rise of deep learning technology, some neural network-based speech enhancement methods have been proposed and applied to Bluetooth earphones. These methods can better adapt to various noise environments through large amounts of data training and have made some progress in improving speech quality. However, these methods still have some limitations. First, they mainly rely on signals collected by air conduction microphones, which are easily disturbed by environmental noise. Second, complex neural network models often bring about high computational complexity, resulting in increased processing delay and affecting real-time call experience.

[0005] In addition, existing Bluetooth earphone speech enhancement technologies mostly use a single signal source, which cannot fully utilize multi-modal information. This single information source processing method limits the performance ceiling of the system, making it difficult to achieve a good balance between noise suppression and speech fidelity.

[0006] In the face of these technical challenges, the industry urgently needs a new Bluetooth earphone speech enhancement method that can effectively improve speech quality, reduce processing delay, and adapt to complex noise environments. The present application was born in such a background and aims to solve the above-mentioned problems in the prior art. SUMMARY

[0007] The present application is a kind of low delay Bluetooth headset multi-modal voice enhancement method proposed for the above technical problems, by innovatively combining bone conduction signal and air conduction signal, and using advanced deep learning model and signal separation technology, it realizes the double breakthrough of speech quality and processing delay. This method can not only effectively suppress various complex environmental noise, but also maintain the naturalness and clarity of speech, while controlling the processing delay at a very low level.

[0008] The present application proposes a kind of low delay Bluetooth headset multi-modal voice enhancement method, which comprises the following steps:

[0009] S1: microphone collects bone conduction signal of earphone when user speaks, and records environmental sound and user voice;

[0010] S2: data preprocessing; all collected bone conduction signals and recorded sounds are preprocessed, such as framing and windowing, and the bone conduction signals without speech and the recorded environmental sounds are divided for training and testing of deep learning model;

[0011] S3: end-to-end deep learning module; a multi-path deep learning model is proposed, which takes preprocessed bone conduction signal and recorded recording as input and directly outputs enhanced user voice for improving voice clarity;

[0012] S4: signal separation module; an effective signal separation algorithm is proposed, which differentiates the input signal and the extracted mixed noise, effectively realizes noise suppression, and is used to exclude noise signal interference;

[0013] S5: deep learning and signal separation module fusion; the idea of denoising first and then filtering is adopted, after the noise component is extracted in the signal separation algorithm, the input signal is subtracted from the noise component in the signal separation and the enhanced voice part in the previous step deep learning module, which is the final voice signal, used to restore the target signal and suppress background environmental noise;

[0014] The microphone, bone conduction microphone and earphone recording end in step S1 collect environmental noise and user voice as follows: adjust the connection mode of bone conduction microphone and earphone recording end, place the bone conduction signal collection end above the earphone and microphone in the metal shielding box, place the recording system beside the earphone in the metal shielding box, and collect the recorded voice by simulating the user's use scene, and the recorded user voice is used as reference signal.

[0015] Preferably, the data preprocessing process of collected information in step S2 is as follows:

[0016] The collected information is divided into two paths, one is the bone conduction signal, and the other is the external recorded environmental sound signal, the user's voice part is intercepted, and the bone conduction signal and the environmental noise sound obtained by external recording are preprocessed respectively;

[0017] First, the collected signal is divided into multiple frames using an overlapping frame method, then each frame is processed using a windowing method, the next frame is spliced to the tail of the window, and finally a new segment is obtained as the model input;

[0018] Meanwhile, the signal segments without user speech and noise segments are deleted, wherein the segment length needs to be adjusted according to the model hyperparameters.

[0019] Preferably, in step S2, the data preprocessing and the method of noise reduction and filtering in step S5 are as follows: different weight factors are added to the sound signal and the noise signal during superposition, and the L1 norm is used for weighted superposition, wherein the formula for weighted superposition is:

[0020] Y = a * S + b * N

[0021] Where Y is the weighted superimposed signal, S is the sound signal, N is the noise signal, a and b are weight factors, and a + b = 1.

[0022] The L1 norm calculation formula is:

[0023] ||Y||1 = å |Y i |

[0024] Where ||Y||1 represents the L1 norm of Y, Y i is the i-th element in Y.

[0025] Preferably, step S3 includes the following process, the structure of the fused multi-path signal includes a multi-path signal input module, multiple deep learning paths and a fusion module;

[0026] The multiple deep learning models use different loss function parameters for adjustment, the model is evaluated according to the multi-path input, and the appropriate model is selected according to the experimental results, and the feature vectors output by the multi-path model are fused through the fusion module.

[0027] Wherein the model is a human voice enhancement model based on an Encoder-Decoder structure, the input signal is first processed by an encoder for feature extraction, then through a decoder for information recovery, and finally the human voice enhanced speech is obtained.

[0028] Preferably, the encoder and decoder in step S3 use a Transformer structure based on a self-attention mechanism to extract the features of the original signal and re-encode the input data. The input signal is encoded by the encoder in the Transformer and then decoded by the decoder to obtain the enhanced vocal signal. The specific steps are as follows:

[0029] S21: The original signal data obtained in step S2 is input into the Transformer encoder of each path to obtain the feature information of the data:

[0030] S22: After obtaining the enhancement information, the output of the encoder is spliced and then input into the fusion module for re-encoding and feature information extraction;

[0031] S23: The results of each enhancement path are fused to form the final output using a multi-path encoding and single-path decoding method;

[0032] S24: The encoder and decoder extract the signal features and re-encode them;

[0033] The multiple deep learning models use different loss functions. In the encoder, an L1 norm loss function is used to linearly map the input data, calculate the error between the encoded path features and the target features, and compare the features with the target features to obtain a set of prediction vectors through supervised learning. The prediction vectors are scaling factors in the self-attention mechanism of the Transformer;

[0034] In the decoder, a signal reconstruction error loss function is used, and an L1 norm loss function is used for further feature compensation. The prediction results of the decoder are compared with the reference signal to obtain the difference between the predicted value and the true value.

[0035] Preferably, the noise signal separation module in step S4 introduces a noise perception feature encoder and additional training signals into the input noise and target signal. By learning, the features related to the noise are obtained. The additional training signals include pure signals and noise signals, and the encoded signals are differentiated for training to obtain the relevant features of the noise signal. The larger the value of the signal filtering module, the closer the signal is to the noise signal. The output signal is subtracted from the noise signal to obtain the enhanced vocal signal. The specific steps are as follows:

[0036] S31: Obtain the target signal features and the relevant features of the target signal and the noise signal through the training data, and extract the self-attention features of the target signal;

[0037] S32: The target signal feature is compared with the target signal self-attention feature using a signal separation loss function to obtain a mixed signal feature.

[0038] S33: The mixed signal is blocked and input into a signal decomposition to obtain a plurality of small blocks. Each small block is input into a filtering module. After obtaining the signal and noise related features, the final feature is obtained by weighting. After obtaining the feature, the noise related feature is removed to obtain the target signal related feature. Through forward calculation, the expected target signal is obtained to realize feature differentiation coding.

[0039] Preferably, in step S31, the input and output of the training signal are associated with the expected feature of the output, which is associated with the target signal and the self-attention feature of the target signal; the mixed signal feature is associated with the noise related feature, and the target signal and the self-attention feature of the target signal are used to normalize the two features, and the similarity between the features is calculated by L1 norm loss function.

[0040] Preferably, in step S33, after obtaining the mixed signal feature and the noise feature, the noise signal feature is obtained through a convolution layer, which encodes each mixed signal feature and extracts feature information using convolution, then combines the feature information with the noise signal feature, and uses a fully connected layer to obtain the weight of the feature. The weight is used to code the difference feature of the target signal and the self-attention related feature of the target signal, and finally the feature of the output signal is obtained.

[0041] Preferably, for different environmental noise, a convolution operation based on signal separation is proposed, which only retains the noise related feature, encodes through an encoder first, then performs differential training, separates the feature information of the noise signal by weighting, calculates the similarity of the target signal and the target signal feature and normalizes it, then extracts through a convolution layer, combines the mixed signal feature with the noise signal feature, thereby realizing differential enhancement.

[0042] Preferably, the model optimization method in steps S4 and S5 comprises the following steps: optimizing the model to improve the model training effect, the model optimization method is to use gradient descent method to optimize, optimize the loss function, obtain the optimal solution of the minimum loss function by multiple iterations; when the error function is minimum, complete the optimal model training, first calculate the direction of gradient descent, use first-order difference to approximate second-order difference to calculate second-order derivative, determine the sensitivity of error in neural network prediction, get the partial derivative of weight and error function, multiply the result and input it layer by layer, iterate the partial derivative of weight and error, get the optimal solution, the optimal solution is that when the error is minimum, the gradient is 0; the error calculation function is calculated by cross-entropy loss function, the cross-entropy loss function is calculated by the real signal of the model and the output signal of the model; the forward convolution process of the model is divided into convolution calculation of the encoder and the decoder, the encoder and decoder structure uses the Transformer structure based on self-attention mechanism, the input signal and feature mapping first use the multi-head attention mechanism of the encoder to extract information, and the extracted information is used as the decoder.

[0043] The beneficial effects of the present application mainly manifest in the following aspects:

[0044] Specifically, the present application solves the following key technical problems: First, through multi-modal signal fusion, the system's ability to capture target speech is improved, effectively reducing the interference of environmental noise. Second, by using an innovative deep learning structure, the computational complexity is significantly reduced while improving the speech enhancement effect, achieving low-latency processing. Third, by introducing differentiated training and adaptive signal separation technology, the system's adaptability to various complex noise environments is improved.

[0045] The implementation of the present application brings significant technical effects. In terms of speech quality, compared with traditional methods, the present application has achieved significant improvement in key indicators such as PESQ, SNR gain and STOI, with an increase of 18.75%, 50% and 8.24% respectively. This means that users will enjoy a clearer and more natural communication experience when using Bluetooth earphones equipped with this technology in noisy environments. In terms of processing delay, the present application reduces the delay time to 15ms, a 40% reduction compared with traditional methods, effectively solving the delay problem in real-time communication.

[0046] More importantly, the present application opens up a new direction for the development of Bluetooth earphone technology. Through multi-modal information fusion and advanced signal processing technology, the present application demonstrates the possibility of breaking through the limitations of a single signal source, providing a new approach for the development of future intelligent audio devices. This innovation not only improves the user's daily communication experience, but also has the potential to play an important role in remote medical care, intelligent customer service and other fields, promoting the development of voice interaction technology to a higher level.

[0047] Overall, the multimodal voice enhancement method of the present application has shown significant advantages in improving speech quality, reducing processing delay, adapting to complex noise environment, etc., providing a new solution to the long-standing technical bottleneck in the field of Bluetooth earphones. This breakthrough not only meets the needs of users for high-quality call experience, but also injects new impetus for the technological innovation and upgrading of related industries. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The overall method logic diagram of the present application.

[0049] Figure 2 The signal acquisition logic diagram of the present application.

[0050] Figure 3 The data preprocessing logic diagram of the present application.

[0051] Figure 4 The deep learning module processing logic diagram of the present application.

[0052] Figure 5 The signal separation logic diagram of the present application.

[0053] Figure 6 The module fusion logic diagram of the present application. DETAILED DESCRIPTION

[0054] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined object of the invention, the specific embodiments, structures, features and effects thereof are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0056] Embodiment 1

[0057] Reference Figures 1-6 The present application relates to a multimodal voice enhancement method for low-latency Bluetooth earphones. The method aims to improve the speech intelligibility of Bluetooth earphones in noisy environments while maintaining low latency performance. The specific embodiments of the present application will be described in detail below. The method includes the following steps:

[0058] S1: The microphone collects the bone conduction signal of the earphone when the user speaks, and records the environmental sound and user voice at the same time;

[0059] S2: data preprocessing; frame, window, and other preprocessing are performed on all collected bone conduction signals and recorded sounds, and bone conduction signals without speech and recorded environmental sounds are removed for division, for training and testing of a deep learning model;

[0060] S3: end-to-end deep learning module; a multi-path deep learning model is proposed, which takes the preprocessed bone conduction signals and recorded sounds as input and directly outputs enhanced user speech for improving voice clarity;

[0061] S4: signal separation module; an effective signal separation algorithm is proposed, which differentiates the input signal and the extracted mixed noise, effectively realizes noise suppression, and is used to exclude noise signal interference;

[0062] S5: deep learning and signal separation module fusion; the idea of denoising first and then filtering is adopted, after the noise component is extracted in the signal separation algorithm, the input signal is subtracted from the noise component in the signal separation and the enhanced human voice part in the previous step deep learning module, which is the final human voice signal, for restoring the target signal and suppressing the background environmental noise;

[0063] The process of collecting environmental noise and user speech in step S1 is as follows: adjust the connection mode of the bone conduction microphone and the earphone recording end, place the bone conduction signal collection end above the earphone and microphone in the metal shielding box, place the recording system next to the earphone in the metal shielding box, collect the recorded speech by simulating the user's use scene, and record the user's speech as a reference signal.

[0064] This is the core step of the application. First, the bone conduction signal and environmental sound are collected simultaneously by the microphone, which provides a rich information source for subsequent processing. Second, the collected signals are preprocessed, including framing and windowing, which helps to improve the effect of subsequent processing. Then, the preprocessed signals are processed by a deep learning model, which directly outputs enhanced user speech. Next, the noise is further removed by a signal separation module. Finally, the results of the deep learning module and the signal separation module are fused to obtain the final enhanced speech.

[0065] Preferably, the present application adopts a special layout during the signal collection phase. The bone conduction signal collection end is placed directly above the earphone and microphone in the metal shielding box, while the recording system is placed beside the earphone in the metal shielding box. The advantage of this layout is that it can effectively reduce external electromagnetic interference and improve the purity of the signal. For example, in a noisy street environment, a traditional Bluetooth earphone may have difficulty distinguishing the user's voice, while the present application can better capture the user's voice characteristics by simultaneously collecting bone conduction signals and air conduction signals, thereby improving the subsequent processing effect.

[0066] The data preprocessing process of the collected information in step S2 is as follows: the collected information is divided into two paths, one is the bone conduction signal and the other is the externally recorded environmental sound signal, the user's speech part is intercepted, and the bone conduction signal and the externally recorded environmental noise sound are preprocessed respectively; first, the collected signal is divided into multiple frames using an overlapping frame method, then each frame is processed using a windowing method, the next frame is spliced to the tail of the window, and finally a new segment is obtained as the model input; at the same time, the signal segments without user speech and noise segments are deleted, wherein the segment length needs to be adjusted according to the model hyperparameters.

[0067] This is the data preprocessing process. First, the collected information is divided into bone conduction signal and environmental sound signal. Then, the two signals are preprocessed respectively. The key steps of preprocessing include overlapping frame, windowing processing and frame splicing. The advantage of this preprocessing method is that it can effectively reduce the discontinuity between frames and improve the accuracy of subsequent processing.

[0068] In an embodiment of the present application, the frame length of the overlapping frame is usually set to 20-30 milliseconds, and the overlapping part is about 50% of the frame length. Such a setting can ensure the continuity of the signal while also capturing the short-time characteristics of the speech. For example, when the frame length is set to 25 milliseconds and the overlapping part is 12.5 milliseconds, the characteristics of most speech phonemes can be effectively captured, while the smooth transition between frames is ensured.

[0069] In addition, the step of deleting signal segments without user speech and noise segments can significantly improve the efficiency and accuracy of subsequent processing. Generally, we can set an energy threshold, if the energy of a segment is lower than the threshold, it is considered to be a noise segment or a segment without speech. This threshold can be dynamically adjusted according to the actual application scenario, for example, it can be set to a lower value in a quieter environment, while it needs to be set to a higher value in a noisy environment.

[0070] In step S2, the data preprocessing and the method of denoising and filtering in step S5 are as follows: different weight factors are added to the sound signal and noise signal during superposition, and the L1 norm is used for weighted superposition.

[0071] This is an innovative signal processing method. In the data preprocessing and deep learning module fusion stage, the invention introduces the idea of weighted superposition. Specifically, different weights are given to the sound signal and noise signal, and then weighted superposition is performed through L1 norm.

[0072] The formula for weighted superposition is Y = α * S + β * N

[0073] Where Y is the weighted superposition signal, S is the sound signal, N is the noise signal, and α and β are weight factors, and α + β = 1.

[0074] The L1 norm calculation formula is:

[0075] ||Y||1 = ∑|Y i |

[0076] Where ||Y||1 represents the L1 norm of Y, Y i is the i-th element in Y.

[0077] The advantage of this method is that it can better balance the target speech signal and background noise, thereby improving the effect of voice enhancement. For example, when processing street traffic noise, the weight factor can be dynamically adjusted according to the characteristics of the noise. Generally, the value of α is in the range of 0.6-0.8, and the value of β is in the range of 0.2-0.4. Such weight settings can retain most of the speech signal while appropriately introducing some background information, making the enhanced speech sound more natural.

[0078] Step S3 includes the following processes: the structure of the fusion multi-channel signal includes a multi-channel signal input module, multiple deep learning paths, and a fusion module; multiple deep learning models use different loss function parameters for adjustment, and according to the model evaluation of the multi-channel input, the appropriate model is selected according to the experimental effect, and the feature vectors output by the multi-channel model are fused through the fusion module; wherein the model is a human voice enhancement model based on the Encoder-Decoder structure, the input signal is first processed by the encoder for feature extraction, then through the decoder for information recovery, and finally the human voice enhanced speech is obtained.

[0079] This claim describes the specific structure and workflow of the deep learning module of the invention. First, this module uses multi-channel signal input, which means it can process bone conduction signals and environmental sound signals simultaneously. Then, multiple deep learning paths are used to process these signals. Each deep learning path may use different model structures and loss functions, and the advantage of this design is that it can capture different features of the signal.

[0080] In one embodiment of the present application, the deep learning model adopts a human voice enhancement model based on an Encoder-Decoder structure. The advantage of this structure is that it can effectively capture the time-frequency features of the signal, thereby achieving more accurate human voice enhancement. For example, when processing a speech signal containing a music background, the encoder can learn the different characteristics of speech and music, and the decoder can reconstruct a clear speech signal according to these characteristics.

[0081] Preferably, the fusion of multiple deep learning models can adopt a weighted average method. weights=softmax(scores), where scores is the score of each model, which can be determined by the performance on the validation set. Then, the final output can be calculated by output=∑(weights i *model i_output This fusion method can adaptively adjust the importance of different models, thereby achieving good results in different noise environments.

[0082] The encoder and the decoder described in step S3 use a Transformer structure based on a self-attention mechanism to extract the features of the original signal and re-encode the input data. The input signal is encoded by the encoder in the Transformer and then decoded by the decoder to obtain the enhanced human voice. The specific steps are as follows: S21: input the original signal data obtained in step S2 into each path's Transformer encoder to obtain the feature information of the data; S22: after obtaining the enhancement information, concatenate the output of the encoder and input it into the fusion module for re-encoding and feature information extraction; S23: use a multi-path encoding and single-path decoding method to fuse the results of each enhancement path to form the final output; S24: the encoder and the decoder extract the signal features and re-encode them; multiple deep learning models use different loss functions; in the encoder, a L1 norm loss function is used to linearly map the input data, calculate the error between the encoded path features and the target features, and compare the features with the target features to obtain a set of prediction vectors through supervised learning, the prediction vectors being scaling factors in the self-attention mechanism of the Transformer; in the decoder, a signal reconstruction error loss function is used, and a L1 norm loss function is used for further feature compensation, to compare the prediction results of the decoder with the reference signal and obtain the difference between the predicted value and the true value.

[0083] This is the working principle of the encoder and decoder based on the Transformer structure. The core of the Transformer structure is the self-attention mechanism, which can effectively capture long-range dependencies in the signal and significantly improve the performance of voice enhancement.

[0084] Specifically, the multi-head attention mechanism in the Transformer is defined as: MultiHead(Q, K, V) = Concat(head1, …, head h )W O , where Q, K, V are the query, key, and value matrices, respectively, and W Q , W K , W V , W O are learnable parameter matrices, and d k is the dimension of the key vector.

[0085] In practical applications, the number of heads h is usually set to 8 or 16, which allows the model to learn information from different representation subspaces. For example, when dealing with complex environments containing multiple types of noise, different attention heads may focus on different types of noise features, enabling more accurate noise separation and speech enhancement.

[0086] In this invention, the encoder uses the L1 norm loss function, which is defined as: L1(x, y) = ∑|x i -y i |, where x is the predicted value, y is the true value, and i is the sample index. The advantage of the L1 norm loss function is that it is not sensitive to outliers, which is particularly useful when dealing with noisy signals. For example, when a short and high-decibel noise suddenly appears in the environment, the L1 norm loss function can reduce the impact of such abnormal noise on model training.

[0087] The decoder uses the reconstruction error loss function of the signal, and further compensates for the features using the L1 norm loss function. The reconstruction error loss function is defined as: where x is the original signal, is the reconstructed signal, and ||·|| 2 represents the L2 norm. This combined loss function design can ensure the overall quality of the reconstructed signal while also focusing on the restoration of local details.

[0088] The noise signal separation module in step S4 introduces a noise-aware feature encoder and additional training signals into the input noise and target signal, learns the features related to the noise by learning the features related to the noise, the additional training signals include pure signals and noise signals, and differentiates the encoded signals to obtain the features related to the noise signals, which are used to train a signal filtering module, the larger the value of the filtering module, the closer the signal is to the noise signal, and the output signal is subtracted from the noise signal to obtain an enhanced human voice signal, and the specific steps are as follows: S31: obtaining target signal features and target signal and noise signal related features through training data, and extracting target signal self-attention features; S32: comparing target signal features and target signal self-attention features by using a signal separation loss function to obtain mixed signal features; S33: dividing the mixed signal into blocks, inputting the signal into signal decomposition to obtain a plurality of small blocks, inputting each small block into the filtering module, obtaining the final features after weighting the signal and noise related features, obtaining the target signal related features after removing the noise related features, and obtaining the expected target signal through forward calculation to realize feature differential encoding.

[0089] This is the working principle of the noise signal separation module of the present application. The innovation of this module lies in the introduction of a noise-aware feature encoder and additional training signals. The advantage of this design is that it can more accurately identify and separate noise signals, thereby improving the effect of human voice enhancement.

[0090] In an embodiment of the present application, the signal separation loss function is defined as: wherein the signal separation loss function is defined as:

[0091]

[0092] wherein s is the target signal, is the estimated target signal, n is the noise signal, L SDR is the signal distortion ratio loss, L SI-SNR is the scale-invariant signal-to-noise ratio loss, L spec is the spectral loss, and α, β, γ are weight coefficients.

[0093] This composite loss function design can evaluate the signal separation effect from multiple angles. For example, when processing speech containing music background, L SDR can evaluate the overall signal quality, L SI-SNR can focus on the intelligibility of speech, and L spec can ensure that the features of the separated speech in the frequency domain are preserved. Generally, the values of α, β, and γ can be adjusted according to specific application scenarios, for example, in situations where speech intelligibility needs to be particularly focused on, the value of β can be appropriately increased.

[0094] In step S31, the input and output of the training signal are outputted, and the expected characteristics of the output are associated with the target signal and the self-attention characteristics of the target signal; the mixed signal characteristics are associated with the noise-related characteristics, and the two characteristics are normalized using the target signal and the self-attention characteristics of the target signal, and the similarity between the characteristics is calculated by L1 norm to calculate the loss function. The self-attention characteristics of the target signal are innovatively introduced into the training process, which can help the model better understand and capture the internal structure of the target signal.

[0095] Preferably, the present application adopts a batch normalization (Batch Normalization) method to normalize the characteristics. The definition is as follows:

[0096] Wherein, x is the input feature, y is the normalized output, μ B is the batch mean, is the batch variance, γ and β are the learnable scaling and translation parameters, ∈ is a small constant to prevent division by zero. Batch normalization can accelerate the convergence process of the model and play a role in regularization to some extent, which helps to improve the generalization ability of the model.

[0097] The characteristic similarity loss function is defined as: L sin (f s ,f a )=||f s -f a ||1

[0098] Wherein, f s is the target signal feature, f a is the self-attention feature of the target signal, and ||·||1 represents the L1 norm. This loss function can measure the difference between the target signal feature and its self-attention feature, thereby guiding the model to learn more representative features.

[0099] In step S33, after obtaining the mixed signal characteristics and noise characteristics, the noise signal characteristics are obtained through a convolution layer, which encodes each mixed signal characteristic, extracts feature information using convolution, then combines these feature information with noise signal characteristics, and finally uses a fully connected layer to obtain the weight of the feature. The difference feature encoding of the target signal and the self-attention related characteristics of the target signal is obtained using the weight, and finally the feature of the output signal is obtained.

[0100] This is an innovative noise feature extraction method. The present application encodes the mixed signal characteristics through a convolution layer, then combines these characteristics with noise signal characteristics, and finally obtains the weight of the feature through a fully connected layer. The advantage of this method is that it can more accurately identify and separate noise signals.

[0101] In one embodiment of the present application, the operation of the convolutional layer is defined as: F(i,j) = (I*K)(i,j) = ∑ m ∑ n I(i+m,j+n)*K(m,n)

[0102] where I is the input feature map, K is the convolution kernel, F is the output feature map, (i,j) is the coordinate on the feature map, and (m,n) is the coordinate on the convolution kernel. The operation of the fully connected layer is defined as:

[0103] y = W*x + b

[0104] where x is the input vector, W is the weight matrix, b is the bias vector, and y is the output vector. Generally, the size of the convolution kernel is set to 3x3 or 5x5, so that local features can be captured while not introducing excessive computational complexity. The fully connected layer can capture complex nonlinear relationships between features, which helps to improve the expressive power of the model.

[0105] For different environmental noise, a convolution operation based on signal separation is proposed, which only retains the features related to noise. First, the encoder is encoded, then the differential training is carried out, the features of the noise signal are separated by weighting, the similarity between the target signal and the target signal features is calculated and normalized, and then the convolution layer is extracted, the mixed signal features and the noise signal features are combined, so as to realize differential enhancement.

[0106] The signal separation method for different environmental noise. The present application innovatively proposes a convolution operation based on signal separation, which only retains the features related to noise. This method can better adapt to different noise environments.

[0107] Preferably, the loss function of differential training is defined as: L diff = L target +λ*L noise

[0108] , where L target is the reconstruction loss of the target signal, L noise is the reconstruction loss of the noise signal, and λ is the weighting factor. This loss function design can ensure the reconstruction quality of the target signal while also focusing on the reconstruction of the noise signal, thereby improving the adaptability of the model to different types of noise.

[0109] The feature similarity calculation uses cosine similarity:

[0110] where a and b are two feature vectors, · denotes the dot product, and ||·|| denotes the L2 norm. Cosine similarity can effectively measure the directional similarity of two vectors and is not affected by the absolute size of the vectors, which is particularly useful when comparing features of different scales.

[0111] The model optimization method in steps S4 and S5 comprises the following steps: optimizing the model to improve the model training effect, the model optimization method is to use gradient descent method to optimize, optimize loss function, obtain the optimal solution of the minimum loss function by multiple iterations;When the error function is minimum, the optimal model training is completed, first calculate the direction of gradient descent, use first-order difference to approximate second-order difference to calculate second-order derivative, determine the sensitivity of error in neural network prediction, get the partial derivative of weight and error function, multiply the result by layer feedback input, iterate the partial derivative of weight and error, get the optimal solution, the optimal solution is that the gradient is 0 when the error is minimum;The error calculation function is calculated by cross-entropy loss function, the cross-entropy loss function is calculated by model real signal and model output signal;The forward convolution process of the model is divided into convolution calculation of encoder and decoder, the encoder and decoder structure uses the Transformer structure based on self-attention mechanism, the input signal and feature mapping are first used for information extraction by the multi-head attention mechanism of the encoder, and the extracted information is used as the decoder.

[0112] The gradient descent method is used to optimize the loss function, which is a classic and effective optimization method.

[0113] Specifically, the update formula of the gradient descent method is:

[0114] Where θ is the model parameter, α is the learning rate, is the gradient of the loss function J(θ) with respect to θ. In practical applications, the learning rate α is usually set to a small value, such as 0.001 or 0.0001, to ensure that the model can converge stably.

[0115] In an embodiment of the present application, the cross-entropy loss function is defined as:

[0116] Where y i is the true label, is the model prediction output, and i is the sample index. In a binary classification problem, the cross-entropy loss function can be simplified as:

[0117] Where y is the true label (0 or 1), is the probability predicted by the model.

[0118] The advantage of the cross-entropy loss function is that it penalizes incorrect predictions more heavily, which can encourage the model to learn correct predictions more quickly. For example, when distinguishing between speech and background noise, if the model incorrectly identifies a clear speech segment as noise, the cross-entropy loss function will impose a larger penalty, prompting the model to quickly adjust its parameters.

[0119] In summary, the multi-modal voice enhancement method for low-latency Bluetooth earphones proposed in this invention effectively improves the speech intelligibility of Bluetooth earphones in noisy environments while maintaining low latency performance. The innovation of this method lies not only in the fusion of multi-modal signals but also in the organic combination of deep learning and traditional signal processing methods, providing a new approach and direction for voice enhancement technology in Bluetooth earphones.

[0120] For example, in a noisy coffee shop environment, traditional Bluetooth earphones may have difficulty filtering out background music and human voice noise. However, the invention can effectively extract the user's voice by combining bone conduction signals and air conduction signals, and then processing them through a deep learning model and a signal separation module, greatly improving call quality. At the same time, by using an end-to-end deep learning model and an efficient signal processing algorithm, the invention can control the processing delay within 20 milliseconds while ensuring the speech enhancement effect, meeting the real-time call requirements.

[0121] In addition, the method of the invention has good adaptability and robustness. By introducing differential training and multi-head attention mechanism, the method can cope with various complex noise environments such as street traffic, indoor echo, and multiple people speaking. For example, in a complex acoustic environment such as an airport, the method of the invention can effectively suppress the low-frequency noise of airplane takeoff and landing, broadcast sound, and crowd noise, ensuring the clarity of the user's voice.

[0122] In summary, the invention provides a comprehensive and efficient solution for voice enhancement technology in low-latency Bluetooth earphones, which is expected to greatly improve the user's call experience in various environments.

[0123] In this embodiment, we use the multi-modal voice enhancement method for low-latency Bluetooth earphones proposed in this invention. This method combines bone conduction signals and air conduction signals and processes them through a deep learning model and a signal separation module. We chose a typical noisy environment - a busy coffee shop scene for testing.

[0124] Comparative Example 1: As a comparison, we chose a traditional single-channel noise reduction algorithm that only uses air conduction signals and uses classic Wiener filtering for noise suppression. This method has been used in the industry for many years and is representative.

[0125] To comprehensively evaluate the performance of both methods, we selected the following key indicators:

[0126] 1. Perceptual Evaluation of Speech Quality (PESQ): Range from -0.5 to 4.5, higher is better.

[0127] 2. Signal-to-Noise Ratio Gain (SNR Gain): Unit in dB, higher is better.

[0128] 3. Speech Intelligibility (STOI): Range from 0 to 1, higher is better.

[0129] 4. Processing Delay: Unit in milliseconds, lower is better.

[0130] We used professional acoustic testing equipment to conduct tests in a simulated coffee shop environment. The test audio contained various common background noises such as human voices, coffee machine sounds, and dish collision sounds. We processed the same set of test audio using both the invented method and the comparative method, and then evaluated the processing results.

[0131] Indicators The inventive method The comparative method Percentage improvement PESQ 3.8 3.2 18.75％ SNRGain (dB) 15 10 50％ STOI 0.92 0.85 8.24％ Processing delay (ms) 15 25 40％

[0132] From the test results, it is clear that the invented method significantly outperforms the traditional single-channel noise reduction algorithm in all indicators. Let's analyze these results one by one:

[0133] 1. Perceptual Evaluation of Speech Quality (PESQ): The PESQ value of the invented method reached 3.8, which is 0.6 higher than the comparative method, an increase of 18.75%. This means that users will hear clearer and more natural speech through the Bluetooth headset of the invention, greatly improving user experience.

[0134] 2. Signal-to-Noise Ratio Gain (SNR Gain): The SNR gain of the invented method reached 15 dB, which is 5 dB higher than the comparative method, an increase of 50%. This significant improvement indicates that the invented method can more effectively suppress background noise, making the target speech more "prominent" in a noisy environment.

[0135] 3. Speech Intelligibility (STOI): The STOI value of the invented method reached 0.92, which is 0.07 higher than the comparative method, an increase of 8.24%. Although the increase seems small, considering the range of the STOI indicator (0 to 1), this increase is actually quite significant. It means that in a complex coffee shop environment, users can more clearly understand the conversation content using the Bluetooth headset of the invention.

[0136] 4. Processing delay: The processing delay of the method of the present application is only 15 ms, which is reduced by 10 ms compared with the comparative method, and is improved by 40%. This result is particularly exciting because it not only proves the advantage of the method of the present application in sound quality, but also demonstrates its excellent performance in real-time performance. Low delay is crucial to improve the call experience, especially in scenarios such as video calls.

[0137] These test results fully demonstrate the superiority of the method of the present application. By combining bone conduction signals and air conduction signals, supplemented by advanced deep learning models and signal separation techniques, the present application not only achieves significant results in noise suppression and speech enhancement, but also achieves lower processing delay. This means that users will enjoy an unprecedented clear call experience when using Bluetooth earphones equipped with this technology, even in noisy environments.

[0138] It is particularly worth mentioning that the method of the present application not only maintains high sound quality, but also achieves low delay, which provides an innovative solution to the long-standing problem of "sound quality and delay being difficult to balance" in the field of Bluetooth earphones. This breakthrough is expected to promote the entire Bluetooth earphone industry to develop towards higher quality and lower delay, and bring users a better product experience.

[0139] In summary, the multi-modal human voice enhancement method of the low-latency Bluetooth earphone of the present application not only has innovation in technology, but also exhibits obvious advantages in practical application. It opens up a new way for improving the call quality of Bluetooth earphones in complex noise environments, and is expected to be widely used in future intelligent audio devices.

[0140] It should be noted that: the above-mentioned is only the preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal voice enhancement method for low latency Bluetooth earphones, characterized in that, The method comprises the following steps: S1: the microphone collects the bone conduction signal of the earphone when the user speaks, and records the environmental sound and the user's voice; S2: data preprocessing; all collected bone conduction signals and recorded sounds are framed, windowed and preprocessed, and bone conduction signals without voice and recorded environmental sounds are removed for division, training and testing of the deep learning model; S3: an end-to-end deep learning module is used; A multi-path deep learning model is proposed, which takes the preprocessed bone conduction signal and the recorded recording as input and directly outputs the enhanced user voice to improve the intelligibility of the human voice; S4: a signal separation module is used; an effective signal separation algorithm is proposed, which differentiates the input signal and the extracted mixed noise, effectively realizes noise suppression, and is used to exclude noise signal interference; S5: deep learning and signal separation module fusion; the idea of denoising first and then filtering is adopted, the noise component is extracted in the signal separation algorithm, the input signal is subtracted from the noise component in the signal separation and the human voice part enhanced by the deep learning module in the previous step, which is the final human voice signal, used to restore the target signal and suppress the background environmental noise; The process of collecting environmental noise and user voice in the microphone, bone conduction microphone and earphone recording end in step S1 is as follows: adjust the connection mode of the bone conduction microphone and the earphone recording end, place the bone conduction signal collection end above the earphone and microphone in the metal shielding box, place the recording system beside the earphone in the metal shielding box, and collect the recorded voice by simulating the user's use scene, and the recorded user voice is used as the reference signal.

2. The multi-modal voice enhancement method for low latency Bluetooth earphone according to claim 1, wherein, The data preprocessing process of the collected information in step S2 is as follows: The collected information is divided into two paths, one is the bone conduction signal, and the other is the externally recorded environmental sound signal, the user voice part is intercepted, and the bone conduction signal and the externally recorded environmental noise sound are preprocessed respectively; First, the collected signal is divided into multiple frames using an overlapping framing method, then each frame is windowed, the next frame is spliced to the tail of the window, and finally a new segment is obtained as the model input; At the same time, the signal segments without user voice and noise segments are deleted, wherein the segment length needs to be adjusted according to the model hyperparameters.

3. The multi-modal voice enhancement method for low latency Bluetooth earphone according to claim 2, wherein, In step S2, data preprocessing, and in step S5, the method of denoising first and then filtering is as follows: different weight factors are added to the sound signal and the noise signal during superposition, and the L1 norm is used for weighted superposition, wherein the formula for weighted superposition is: Y = a * S + b * N Where Y is the weighted superposition signal, S is the sound signal, N is the noise signal, a and b are weight factors, and a + = 1; The L1 norm calculation formula is: ||Y||1 = ∑|Y i | where ||Y||1represents the L1 norm of Y, Y i is the i-th element in Y.

4. The multi-modal voice enhancement method for low latency Bluetooth earphone according to claim 1, wherein, Step S3 includes the following processes, the structure of the fused multi-path signal, including a multi-path signal input module, multiple deep learning paths and a fusion module; Multiple deep learning models use different loss function parameter adjustments, model evaluation of multiple inputs, and selection of appropriate models according to experimental results, and the feature vectors output by the multi-path model are fused through the fusion module; The model is a human voice enhancement model based on an encoder-decoder structure. The input signal is first subjected to feature extraction by the encoder, and then information recovery by the decoder, to obtain the human voice enhanced speech.

5. The multi-modal voice enhancement method of a low latency Bluetooth headset according to claim 4, wherein, The encoder and decoder in step S3 use a Transformer structure based on a self-attention mechanism to extract the features of the original signal and re-encode the input data. The input signal is encoded by the encoder in the Transformer and then decoded by the decoder to obtain the enhanced human voice. The specific steps are as follows: S21: The original signal data obtained in step S2 is input into the Transformer encoder of each path to obtain the feature information of the data: S22: After obtaining the enhancement information, the output of the encoder is spliced and then input into the fusion module for re-encoding and feature information extraction; S23: The results of each enhancement path are fused to form the final output using a multi-path encoding and single-path decoding method; S24: The encoder and decoder extract the signal features and re-encode them; Multiple deep learning models use different loss functions. In the encoder, the L1 norm loss function is used to linearly map the input data, calculate the error between the encoded path features and the target features, and compare the features with the target features to obtain a set of prediction vectors through supervised learning. The prediction vector is the scaling factor in the self-attention mechanism of the Transformer; In the decoder, the reconstruction error loss function of the signal is used, and the L1 norm loss function is used for further feature compensation. The prediction results of the decoder are compared with the reference signal to obtain the difference between the predicted value and the true value.

6. The multi-modal voice enhancement method of a low latency Bluetooth headset of claim 1, wherein, The noise signal separation module in step S4 introduces a noise perception feature encoder and additional training signals into the input noise and target signal. By learning, the features related to the noise are obtained. The additional training signals include pure signals and noise signals, and the encoded signals are subjected to differential training to obtain the related features of the noise signals. The larger the value of the filter module, the closer the signal is to the noise signal. The output signal is subtracted from the noise signal to obtain the enhanced human voice signal. The specific steps are as follows: S31: Obtain the target signal features and the related features of the target signal and the noise signal through training data, and extract the self-attention features of the target signal; S32: Use the signal separation loss function to compare the target signal features with the target signal self-attention features to obtain the mixed signal features; S33: The mixed signal is divided into multiple small blocks, and the input signal is decomposed to obtain multiple small blocks. Each small block is input into the filter module to obtain the final feature after weighting. After obtaining the feature, the noise-related feature is removed to obtain the related feature of the target signal. Through forward calculation, the expected target signal is obtained to realize differential encoding of the features.

7. The multi-modal voice enhancement method of claim 5, wherein, In step S31, the input and output of the training signal are associated with the target signal and the self-attention feature of the target signal, and the mixed signal feature is associated with the noise-related feature, and the similarity between the two features is calculated by using the L1 norm to calculate the loss function.

8. The multi-modal voice enhancement method of a low latency Bluetooth headset according to claim 6, wherein, In step S33, after obtaining the mixed signal feature and the noise feature, the noise signal feature is obtained through a convolution layer, which encodes each mixed signal feature and extracts feature information using convolution, then combines the feature information with the noise signal feature, and uses a fully connected layer to obtain the weight of the feature, and uses the weight to encode the difference feature of the target signal and the self-attention-related feature of the target signal, and finally obtains the feature of the output signal.

9. The multi-modal voice enhancement method of claim 8, wherein, For different environmental noise, a convolution operation based on signal separation is proposed, which only retains the noise-related feature, encodes through an encoder, and then performs differential training, separates the feature information obtained by the noise signal through weighting, calculates the similarity between the target signal and the target signal feature and normalizes, and then extracts through a convolution layer to combine the mixed signal feature and the noise signal feature, thereby realizing differential enhancement.

10. The multi-modal voice enhancement method of a low latency Bluetooth headset of claim 1, wherein, The model optimization method in steps S4 and S5 includes the following steps: optimizing the model to improve the model training effect, the model optimization method is to use gradient descent method to optimize, optimize the loss function, and obtain the optimal solution of the loss function by multiple iterations; when the error function is minimum, the optimal model training is completed, first calculate the direction of gradient descent, use first-order difference to approximate second-order difference to calculate second-order derivative, determine the sensitivity of error in neural network prediction, get the partial derivative of weight and error function, multiply the result by layer by layer feedback input, iterate the partial derivative of weight and error, get the optimal solution, the optimal solution is that when the error is minimum, the gradient is 0; the error calculation function is calculated by cross-entropy loss function, the cross-entropy loss function is calculated by model real signal and model output signal; the forward convolution process of the model is divided into convolution calculation of encoder and decoder, the encoder and decoder structure uses the Transformer structure based on self-attention mechanism, the input signal and feature mapping first use the multi-head attention mechanism of the encoder to extract information, and the extracted information is used as the decoder.

Images