Bone conduction assisted air conduction speech processing method, device, medium and equipment

Abstract

The application discloses a bone conduction assisted air conduction speech processing method, device, medium and equipment, and belongs to the technical field of audio coding. The method mainly comprises the following steps: performing windowing processing and low-delay improved discrete cosine transformation processing on bone conduction audio and noisy air conduction audio respectively to obtain bone conduction spectrum coefficients and air conduction spectrum coefficients; performing mel-frequency cepstrum calculation on the bone conduction spectrum coefficients and the air conduction spectrum coefficients respectively to obtain bone conduction features and noisy air conduction features; processing the bone conduction features and the noisy air conduction features by using a preprocessed neural network model to obtain sub-band masks; calculating enhanced spectrum coefficients by using the sub-band masks and the air conduction spectrum coefficients, and calculating enhanced speech audio code streams by using the enhanced spectrum coefficients. The application can realize enhancement of speech signals with small time delay under the condition of saving computing power and storage space.

Description

Technical Field

[0001] This application relates to the field of audio encoding and decoding technology, and in particular to a bone conduction-assisted air conduction speech processing method, device, medium and equipment. Background Technology

[0002] Traditional microphones (also known as air conduction microphones) capture speech through air conduction, making them susceptible to background noise interference. Bone conduction microphones, on the other hand, use the vibrations of human bones to generate electrical signals for speech capture. Bone conduction microphones effectively avoid background noise interference and have strong noise immunity. However, using a bone conduction microphone alone, compared to using an air conduction microphone alone, still presents the following problems: 1. While bone conduction microphones offer strong interference immunity, they suffer from a lack of high-frequency response and a heavy mid-frequency response, resulting in insufficient clarity and brightness in the captured speech. 2. Air conduction microphones are widely used, offer wide bandwidth and high-quality speech capture, but are easily affected by background noise.

[0003] To address the problems associated with using bone conduction microphones and air conduction microphones alone, existing technologies typically utilize transfer functions in servers to enhance speech signals, or compare the signal-to-noise ratio of the air conduction audio signal with a predetermined threshold to determine how to enhance the audio signal, or employ pre-trained deep neural networks to enhance the audio signal.

[0004] However, the methods used in the existing technology still have problems such as large latency, large bandwidth consumption, large amount of computation and large storage space requirements when applied to Bluetooth Low Energy. In addition, the enhanced audio signal has poor sound quality and poor user experience. Summary of the Invention

[0005] To address the problems of large latency, large bandwidth consumption, large computational load, and large storage space requirements in existing technologies, this application mainly provides a bone conduction-assisted air conduction speech processing method, device, medium, and equipment.

[0006] To achieve the above objectives, one technical solution adopted in this application is: providing a bone-guided assisted air-guided speech processing method, which includes: performing windowing processing and low-latency improved discrete cosine transform processing on bone-guided audio and noisy air-guided audio respectively to obtain bone-guided spectral coefficients and air-guided spectral coefficients; calculating Mel-frequency cepstrum using bone-guided spectral coefficients and air-guided spectral coefficients respectively to obtain bone-guided features and noisy air-guided features; processing bone-guided features and noisy air-guided features using a pre-processed neural network model to obtain a sub-band mask; calculating enhancement spectral coefficients using the sub-band mask and air-guided spectral coefficients, and calculating enhanced speech audio bitstream using the enhancement spectral coefficients.

[0007] Optionally, bone conduction audio can be resampled, subjected to long-term post-filtering, impulse detection, and bandwidth detection.

[0008] Optionally, the air-guided speech and corresponding bone-guided audio are acquired, and the air-guided speech is mixed with noise according to a predetermined signal-to-noise ratio to obtain noisy air-guided audio. Pure air-guided features, noisy air-guided features, and bone-guided features are calculated using the air-guided speech, bone-guided audio, and noisy air-guided audio, respectively. The bone-guided features and noisy air-guided features are concatenated to obtain concatenated features, which are then input into a neural network model to calculate a sub-band mask. Enhancement spectral coefficients are calculated using the sub-band mask and the discrete cosine transform spectral coefficients of the noisy air-guided audio. The source distortion ratio between the air-guided speech and the enhanced speech audio is calculated based on the enhancement spectral coefficients and the pure air-guided features. The model parameters of the neural network model are adjusted according to the source distortion ratio until a pre-processed neural network model is obtained.

[0009] Optionally, the air-guided speech is processed by a low-delay improved discrete cosine transform to obtain pure air-guided features; the noisy air-guided audio is processed by Mel frequency cepstrum calculation to obtain noisy air-guided features; and the bone-guided audio is processed by Mel frequency cepstrum calculation and long-term post-filter processing to obtain bone-guided features.

[0010] Optionally, when the source distortion ratio is greater than a preset threshold, the neural network model and its corresponding model parameters are locked, and the locked neural network model is used as the pre-processed neural network model.

[0011] Optionally, the subband mask and the air conduction spectral coefficients are multiplied to obtain the enhancement spectral coefficients; the enhancement spectral coefficients are then subjected to processing steps including transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding, and bitstream encapsulation to obtain the enhanced speech audio bitstream.

[0012] Another technical solution adopted in this application is: providing a bone conduction-assisted air conduction speech processing device, which includes: a spectral coefficient acquisition module, used to perform windowing processing and low-latency improved discrete cosine transform processing on bone conduction audio and noisy air conduction audio respectively to obtain bone conduction spectral coefficients and air conduction spectral coefficients; a feature acquisition module, used to calculate Mel frequency cepstrum using bone conduction spectral coefficients and air conduction spectral coefficients respectively to obtain bone conduction features and noisy air conduction features;

[0013] The sub-band mask acquisition module is used to process bone conduction features and noisy air conduction features using a pre-processed neural network model to obtain the sub-band mask; the enhanced speech audio bitstream acquisition module is used to calculate the enhancement spectral coefficients using the sub-band mask and air conduction spectral coefficients, and then use the enhancement spectral coefficients to calculate the enhanced speech audio bitstream.

[0014] Optional modules for resampling, long-term post-filtering, impulse detection, and bandwidth detection of bone conduction audio.

[0015] Optionally, a pre-processed neural network model acquisition module is used to acquire air-guided speech and the corresponding bone-guided audio, and mix the air-guided speech with noise according to a predetermined signal-to-noise ratio to obtain noisy air-guided audio; using the air-guided speech, bone-guided audio, and noisy air-guided audio, clean air-guided features, noisy air-guided features, and bone-guided features are calculated respectively; the bone-guided features and noisy air-guided features are concatenated to obtain concatenated features, and the concatenated features are input into the neural network model to calculate a sub-band mask; using the sub-band mask and the discrete cosine transform spectral coefficients of the noisy air-guided audio, enhancement spectral coefficients are calculated; based on the enhancement spectral coefficients and the clean air-guided features, the source distortion ratio between the air-guided speech and the enhanced speech audio is calculated; based on the source distortion ratio, the model parameters of the neural network model are adjusted until the pre-processed neural network model is obtained.

[0016] Optionally, there are modules for performing low-delay improved discrete cosine transform processing on air-guided speech to obtain pure air-guided features; calculating the Mel frequency cepstral spectrum of noisy air-guided audio to obtain noisy air-guided features; and calculating the Mel frequency cepstral spectrum and processing with a long-term post-filter on bone-guided audio to obtain bone-guided features.

[0017] Optionally, when the source distortion ratio is greater than a preset threshold, the neural network model and the corresponding model parameters of the neural network model are locked, and the locked neural network model is used as a module of the pre-processed neural network model.

[0018] Optionally, a module is used to multiply the subband mask and the air conduction spectral coefficients to obtain the enhancement spectral coefficients; and to perform processing steps on the enhancement spectral coefficients, including transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding, and bitstream encapsulation, to obtain the enhanced speech audio bitstream.

[0019] Another technical solution adopted in this application is: providing a computer-readable storage medium storing computer instructions that are operated to execute the bone-guided assisted air-guided speech processing method in Solution 1.

[0020] Another technical solution adopted in this application is: providing a computer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer instructions executable by the at least one processor, and the at least one processor operates the computer instructions to execute the bone conduction-assisted air conduction speech processing method in Solution 1.

[0021] The beneficial effects that the technical solution of this application can achieve are: it can enhance the audio signal with a small amount of bandwidth while reducing the latency time, and the algorithm structure of this application is simple and requires less computing power, which can make the enhanced audio sound quality better and more suitable for Bluetooth Low Energy. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a specific embodiment of a bone conduction-assisted air conduction speech processing method according to this application;

[0024] Figure 2 This is a schematic diagram illustrating the application of a bone conduction-assisted air conduction speech processing method of this application in LC3;

[0025] Figure 3 This is a schematic diagram of the offline training and online enhancement process of the neural network model of the bone-guided assisted air-guided speech processing method of this application;

[0026] Figure 4 This is a schematic diagram of a specific embodiment of a bone conduction-assisted air conduction speech processing device according to this application.

[0027] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0028] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings, so that the advantages and features of this application can be more easily understood by those skilled in the art, thereby providing a clearer and more definite definition of the scope of protection of this application.

[0029] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0030] To address the problems associated with using bone conduction microphones and air conduction microphones alone, existing technologies typically employ the following methods:

[0031] 1. The bone conduction signal is acquired through a bone conduction microphone on the terminal device; the bone conduction signal is sent to the server; the server, based on the received bone conduction signal, acquires the low frequency that matches the initial audio and the high frequency transfer function corresponding to the low frequency transfer function stored in the cloud database, and sends the acquired low frequency transfer function and high frequency transfer function to the terminal device; finally, the terminal performs frequency domain expansion on the bone conduction audio signal based on the low frequency transfer function and high frequency transfer function to obtain an enhanced audio signal.

[0032] Theoretically, this method can produce enhanced audio signals with good sound quality, but it is difficult to apply to Bluetooth Low Energy (BLE). This is because the method requires interaction with a server to enhance the audio signal, which not only consumes significant communication bandwidth but also causes substantial system latency, failing to meet the low-latency requirements of BLE and further contributing to a poor user experience. Secondly, while the high-frequency transfer function used in this method is theoretically applicable, in reality, due to the limited precision of processors, it is difficult to implement the conversion using a high-frequency transfer function.

[0033] 2. Use a bone conduction microphone to collect a first audio signal and an air conduction microphone to collect a second audio signal; calculate the signal-to-noise ratio (SNR) of the second audio signal and compare it with a predetermined threshold; when the SNR is greater than the threshold, directly output the second audio signal; when the SNR is not greater than the threshold, fuse the first audio signal and the second audio signal and output them.

[0034] The drawback of this method is that the low-frequency part of the fused audio signal comes from the gain-adjusted bone conduction signal, so the quality is good. However, the high-frequency part of the fused audio signal comes from the high-frequency part of the air conduction signal. Since the signal-to-noise ratio of the second audio signal is low, when the ambient noise is a broadband signal, the signal-to-noise ratio of the high-frequency part of the air conduction signal is also very low. At this time, even if there is a noise reduction module in the subsequent steps, directly using the high-frequency part of the air conduction signal as the high-frequency part of the fused audio signal still cannot change the problem of poor sound quality of the high-frequency part of the fused audio signal.

[0035] 3. Simultaneously record bone conduction audio and air conduction speech, and add environmental noise to the air conduction speech to obtain noisy air conduction speech. Then, use the bone conduction audio, air conduction speech, and noisy air conduction speech to train a deep neural network, thereby using the trained deep learning neural network to calculate the enhanced speech signal. The specific processing steps of the training and inference processes are as follows: Training phase: Train the deep neural network based on bone conduction audio and noisy air conduction speech, and solidify the neural network when the signal-to-noise ratio reaches the desired level. Inference phase: Use the trained neural network, input bone conduction audio and noisy air conduction speech, and output the enhanced speech signal.

[0036] While this end-to-end method can achieve good sound quality, the large length of the audio data input during training and inference causes latency. Furthermore, time-domain-based end-to-end enhancement techniques struggle to optimize data length, making it unsuitable for Bluetooth, especially Bluetooth Low Energy audio. Although time-domain-based end-to-end enhancement is structurally simple, its computational and storage requirements are substantial, which cannot meet the needs of Bluetooth Low Energy applications.

[0037] 4. Extract the amplitude spectrum of STFT (Short Time Fourier Transform) from bone conduction audio and air conduction speech respectively as bone conduction features and air conduction features; input the bone conduction features and air conduction features into a trained neural network to obtain enhanced speech features; perform inverse short time fourier transform on the enhanced speech features to obtain enhanced speech signal.

[0038] The drawback of this method is that the neural network can only enhance the amplitude spectrum, while the phase spectrum is still the phase spectrum of the noisy air-conducting signal, resulting in a relatively small improvement in sound quality. Furthermore, the audio processing based on short-time Fourier transform and inverse short-time Fourier transform requires an overlapping addition step to avoid sound quality degradation caused by waveform discontinuity. However, overlapping addition increases the algorithm latency of the system, making it difficult to apply in some situations where latency requirements are high.

[0039] All of the above technologies exist as independent modules, typically belonging to the speech preprocessing module in specific scenarios. A typical processing flow involves: at the signal transmitting end, the input audio signal is first acquired through a microphone, then undergoes speech preprocessing, the processed audio signal is encoded, and the bitstream is transmitted via radio frequency through baseband; at the signal receiving end, the bitstream is first received by an RF receiving module, then decoded, the decoded speech signal undergoes post-processing, and finally output through a speaker. The speech preprocessing module in this process not only increases algorithm latency and computational power but also leads to system latency, further complicating the optimization process for system latency issues.

[0040] Therefore, existing technologies mainly suffer from large algorithm latency due to the introduction of independent modules, high computational and storage requirements due to the difficulty in reusing modules, and complex neural network structures and high computing power requirements due to the large amounts of input and output data, making them unsuitable for application on low-power Bluetooth platforms. To address these technical problems, this application proposes a bone conduction-assisted air conduction speech processing method, device, medium, and equipment.

[0041] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. The specific embodiments described below can be combined with each other to form new embodiments. The same or similar ideas or processes described in one embodiment may not be repeated in other embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0042] Figure 1 This invention illustrates one embodiment of a bone conduction-assisted air conduction speech processing method according to this application.

[0043] Figure 1 The bone-guided assisted air-guided speech processing method shown includes: step S101, performing windowing processing and low-latency improved discrete cosine transform processing on the bone-guided audio and the noisy air-guided audio respectively to obtain the bone-guided spectral coefficients and the air-guided spectral coefficients;

[0044] Step S102: Calculate the Mel frequency cepstral using the bone conduction spectral coefficient and the air conduction spectral coefficient respectively to obtain the bone conduction characteristics and the noisy air conduction characteristics.

[0045] Step S103: Using a pre-processed neural network model, process the bone conduction features and noisy air conduction features to obtain a sub-band mask;

[0046] Step S104: The enhancement spectral coefficients are calculated using the sub-band mask and air conduction spectral coefficients, and the enhanced speech audio bitstream is calculated using the enhancement spectral coefficients. This specific implementation can enhance the audio signal with a smaller bandwidth while reducing latency. Furthermore, the algorithm structure of this application is simple and requires less computing power, resulting in better sound quality of the calculated enhanced audio, making it more suitable for Bluetooth Low Energy.

[0047] Specifically, such as Figure 2 In Bluetooth Low Energy (BLE), the existing low-latency improved discrete cosine transform (DCT) module is used to perform windowing and DCT processing on bone conduction audio and noisy air conduction audio respectively, obtaining bone conduction spectral coefficients and air conduction spectral coefficients. The Mel-frequency cepstrum of the bone conduction and air conduction spectral coefficients is calculated, and the corresponding calculation results are used as bone conduction features and noisy air conduction features, which are then input into a pre-processed neural network model. The pre-processed neural network model uses the bone conduction features and noisy air conduction features to calculate a sub-band mask. The sub-band mask is multiplied by the air conduction spectral coefficients to obtain the enhancement spectral coefficients. These enhancement spectral coefficients are then processed through transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding, and bitstream encapsulation steps to obtain the enhanced speech audio bitstream. This method avoids secondary calculations by utilizing the existing functional modules in the LC3 encoder, thereby saving computing power and storage space. At the same time, it uses a deep learning neural network model to improve the sound quality of the enhanced speech audio. The neural network complexity of this application is low, avoiding the problem of complex neural network models and large computational load caused by direct end-to-end output of spectral coefficients in the prior art. Furthermore, all processing in this application is implemented in the embedded terminal, avoiding the high latency caused by interaction with the server.

[0048] exist Figure 1 In the illustrated embodiment, the bone-guided assisted air-guided speech processing method includes step S101, which involves windowing and low-latency improved discrete cosine transform processing of the bone-guided audio and the noisy air-guided audio respectively to obtain bone-guided spectral coefficients and air-guided spectral coefficients. This specific embodiment obtains the bone-guided spectral coefficients and air-guided spectral coefficients based on the low-latency improved discrete cosine transform, while simultaneously acquiring the amplitude and phase spectra of the bone-guided audio and the noisy air-guided audio. This lays the foundation for ensuring the sound quality of the enhanced audio and avoids algorithm latency caused by introducing speech enhancement functions by utilizing the overlapping and adding structure in the decoder.

[0049] Specifically, the bone conduction audio and noisy air conduction audio are input into the Bluetooth encoder frame by frame. The low-latency improved discrete cosine transform module in the encoder is used to calculate the spectral coefficients corresponding to the bone conduction audio and noisy air conduction audio as the bone conduction spectral coefficients and air conduction spectral coefficients.

[0050] For example, bone conduction audio and noisy air conduction audio, both configured with an 8kHz sampling rate and a frame length of 10ms, are input frame by frame into a Bluetooth encoder. Each frame corresponds to 80 sampling points. Using the low-latency improved discrete cosine transform module in the LC3 encoder, the input bone conduction audio frames and noisy air conduction audio frames are converted from time to frequency to obtain the corresponding audio spectral coefficients. That is, the spectral coefficients corresponding to the output bone conduction audio are X. BC (k), the audio spectral coefficients corresponding to the noisy air-guided audio frame output are X AC-Noise (k), where k = 0…N F -1, N F =80.

[0051] exist Figure 1 In the specific embodiment shown, the bone conduction-assisted air conduction speech processing method further includes step S102, which calculates the Mel frequency cepstrum using the bone conduction spectral coefficients and air conduction spectral coefficients respectively to obtain bone conduction features and noisy air conduction features. This specific embodiment, by deploying feature extraction and its inverse transformation at the Bluetooth transmitter and Bluetooth receiver respectively, ensures a more even distribution of computational power consumption and reduces bandwidth usage.

[0052] Specifically, Mel frequency cepstral (MFCC) is calculated using bone conduction spectral coefficients and air conduction spectral coefficients, respectively. When extracting features from noisy air conduction spectral coefficients and bone conduction spectral coefficients, the audio frame is segmented into Bark frequencies, and noisy air conduction features and bone conduction features are calculated based on the sub-bands obtained from the segmentation. The MFCC features of the noisy air conduction spectral coefficients include 13 MFCCs, 3 first-order differences of MFCCs, and 3 second-order differences of MFCCs.

[0053] Since the effective bandwidth of bone conduction audio is typically below 2kHz, the MFCC features of bone conduction audio only need to be calculated for the first 8 subbands. By reducing the number of subbands for MFCC feature calculation, the number of input features for the neural network model is reduced, further effectively lowering the model complexity and computational requirements of the neural network model. The MFCC features of bone conduction audio include 8 MFCCs, 3 first-order differences, and 3 second-order differences.

[0054] For example, 80 noisy gas conduction spectral coefficients, i.e., improved discrete cosine transform (MDCT) spectral coefficients, are divided into 13 sub-bands. Each sub-band has the following distribution of spectral coefficients: 4, 4, 4, 4, 4, 4, 4, 8, 8, 8, 8, 16. Mel frequency cepstrum is calculated using these sub-bands derived from the MDCT spectral coefficients. Furthermore, calculating MFCC from spectral coefficients is a relatively mature technique, and will not be elaborated upon in this application.

[0055] exist Figure 1In the specific embodiment shown, the bone-guided assisted air-guided speech processing method further includes step S103, which uses a pre-processed neural network model to process bone-guided features and noisy air-guided features to obtain a sub-band mask. This specific embodiment trains a deep neural network model based on frequency domain features and utilizes the anti-interference capability of bone-guided audio to assist air-guided speech signal enhancement in the trained deep neural network, avoiding the high computational power and high storage space requirements of signal enhancement based on time-domain signals.

[0056] Specifically, bone conduction features and noisy air conduction features are input into a pre-processed neural network model, which then outputs the corresponding sub-band mask. The neural network used in this invention is not limited; for example, it can use an LSTM network with fully connected layers as input, three intermediate layers, and a fully connected layer as output.

[0057] For example, if 38 features are input into a pre-processed neural network model, the neural network model will output 13 sub-band masks. Here, the 38 features include bone conduction features and noisy air conduction features.

[0058] In one specific embodiment of this application, the subband mask is calculated using a pre-processed neural network model. The subband mask is a commonly used concept in audio signal processing, primarily applied to speech noise reduction. In a conventional noise reduction process, the subband mask is calculated as follows: First, the time-domain audio signal is transformed to the frequency domain. Then, the audio spectral coefficients in the frequency domain are divided into subbands to obtain multiple subbands, and the signal-to-noise ratio (SNR) of each subband is calculated. Finally, the SNR value of each subband is compared with a preset threshold value to obtain the subband mask corresponding to that subband. That is, when the signal-to-noise ratio (SNR) of a sub-band is lower than a preset threshold, it indicates that the current sub-band is mainly noise with very little speech, and the sub-band mask corresponding to the sub-band is 0; when the SNR of a sub-band is greater than a preset threshold, it indicates that the current sub-band is mainly speech with very little noise, and the sub-band mask corresponding to the sub-band is 1; when the SNR of a sub-band is within a predetermined range of the preset threshold, it indicates that the current sub-band contains both speech and noise, and the sub-band mask corresponding to the sub-band is set to a value between 0 and 1 according to its corresponding SNR.

[0059] In one specific embodiment of this application, the training process of the neural network model includes: acquiring air-guided speech and the corresponding bone-guided audio, and mixing the air-guided speech with noise according to a predetermined signal-to-noise ratio to obtain noisy air-guided audio; calculating clean air-guided features, noisy air-guided features, and bone-guided features using the air-guided speech, bone-guided audio, and noisy air-guided audio, respectively; concatenating the bone-guided features and noisy air-guided features to obtain concatenated features, and inputting the concatenated features into the neural network model to calculate a sub-band mask; calculating enhancement spectral coefficients using the sub-band mask and the discrete cosine transform spectral coefficients of the noisy air-guided audio; calculating the source distortion ratio between the air-guided speech and the enhanced speech audio based on the enhancement spectral coefficients and the clean air-guided features; and adjusting the model parameters of the neural network model according to the source distortion ratio until a pre-processed neural network model is obtained.

[0060] Furthermore, a low-delay improved discrete cosine transform is applied to the air-guided speech to obtain pure air-guided features; Mel frequency cepstrum is calculated for the noisy air-guided audio to obtain noisy air-guided features; and Mel frequency cepstrum is calculated and long-term post-filter is applied to the bone-guided audio to obtain bone-guided features.

[0061] Furthermore, when the source distortion ratio exceeds a preset threshold, the neural network model and its corresponding model parameters are locked, and the locked neural network model is used as the pre-processed neural network model. This specific embodiment reduces the system's buffer space requirements and system latency by training and inferring audio signals of a predetermined frame length.

[0062] Specifically, such as Figure 3 Speech audio is recorded simultaneously using bone conduction and air conduction microphones to obtain clean air conduction and bone conduction audio. The air conduction audio is then mixed with noise at a certain signal-to-noise ratio to simulate noisy air conduction speech in real-world scenarios. The signal-to-noise ratio can be adjusted as needed, and its specific value is not limited in this application.

[0063] A low-latency improved discrete cosine transform (LCT) is applied to air-guided speech to obtain clean air-guided features. Mel-frequency cepstrum (MFCC) calculation is performed on noisy air-guided audio to obtain noisy air-guided features. Mel-frequency cepstrum calculation and long-term post-filter (LTPF) processing are applied to bone-guided audio to obtain bone-guided features. The step of obtaining LTPF features by processing bone-guided audio with an existing LTPF in the LC3 encoder is described in detail below. The obtained LTPF features are used for neural network inference and also serve as the output data of the LTPF module in the LC3 encoding process for subsequent encoding. The LTPF features mainly include: an indication of whether a valid pitch is detected in the current frame, the integer value of the pitch, the decimal value of the pitch, a coarse normalized correlation value, and a fine normalized correlation value. The pitch-related information is typically used as an important reference for VAD (Voice Amplification) and is crucial reference information in speech enhancement. When performing speech enhancement based on a single microphone, it is usually necessary to calculate LTPF features based on noisy frequencies. However, the accuracy of the calculated LTPF features is often not high due to noise interference. This application calculates LTPF features based on bone conduction audio, which is more accurate.

[0064] The neural network model is trained based on bone conduction features, noisy air conduction features, and pure air conduction features until the desired neural network model is obtained. The neural network model and its corresponding parameters are then fixed, and this model is used as a pre-processed neural network model for calculating subband masks during online processing. The training process is as follows: First, bone conduction features and noisy air conduction features are concatenated and input into the neural network model. The neural network outputs a subband mask. The calculated subband mask is multiplied by the MDCT spectral coefficients of the noisy air conduction audio to obtain new MDCT spectral coefficients, i.e., enhancement spectral coefficients. The source distortion ratio is calculated using the enhancement spectral coefficients and pure air conduction features. The calculated source distortion ratio is compared with a predetermined threshold. The backpropagation data results continuously adjust the model parameters of the neural network model until the source distortion ratio exceeds the preset threshold. At this point, the neural network model and its corresponding parameters are locked, and the locked neural network model is used as the pre-processed neural network model. The size of the preset threshold is not limited in this application. Preferably, when the source distortion ratio no longer increases, the neural network model and its corresponding parameters are locked. Here, "splicing" refers to placing bone-guided features and noisy air-guided features into a one-dimensional array in a certain order to obtain spliced ​​features.

[0065] Considering the nonlinear nature of human hearing, the source distortion ratio is used as a metric to measure the L2 norm loss (reflecting the least squares error between the reference spectral coefficients (pure air conduction characteristics) and the enhancement spectral coefficients). The training objective of this application's model is to obtain the largest possible source distortion ratio. A higher source distortion ratio results in a smaller L2 norm loss in the enhancement spectral coefficients, leading to better enhancement of the audio calculated using those coefficients. The formula for calculating the source distortion ratio is: Where X SE (k) represents the enhancement spectral coefficient.

[0066] exist Figure 1 In the specific embodiment shown, the bone conduction-assisted air conduction speech processing method further includes step S104, calculating the enhancement spectral coefficient using a sub-band mask and air conduction spectral coefficients, and then calculating the enhanced speech audio bitstream using the enhancement spectral coefficients. This specific embodiment can enhance the audio signal with a smaller bandwidth while reducing latency, and its simple algorithm structure requires less computing power, resulting in better sound quality of the calculated enhanced audio, making it more suitable for Bluetooth Low Energy.

[0067] In one specific embodiment of this application, step S104 includes multiplying the subband mask and the air conduction spectral coefficients to obtain the enhancement spectral coefficients; and performing processing steps on the enhancement spectral coefficients including transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding, and bitstream encapsulation to obtain the enhanced speech audio bitstream.

[0068] In one specific embodiment of this application, the bone-guided assisted air-guided speech processing method further includes resampling processing, long-term post-filtering processing, impulse detection processing, and bandwidth detection processing of the bone-guided audio. This specific embodiment processes the bone-guided audio to obtain information during the encoding process through modules in the encoder, rather than processing the noisy air-guided audio. This reduces computational load while improving the sound quality of the encoded enhanced speech, thus enhancing the user experience.

[0069] Specifically, when enhancing noisy air-conducted audio using bone-conducted audio, the resampling module and long-term post-filter in the encoder are used to process the bone-conducted audio, thereby obtaining the relevant parameters of the long-term post-filter. Bone-conducted audio exhibits good speech performance below 2kHz, and the pitch component is mainly concentrated in the low-frequency range below 2kHz. Therefore, using bone-conducted audio to calculate the relevant parameters of the long-term post-filter can effectively enhance the pitch component.

[0070] By using impulse detection in the encoder to process bone conduction audio, impulse detection during the encoding process is completed based on the bone conduction audio, avoiding interference from non-stationary background noise to this module and effectively preventing false detection.

[0071] The bone conduction audio is processed using bandwidth detection in the encoder. While the bandwidth detection module in the encoder primarily targets voice signals from mobile communication networks, the goal of this invention is to better fuse bone conduction and air conduction audio signals, resulting in a full-bandwidth enhanced signal. Therefore, in practical applications, users can choose to ignore this module or modify its operating mode. Preferably, in practical applications, the bandwidth detection module does not perform actual detection but directly outputs a full-bandwidth indicator, further saving computational power.

[0072] Figure 4 This paper illustrates a specific embodiment of a bone conduction-assisted air conduction speech processing device according to this application.

[0073] exist Figure 4 In the specific embodiment shown, the bone conduction-assisted air conduction speech processing device mainly includes: a spectral coefficient acquisition module 401, which is used to perform windowing processing and low-latency improved discrete cosine transform processing on the bone conduction audio and the noisy air conduction audio respectively to obtain the bone conduction spectral coefficient and the air conduction spectral coefficient.

[0074] The feature acquisition module 402 is used to calculate the Mel frequency cepstrum using the bone conduction spectral coefficient and the air conduction spectral coefficient respectively, so as to obtain the bone conduction feature and the noisy air conduction feature.

[0075] Subband mask acquisition module 403 is used to process bone conduction features and noisy air conduction features using a pre-processed neural network model to obtain a subband mask;

[0076] The enhanced speech audio bitstream acquisition module 404 is used to calculate the enhancement spectral coefficient using the sub-band mask and the air conduction spectral coefficient, and to calculate the enhanced speech audio bitstream using the enhancement spectral coefficient.

[0077] In one specific embodiment of this application, the bone conduction-assisted air conduction speech processing device includes modules for resampling bone conduction audio, long-term post-filtering, impulse detection, and bandwidth detection.

[0078] In one specific embodiment of this application, the subband mask acquisition module includes a pre-processed neural network model acquisition module, which is used to acquire air-guided speech and the corresponding bone-guided audio, and mix the air-guided speech with noise according to a predetermined signal-to-noise ratio to obtain noisy air-guided audio; using the air-guided speech, bone-guided audio, and noisy air-guided audio, pure air-guided features, noisy air-guided features, and bone-guided features are calculated respectively; the bone-guided features and noisy air-guided features are concatenated to obtain concatenated features, and the concatenated features are input into the neural network model to calculate the subband mask; using the discrete cosine transform spectral coefficients of the subband mask and the noisy air-guided audio, the enhancement spectral coefficients are calculated; based on the enhancement spectral coefficients and the pure air-guided features, the source distortion ratio between the air-guided speech and the enhanced speech audio is calculated; based on the source distortion ratio, the model parameters of the neural network model are adjusted until the pre-processed neural network model is obtained.

[0079] In one specific embodiment of this application, the preprocessed neural network model acquisition module includes modules for performing low-latency improved discrete cosine transform processing on air-guided speech to obtain pure air-guided features; calculating the Mel frequency cepstral spectrum of noisy air-guided audio to obtain noisy air-guided features; and calculating the Mel frequency cepstral spectrum and processing with a long-term post-filter on bone-guided audio to obtain bone-guided features.

[0080] In one specific embodiment of this application, the preprocessed neural network model acquisition module includes a module for locking the neural network model and the corresponding model parameters of the neural network model when the source distortion ratio is greater than a preset threshold, and using the locked neural network model as the preprocessed neural network model.

[0081] In one specific embodiment of this application, the enhanced speech audio bitstream acquisition module includes a module for multiplying the subband mask and the air conduction spectral coefficients to obtain enhanced spectral coefficients; and performing processing steps on the enhanced spectral coefficients including transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding, and bitstream encapsulation to obtain the enhanced speech audio bitstream.

[0082] The bone-guided air-guided speech processing device provided in this application can be used to execute the bone-guided air-guided speech processing method described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0083] In one specific embodiment of this application, the functional modules of the bone conduction-assisted air conduction speech processing device of this application can be directly in hardware, in software modules executed by a processor, or in a combination of both.

[0084] Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in this art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium.

[0085] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, but alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium can reside as discrete components in the user terminal.

[0086] In another specific embodiment of this application, a computer-readable storage medium stores computer instructions that are operated to perform the bone-guided assisted air-guided speech processing method described in the above embodiments.

[0087] In one specific embodiment of this application, a computer device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer instructions executable by the at least one processor, and the at least one processor operates the computer instructions to perform the bone conduction-assisted air conduction speech processing method described in the above embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A bone conduction-assisted air conduction speech processing method, characterized in that, include: The bone conduction audio and the noisy air conduction audio were windowed and processed by a low-delay improved discrete cosine transform to obtain the bone conduction spectral coefficients and the air conduction spectral coefficients, respectively. The bone conduction spectral coefficient and the air conduction spectral coefficient are used to calculate the Mel frequency cepstrum to obtain the bone conduction characteristics and the noisy air conduction characteristics. Using a pre-processed neural network model, the bone conduction features and the noisy air conduction features are processed to obtain a sub-band mask; The enhancement spectral coefficients are calculated using the sub-band mask and the air conduction spectral coefficients, and the enhanced speech audio bitstream is calculated using the enhancement spectral coefficients. The process of using a pre-processed neural network model to process the bone conduction features and the noisy air conduction features to obtain a sub-band mask includes: Acquire air conduction speech and corresponding bone conduction audio, and mix the air conduction speech with noise according to a predetermined signal-to-noise ratio to obtain noisy air conduction audio; The clean air conduction features, the noisy air conduction features, and the bone conduction features are calculated using the air conduction speech, the bone conduction audio, and the noisy air conduction audio, respectively. The bone conduction feature and the noisy air conduction feature are spliced ​​together to obtain the spliced ​​feature, and the spliced ​​feature is input into a neural network model to calculate the sub-band mask; The enhancement spectral coefficients are calculated using the discrete cosine transform spectral coefficients of the sub-band mask and the noisy air-guided audio frequency. Based on the enhancement spectral coefficients and the pure air conduction characteristics, the source distortion ratio between the air conduction speech and the enhanced speech audio is calculated; Based on the source distortion ratio, the model parameters of the neural network model are adjusted until the pre-processed neural network model is obtained.

2. The bone conduction-assisted air conduction speech processing method according to claim 1, characterized in that, Also includes: The bone conduction audio is subjected to resampling, long-term post-filtering, impulse detection, and bandwidth detection.

3. The bone conduction-assisted air conduction speech processing method according to claim 1, characterized in that, The step of calculating clean air conduction features, noisy air conduction features, and bone conduction features using the air conduction speech, the bone conduction audio, and the noisy air conduction audio, respectively, further includes: The air-conducted speech is processed by a low-latency improved discrete cosine transform to obtain the pure air-conducted features; The noisy air conduction audio signal is subjected to Mel-frequency cepstral calculation to obtain the noisy air conduction characteristics; and, The bone conduction characteristics are obtained by calculating the Mel frequency cepstral spectrum and processing it with a long-term post-filter on the bone conduction audio.

4. The bone conduction-assisted air conduction speech processing method according to claim 1, characterized in that, The step of adjusting the model parameters of the neural network model according to the source distortion ratio includes: When the source distortion ratio is greater than a preset threshold, the neural network model and the corresponding model parameters are locked.

5. The bone conduction-assisted air conduction speech processing method according to claim 1, characterized in that, The step of calculating the enhancement spectral coefficient using the sub-band mask and the air conduction spectral coefficient, and then calculating the enhanced speech audio bitstream using the enhancement spectral coefficient, includes: Multiply the sub-band mask and the gas conduction spectral coefficient to obtain the enhancement spectral coefficient; The enhanced spectral coefficients are processed through steps including transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding, residual coding, and bitstream encapsulation to obtain an enhanced speech audio bitstream.

6. A bone conduction-assisted air conduction speech processing device, characterized in that, include: The spectral coefficient acquisition module is used to perform windowing and low-delay improved discrete cosine transform processing on bone conduction audio and noisy air conduction audio respectively to obtain bone conduction spectral coefficients and air conduction spectral coefficients. The feature acquisition module is used to calculate the Mel frequency cepstrum using the bone conduction spectral coefficient and the air conduction spectral coefficient respectively, to obtain bone conduction features and noisy air conduction features; The sub-band mask acquisition module is used to process the bone conduction features and the noisy air conduction features using a pre-processed neural network model to obtain a sub-band mask. An enhanced speech audio bitstream acquisition module is used to calculate the enhancement spectral coefficient using the sub-band mask and the air conduction spectral coefficient, and to calculate the enhanced speech audio bitstream using the enhancement spectral coefficient; The process of using a pre-processed neural network model to process the bone conduction features and the noisy air conduction features to obtain a sub-band mask includes: Acquire air conduction speech and corresponding bone conduction audio, and mix the air conduction speech with noise according to a predetermined signal-to-noise ratio to obtain noisy air conduction audio; The clean air conduction features, the noisy air conduction features, and the bone conduction features are calculated using the air conduction speech, the bone conduction audio, and the noisy air conduction audio, respectively. The bone conduction feature and the noisy air conduction feature are spliced ​​together to obtain the spliced ​​feature, and the spliced ​​feature is input into a neural network model to calculate the sub-band mask; The enhancement spectral coefficients are calculated using the discrete cosine transform spectral coefficients of the sub-band mask and the noisy air-guided audio frequency. Based on the enhancement spectral coefficients and the pure air conduction characteristics, the source distortion ratio between the air conduction speech and the enhanced speech audio is calculated; Based on the source distortion ratio, the model parameters of the neural network model are adjusted until the pre-processed neural network model is obtained.

7. The bone conduction-assisted air conduction speech processing device according to claim 6, characterized in that, The enhanced voice audio stream acquisition module includes: A module is used to multiply the sub-band mask and the air conduction spectral coefficient to obtain the enhancement spectral coefficient; and to perform processing steps including transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding and bitstream encapsulation on the enhancement spectral coefficient to obtain the enhanced speech audio bitstream.

8. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are operated to perform the bone conduction-assisted air conduction speech processing method according to any one of claims 1-5.

9. A computer device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores computer instructions executable by the at least one processor, which operates the computer instructions to perform the bone conduction-assisted air conduction speech processing method as described in any one of claims 1-5.

Images