Bone-conduction speech enhancement method and device, storage medium and equipment
By combining deep learning and non-negative matrix decomposition technology, bone-directed voice enhancement and LC3 encoding are solved, and the problems of high-frequency loss and mid-frequency heavy frequency of bone-directed voice acquisition are improved, achieving sound quality and improvement of user experience.
Patent Information
- Application Number
- CN202411579182.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, the high-frequency part of the voice collected by the bone conduction microphone is missing and the mid-frequency part is thick, resulting in insufficient clarity and brightness of the voice, and it is impossible to directly encode the bone conduction voice using LC3.
Combining deep learning and non-negative matrix decomposition technology, bone-directed speech enhancement is achieved while LC3 encoding. The specific steps include: inputting bone-directed speech PCM data, performing low-latency improved discrete cosine transformation, extracting features, inputting pre-trained neural network for enhancement, performing non-negative matrix decomposition, and finally performing standard LC3 encoding.
By compensating high-frequency components and improving mid-frequency components, the sound quality of bone-directed voice is improved, the user experience needs are met, and the problem of bone-directed voice cannot be directly encoded by LC3.
Smart Images
Figure CN119446168A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of Bluetooth audio technology, and in particular, relates to a bone conduction speech enhancement method, device, storage medium and equipment. Background Art
[0002] The current mainstream Bluetooth audio encoders are as follows: SBC: mandatory requirement of the A2DP protocol, the most widely used, and must be supported by all Bluetooth audio devices, but the sound quality is average; AAC-LC: better sound quality and more widely used, many mainstream mobile phones support it, but compared with SBC, it occupies a larger memory and has a higher computational complexity. Many Bluetooth devices are based on embedded platforms with limited battery capacity, poor processor computing power and limited memory, and its patent fees are high; aptX series: better sound quality, but very high bit rate, aptX requires a bit rate of 384kbps, while aptX -HD has a bit rate of 576kbps, and it is a unique technology of Qualcomm and is relatively closed; LDAC: The sound quality is good, but the bit rate is also very high, which are 330kbps, 660kbps and 990kbps respectively. Since the wireless environment of Bluetooth devices is particularly complex, it is difficult to stably support such a high bit rate, and it is a unique technology of Sony and is also very closed; LHDC: The sound quality is good, but the bit rate is also very high, typically including 400kbps, 600kbps and 900kbps. Such a high bit rate puts high demands on the baseband / RF design of Bluetooth. Based on the above reasons, the Bluetooth International Alliance Bluetooth Sig and many manufacturers launched LC3, which is mainly for low-power Bluetooth and can also be used for classic Bluetooth. It has the advantages of lower latency, higher sound quality and coding gain, and no patent fees in the Bluetooth field, and has attracted the attention of many manufacturers.
[0003] The voice collected by traditional microphones (Air Conduction, AC, also known as air conduction) is conducted through air and is easily interfered by background noise. Bone conduction (BC) microphones are voice collection devices that use the vibration of human bones to form electrical signals. They can effectively avoid background noise interference and have strong anti-noise performance. They have been widely used. In the field of Bluetooth, there are also many application scenarios, such as Bluetooth headsets with bone conduction microphones. At present, there are the following problems when using bone conduction microphones on Bluetooth headsets: 1. The high-frequency part of the voice collected by the bone conduction microphone is missing, the mid-frequency part is heavy, and the clarity and brightness of the voice are not enough, which reduces the user experience; 2. In LE Audio, LC3 will become the mainstream codec, in which the bandwidth detection module can detect the real bandwidth of the input voice and adjust the encoding strategy according to the real bandwidth to avoid spectrum leakage. The minimum bandwidth that can be detected by the bandwidth detection in LC3 is 4kHz, while the real bandwidth of the voice collected by the bone conduction microphone is usually around 2kHz, so LC3 cannot be used directly to encode bone conduction voice.
[0004] In order to improve the sound quality of bone conduction speech, the deep learning-based method is the mainstream in the prior art. It uses the powerful learning ability of neural networks to establish a nonlinear mapping relationship between bone conduction signals and air conduction signals through a large amount of training. However, during the training process of the neural network, the weight is adjusted based on the error through the back propagation algorithm. The calculation of this error is based on the new amplitude spectrum generated by the nonlinear transformation of the air conduction amplitude spectrum and the bone conduction amplitude spectrum through the neural network. As a result, the data generated in actual application is relatively smooth, which is different from the characteristics of speech data, resulting in unnatural sound quality. Summary of the invention
[0005] In response to the above-mentioned technical problems existing in the prior art, the present application provides a bone conduction speech enhancement method, device, storage medium and equipment. By combining deep learning and non-negative matrix decomposition technology, bone conduction speech is enhanced while LC3 encoding is performed on the bone conduction speech, thereby improving the user experience.
[0006] In order to achieve the above-mentioned purpose, the first technical solution adopted by the present application is: to provide a bone conduction speech enhancement method, comprising: inputting bone conduction speech PCM data, performing a low-delay improved discrete cosine transform, and outputting spectrum coefficients; performing feature extraction according to the spectrum coefficients to obtain bone conduction speech features; inputting the bone conduction speech features into a pre-trained neural network, outputting an enhanced bone conduction speech logarithmic amplitude spectrum, and processing the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum; performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectrum coefficients; and continuing to perform a standard LC3 encoding process on the enhanced spectrum coefficients to output an enhanced speech code stream.
[0007] The second technical solution adopted in the present application is: to provide a bone conduction speech enhancement device, including: a module for inputting bone conduction speech PCM data, performing a low-delay improved discrete cosine transform, and outputting spectral coefficients; a module for performing feature extraction according to the spectral coefficients to obtain bone conduction speech features; a module for inputting the bone conduction speech features into a pre-trained neural network, outputting an enhanced bone conduction speech logarithmic amplitude spectrum, and processing the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum; a module for performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectral coefficients; and a module for continuing to perform a standard LC3 encoding process on the enhanced spectral coefficients and outputting an enhanced speech code stream.
[0008] The third technical solution adopted in the present application is: providing a computer-readable storage medium storing computer instructions, wherein the computer instructions are operated to execute the bone conduction speech enhancement method in solution one.
[0009] The fourth technical solution adopted in the present application is: to provide a computer device, which includes a processor and a memory, the memory stores computer instructions, wherein the processor operates the computer instructions to execute the bone conduction speech enhancement method in solution one.
[0010] The beneficial effects that can be achieved by the technical solution of this application are: this application can be applied to the field of Bluetooth audio as well as other fields, and can be applied to the configuration of frame length 10ms as well as the configuration of frame length 7.5ms. By combining deep learning with non-negative matrix decomposition technology, bone conduction speech is enhanced while LC3 encoding is performed on bone conduction speech, thereby improving user experience. Based on deep learning, high-frequency components can be effectively compensated and intermediate-frequency components can be improved at the same time; based on non-negative matrix decomposition, the problem of over-smoothing of speech amplitude spectrum can be improved, thereby making the sound quality natural. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a flowchart of a specific implementation method of the bone conduction speech enhancement method of the present application;
[0012] Figure 2 is a schematic diagram of a specific example of offline training and online reasoning of a pre-trained neural network in this application;
[0013] Figure 3 is a schematic diagram of a specific example of bone conduction speech enhancement of the present application;
[0014] Figure 4 It is a schematic diagram of a specific implementation of the bone conduction speech enhancement device of the present application. DETAILED DESCRIPTION
[0015] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present application.
[0016] The following describes in detail the specific implementation of the present application by taking the configuration of 8 kHz sampling rate as an example. The principles of other sampling rates are similar to those of the 8 kHz sampling rate.
[0017] Figure 1 It is a flowchart of a specific implementation method of the bone conduction speech enhancement method of the present application.
[0018] exist Figure 1 In a specific embodiment shown, the bone conduction speech enhancement method of the present application includes process S101, inputting bone conduction speech PCM data, performing low-delay improved discrete cosine transform, and outputting spectral coefficients.
[0019] In this specific implementation, at the LC3 encoder end, audio data in PCM format is input, a low-delay improved discrete cosine transform is performed, and spectral coefficients are output. Specifically,
[0020] t(n)=x s (ZN F +n), for n=0...2·N F -1-Z
[0021] t(2N F -Z+n)=0,for n=0...Z-1
[0022]
[0023] Among them, x s (n) is the input time domain audio PCM signal. Taking the configuration of 8kHz and 10ms as an example, N F =80 is the length of a frame, X(k) is the frequency domain spectrum coefficient after discrete cosine transform, which contains N F The effective spectral coefficients.
[0024] exist Figure 1 In a specific embodiment shown, the bone conduction speech enhancement method of the present application includes process S102, performing feature extraction according to spectral coefficients to obtain bone conduction speech features.
[0025] In a specific embodiment of the present application, feature extraction is performed based on spectral coefficients to obtain bone conduction speech features, including: obtaining a logarithmic amplitude spectrum based on the spectral coefficients, and calculating the mean and variance of the logarithmic amplitude spectrum; and normalizing the logarithmic amplitude spectrum based on the mean and variance to obtain the bone conduction speech features.
[0026] In this specific embodiment, the spectral coefficients X are first input. BC (k), take the logarithmic magnitude spectrum log|X BC (k)|; Secondly, calculate the mean of the logarithmic amplitude spectrum and variance σ X ; Finally, perform normalization to obtain the bone conduction speech feature BC is the abbreviation of Bone Conduction.
[0027] exist Figure 1 In a specific embodiment shown, the bone conduction speech enhancement method of the present application includes process S103, inputting the bone conduction speech feature into a pre-trained neural network, outputting an enhanced bone conduction speech logarithmic amplitude spectrum, and processing the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum.
[0028] In this specific implementation, the bone conduction speech feature, that is, the normalized logarithmic amplitude spectrum log Norm |X BC (k)|, input into the pre-trained neural network to generate the enhanced bone conduction speech logarithmic amplitude spectrum log Norm |X NN (k)|, and by processing the enhanced bone conduction speech logarithmic amplitude spectrum, the linear amplitude spectrum|X Linear (k)|. The enhanced bone conduction speech logarithmic amplitude spectrum generated by the neural network processing can compensate for the high-frequency components and improve the mid-frequency components.
[0029] In a specific embodiment of the present application, the training process of the pre-trained neural network includes: obtaining air conduction speech and bone conduction speech, and obtaining air conduction speech spectrum coefficients and bone conduction speech spectrum coefficients; performing feature extraction according to the air conduction speech spectrum coefficients and the bone conduction speech spectrum coefficients, respectively, to obtain air conduction features and bone conduction features; inputting the air conduction features and the bone conduction features into the neural network for training, and calculating the error; and when the error reaches an expected value, freezing the neural network to obtain the pre-trained neural network.
[0030] Specifically, first, obtain the air conduction speech and bone conduction speech, and obtain the spectral coefficients corresponding to each of them, that is, the air conduction speech spectral coefficient X AC (k) and the bone conduction speech spectrum coefficient X BC (k), wherein air conduction speech is speech recorded by a conventional air conduction microphone, AC is the abbreviation of Air Conduction; bone conduction speech is speech recorded by a bone conduction microphone.
[0031] Secondly, according to the air conduction speech spectrum coefficient X AC (k) and the bone conduction speech spectrum coefficient X BC(k) Perform feature extraction and output air conduction features and bone conduction features. The following is a brief description of the process of obtaining bone conduction features as an example:
[0032] (1) Take the logarithmic amplitude spectrum log|X BC (k)|;
[0033] (2) Calculate the mean of the logarithmic amplitude spectrum With variance σ X ;
[0034] (3) Perform normalization to obtain bone conduction features
[0035] Among them, normalization is a common strategy in deep learning, which can effectively speed up the convergence and enhance the stability of training during training.
[0036] Similarly, we can get the air conduction characteristic log Norm |X AC (k)|.
[0037] Again, the air conduction characteristic log Norm |X AC (k)| and bone conduction characteristics log Norm |X BC (k)| Input the model into the neural network for training and calculate the error. Then adjust the weight of the neural network through the back propagation algorithm. When the loss level reaches the expected level, freeze the neural network to obtain a pre-trained neural network. The error calculation method is:
[0038]
[0039] This application does not limit the choice of neural network, but considering the context-correlation characteristics of speech frames, a recurrent neural network (RNN) is preferably selected.
[0040] Figure 2 It is a schematic diagram of a specific example of offline training and online reasoning of a pre-trained neural network in this application.
[0041] like Figure 2As shown, above the dotted line is the offline training process, in which the neural network is trained based on air conduction speech and bone conduction speech, and feature extraction is performed on air conduction speech and bone conduction speech respectively to obtain air conduction features and bone conduction features, and then the neural network is trained according to the air conduction features and bone conduction features, and the parameters of the neural network are continuously adjusted to finally obtain the pre-trained neural network; below the dotted line is the online reasoning process, in which the bone conduction speech is input into the LC3 encoder, and feature extraction is performed, and the extracted features are input into the pre-trained neural network, and enhanced features are output, and non-negative matrix decomposition is performed based on the enhanced features to improve the over-smoothing problem, and then new spectral coefficients are obtained, and then the remaining encoding modules of the standard LC3 are continued to be executed, and an enhanced voice code stream is output, and then the enhanced voice code stream can be sent to the decoding end for decoding through the Bluetooth communication module, thereby obtaining enhanced speech.
[0042] In a specific embodiment of the present application, the enhanced bone conduction speech logarithmic amplitude spectrum is processed to obtain a linear amplitude spectrum, including: performing denormalization and exponential calculation on the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum.
[0043] In a specific embodiment of the present application, denormalization and exponential calculation are performed on the enhanced bone conduction speech logarithmic amplitude spectrum, including: denormalization and exponential calculation are performed on the enhanced bone conduction speech logarithmic amplitude spectrum according to the mean and variance of the air conduction speech logarithmic amplitude spectrum obtained during the training process of the pre-trained neural network.
[0044] Specifically, the linear amplitude spectrum is:
[0045]
[0046] in, is the mean of the logarithmic amplitude spectrum of air-conduction speech calculated when training the neural network, is the corresponding variance.
[0047] exist Figure 1 In a specific embodiment shown, the bone conduction speech enhancement method of the present application includes process S104, performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectrum coefficients.
[0048] In this specific implementation, based on non-negative matrix decomposition, the problem of over-smoothing of speech amplitude spectrum is improved.
[0049] |X Linear (k)|, output improved smoothed speech amplitude spectrum |X NMF (k)|, and then the enhanced spectrum coefficient X is obtained by calculation NMF (k).
[0050] In a specific embodiment of the present application, non-negative matrix decomposition is performed according to the linear amplitude spectrum to obtain enhanced spectral coefficients, including: constructing an amplitude spectrum observation matrix according to the linear amplitude spectrum; performing non-negative matrix decomposition on the amplitude spectrum observation matrix according to a basis matrix obtained by pre-training to obtain an activation matrix; and obtaining enhanced spectral coefficients according to the basis matrix and the activation matrix.
[0051] In this specific embodiment, non-negative matrix decomposition is used to improve the over-smoothing problem, and the process is briefly described as follows:
[0052] (1) According to the linear amplitude spectrum |X Linear (k)|Construct the amplitude spectrum observation matrix V BC ;
[0053] (2) Based on the basis matrix W obtained in the training phase BC Perform non-negative matrix factorization to obtain the activation matrix H BC ;
[0054] (3) According to the basis matrix W BC and the activation matrix H BC Compute the new magnitude spectrum:
[0055] |X NMF (k)|=W BC H BC
[0056] Then calculate the new spectral coefficients, that is, the enhanced spectral coefficients:
[0057] X NMF (k)=sign(X BC (k))|X NMF (k)|
[0058] For a non-negative matrix V, non-negative matrix decomposition is to find the optimal solution so that the product of W and H obtained by decomposition is an approximation of V. The steps of non-negative matrix decomposition are briefly described as follows:
[0059] (1) Construct the observation matrix V KL , there are K rows and L columns in total, each column is the amplitude spectrum of a frame, and there are L columns in total.
[0060] (2) Initialize the basis matrix W KM .
[0061] (3) Initialize the activation matrix H ML ,In a typical scenario, the above parameters can be K=80, M=40, L=5~10.
[0062] (4) Iterative solution based on KL divergence criterion is briefly described as follows:
[0063]
[0064] in represents element-wise multiplication, T represents the transpose operation of the matrix, and I represents the identity matrix.
[0065] During the iteration process, the KL divergence of V and WH is calculated, and the process stops when it is less than the preset threshold. In order to avoid long calculation time caused by abnormal conditions, the number of iterations can also be preset, and the process stops when the preset number is reached, thereby obtaining the basis matrix and activation matrix.
[0066] exist Figure 1 In a specific implementation manner shown, the bone conduction speech enhancement method of the present application includes process S105, continuing to perform the standard LC3 encoding process on the enhanced spectral coefficients to output an enhanced speech code stream.
[0067] In a specific embodiment of the present application, the standard LC3 encoding process is continued to be performed on the enhanced spectral coefficients to output an enhanced speech code stream, including: the standard LC3 encoding process including impact detection, bandwidth detection, long-term post filter, transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding and code stream encapsulation is continued to be performed on the enhanced spectral coefficients to output an enhanced speech code stream.
[0068] Figure 3 It is a schematic diagram of a specific example of bone conduction speech enhancement of the present application.
[0069] like Figure 3 As shown, the bone conduction speech enhancement process of the present application is: first, input audio data (PCM), perform low-delay improved discrete cosine transform to obtain spectral coefficients, then perform feature extraction, and then input the extracted features into the trained neural network to obtain enhanced bone conduction features, and then perform non-negative matrix decomposition to improve the over-smoothing problem, and then perform the remaining steps of standard LC3 encoding, including impact detection, bandwidth detection, long-term post-filter, transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding and bitstream encapsulation, and finally output enhanced bitstream. The present application combines deep learning with non-negative matrix decomposition technology to achieve bone conduction speech enhancement while performing LC3 encoding on bone conduction speech.
[0070] In the bone conduction speech enhancement method of the present application, by combining deep learning and non-negative matrix decomposition technology, bone conduction speech is enhanced while LC3 encoding is performed on the bone conduction speech, thereby improving the user experience. Based on deep learning, the high-frequency components can be effectively compensated and the intermediate-frequency components can be improved at the same time; based on non-negative matrix decomposition, the problem of over-smoothing of the speech amplitude spectrum can be improved, thereby making the sound quality natural.
[0071] Figure 4It is a schematic diagram of a specific implementation of the bone conduction speech enhancement device of the present application.
[0072] exist Figure 4 In a specific embodiment shown, the bone conduction speech enhancement device of the present application includes: a module 401 for inputting bone conduction speech PCM data, performing a low-delay improved discrete cosine transform, and outputting spectrum coefficients; a module 402 for performing feature extraction according to the spectrum coefficients to obtain bone conduction speech features; a module 403 for inputting the bone conduction speech features into a pre-trained neural network, outputting an enhanced bone conduction speech logarithmic amplitude spectrum, and processing the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum; a module 404 for performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectrum coefficients; and a module 405 for continuing to perform a standard LC3 encoding process on the enhanced spectrum coefficients and outputting an enhanced speech code stream.
[0073] In a specific embodiment of the present application, feature extraction is performed based on spectral coefficients to obtain bone conduction speech features, including: obtaining a logarithmic amplitude spectrum based on the spectral coefficients, and calculating the mean and variance of the logarithmic amplitude spectrum; and normalizing the logarithmic amplitude spectrum based on the mean and variance to obtain the bone conduction speech features.
[0074] In a specific embodiment of the present application, the training process of the pre-trained neural network includes: obtaining air conduction speech and bone conduction speech, and obtaining air conduction speech spectrum coefficients and bone conduction speech spectrum coefficients; performing feature extraction according to the air conduction speech spectrum coefficients and the bone conduction speech spectrum coefficients, respectively, to obtain air conduction features and bone conduction features; inputting the air conduction features and the bone conduction features into the neural network for training, and calculating the error; and when the error reaches an expected value, freezing the neural network to obtain the pre-trained neural network.
[0075] In a specific embodiment of the present application, the enhanced bone conduction speech logarithmic amplitude spectrum is processed to obtain a linear amplitude spectrum, including: performing denormalization and exponential calculation on the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum.
[0076] In a specific embodiment of the present application, denormalization and exponential calculation are performed on the enhanced bone conduction speech logarithmic amplitude spectrum, including: denormalization and exponential calculation are performed on the enhanced bone conduction speech logarithmic amplitude spectrum according to the mean and variance of the air conduction speech logarithmic amplitude spectrum obtained during the training process of the pre-trained neural network.
[0077] In a specific embodiment of the present application, non-negative matrix decomposition is performed according to the linear amplitude spectrum to obtain enhanced spectral coefficients, including: constructing an amplitude spectrum observation matrix according to the linear amplitude spectrum; performing non-negative matrix decomposition on the amplitude spectrum observation matrix according to a basis matrix obtained by pre-training to obtain an activation matrix; and obtaining enhanced spectral coefficients according to the basis matrix and the activation matrix.
[0078] In a specific embodiment of the present application, the standard LC3 encoding process is continued to be performed on the enhanced spectral coefficients to output an enhanced speech code stream, including: the standard LC3 encoding process including impact detection, bandwidth detection, long-term post filter, transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding and code stream encapsulation is continued to be performed on the enhanced spectral coefficients to output an enhanced speech code stream.
[0079] In the bone conduction speech enhancement device of the present application, by combining deep learning and non-negative matrix decomposition technology, bone conduction speech is enhanced while LC3 encoding is performed on the bone conduction speech, thereby improving the user experience. Based on deep learning, the high-frequency components can be effectively compensated and the intermediate-frequency components can be improved at the same time; based on non-negative matrix decomposition, the problem of over-smoothing of the speech amplitude spectrum can be improved, thereby making the sound quality natural.
[0080] In a specific embodiment of the present application, a computer-readable storage medium stores computer instructions, which are operated to execute the bone conduction speech enhancement method described in any embodiment. The storage medium can be directly in hardware, in a software module executed by a processor, or in a combination of the two.
[0081] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium.
[0082] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In an alternative, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.
[0083] In a specific embodiment of the present application, a computer device includes a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the bone conduction speech enhancement method described in any embodiment.
[0084] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0086] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structural transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A bone conduction speech enhancement method, characterized in that: include: Input bone conduction speech PCM data, perform low-latency modified discrete cosine transform, and output spectrum coefficients; Perform feature extraction according to the spectral coefficients to obtain bone conduction speech features; Inputting the bone conduction speech feature into a pre-trained neural network, outputting an enhanced bone conduction speech logarithmic amplitude spectrum, and processing the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum; performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectrum coefficients; as well as The standard LC3 encoding process is continued to be performed on the enhanced spectral coefficients to output an enhanced speech code stream.
2. The bone conduction speech enhancement method according to claim 1, characterized in that: The performing feature extraction according to the spectral coefficients to obtain bone conduction speech features includes: According to the spectrum coefficients, a logarithmic amplitude spectrum is obtained, and a mean and a variance of the logarithmic amplitude spectrum are calculated; and The logarithmic amplitude spectrum is normalized according to the mean and the variance to obtain a bone conduction speech feature.
3. The bone conduction speech enhancement method according to claim 1, characterized in that: The training process of the pre-trained neural network includes: Acquire air conduction speech and bone conduction speech, and acquire air conduction speech spectrum coefficients and bone conduction speech spectrum coefficients; Performing feature extraction according to the air conduction speech spectrum coefficient and the bone conduction speech spectrum coefficient respectively to obtain air conduction features and bone conduction features; Inputting the air conduction feature and the bone conduction feature into a neural network for training, and calculating an error; and When the error reaches an expected value, the neural network is frozen to obtain the pre-trained neural network.
4. The bone conduction speech enhancement method according to claim 1, characterized in that: The performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectrum coefficients includes: Constructing an amplitude spectrum observation matrix according to the linear amplitude spectrum; Performing non-negative matrix decomposition on the amplitude spectrum observation matrix according to the basis matrix obtained by pre-training to obtain an activation matrix; and The enhanced spectral coefficients are obtained according to the base matrix and the activation matrix.
5. The bone conduction speech enhancement method according to claim 1, characterized in that: The enhanced bone conduction speech logarithmic amplitude spectrum is processed to obtain a linear amplitude spectrum, including: Denormalization and exponential calculation are performed on the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum.
6. The bone conduction speech enhancement method according to claim 5, characterized in that: The step of performing denormalization and exponential calculation on the enhanced bone conduction speech logarithmic amplitude spectrum comprises: Denormalization and exponential calculation are performed on the enhanced bone conduction speech logarithmic amplitude spectrum according to the mean and variance of the air conduction speech logarithmic amplitude spectrum obtained during the training process of the pre-trained neural network.
7. The bone conduction speech enhancement method according to claim 1, characterized in that: The step of continuing to perform a standard LC3 encoding process on the enhanced spectral coefficients and outputting an enhanced speech code stream comprises: The enhanced spectral coefficients are further subjected to a standard LC3 coding process including impact detection, bandwidth detection, long-term post-filter, transform domain noise shaping, time domain noise shaping, quantization, noise level estimation, arithmetic coding residual coding and bitstream encapsulation to output the enhanced speech bitstream.
8. A bone conduction speech enhancement device, characterized in that: include: A module for inputting bone conduction speech PCM data, performing low-latency modified discrete cosine transform, and outputting spectral coefficients; A module for performing feature extraction according to the spectral coefficients to obtain bone conduction speech features; A module for inputting the bone conduction speech feature into a pre-trained neural network, outputting an enhanced bone conduction speech logarithmic amplitude spectrum, and processing the enhanced bone conduction speech logarithmic amplitude spectrum to obtain a linear amplitude spectrum; A module for performing non-negative matrix decomposition according to the linear amplitude spectrum to obtain enhanced spectral coefficients; A module for continuing to perform a standard LC3 encoding process on the enhanced spectral coefficients to output an enhanced speech code stream.
9. A computer-readable storage medium storing computer instructions, wherein the computer instructions are operated to execute the bone conduction speech enhancement method according to any one of claims 1 to 7.
10. A computer device comprising a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the bone conduction speech enhancement method according to any one of claims 1 to 7.