Speech enhancement methods, neural network training methods, devices, equipment and media

Abstract

This application provides a speech enhancement method, neural network training method, apparatus, device, and medium, relating to the field of speech processing technology. By performing speech environment complexity analysis, it determines the noise estimate corresponding to the speech data. When the noise estimate exceeds a target threshold, image-assisted speech enhancement is employed. This allows for targeted determination of whether more information is needed through image data based on the speech environment complexity, effectively achieving speech enhancement in complex environments. This adapts to different environmental complexities and meets various speech enhancement requirements. Compared to single-modal speech enhancement, using image data to assist speech data enhancement, by incorporating noise-insensitive image signals, results in better enhancement effects, improved environmental noise tolerance, and stronger robustness of the enhanced speech data. This, in turn, helps reduce the computational power consumption and complexity of multimodal fusion speech enhancement while ensuring speech enhancement performance.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and more specifically, to a speech enhancement method, a neural network training method, apparatus, device, and medium for speech enhancement. Background Technology

[0002] With the continuous development of social technology, the demand for speech enhancement tasks in complex environments is also increasing. Combining speech enhancement with other modalities is crucial for improving the accuracy of speech recognition and understanding. In noisy environments, a single audio signal often fails to provide sufficient information to ensure effective speech processing. However, by integrating other data, the quality of the speech signal can be significantly improved, enhancing the robustness of the speech system in complex environments.

[0003] When performing speech enhancement, most methods use digital signal processors (DSPs). However, this method has limited enhancement capabilities and poor enhancement effects in complex interference environments. Summary of the Invention

[0004] In view of this, this application provides a speech enhancement method, a neural network training method, apparatus, medium, and device for speech enhancement, which can determine whether more information needs to be provided by image data based on the complexity of the speech environment, effectively achieving speech enhancement in complex environments and improving environmental noise tolerance.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of this application, a speech enhancement method is provided, the method comprising: acquiring speech data from a target video; extracting multidimensional speech features from the speech data; performing speech environment complexity analysis based on the multidimensional speech features to determine a noise estimate corresponding to the speech data; if the noise estimate is greater than a target threshold, acquiring image data from the target video, the image data including an image of the mouth region of the speaker who generated the speech data; and enhancing the speech data based on the image data to obtain enhanced speech data.

[0007] In one optional implementation, the step of performing speech environment complexity analysis based on the multidimensional speech features to determine the noise estimate corresponding to the speech data includes: determining the contribution value of the speech features in each dimension, wherein the contribution value is used to characterize the degree of influence of the speech features in that dimension on the speech environment complexity analysis; selecting target speech features from the multidimensional speech features according to the contribution values ​​of the speech features in each dimension; and performing speech environment complexity analysis based on the target speech features to determine the noise estimate corresponding to the speech data.

[0008] In one optional implementation, the step of selecting target speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension includes: deleting a preset number of speech features with the smallest contribution values ​​in each dimension to obtain remaining speech features; determining the contribution value of each remaining speech feature, repeating the above steps until a loss condition is met; and determining the speech features that meet the loss condition as the target speech features.

[0009] In one optional implementation, the step of extracting multidimensional speech features from the speech data includes: performing feature extraction on the speech data to obtain time-frequency features of the speech data, wherein the time-frequency features include amplitude features and phase features; and performing dimensionality enhancement processing on the time-frequency features to obtain multidimensional speech features of the speech data.

[0010] In one optional implementation, the step of enhancing the speech data based on the image data to obtain enhanced speech data includes: extracting multidimensional image features from the image data; aligning and fusing the multidimensional image features and the multidimensional speech features in the time dimension to generate fused features; generating a spectral mask based on the fused features, the spectral mask being used to predict the distribution differences between speech and noise; and generating enhanced speech data based on the spectral mask and the time-frequency features.

[0011] In one optional implementation, generating a spectral mask based on the fusion features includes: performing frequency correlation analysis on the fusion features to obtain frequency correlation analysis results; performing time correlation analysis on the fusion features to obtain time correlation analysis results; and generating the spectral mask based on the frequency correlation analysis results, the time correlation analysis results, and the fusion features.

[0012] In one optional implementation, generating enhanced speech data based on the spectral mask and the time-frequency features includes: determining the enhanced amplitude spectrum based on the spectral mask and the amplitude features; and reconstructing the waveform using the enhanced amplitude spectrum and the phase features to obtain the enhanced speech data.

[0013] According to a second aspect of this application, a neural network training method for speech enhancement is provided, comprising: acquiring sample speech data from a sample video, wherein the sample speech data corresponds to sample clean speech data, and the sample clean speech data does not contain noise; extracting multidimensional speech features from the sample speech data, performing speech environment complexity analysis based on the multidimensional speech features, and determining a noise estimate corresponding to the sample speech data; when the noise estimate is greater than a target threshold, acquiring sample image data from the sample video, wherein the sample image data includes an image of the speaker's mouth region that generated the sample speech data; performing enhancement processing on the sample speech data based on the sample image data to obtain enhanced sample speech data; adjusting the parameters of the neural network for speech enhancement according to the error loss between the enhanced sample speech data and the sample clean data until a training cutoff condition is met; wherein the neural network includes a feature extraction module, a noise estimation module, and a speech enhancement module, wherein the feature extraction module is used to extract features from the sample speech data, the noise estimation module is used to determine the noise estimate corresponding to the sample speech data, and the speech enhancement module is used to perform enhancement processing on the sample speech data based on the sample image data.

[0014] According to a third aspect of this application, a speech enhancement device is provided, the device comprising: a speech acquisition module for acquiring speech data from a target video; a feature extraction module for extracting multi-dimensional speech features from the speech data; a noise estimation module for performing speech environment complexity analysis based on the multi-dimensional speech features to determine a noise estimate corresponding to the speech data; an image acquisition module for acquiring image data from the target video when the noise estimate is greater than a target threshold, the image data including an image of the mouth region of the speaker who generated the speech data; and a multi-modal enhancement module for enhancing the speech data based on the image data to obtain enhanced speech data.

[0015] According to a fourth aspect of this application, a neural network training apparatus for speech enhancement is provided, the apparatus comprising: a sample speech acquisition module for acquiring sample speech data from a sample video, the sample speech data corresponding to sample clean speech data, the sample clean speech data being free of noise; a sample feature extraction module for extracting multidimensional speech features from the sample speech data; a sample noise estimation module for performing speech environment complexity analysis based on the multidimensional speech features to determine a noise estimate corresponding to the sample speech data; and a sample image acquisition module for acquiring sample image data from the sample video when the noise estimate is greater than a target threshold, the sample image data including data generated from the sample speech data. The system comprises: an image of the speaker's mouth area; a sample multi-modal enhancement module for enhancing the sample speech data based on the sample image data to obtain enhanced sample speech data; and a parameter adjustment module for adjusting the parameters of the neural network used for speech enhancement based on the error loss between the enhanced sample speech data and the clean sample data, until the training cutoff condition is met. The neural network includes a feature extraction module, a noise estimation module, and a speech enhancement module. The feature extraction module extracts features from the sample speech data, the noise estimation module determines the noise estimate corresponding to the sample speech data, and the speech enhancement module enhances the sample speech data based on the sample image data.

[0016] According to a fifth aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the speech enhancement method described in the first aspect or the steps of the neural network training method for speech enhancement described in the second aspect.

[0017] According to a sixth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech enhancement method described in the first aspect or the steps of the neural network training method for speech enhancement described in the second aspect.

[0018] The speech enhancement method, neural network training method, apparatus, device, and medium provided in this application embodiment determine the noise estimate corresponding to the speech data by performing speech environment complexity analysis. When the noise estimate is greater than the target threshold, image-assisted speech enhancement is adopted. It can determine whether more information needs to be brought in by image data based on the speech environment complexity, effectively realizing speech enhancement in complex environments. This adapts to different environmental complexities and meets different speech enhancement requirements. Compared with single-modal speech enhancement, using image data to assist in speech data enhancement, by incorporating image signals that are not sensitive to noise, the enhancement effect is better, the environmental noise tolerance is improved, and the enhanced speech data has stronger robustness. Thus, while ensuring speech enhancement performance, it helps to reduce the computational power consumption and computational complexity of multimodal fusion speech enhancement.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0020] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a neural network training method for speech enhancement according to an exemplary embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a neural network structure shown in an exemplary embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating a neural network training process for speech enhancement, as shown in an exemplary embodiment of this application.

[0024] Figure 4 This is a flowchart illustrating a speech enhancement method according to an exemplary embodiment of this application;

[0025] Figure 5 This is a schematic diagram illustrating a process for performing speech environment complexity analysis, as shown in an exemplary embodiment of this application.

[0026] Figure 6 This is a schematic diagram illustrating a process for filtering target speech features according to an exemplary embodiment of this application;

[0027] Figure 7 This is a schematic diagram illustrating a single-mode speech enhancement process according to an exemplary embodiment of this application;

[0028] Figure 8aThis is a spectrogram of noisy speech data illustrated in an exemplary embodiment of this application;

[0029] Figure 8b This is a time-domain waveform diagram of noisy speech data illustrated in an exemplary embodiment of this application;

[0030] Figure 8c This is a spectrogram of clean speech data illustrated in an exemplary embodiment of this application;

[0031] Figure 8d This is a time-domain waveform diagram of clean speech data illustrated in an exemplary embodiment of this application;

[0032] Figure 8e This is a spectrogram of single-mode enhanced speech data illustrated in an exemplary embodiment of this application;

[0033] Figure 8f This is a time-domain waveform diagram of single-mode enhanced speech data illustrated in an exemplary embodiment of this application;

[0034] Figure 8g This is a spectrogram of multimodal enhanced speech data illustrated in an exemplary embodiment of this application;

[0035] Figure 8h This is a time-domain waveform diagram of multimodal enhanced speech data illustrated in an exemplary embodiment of this application;

[0036] Figure 9 This is a schematic diagram illustrating a speech enhancement device according to an exemplary embodiment of this application;

[0037] Figure 10 This is a schematic diagram of a neural network training apparatus for speech enhancement, as illustrated in an exemplary embodiment of this application.

[0038] Figure 11 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0040] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0042] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0043] Research has shown that speech enhancement devices, such as hearing aids, are mostly composed of multiple key components that work together to amplify and optimize sound signals, enabling the wearer to hear surrounding sounds better. These typically include an analog signal acquisition front-end, an analog-to-digital converter (ADC), a digital signal processing unit (DSP), a digital-to-analog converter (DAC), and finally, analog signal drive circuitry. The DSP is the core component of the hearing aid, used for complex processing and enhancement of digital signals, including noise reduction, echo cancellation, feedback suppression, and dynamic range compression. The DSP improves speech clarity, reduces background noise, and adjusts sound frequencies to suit the needs of the hearing-impaired individual, thus determining the performance of the hearing aid.

[0044] Common speech enhancement methods in DSPs can be categorized into three types. The first method uses adaptive filters to perform frequency domain filtering on the speech signal, then converts the filtered audio signal into a time domain output. Although this feature extraction method has low computational complexity, its speech enhancement performance is poor due to the lack of time-frequency correlation analysis. The second method focuses on the time-frequency spatial correlation of the speech signal, using a Convolutional Neural Network (CNN)-based approach to extract spatial correlations and predict the ideal mask for the Fourier spectrum of the speech signal based on these correlations. This method employs deep learning techniques, effectively improving speech enhancement performance, but still lacks information on the time-frequency temporal correlation of the speech signal. The third method incorporates both time-frequency spatial and temporal correlations of the speech signal and uses Long Short-Term Memory (LSTM) networks (Time-dimension GRU, Frequency-dimension GRU) to analyze the signal, further improving the quality of speech enhancement. The design achieved a higher score on objective evaluation metrics than previous speech enhancement methods, demonstrating its superior performance in real-time speech enhancement. This single-mode speech enhancement method makes full use of the spatial and temporal relationships of speech time-frequency signals, and the input speech signals all adopt the method of extracting time-frequency features. However, it still has the problem of limited enhancement capability under complex interference environments, and the computational load for speech enhancement is large, resulting in high computational power consumption.

[0045] Based on the above research, this application provides a speech enhancement method and a neural network training method for speech enhancement. By performing speech environment complexity analysis, the noise estimate corresponding to the speech data is determined. When the noise estimate is greater than the target threshold, image-assisted speech enhancement is adopted. It can determine whether more information needs to be brought by image data assistance according to the speech environment complexity, effectively realizing speech enhancement in complex environments. This adapts to different environmental complexities and meets different speech enhancement requirements. Thus, while ensuring speech enhancement performance, it helps to reduce the computational power consumption and computational complexity of multimodal fusion speech enhancement.

[0046] To facilitate understanding of this embodiment, a detailed description of the speech enhancement method and the neural network training method for speech enhancement disclosed in this application embodiment will be provided first. The execution entity of the speech enhancement method and the neural network training method for speech enhancement provided in this application embodiment is generally an electronic device with a certain computing power. This electronic device can be a server, which can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In some possible implementations, the speech enhancement method and the neural network training method for speech enhancement can be implemented by a processor calling computer-readable instructions stored in memory.

[0047] The following description, in conjunction with the accompanying drawings, illustrates a neural network training method for speech enhancement provided in an embodiment of this application.

[0048] See Figure 1 The diagram shown is a flowchart illustrating a neural network training method for speech enhancement, as presented in an exemplary embodiment of this application. Figure 1 As shown in the embodiments of this disclosure, the neural network training method for speech enhancement includes steps S101 to S105, wherein:

[0049] S101: Sample speech data is collected from the sample video. The sample speech data corresponds to sample clean speech data, which does not contain noise.

[0050] In this step, in order to train the neural network, multiple sample videos can be collected in different environments to form training samples corresponding to different environmental complexities.

[0051] Sample speech data can be collected from the sample video according to a preset sampling rate and preset duration. The specific values ​​of the preset sampling rate and preset duration can be determined according to training needs, and are not specifically limited here.

[0052] For example, 16kHz, 32ms audio frames can be captured as sample speech data. Here, since the sample speech data is directly captured from the sample video, the sample speech data often contains noise.

[0053] S102: Extract multidimensional speech features from the sample speech data.

[0054] Here, feature extraction can be performed on the sample speech data to obtain multidimensional speech features.

[0055] S103: Based on the multidimensional speech features, perform speech environment complexity analysis to determine the noise estimate corresponding to the sample speech data.

[0056] In comparison, unimodal speech enhancement has lower computational complexity than multimodal speech enhancement. Therefore, in environments with low noise interference, the performance difference between unimodal and multimodal speech enhancement is small. This embodiment performs dynamic calculations, analyzes the speech environment complexity based on the multidimensional speech features, determines the noise estimate corresponding to the sample speech data, and uses the noise estimate corresponding to the sample speech data as the criterion for whether to perform multimodal speech enhancement. Here, unimodal speech enhancement refers to speech enhancement using audio signals.

[0057] S104: If the noise estimate is greater than the target threshold, sample image data is acquired from the sample video, the sample image data including an image of the speaker's mouth area that generated the sample speech data.

[0058] In this step, if the noise estimate exceeds the target threshold, it is considered that the speech environment has significant noise interference, requiring multimodal speech enhancement. Sample image data is acquired from the sample video, including images of the speaker's mouth area that generated the sample speech data. Understanding the speaker's mouth movements can better aid in comprehending the speech content. The specific value of the target threshold can be set according to training needs and is not specifically limited here.

[0059] S105: Enhance the sample speech data based on the sample image data to obtain enhanced sample speech data.

[0060] Here, based on the sample image data, an auxiliary process is used to enhance the sample speech data to obtain enhanced sample speech data. Compared with the original sample speech data, the enhanced sample speech data includes more information and has stronger robustness.

[0061] S106: Based on the error loss between the enhanced sample speech data and the clean sample data, adjust the parameters of the neural network used for speech enhancement until the training cutoff condition is met; wherein, the neural network includes a feature extraction module, a noise estimation module, and a speech enhancement module, the feature extraction module is used to extract features from the sample speech data, the noise estimation module is used to determine the noise estimation value corresponding to the sample speech data, and the speech enhancement module is used to enhance the sample speech data based on sample image data.

[0062] This embodiment uses a neural network for speech enhancement to enhance the sample speech data. The neural network outputs the enhanced sample speech data, and the error loss between the enhanced sample speech data and the clean sample data can be calculated. The parameters of the neural network for speech enhancement are adjusted according to the error loss, and the above neural network training process is iterated until the training cutoff condition is met.

[0063] Optionally, the error loss between the enhanced sample speech data and the clean sample data can be calculated using the mean square error function, or the error loss can be calculated in other ways.

[0064] The training cutoff condition may be that the error loss between the enhanced sample speech data and the pure sample data is less than a preset error, or that the number of training iterations of the neural network reaches a preset number, etc.

[0065] To better understand this embodiment, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a neural network as an exemplary embodiment of this application. Figure 2 As shown, the neural network includes a neural network encoder (i.e., a feature extraction module), a noise estimation module, and a speech enhancement module. The speech enhancement module includes a fusion module, a frequency correlation analysis module, a time correlation analysis module, a mask generation module, and an amplitude spectrum generation module.

[0066] See Figure 3 , Figure 3 This is a schematic diagram illustrating a neural network training process for speech enhancement, as shown in an exemplary embodiment of this application. Figure 3 As shown in the image, this example uses a deep neural network (DNN) to construct a neural network for speech enhancement. When training the neural network, sample speech data (i.e., ...) is first prepared. Figure 3 Noisy speech data and clean speech data (i.e.) Figure 3 (The clean speech in the sample). Feature extraction is performed on the sample speech data to obtain its time-frequency features. These features are then input into a speech enhancement neural network (DNN). The DNN enhances the sample speech data and outputs the enhanced amplitude spectrum.

[0067] Specifically, the neural network (DNN) used for speech enhancement utilizes a convolutional network (i.e., a corresponding...) Figure 2The neural network encoder in the dataset performs dimensionality enhancement on the time-frequency features of the sample speech data to obtain multi-dimensional speech features. These time-frequency features include amplitude and phase features. Then, the noise estimation module performs speech environment complexity analysis based on the multi-dimensional speech features of the sample speech data to determine the noise estimate corresponding to the sample speech data. If the noise estimate corresponding to the sample speech data is greater than a target threshold, another convolutional network performs dimensionality enhancement on the image features of the sample image data to obtain multi-dimensional image features. Modality fusion is performed on the multi-dimensional speech features and the multi-dimensional image features of the sample image data to generate sample fusion features. A sample spectrum mask is generated based on the sample fusion features. The enhanced sample amplitude spectrum is determined based on the sample spectrum mask and the amplitude features of the sample speech data.

[0068] Waveform reconstruction is performed using the enhanced sample amplitude spectrum and the phase features of the sample speech data to obtain enhanced sample speech data. Then, based on the error loss between the enhanced sample speech data and the clean sample data, the parameters of the neural network used for speech enhancement are adjusted until the training cutoff condition is met.

[0069] The neural network training method for speech enhancement provided in this application analyzes the complexity of the speech environment to determine the noise estimate corresponding to the sample speech data. When the noise estimate is greater than the target threshold, image-assisted speech enhancement is used. This method can determine whether more information needs to be provided by image data based on the complexity of the speech environment, effectively achieving speech enhancement in complex environments. This adapts to different environmental complexities and meets different speech enhancement requirements. By using image data to assist in speech data enhancement, the trained neural network can significantly improve the quality of speech data and enhance the robustness of speech enhancement in complex speech environments. This helps to reduce the computational power consumption and computational complexity of multimodal fusion speech enhancement while ensuring speech enhancement performance.

[0070] Here, the training and application processes of neural networks are similar. Next, we will explain in detail the specific execution process of each module of the neural network in conjunction with its application.

[0071] It is understood that after training the neural network, the trained neural network can be used for speech enhancement. Therefore, this disclosure also provides a speech enhancement method; please refer to [link to relevant documentation]. Figure 4 , Figure 4 A flowchart illustrating a speech enhancement method provided in an embodiment of this disclosure. Figure 4As shown in the figure, the speech enhancement method provided in this embodiment includes steps S401 to S404, wherein:

[0072] S401: Acquire audio data from the target video.

[0073] In this step, when speech enhancement is required, the target video to be enhanced and the neural network trained according to the neural network training method for speech enhancement described above can be obtained.

[0074] Voice data can be collected from the target video according to a preset sampling rate and preset duration. The specific values ​​of the preset sampling rate and preset duration can be determined according to actual needs and are not specifically limited here.

[0075] For example, 16kHz, 32ms audio frames can be captured as speech data. Here, since the speech data is captured directly from the target video, the speech data often contains noise.

[0076] In some possible implementations, the parameters of the neural network can be adjusted to suit specific application requirements (e.g., computational cache, computing power, etc.).

[0077] S402: Extract multidimensional speech features from the speech data.

[0078] Here, feature extraction can be performed on the speech data to obtain multidimensional speech features.

[0079] S403: Based on the multidimensional speech features, perform speech environment complexity analysis to determine the noise estimate corresponding to the speech data.

[0080] In comparison, single-modal speech enhancement has lower computational complexity than multimodal speech enhancement. Therefore, when there is little noise interference in the environment, the performance difference between using single-modal speech enhancement and using multimodal speech enhancement is small. In this embodiment, dynamic calculation is performed, and the speech environment complexity is analyzed based on the multidimensional speech features to determine the noise estimate corresponding to the speech data. The noise estimate corresponding to the speech data is used as the criterion for whether to perform multimodal speech enhancement.

[0081] In some possible implementations, the extraction of multidimensional speech features from the speech data includes:

[0082] Feature extraction is performed on the speech data to obtain the time-frequency features of the speech data, which include amplitude features and phase features;

[0083] The time-frequency features are subjected to dimensionality enhancement processing to obtain multidimensional speech features of the speech data.

[0084] See again Figure 2 First, feature extraction is performed on the speech data to obtain the time-frequency features of the speech data. Then, the time-frequency features are subjected to dimensionality enhancement processing through a neural network encoder to obtain the multidimensional speech features of the speech data.

[0085] Optionally, when extracting features from the speech data to obtain the time-frequency features of the speech data, the features can be extracted using a Mel-frequency cepstral coefficients (MFCC) feature extractor or by means of filter bank energy (FBank).

[0086] Specifically, the time-frequency characteristics can be Mel spectrum, MFCC, etc.

[0087] See again Figure 2 The time-frequency features can be augmented using convolutional layers in the neural network to obtain multidimensional speech features from the speech data.

[0088] The multidimensional speech features are at a higher level than the time-frequency features, and are high-level speech features. For example, the multidimensional speech features can be 8*64 dimensions, for a total of 512 dimensions, with one feature in each dimension, for a total of 512 features. Here, 8*64 represents 8 frequency axes and 64 frequency vectors.

[0089] In this way, by first extracting the time-frequency features of speech and then generating high-dimensional speech features through dimensionality enhancement, the spectral differences between noise and speech can be represented more precisely, significantly enhancing the distinguishability between noise and speech. This makes the subsequent speech environment complexity analysis more accurate, thereby ensuring that image-assisted enhancement can be triggered accurately and avoiding accidental or missed triggers.

[0090] In various speech environments, multimodal speech enhancement processing, compared to single-modal speech enhancement processing, requires the extraction and calculation of multi-dimensional speech features and multi-dimensional image features, which necessitates a greater computational load and storage space.

[0091] Considering that in practical applications, when the ambient noise is low, the speech signal already contains a lot of information, and the performance difference between using single-modal speech enhancement and multi-modal speech enhancement is small, in order to reduce computational power consumption and computational complexity, this embodiment of the present disclosure does not require the introduction of image-assisted enhancement when the ambient noise is low, thereby reducing the amount of computation required to process image data. When the ambient noise is high, image-assisted enhancement is introduced.

[0092] This embodiment proposes the concept of dynamic computation, suggesting a neural network whose structure or computational flow adjusts at runtime. Unlike traditional static neural networks (which have a fixed structure and computational load during training and inference), dynamic neural networks select different computational paths or levels based on the characteristics of the input, thereby adapting to different data complexities and computational requirements. This embodiment utilizes extracted high-dimensional speech features to determine the complexity of the speech environment, thereby deciding whether to employ multimodal speech enhancement.

[0093] Combination Figure 2 As can be seen, this embodiment proposes a method for calculating environmental noise levels based on transfer learning. The high-dimensional speech features extracted by the encoder are directly transferred to the noise estimation module. The noise estimation module performs speech environment complexity analysis to determine the noise estimation value corresponding to the speech data.

[0094] Optionally, the noise estimation module may be set in a fully connected layer of the neural network.

[0095] In some possible implementations, the step of performing speech environment complexity analysis based on the multidimensional speech features to determine the noise estimate corresponding to the speech data includes: determining the contribution value of the speech features in each dimension, wherein the contribution value is used to characterize the degree of influence of the speech features in that dimension on the speech environment complexity analysis; selecting target speech features from the multidimensional speech features according to the contribution values ​​of the speech features in each dimension; and performing speech environment complexity analysis based on the target speech features to determine the noise estimate corresponding to the speech data.

[0096] Here, because multidimensional speech features are used, the content of speech features in each dimension is extremely rich, so it is not necessary to use speech features in all dimensions to accurately estimate noise. Therefore, this embodiment adopts the pruning idea, based on the contribution value of the speech features in each dimension, to select target speech features from the multidimensional speech features, and only uses the target speech features for speech environment complexity analysis.

[0097] You can refer to this section as well. Figure 5 , Figure 5 This is a schematic diagram illustrating a process for performing speech environment complexity analysis, as shown in an exemplary embodiment of this application. Figure 5As shown, a neural network encoder can obtain 8*64 dimensional multidimensional speech features, specifically including 8 frequency axes (0-7) and 64 frequency vectors (0-63). These high-dimensional speech features are then transferred to a noise estimation module. The noise estimation module determines the contribution value of the speech features in each dimension. Based on the contribution values ​​of the speech features in each dimension, target speech features are selected from the multidimensional speech features. Based on the target speech features, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the speech data. Figure 5 W can be seen in 0_1 W 0_63 W 7_0 These speech features have been removed and will not be used as target speech features.

[0098] After selecting the target speech features, the noise coefficient of the speech environment can be fitted using the target speech features to obtain the noise estimate.

[0099] In this way, based on the contribution value of speech features under each dimension, redundant speech features are eliminated on the basis of the original high-dimensional speech features, while the target speech features with high influence on noise estimation are retained. This effectively reduces the computational power consumption and computational load of noise estimation while ensuring the accuracy of noise estimation.

[0100] In some possible implementations, the step of selecting target speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension includes: deleting a preset number of speech features with the smallest contribution values ​​in each dimension to obtain remaining speech features; determining the contribution value of each remaining speech feature, repeating the above steps until a loss condition is met; and determining the speech features that meet the loss condition as the target speech features.

[0101] In the above steps, the speech features under each dimension can be sorted from largest to smallest contribution value, and a predetermined number of speech features with lower rankings in the sequence can be deleted to obtain the remaining speech features. Alternatively, the speech features under each dimension can be sorted from smallest to largest contribution value, and a predetermined number of speech features with higher rankings in the sequence can be deleted to obtain the remaining speech features. Or, alternatively, a predetermined number of speech features with the smallest contribution value can be selected from the speech features under each dimension and deleted to obtain the remaining speech features. The specific value of the predetermined number can be determined according to actual needs and is not specifically limited here.

[0102] Then, determine the contribution value of each of the remaining speech features, repeat the above steps until the loss condition is met, and determine the speech feature that meets the loss condition as the target speech feature.

[0103] The loss condition refers to the fact that if speech features are continued to be deleted, the impact on noise estimation will be significant.

[0104] You can refer to this section as well. Figure 6 , Figure 6 This is a schematic diagram illustrating a process for filtering target speech features, as shown in an exemplary embodiment of this application. Figure 6 As shown, the contribution value of each speech feature in each dimension is determined. The speech features in each dimension are then sorted from largest to smallest contribution value. The bottom K speech features in the sequence are deleted to obtain the remaining speech features. It is then determined whether further deletion of speech features is possible based on these remaining features. If deletion can continue, the loss condition is considered not met. The contribution value of each remaining speech feature is determined, and the above steps are repeated until the loss condition is met. If deletion cannot continue, the loss condition is considered met. The speech features that meet the loss condition are determined as the target speech features.

[0105] For example, if 64 speech features are ultimately obtained as the target speech features, the number of multiplication and addition operations introduced is only 65, which is relatively low in computational cost.

[0106] In this way, by eliminating low-contribution speech feature value dimensions layer by layer until the loss condition is met, noise estimation accuracy that approximates the original high-dimensional space can be achieved with the fewest speech features. Furthermore, the contribution value is recalculated in each iteration, which can dynamically adapt to changes in sudden noise types.

[0107] During the training of the noise estimation module, the encoder parameters are frozen, and the training process only updates the parameters of the fully connected layer containing the noise estimation module. During training, input neurons corresponding to speech features with smaller contributions are disabled, while neuron channels corresponding to speech features with larger contributions are retained.

[0108] S404: If the noise estimate is greater than the target threshold, image data is acquired from the target video, the image data including an image of the mouth region of the speaker who generated the speech data.

[0109] Here, image data can be acquired from the target video according to a preset frame rate and preset resolution. The specific values ​​of the preset frame rate and preset resolution can be determined according to actual needs and are not specifically limited here.

[0110] For example, a face image with a frame rate of 31.25fps and a resolution of 200*200 can be captured.

[0111] The specific value of the target threshold can be set according to actual needs, and no specific limitation is made here.

[0112] S405: Enhance the speech data based on the image data to obtain enhanced speech data.

[0113] Here, based on the image data, the speech data is enhanced to obtain enhanced speech data. Compared with the original speech data, the enhanced speech data includes more information and has stronger robustness.

[0114] In some possible implementations, the step of enhancing the speech data based on the image data to obtain enhanced speech data includes: extracting multidimensional image features from the image data; aligning and fusing the multidimensional image features and the multidimensional speech features in the time dimension to generate fused features; generating a spectral mask based on the fused features, the spectral mask being used to predict the distribution differences between speech and noise; and generating enhanced speech data based on the spectral mask and the time-frequency features.

[0115] When extracting multidimensional image features from the image data, specifically, feature extraction is performed on the image data to obtain the image features of the image data; the image features are then subjected to dimensionality enhancement processing to obtain the multidimensional image features of the image data.

[0116] Optionally, when extracting features from the image data to obtain the image features, the extraction can be performed by methods such as extracting the Region of Interest (ROI). Specifically, the image features can be visual features of the speaker's mouth movements, such as lip shape changes and temporal movements.

[0117] See again Figure 2 The image features can be augmented by a neural network encoder located in the convolutional layer of the neural network to obtain multidimensional image features of the image data.

[0118] Here, the neural network encoder that performs dimension upscaling on the time-frequency features and the neural network encoder that performs dimension upscaling on the image features can be the same type of encoder or different types of encoders; there is no limitation here.

[0119] The multidimensional image features are at a higher level than the image features, and are high-level image features. For example, combined with... Figure 3 The multidimensional speech features can be 8*32 dimensions, for a total of 256 dimensions, with one feature per dimension, for a total of 256 features. Here, 8*32 represents 8 frequency axes and 32 frequency vectors.

[0120] The fusion module located in the fusion layer of the neural network aligns and fuses the multidimensional image features and the multidimensional speech features in the time dimension to generate fused features, ensuring that lip movements are synchronized with speech.

[0121] It should be noted that the multidimensional speech features used in this fusion process are all the speech features output by the neural network encoder, not the speech features after pruning.

[0122] Optionally, the fusion module can also be implemented based on a fully connected layer, a convolutional neural network, or the self-attention mechanism in Transformer.

[0123] Specifically, the fusion methods include: 1) Feature concatenation: connecting multidimensional speech features and multidimensional image features into joint features. 2) Attention mechanism fusion: projecting one feature from the multidimensional speech features and multidimensional image features onto the other feature to dynamically weight the contribution of multidimensional image features to multidimensional speech features, for example, assigning weights based on the importance of lip movements.

[0124] Then, based on the fusion features, a spectral mask is generated. The size of the spectral mask is the same as the size of the speech spectrum, with a numerical range of [0,1]. The spectral mask is used to predict the distribution differences between speech and noise. Ideally, the spectral mask can amplify speech components and suppress noise regions.

[0125] Based on the spectral mask and the time-frequency features, enhanced speech data can be generated.

[0126] In this way, by aligning and fusing multidimensional image features with multidimensional speech features in the time dimension, a spectral mask can be accurately generated, which can still accurately distinguish speech from noise when there are sudden noise changes or weak energy, thereby ensuring the stability and effectiveness of speech enhancement.

[0127] In some possible implementations, generating a spectral mask based on the fusion features includes: performing frequency correlation analysis on the fusion features to obtain frequency correlation analysis results; performing time correlation analysis on the fusion features to obtain time correlation analysis results; and generating the spectral mask based on the frequency correlation analysis results, the time correlation analysis results, and the fusion features.

[0128] See again Figure 2The frequency correlation analysis module set in the recurrent neural network layer of the neural network can be used to perform frequency correlation analysis on the fused features to obtain frequency correlation analysis results; the time correlation analysis module set in the recurrent neural network layer of the neural network can be used to perform time correlation analysis on the fused features to obtain time correlation analysis results; and the mask generation module set in the fully connected layer of the neural network can be used to generate the spectral mask based on the frequency correlation analysis results, the time correlation analysis results, and the fused features.

[0129] Optionally, the spectral mask can also be generated by a mask generation module set in the convolutional layer of the neural network, based on the frequency correlation analysis results, the time correlation analysis results, and the fusion features.

[0130] The frequency correlation analysis results may include the correlation between each frequency axis, and the time correlation analysis results may include the correlation between each frequency axis and time.

[0131] In this way, by introducing frequency correlation analysis and event correlation analysis, the generation of the spectral mask no longer relies solely on instantaneous fusion features, but also considers the energy coupling between frequency bands and the consistency of temporal context. On the one hand, the results of frequency correlation analysis can avoid misjudging narrowband noise similar to speech harmonic structure as speech; on the other hand, the results of temporal correlation analysis can utilize the continuity of mouth movements and the correlation between speech frames to quickly truncate abrupt noise (such as keyboard tapping, door closing sounds), preventing the mask from being over-amplified at short-term high-energy noise points. The two work together to ensure that the spectral mask remains smooth and coherent under complex non-stationary noise, significantly reducing musicality residue and speech distortion, and improving speech enhancement effects.

[0132] In some possible implementations, generating enhanced speech data based on the spectral mask and the time-frequency features includes: determining the enhanced amplitude spectrum based on the spectral mask and the amplitude features; and reconstructing the waveform using the enhanced amplitude spectrum and the phase features to obtain the enhanced speech data.

[0133] Continue reading Figure 2 The amplitude features of the original noisy speech are multiplied element-wise with the spectral mask to determine the enhanced amplitude spectrum, which is calculated as: Enhanced amplitude spectrum = Spectral mask * Amplitude features.

[0134] Specifically, the amplitude feature can be in the form of an amplitude spectrum.

[0135] Here, the spectral mask acts like a "filter" to preserve the speech-dominated time-frequency region (mask value ≈ 1) and attenuate the noise-dominated region (mask value ≈ 0). After adding the amplitude feature, an enhanced amplitude spectrum can be obtained.

[0136] However, the enhanced amplitude spectrum lacks phase information, thus the enhanced speech data can be obtained by reconstructing the waveform using the enhanced amplitude spectrum and the phase features. Specifically, the phase features can be in the form of phase angles.

[0137] During waveform reconstruction, phase integration is performed first, combining the enhanced amplitude spectrum with the phase features to form a complex spectrum. Here, the amplitude is the enhanced value, and the phase is the original value. Then, spectrum to waveform conversion is performed: the complex spectrum is converted back to a time-domain waveform using a vocoder or inverse short-time fourier transform (ISTFT) to obtain the enhanced speech data.

[0138] It is understandable that in the short-time Fourier transform, the speech signal is decomposed into a complex spectrum:

[0139] Spectrum = Amplitude × exp(j * phase angle).

[0140] The phase angle represents the waveform offset angle of each time-frequency bin, which determines the timing alignment and shape of the waveform.

[0141] By combining spectral masks and amplitude features, the high resolution of the spectral masks is used to strip away noise energy from the amplitude features, and the waveform is reconstructed using the original phase features. This results in enhanced speech that is neither metallic due to phase distortion nor hollow due to excessive suppression, and the overall timbre remains natural and coherent.

[0142] In some possible implementations, the method further includes: generating a spectral mask based on the multidimensional speech features when the noise estimate is less than or equal to the target threshold; and generating enhanced speech data based on the spectral mask and the time-frequency features.

[0143] In the above steps, when the noise estimate is less than or equal to the target threshold, there is no need to extract and fuse high-dimensional image features; the spectral mask and enhanced speech data are generated directly through multi-dimensional speech features.

[0144] The specific steps of generating a spectral mask based on the multidimensional speech features, and generating enhanced speech data based on the spectral mask and the time-frequency features, are similar to the aforementioned steps of generating a spectral mask based on the fusion features, and generating enhanced speech data based on the spectral mask and the time-frequency features. For a detailed description, please refer to the aforementioned embodiments, which will not be repeated here.

[0145] See again Figure 5 In the case of single-modal speech enhancement, the computational complexity is relatively low, the speech enhancement capability is relatively weak, the parameters are 0.4K, and the computation amount per frame is 0.53M; in the case of multimodal speech enhancement, the computational complexity is relatively high, the speech enhancement capability is relatively strong, the parameters are 0.8K, and the computation amount per frame is 1.2M.

[0146] Thus, this embodiment proposes a dynamic calculation method. By determining the noise estimate of the speech environment, it decides whether to use multimodal speech enhancement based on the noise estimate, thereby adopting different calculation paths and levels to adapt to different data complexities and computational requirements. While ensuring speech enhancement performance, it greatly reduces the computational power consumption and computational complexity of multimodal fusion speech enhancement.

[0147] The enhanced speech data obtained through this embodiment has significantly improved intelligibility and environmental noise tolerance compared to the original speech data.

[0148] To more clearly illustrate the speech enhancement process when the noise estimate is less than or equal to the target threshold, please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram illustrating a single-mode speech enhancement process, as shown in an exemplary embodiment of this application. Figure 7As shown in the illustration, the application of the neural network for speech enhancement in this embodiment to a hearing aid is used as an example. For instance, the input speech timing signal is data sampled at 2 seconds and 16 kHz, totaling 320,000 data points. The input speech data is framed into 512 points and then subjected to 512-point FFT feature extraction to obtain spectral data with 257 feature points. This extracted spectral data is then filtered by a 128-dimensional MEL filter to obtain 128 channels of MEL features, which are the time-frequency features of the speech data and serve as the input to the neural network. The neural network first preprocesses the input features using standard convolution, resulting in 16 channels of feature data. To reduce the network parameters, depthwise separable convolution is then used for further feature extraction. After encoder processing, a 64*8 feature data set, i.e., high-dimensional speech feature data, is obtained. This high-dimensional speech feature is then processed by Frequency Domain Relevance Feature Extraction (FGRU), followed by pointwise convolution, batch normalization (BN), and corrected linear units (ReLU). After passing through the FGRU, the temporal correlation of speech features is extracted to enhance the speech signal. Following frequency and temporal correlation extraction, the network generates a spectral mask through a fully connected layer with 257 output points, thus enhancing the 257-point spectrum. Finally, the enhanced speech features undergo a 512-point inverse FFT to obtain the enhanced amplitude spectrum, which is then combined with phase features for waveform reconstruction, yielding the enhanced speech data.

[0149] For speech data with high environmental complexity, the embodiments of this disclosure employ image data to assist speech enhancement, which can significantly improve the enhancement effect compared to single-modal speech enhancement.

[0150] For example, see Figures 8a-8h , Figure 8a This is a spectrogram of noisy speech data illustrated in an exemplary embodiment of this application. Figure 8b This is a time-domain waveform diagram of noisy speech data illustrated in an exemplary embodiment of this application. Figure 8c This is a spectrogram of clean speech data illustrated in an exemplary embodiment of this application. Figure 8d This is a time-domain waveform diagram of clean speech data illustrated in an exemplary embodiment of this application. Figure 8e This is a spectrogram of single-mode enhanced speech data illustrated in an exemplary embodiment of this application. Figure 8f This is a time-domain waveform diagram of single-mode enhanced speech data illustrated in an exemplary embodiment of this application. Figure 8g This is a spectrogram of multimodal enhanced speech data illustrated in an exemplary embodiment of this application. Figure 8hThis is a time-domain waveform diagram of multimodal enhanced speech data as illustrated in an exemplary embodiment of this application.

[0151] As can be seen, image fusion-based multimodal speech enhancement exhibits better performance than single-mode speech enhancement. In a -13dB environment, the Perceptual Evaluation of Speech Quality (PESQ) improves by 1.07, and the Short-Time Objective Intelligibility (STOI) improves by 0.06. However, as the Signal-to-Interference plus Noise Ratio (SNR) decreases, indicating increased environmental interference, the performance gap between image fusion-based multimodal speech enhancement and single-mode speech enhancement widens, demonstrating that image fusion-based multimodal speech enhancement outperforms single-mode speech enhancement.

[0152] The speech enhancement method provided in this application analyzes the complexity of the speech environment to determine the noise estimate corresponding to the speech data. When the noise estimate exceeds a target threshold, image-assisted speech enhancement is employed. This method can determine whether more information needs to be provided through image data based on the complexity of the speech environment, effectively achieving speech enhancement in complex environments. It adapts to different environmental complexities and meets various speech enhancement requirements by using image data to assist in speech data enhancement. Compared to single-modal speech enhancement, the fusion of noise-insensitive image signals results in better enhancement effects, improved environmental noise tolerance, and stronger robustness of the enhanced speech data. This, in turn, helps reduce the computational power consumption and complexity of multimodal fusion speech enhancement while ensuring speech enhancement performance.

[0153] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0154] In some possible implementations, the speech enhancement method and the neural network training method for speech enhancement can be deployed using a microcontroller unit (MCU), a dedicated signal processor (DSP), or an application-specific integrated circuit (ASIC).

[0155] Corresponding to the embodiments of the aforementioned speech enhancement method, this application also provides embodiments of a speech enhancement device.

[0156] Please see Figure 9 This is a schematic diagram illustrating a speech enhancement device according to an exemplary embodiment of this application. Figure 9 As shown in the illustration, the speech enhancement device 900 provided in this application embodiment includes: a speech acquisition module 901, used to acquire speech data from a target video; a feature extraction module 902, used to extract multi-dimensional speech features from the speech data; a noise estimation module 903, used to perform speech environment complexity analysis based on the multi-dimensional speech features to determine the noise estimation value corresponding to the speech data; an image acquisition module 904, used to acquire image data from the target video when the noise estimation value is greater than a target threshold, the image data including an image of the speaker's mouth area that generated the speech data; and a multi-mode enhancement module 905, used to enhance the speech data based on the image data to obtain enhanced speech data.

[0157] In some possible implementations, the noise estimation module 903 is specifically used to: determine the contribution value of the speech feature in each dimension, the contribution value being used to characterize the degree of influence of the speech feature in that dimension on the speech environment complexity analysis; select target speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension; and perform speech environment complexity analysis based on the target speech features to determine the noise estimation value corresponding to the speech data.

[0158] In some possible implementations, when the noise estimation module 903 is used to filter target speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension, it is specifically used to: delete a preset number of speech features with the smallest contribution values ​​in each dimension to obtain the remaining speech features; determine the contribution value of each of the remaining speech features, repeat the above steps until the loss condition is met; and determine the speech features that meet the loss condition as the target speech features.

[0159] In some possible implementations, the feature extraction module 902 is specifically used to: extract features from the speech data to obtain time-frequency features of the speech data, the time-frequency features including amplitude features and phase features; and perform dimensionality enhancement processing on the time-frequency features to obtain multidimensional speech features of the speech data.

[0160] In some possible implementations, the multi-mode enhancement module 905 is specifically used to: extract multi-dimensional image features from the image data; align and fuse the multi-dimensional image features and the multi-dimensional speech features in the time dimension to generate fused features; generate a spectral mask based on the fused features, the spectral mask being used to predict the distribution differences between speech and noise; and generate enhanced speech data based on the spectral mask and the time-frequency features.

[0161] In some possible implementations, when the multimode enhancement module 905 generates a spectral mask based on the fusion features, it is specifically configured to: perform frequency correlation analysis on the fusion features to obtain frequency correlation analysis results; perform time correlation analysis on the fusion features to obtain time correlation analysis results; and generate the spectral mask based on the frequency correlation analysis results, the time correlation analysis results, and the fusion features.

[0162] In some possible implementations, when the multimode enhancement module 905 generates enhanced speech data based on the spectral mask and the time-frequency features, it is specifically used to: determine the enhanced amplitude spectrum based on the spectral mask and the amplitude features; and reconstruct the waveform using the enhanced amplitude spectrum and the phase features to obtain the enhanced speech data.

[0163] Corresponding to the aforementioned embodiments of neural network training methods for speech enhancement, this application also provides embodiments of neural network training apparatus for speech enhancement.

[0164] Please see Figure 10 This is a schematic diagram illustrating a neural network training device for speech enhancement, as shown in an exemplary embodiment of this application. Figure 10As shown in the figure, the neural network training device 1000 for speech enhancement provided in this application embodiment includes: a sample speech acquisition module 1001, used to acquire sample speech data from sample videos, wherein the sample speech data corresponds to sample clean speech data, and the sample clean speech data does not contain noise; a sample feature extraction module 1002, used to extract multi-dimensional speech features from the sample speech data; a sample noise estimation module 1003, used to perform speech environment complexity analysis based on the multi-dimensional speech features to determine the noise estimation value corresponding to the sample speech data; and a sample image acquisition module 1004, used to acquire sample image data from the sample videos when the noise estimation value is greater than a target threshold, wherein the sample image data includes the data generated by the sample speech data acquisition module 1004. The speech data includes an image of the speaker's mouth area; a sample multi-modal enhancement module 1005, used to enhance the sample speech data based on the sample image data to obtain enhanced sample speech data; and a parameter adjustment module 1006, used to adjust the parameters of the neural network used for speech enhancement according to the error loss between the enhanced sample speech data and the clean sample data, until the training cutoff condition is met; wherein, the neural network includes a feature extraction module, a noise estimation module, and a speech enhancement module, the feature extraction module is used to extract features from the sample speech data, the noise estimation module is used to determine the noise estimation value corresponding to the sample speech data, and the speech enhancement module is used to enhance the sample speech data based on the sample image data.

[0165] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0166] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0167] Based on the same technical concept, this application also provides a computer device 1100, referring to... Figure 11As shown, it includes a processor 1110, a memory 1120, and a bus 1130. The memory 1120 is used to store execution instructions and includes main memory 1121 and external memory 1122. The main memory 1121, also known as internal memory, is used to temporarily store the computational data in the processor 1110 and the data exchanged with external memory 1122 such as a hard disk. The processor 1110 exchanges data with the external memory 1122 through the main memory 1121.

[0168] In this embodiment, the memory 1120 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 1110. That is, when the electronic device 1100 is running, the processor 1110 communicates with the memory 1120 through the bus 1130, or the processor 1110 communicates with the memory 1120 through other means, so that the processor 1110 executes the application code stored in the memory 1120, and then executes the steps of the speech enhancement method or the neural network training method for speech enhancement described in any of the foregoing embodiments.

[0169] The memory 1120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0170] Processor 1110 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0171] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1100. In other embodiments of this application, the electronic device 1100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0172] This disclosure also provides a computer-readable storage medium storing a computer program. When a processor executes the computer program, it performs the steps of the speech enhancement method or the neural network training method for speech enhancement described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0173] This disclosure also provides a computer program product storing a computer program. When the computer program is run by a processor, it executes the steps of the speech enhancement method or the neural network training method for speech enhancement provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here.

[0174] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0175] Furthermore, embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0176] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0177] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0178] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0179] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0180] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0181] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0182] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A speech enhancement method, characterized in that, The method includes: Voice data is collected from the target video; Multidimensional speech features are extracted from the speech data; Based on the multidimensional speech features, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the speech data; If the noise estimate is greater than the target threshold, image data is acquired from the target video, and the image data includes an image of the mouth region of the speaker who generated the speech data; Multimodal speech enhancement processing is performed on the speech data based on the image data to obtain enhanced speech data; If the noise estimate is less than or equal to the target threshold, single-modal speech enhancement processing is performed based on the speech data to obtain enhanced speech data. The step of performing speech environment complexity analysis based on the multidimensional speech features to determine the noise estimate corresponding to the speech data includes: Determine the contribution value of the speech features under each dimension, and the contribution value is used to characterize the degree of influence of the speech features under that dimension on the speech environment complexity analysis; Based on the contribution value of the speech features under each dimension, target speech features are selected from the multidimensional speech features. Based on the target speech features, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the speech data; The step of selecting target speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension includes: The speech features with the smallest contribution value in each dimension are deleted to obtain the remaining speech features. Determine the contribution value of each of the remaining speech features, and repeat the above steps until the loss condition is met; the loss condition is used to indicate the degree of impact of deleting speech features on noise estimation. The speech features that meet the loss condition are determined as the target speech features.

2. The method according to claim 1, characterized in that, The extraction of multidimensional speech features from the speech data includes: Feature extraction is performed on the speech data to obtain the time-frequency features of the speech data, which include amplitude features and phase features; The time-frequency features are subjected to dimensionality enhancement processing to obtain multidimensional speech features of the speech data.

3. The method according to claim 2, characterized in that, The process of performing multimodal speech enhancement processing on the speech data based on the image data to obtain enhanced speech data includes: Multidimensional image features are extracted from the image data; The multidimensional image features and the multidimensional speech features are aligned and fused along the time dimension to generate fused features; Based on the fusion features, a spectral mask is generated, which is used to predict the distribution differences between speech and noise; Enhanced speech data is generated based on the spectral mask and the time-frequency features.

4. The method according to claim 3, characterized in that, The step of generating a spectral mask based on the fusion features includes: Frequency correlation analysis was performed on the fusion features to obtain the frequency correlation analysis results; Temporal correlation analysis was performed on the fusion features to obtain the temporal correlation analysis results; The spectral mask is generated based on the frequency correlation analysis results, the time correlation analysis results, and the fusion features.

5. The method according to claim 3, characterized in that, The process of generating enhanced speech data based on the spectral mask and the time-frequency features includes: Based on the spectral mask and the amplitude characteristics, the enhanced amplitude spectrum is determined; The enhanced speech data is obtained by reconstructing the waveform using the enhanced amplitude spectrum and the phase features.

6. A neural network training method for speech enhancement, characterized in that, The method includes: Sample speech data is collected from sample videos, and the sample speech data corresponds to sample clean speech data, which does not contain noise; Multidimensional speech features are extracted from the sample speech data; Based on the multidimensional speech features, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the sample speech data; If the noise estimate is greater than the target threshold, sample image data is acquired from the sample video, and the sample image data includes an image of the speaker's mouth area that generated the sample speech data; Multimodal speech enhancement processing is performed on the sample speech data based on the sample image data to obtain enhanced sample speech data; If the noise estimate is less than or equal to the target threshold, single-modal speech enhancement processing is performed based on the sample speech data to obtain enhanced sample speech data. Based on the error loss between the enhanced sample speech data and the clean sample speech data, the parameters of the neural network used for speech enhancement are adjusted until the training cutoff condition is met. The neural network includes a feature extraction module, a noise estimation module, and a speech enhancement module. The feature extraction module is used to extract features from the sample speech data. The noise estimation module is used to determine the noise estimation value corresponding to the sample speech data. The speech enhancement module is used to enhance the sample speech data based on sample image data. The step of performing speech environment complexity analysis based on the multidimensional speech features to determine the noise estimate corresponding to the sample speech data includes: Determine the contribution value of the speech features under each dimension, and the contribution value is used to characterize the degree of influence of the speech features under that dimension on the speech environment complexity analysis; Based on the contribution value of the speech features under each dimension, the target sample speech features are selected from the multidimensional speech features. Based on the speech features of the target sample, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the sample speech data; The step of selecting target sample speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension includes: The speech features with the smallest contribution value in each dimension are deleted to obtain the remaining speech features. Determine the contribution value of each of the remaining speech features, and repeat the above steps until the loss condition is met; the loss condition is used to indicate the degree of impact of deleting speech features on noise estimation. The speech features that meet the loss condition are determined as the speech features of the target sample.

7. A voice enhancement device, characterized in that, The device includes: The voice acquisition module is used to acquire voice data from the target video; The feature extraction module is used to extract multidimensional speech features from the speech data; The noise estimation module is used to perform speech environment complexity analysis based on the multidimensional speech features and determine the noise estimation value corresponding to the speech data. An image acquisition module is used to acquire image data from the target video when the noise estimate is greater than a target threshold. The image data includes an image of the mouth region of the speaker who generated the speech data. A multimodal enhancement module is used to perform multimodal speech enhancement processing on the speech data based on the image data to obtain enhanced speech data; A single-mode enhancement module is used to perform single-mode speech enhancement processing on the speech data when the noise estimate is less than or equal to the target threshold, so as to obtain enhanced speech data. The noise estimation module is specifically used for: Determine the contribution value of the speech features under each dimension, and the contribution value is used to characterize the degree of influence of the speech features under that dimension on the speech environment complexity analysis; Based on the contribution value of the speech features under each dimension, target speech features are selected from the multidimensional speech features. Based on the target speech features, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the speech data; When the noise estimation module is used to filter target speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension, it is specifically used for: The speech features with the smallest contribution value in each dimension are deleted to obtain the remaining speech features. Determine the contribution value of each of the remaining speech features, and repeat the above steps until the loss condition is met; the loss condition is used to indicate the degree of impact of deleting speech features on noise estimation. The speech features that meet the loss condition are determined as the target speech features.

8. A neural network training device for speech enhancement, characterized in that, The device includes: The sample speech acquisition module is used to acquire sample speech data from sample videos. The sample speech data corresponds to sample clean speech data, which does not contain noise. The sample feature extraction module is used to extract multidimensional speech features from the sample speech data; The sample noise estimation module is used to perform speech environment complexity analysis based on the multidimensional speech features and determine the noise estimation value corresponding to the sample speech data. The sample image acquisition module is used to acquire sample image data from the sample video when the noise estimate is greater than the target threshold. The sample image data includes an image of the mouth region of the speaker who generated the sample speech data. The sample multimodal enhancement module is used to perform multimodal speech enhancement processing on the sample speech data based on the sample image data to obtain enhanced sample speech data. The sample single-mode enhancement module is used to perform single-mode speech enhancement processing on the sample speech data when the noise estimate is less than or equal to the target threshold, so as to obtain enhanced sample speech data. The parameter adjustment module is used to adjust the parameters of the neural network used for speech enhancement based on the error loss between the enhanced sample speech data and the sample clean speech data, until the training cutoff condition is met. The neural network includes a feature extraction module, a noise estimation module, and a speech enhancement module. The feature extraction module is used to extract features from the sample speech data. The noise estimation module is used to determine the noise estimation value corresponding to the sample speech data. The speech enhancement module is used to enhance the sample speech data based on sample image data. The sample noise estimation module is specifically used for: Determine the contribution value of the speech features under each dimension, and the contribution value is used to characterize the degree of influence of the speech features under that dimension on the speech environment complexity analysis; Based on the contribution value of the speech features under each dimension, the target sample speech features are selected from the multidimensional speech features. Based on the speech features of the target sample, a speech environment complexity analysis is performed to determine the noise estimate corresponding to the sample speech data; When the noise estimation module is used to select target sample speech features from the multidimensional speech features based on the contribution values ​​of the speech features in each dimension, it is specifically used for: The speech features with the smallest contribution value in each dimension are deleted to obtain the remaining speech features. Determine the contribution value of each of the remaining speech features, and repeat the above steps until the loss condition is met; the loss condition is used to indicate the degree of impact of deleting speech features on noise estimation. The speech features that meet the loss condition are determined as the speech features of the target sample.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the speech enhancement method according to any one of claims 1 to 5 or the steps of the neural network training method for speech enhancement according to claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the speech enhancement method according to any one of claims 1 to 5 or the steps of the neural network training method for speech enhancement according to claim 6.

Images