Directional pickup method, system and device and storage medium

Abstract

The invention provides a directional pickup method, system and device and a storage medium, and relates to the field of pickup. The method comprises the following steps: performing MVDR beam forming processing on an initial voice signal input by a microphone array, and then extracting a BFCC feature and a fundamental tone frequency feature of the voice signal; and processing the BFCC features by using a pre-trained voice / noise separation DNN to obtain a time-frequency mask. And according to the fundamental tone frequency characteristics, adaptively designing comb filter parameters, and carrying out comb filtering on the initial voice signal. And taking the time-frequency mask as a weighting coefficient, respectively multiplying the weighting coefficient with the voice signal after comb filtering and the initial voice signal, and estimating target voice energy and noise energy. And calculating a Bark gain coefficient related to the frequency on the Bark frequency scale, wherein the Bark gain coefficient is used for enhancing the voice signal in the time-frequency domain. And finally, converting the enhanced voice signal into a time domain signal through inverse BFCC conversion to obtain a target enhanced voice pointing to the target direction. Through the scheme, the directional pickup effect is improved.

Description

Technical Field

[0001] The present application relates to the field of sound pickup, and in particular to a directional sound pickup method, system, device and storage medium. Background Art

[0002] Speech is one of the main ways for people to communicate and transfer information in daily life. In real-world environments, speech signals are often accompanied by various background noises, such as environmental noise, reverberation, and the voices of other speakers. These noise interferences seriously affect the clarity and intelligibility of speech, bringing huge challenges to applications such as speech interaction and remote communication.

[0003] In order to improve the quality of speech signals and suppress background noise, a variety of speech enhancement methods have been proposed in the prior art. Among them, a common method is beamforming technology based on microphone arrays. This technology spatially filters the multi-channel speech signals collected by the microphone array to form spatial directivity pointing to the target sound source, thereby extracting the target speech and suppressing noise interference in non-target directions. However, traditional beamforming technology usually only considers the spatial characteristics of speech signals, and has limited suppression effects on non-stationary noise and reverberation, resulting in poor sound pickup effects. Summary of the invention

[0004] The present application provides a directional sound pickup method, system, device and storage medium, which can improve the directional sound pickup effect.

[0005] In a first aspect, the present application provides a directional sound pickup method, the method comprising: Performing MVDR beamforming processing on the initial speech signal input by the microphone array to obtain the initial speech signal after beamforming; Extract features of the initial speech signal to obtain BFCC features and fundamental frequency features; Use the pre-trained speech / noise separation DNN to process the BFCC features and obtain the time-frequency mask; Adaptively designing the parameters of the comb filter according to the fundamental frequency characteristics to obtain a target comb filter, and filtering the initial speech signal based on the target comb filter to obtain a speech signal after comb filtering; The time-frequency mask is converted into a speech time-frequency mask, the speech time-frequency mask is used as a weighting coefficient and multiplied with the speech signal after comb filtering to obtain the energy of the weighted signal, and the target speech energy estimation value is determined based on the energy of the weighted signal; The time-frequency mask is used as a weighting coefficient to multiply the initial speech signal to obtain a noise energy estimate; Calculate the frequency-dependent Bark gain coefficient on the Bark frequency scale according to the target speech energy estimate and the noise energy estimate; The speech signal after comb filtering is multiplied by the Bark gain coefficient, and the initial speech signal is enhanced in the time-frequency domain to obtain an enhanced speech signal; The enhanced speech signal is converted from the Bark frequency scale to a time domain signal using an inverse BFCC transform to obtain the target enhanced speech pointing in the target direction.

[0006] By adopting the above technical solution, the initial speech signal input by the microphone array is processed by MVDR beamforming to form a spatial filter pointing in the direction of the target speech, which can effectively suppress noise interference in non-target directions and lay a good foundation for subsequent feature extraction and speech enhancement.

[0007] Then, the BFCC features and fundamental frequency features of the speech signal after beamforming are extracted to make full use of the characteristics of speech in the frequency domain and time domain. Among them, the BFCC feature uses the Bark frequency scale, which is more compatible with the hearing characteristics of the human ear and is helpful for subsequent speech / noise separation and gain adjustment. The fundamental frequency feature provides key parameters for the design of the adaptive comb filter, enabling it to specifically enhance the fundamental frequency and its harmonic components of the speech.

[0008] Next, the BFCC features are processed using a pre-trained speech / noise separation DNN to obtain a time-frequency mask. This DNN can fully exploit the distinguishing features of speech and noise in the time-frequency domain, thereby accurately estimating the time-frequency regions dominated by speech and noise. As an important basis for subsequent energy estimation and gain calculation, the time-frequency mask can more finely control the strength and effect of speech enhancement.

[0009] On this basis, an adaptive comb filter is designed to filter the speech signal, further enhancing the periodic components of the speech and weakening the non-periodic noise. At the same time, the target speech energy and noise energy are estimated respectively according to the time-frequency mask, and the Bark gain coefficient is calculated on the Bark frequency scale. This gain coefficient fully considers the energy distribution of speech and noise in different frequency bands, giving greater gain to the time-frequency area dominated by speech, and more suppression to the time-frequency area dominated by noise, thus retaining the important components of speech to the maximum extent while reducing noise.

[0010] Finally, the speech signal after comb filtering is multiplied by the Bark gain coefficient to enhance the speech, and the enhanced speech signal is converted into the time domain through the inverse BFCC transform to obtain the enhanced speech pointing in the target direction. Compared with the traditional speech enhancement method, the present invention makes full use of the multi-dimensional characteristics of speech and introduces the Bark frequency scale that conforms to the auditory characteristics of the human ear. Therefore, it has obvious advantages in improving speech clarity and subjective listening quality.

[0011] In a second aspect of the present application, a directional sound pickup method system is provided, comprising: A data acquisition module is used to perform MVDR beamforming processing on the initial speech signal input by the microphone array to obtain the initial speech signal after beamforming; The feature extraction module is used to extract features from the initial speech signal to obtain BFCC features and fundamental frequency features; A special processing module is used to process the BFCC features using a pre-trained speech / noise separation DNN to obtain a time-frequency mask; A filtering module is used to adaptively design the parameters of a comb filter according to the fundamental frequency characteristics to obtain a target comb filter, and filter the initial speech signal based on the target comb filter to obtain a speech signal after comb filtering; A target speech energy estimation module, used for converting the time-frequency mask into a speech time-frequency mask, multiplying the speech time-frequency mask with the comb-filtered speech signal as a weighting coefficient to obtain the energy of the weighted signal, and determining the target speech energy estimation value based on the energy of the weighted signal; A noise energy estimation module, used for multiplying the time-frequency mask as a weighting coefficient with the initial speech signal to obtain a noise energy estimation value; A gain coefficient calculation module, used for calculating a frequency-dependent Bark gain coefficient on a Bark frequency scale according to a target speech energy estimate and a noise energy estimate; A first signal enhancement module, used for multiplying the speech signal after comb filtering by the Bark gain coefficient, enhancing the initial speech signal in the time-frequency domain to obtain an enhanced speech signal; The second signal enhancement module is used to convert the enhanced speech signal from the Bark frequency scale into a time domain signal by using an inverse BFCC transform to obtain a target enhanced speech pointing in a target direction.

[0012] In a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above method steps.

[0013] In the fourth aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the above method.

[0014] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. This application obtains the product information of the Chinese medicine to be detected, and determines the data acquisition standard based on the product information and the preset detection rules, and then obtains the standard image data and the standard spectral data. This process fully considers the differences in morphology, composition, etc. of different Chinese medicine varieties, as well as the pertinence and normative requirements of quality inspection, and ensures the accuracy and reliability of subsequent analysis from the source of the data. This application obtains the product information of the Chinese medicine to be detected, and determines the data acquisition standard based on the product information and the preset detection rules, and then obtains the standard image data and the standard spectral data. This process fully considers the differences in morphology, composition, etc. of different Chinese medicine varieties, as well as the pertinence and normative requirements of quality inspection, and ensures the accuracy and reliability of subsequent analysis from the source of the data.

[0015] 2. This application preprocesses the acquired standard image data and standard spectral data, and extracts features respectively to obtain visual feature data and spectral feature data. This process effectively eliminates the redundancy and noise of the original data, extracts the key features that best reflect the characteristics of traditional Chinese medicine, and lays the foundation for subsequent anomaly detection. At the same time, the image data is input into a preset image anomaly detection model to obtain a preliminary abnormal area, which further guides the subsequent feature fusion and discriminant analysis.

[0016] 3. This application performs feature fusion on visual feature data and spectral feature data to obtain an abnormal feature vector. The traditional single feature analysis method is difficult to fully characterize the characteristics of traditional Chinese medicine and easily causes detection blind spots. However, this application fully explores the complementary information between different modal data through the fusion of multi-source heterogeneous features, forming a more comprehensive and accurate description of the characteristics of traditional Chinese medicine, greatly improving the pertinence and reliability of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of a directional sound pickup method provided in an embodiment of the present application; Figure 2 An architectural diagram of a directional sound pickup system provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0018] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0019] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0020] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0021] In order to clearly and completely describe the technical solutions in the embodiments of the present application in combination with the drawings in the embodiments of the present application below, it is obvious that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0022] Based on the above background technology, please refer to Figure 1 , Figure 1 A flow chart of a directional sound pickup method provided in an embodiment of the present application. The system can be implemented by a computer program or can be run as an independent tool application. In a preferred embodiment of the present invention, the method can be applied on a server in an embodiment of the present application, but can also be applied on electronic devices such as servers. A directional sound pickup method includes the following steps: S101, performing MVDR beamforming processing on an initial speech signal input by a microphone array to obtain an initial speech signal after beamforming; In a preferred embodiment of the present invention, firstly, an MVDR (minimum variance distortionless response) beamforming process is performed on an initial speech signal input by a microphone array to obtain an initial speech signal after beamforming.

[0023] MVDR beamforming is a commonly used spatial filtering technology, which aims to minimize interference and noise from other directions while keeping the target direction signal undistorted. Specifically, by weighted summing the multi-channel speech signals collected by the microphone array, a spatial filter pointing to the target sound source is formed, so that its gain in the target direction is 1, and the gain in other directions is as small as possible, thereby achieving the extraction of the target signal and the suppression of noise.

[0024] In this step, the steering vector of the microphone array is first determined based on the geometric arrangement information of the microphone array and the azimuth information of the target speech signal. The vector represents the propagation delay and attenuation difference of the target signal from the sound source to each microphone. Then, the optimal weight coefficient of the MVDR beamformer is calculated using the spatial covariance matrix and steering vector of the speech signal to minimize the variance of its output signal. Finally, the weight coefficient is weighted and summed with the microphone signals of each channel to obtain the initial speech signal after beamforming.

[0025] Through MVDR beamforming processing, the target speech signal can be effectively extracted in the spatial domain, and noise interference in non-target directions can be suppressed, laying a good foundation for subsequent feature extraction and speech enhancement processing. Compared with the traditional delay-sum beamformer, the MVDR beamformer can adaptively adjust the weights of each signal, and obtain a higher noise suppression effect without introducing speech distortion, so that the speech signal after beamforming has a higher signal-to-noise ratio and clarity.

[0026] It should be noted that the spatial covariance matrix is ​​a statistic that describes the spatial correlation between multiple speech signals. Its diagonal elements represent the power of each signal, and the non-diagonal elements represent the correlation coefficients between different signals. By performing eigendecomposition on the spatial covariance matrix, the subspaces of the noise and target signals can be estimated, and then the weight coefficients of the MVDR beamformer can be optimized to achieve adaptive noise suppression.

[0027] Based on the above embodiment, as an optional embodiment, the initial speech signal input by the microphone array is subjected to MVDR beamforming processing to obtain the initial speech signal after beamforming, including: S201, determining an array manifold vector according to geometric arrangement information of the microphone array and azimuth information of an initial speech signal; In a preferred embodiment of the present invention, when performing MVDR beamforming processing on the initial speech signal input by the microphone array, the first step of the method of the present invention is to determine the array manifold vector based on the geometric arrangement information of the microphone array and the azimuth information of the initial speech signal.

[0028] A microphone array is an array composed of multiple microphones arranged in a specific geometric pattern. Common arrangements include linear arrays, circular arrays, and planar arrays. The geometric arrangement information of a microphone array includes parameters such as the number of microphones, position coordinates, and spacing. These parameters determine the spatial sampling characteristics and directional resolution capabilities of the array. The azimuth information of the initial speech signal indicates the azimuth of the speech signal arriving at the microphone array, that is, the angle between the speech signal and the normal direction of the microphone array. The azimuth information can be estimated by a sound source localization algorithm, such as a localization algorithm based on time delay estimation or a localization algorithm based on subspace decomposition.

[0029] The array manifold vector is a vector that represents the response characteristics of the microphone array to the speech signal in a specific direction, and its dimension is equal to the number of microphones. The array manifold vector is closely related to the geometric arrangement of the microphone array and the incident direction of the speech signal. Different geometric arrangements and incident directions will lead to different array manifold vectors. The purpose of determining the array manifold vector is to optimize the weight vector in the subsequent MVDR beamforming so that the beamformed speech signal obtains the maximum gain in the target direction and the minimum gain in other directions, thereby achieving spatial filtering pointing to the target direction.

[0030] The specific implementation process is as follows: First, the steering vector of the microphone array is calculated based on the geometric arrangement information of the microphone array. The steering vector is a complex vector that represents the phase difference between the signals received by each microphone when a plane wave is incident on the microphone array from a specific direction. The calculation formula of the steering vector is: a(θ) = [1, e^(-j2πfτ_1), ..., e^(-j2πfτ_M-1)]^T Where θ is the azimuth of the speech signal, f is the frequency of the speech signal, τ_m is the time delay of the speech signal from the reference microphone to the mth microphone, and M is the number of microphones. The time delay τ_m can be calculated based on the position coordinates of the microphone and the azimuth of the speech signal.

[0031] Then, the calculated steering vector is normalized to obtain a unit steering vector d(θ). The purpose of normalization is to eliminate the amplitude difference of the steering vector and make it a unit vector, which is convenient for subsequent weight calculation and optimization.

[0032] Finally, the unit steering vector d(θ) is used as the array manifold vector to characterize the response characteristics of the microphone array to the speech signal in the target direction.

[0033] By determining the array manifold vector based on the geometric arrangement information of the microphone array and the azimuth information of the initial speech signal, a spatial response model of the microphone array is established, laying the foundation for subsequent MVDR beamforming processing. The array manifold vector accurately describes the gain characteristics of the microphone array for the speech signal in the target direction and is a key factor in achieving spatial filtering and directionality enhancement.

[0034] S202, calculating the weight coefficient of the MVDR beamformer according to the spatial covariance matrix of the initial speech signal and the array manifold vector to obtain the MVDR weight coefficient; In a preferred embodiment of the present invention, after determining the array manifold vector, the next step of the method of the present invention is to calculate the weight coefficients of the MVDR beamformer based on the spatial covariance matrix of the initial speech signal and the array manifold vector to obtain the MVDR weight coefficients.

[0035] The spatial covariance matrix is ​​a matrix that represents the correlation and energy distribution of the speech signal received by the microphone array between different microphones. It contains the statistical characteristics of the speech signal and noise in space, and reflects the spatial distribution pattern of the speech signal and noise on the microphone array. The calculation of the spatial covariance matrix is ​​usually based on the short-term analysis of the speech signal, and is obtained by estimating the samples of the speech signal over a period of time.

[0036] The MVDR (Minimum Variance Distortionless Response) beamformer is an adaptive beamforming algorithm that aims to minimize the variance of the beamforming output signal while keeping the target direction speech signal undistorted, thereby suppressing noise and interference. The core of the MVDR beamformer is the calculation of the weight coefficient, which determines the gain and attenuation of the beamformer for signals in different directions.

[0037] The purpose of calculating the MVDR weight coefficients based on the spatial covariance matrix of the initial speech signal and the array manifold vector is to utilize the spatial distribution characteristics of the speech signal and noise to adaptively adjust the weights of the beamformer so that it can enhance the speech signal in the target direction and suppress the noise in the non-target direction, thereby achieving spatial filtering and speech enhancement.

[0038] The specific implementation process is as follows: First, the spatial covariance matrix is ​​calculated based on the initial speech signal. The speech signal received by the microphone array is divided into several short-time frames, the sample covariance matrix is ​​calculated for each frame, and then the sample covariance matrices of all frames are averaged to obtain the spatial covariance matrix R_xx. The calculation formula of the spatial covariance matrix is: R_xx = E[x(t)x^H(t)] Where x(t) is the speech signal vector received by the microphone array, x^H(t) is the conjugate transpose of x(t), and E[] represents the mathematical expectation.

[0039] Then, the MVDR weight coefficient is calculated using the spatial covariance matrix R_xx and the array manifold vector d(θ). The calculation formula of the MVDR weight coefficient is: w_mvdr = (R_xx^(-1)d(θ)) / (d^H(θ)R_xx^(-1)d(θ)) Where R_xx^(-1) is the inverse matrix of the spatial covariance matrix, d(θ) is the array manifold vector in the target direction, and d^H(θ) is the conjugate transpose of d(θ).

[0040] The MVDR weight coefficient w_mvdr is a complex vector whose dimension is equal to the number of microphones. It represents the weighting coefficient of the beamformer for different microphone signals, so that the output signal of the beamformer obtains the maximum gain in the target direction and the minimum gain in other directions.

[0041] Finally, the calculated MVDR weight coefficient is applied to the beamformer to perform spatial filtering and enhancement on the initial speech signal. The beamformed speech signal y(t) can be expressed as: y(t) = w_mvdr^H * x(t) Where w_mvdr^H is the conjugate transpose of the MVDR weight coefficient, and x(t) is the speech signal vector received by the microphone array.

[0042] Adaptive spatial filtering and speech enhancement are achieved by calculating the weight coefficients of the MVDR beamformer based on the spatial covariance matrix and array manifold vector of the initial speech signal. Compared with the fixed-weight beamformer, the MVDR beamformer can dynamically adjust the weight coefficients according to the spatial distribution characteristics of the speech signal and noise, thereby achieving better noise suppression and speech enhancement effects.

[0043] S203, weighted summing of the initial speech signal is performed using the MVDR weight coefficient to form a spatial filter pointing in the direction of the target speech, and MVDR beamforming processing is performed on the initial speech signal based on the spatial filter to obtain the initial speech signal after beamforming.

[0044] In a preferred embodiment of the present invention, after calculating the MVDR weight coefficient, the next step of the method of the present invention is to use the MVDR weight coefficient to perform weighted summation on the initial speech signal to form a spatial filter pointing in the direction of the target speech, and perform MVDR beamforming processing on the initial speech signal based on the spatial filter to obtain the initial speech signal after beamforming.

[0045] Spatial filter is a signal processing technology based on microphone array. It enhances the speech signal in the target direction and suppresses the noise in the non-target direction by weighted combination of speech signals received by different microphones. The core of spatial filter is weight coefficient, which determines the contribution of different microphone signals in the combination.

[0046] MVDR beamforming is a commonly used spatial filtering technology. It uses the MVDR criterion to calculate the optimal weight coefficient so that the beamformed speech signal obtains the maximum gain in the target direction and the minimum gain in other directions, thereby achieving spatial filtering pointing to the target speech direction.

[0047] The purpose of weighted summing of the initial speech signal using the MVDR weight coefficient is to form the optimal spatial filter according to the MVDR criterion, enhance the speech signal in the target direction, and suppress the noise in the non-target direction. Through weighted summation, the speech signals received by different microphones are assigned different weights, so that the speech signal after beamforming has a higher signal-to-noise ratio and directionality.

[0048] The specific implementation process is as follows: First, the MVDR weight coefficient w_mvdr is weighted and summed with the speech signal x(t) received by the microphone array to obtain the beamformed speech signal y(t). The weighted summation process can be expressed as: y(t) = w_mvdr^H * x(t) Where w_mvdr^H is the conjugate transpose of the MVDR weight coefficient, x(t) is the speech signal vector received by the microphone array, and y(t) is the speech signal after beamforming.

[0049] The weighted summation process can be regarded as a spatial filter, whose frequency response is determined by the MVDR weight coefficient. By selecting the appropriate MVDR weight coefficient, the spatial filter can have a high gain in the direction of the target speech and a low gain in the non-target direction, thereby achieving spatial filtering in the direction of the target speech.

[0050] Then, the beamformed speech signal y(t) is used as the output of the MVDR beamforming process to obtain the initial speech signal after beamforming.

[0051] The essence of MVDR beamforming processing is to optimize the initial speech signal using a spatial filter, so that it can minimize the variance of the output signal while keeping the speech signal in the target direction undistorted, thereby suppressing noise and interference. Adaptive spatial filtering is achieved through the weighted summation of MVDR weight coefficients, so that the speech signal after beamforming has higher quality and clarity.

[0052] Finally, the initial speech signal after beamforming is used as the input of the subsequent speech enhancement algorithm for further processing and optimization.

[0053] S102, extracting features from the initial speech signal to obtain BFCC features and fundamental frequency features; In a preferred embodiment of the present invention, after the initial speech signal is subjected to MVDR beamforming processing, the next step of the method of the present invention is to extract features of the initial speech signal after beamforming to obtain BFCC features and fundamental frequency features.

[0054] BFCC features, namely Bark frequency cepstral coefficients, are a method for extracting frequency domain features of speech signals on the Bark frequency scale. Similar to traditional MFCC (Mel frequency cepstral coefficients), BFCC features are also obtained by a series of transformations and compressions on the spectrum of speech signals. The difference is that BFCC uses the Bark frequency scale, which is more in line with the auditory characteristics of the human ear and is closer to human hearing in terms of frequency resolution and perceived pitch. The specific steps for extracting BFCC features are as follows: First, perform short-time Fourier transform on the initial speech signal after beamforming to obtain the spectrum; then, map the spectrum to the Bark frequency scale to obtain the Bark spectrum; then, take the logarithm of the Bark spectrum to obtain the logarithmic Bark spectrum; finally, perform discrete cosine transform on the logarithmic Bark spectrum to obtain the BFCC feature coefficients. By extracting BFCC features, the characteristics of speech signals on the Bark frequency scale can be fully utilized to provide effective frequency domain feature representation for subsequent speech / noise separation and speech enhancement.

[0055] The fundamental frequency feature, also known as the pitch feature, reflects the basic periodicity of the speech signal. For voiced sounds, the fundamental frequency corresponds to the frequency of the vocal cord vibration and is the lowest harmonic frequency in the speech signal. Extracting the fundamental frequency feature can provide key time domain feature information for subsequent adaptive filtering and speech enhancement. Commonly used fundamental frequency estimation methods include the autocorrelation function method, the average amplitude difference function method, the cepstrum method, etc. Here, the autocorrelation function method is used to estimate the fundamental frequency. The specific steps are as follows: first, the initial speech signal after beamforming is framed; then, the autocorrelation function is calculated for each frame of speech signal; then, the delay time corresponding to the first maximum point in the autocorrelation function is found, which is the fundamental frequency period of the frame of speech; finally, the inverse of the fundamental frequency period is used as the estimated value of the fundamental frequency of the frame of speech. By extracting the fundamental frequency feature, the periodic structure of the speech signal can be accurately characterized, providing important parameters for the subsequent adaptive filter design.

[0056] Based on the above embodiment, as an optional embodiment, feature extraction is performed on the initial speech signal to obtain BFCC features, including: Perform short-time Fourier transform on the initial speech signal to obtain the spectrum; Map the spectrum to the Bark frequency scale to obtain the Bark spectrum; Take the logarithm of the Bark spectrum to obtain the logarithmic Bark spectrum; The logarithmic Bark spectrum is subjected to discrete cosine transform to obtain the BFCC feature.

[0057] In a preferred embodiment of the present invention, the initial speech signal is subjected to a short-time Fourier transform (STFT) to obtain a spectrum. Short-time Fourier transform is a time-frequency analysis method that divides the speech signal into a number of short-time frames, and performs a Fourier transform on each frame to obtain the spectrum of the frame. Through STFT, the frequency components of the speech signal at different time points can be obtained, revealing the time-varying characteristics of the speech signal.

[0058] The spectrum represents the energy distribution of the speech signal at different frequency points, but it does not directly reflect the auditory perception characteristics of the human ear. In order to better characterize the perceptual characteristics of the speech signal, the spectrum needs to be mapped to a frequency scale that matches the auditory characteristics of the human ear.

[0059] Therefore, the next step is to map the spectrum to the Bark frequency scale to obtain the Bark spectrum. The Bark frequency scale is a nonlinear frequency scale based on the critical bandwidth of the human ear. It divides the frequency range into several critical bands, and the frequency components in each critical band have similar contributions to the perception of the human ear. By mapping the spectrum to the Bark frequency scale, the perceptual characteristics of the speech signal can be more accurately characterized, highlighting the frequency components that the human ear is sensitive to.

[0060] The process of mapping to the Bark frequency scale can be achieved through the Bark frequency conversion formula, which converts the linear frequency into the Bark frequency. The Bark frequency conversion formula is: B(f) = 13arctan(0.00076f) + 3.5arctan((f / 7500)^2) Where f is the linear frequency and B(f) is the corresponding Bark frequency.

[0061] After obtaining the Bark spectrum, in order to further highlight the dynamic changes in energy in different frequency bands, the logarithm of the Bark spectrum is taken to obtain the logarithmic Bark spectrum. The logarithmic operation can compress the dynamic range of the spectrum, making the energy difference between different frequency bands smoother, which conforms to the loudness perception law of the human ear. At the same time, the logarithmic operation can also convert the multiplication operation of the spectrum into an addition operation, simplifying subsequent processing.

[0062] Finally, the logarithmic Bark spectrum is subjected to discrete cosine transform (DCT) to obtain the BFCC feature. Discrete cosine transform is an orthogonal transform that can compress spectral information into a few low-order coefficients while having good decorrelation and energy concentration. Through DCT, compact and independent BFCC feature coefficients can be obtained to effectively characterize the timbre, pitch and other characteristics of the speech signal.

[0063] The calculation formula of BFCC characteristic is: BFCC(i) = sqrt(2 / N) * sum(log(Bark(j)) * cos(pi i (j-0.5) / N)), i=1,2,...,M Where N is the number of Bark bands, M is the order of the BFCC feature, and Bark(j) is the energy value of the jth Bark band.

[0064] S103, using a pre-trained speech / noise separation DNN to process the BFCC features to obtain a time-frequency mask; In a preferred embodiment of the present invention, after obtaining the BFCC features of the initial speech signal, the next step of the method of the present invention is to process the BFCC features using a pre-trained speech / noise separation deep neural network (DNN) to obtain a time-frequency mask.

[0065] Speech / noise separation DNN is a speech enhancement model based on deep learning, which aims to separate clear target speech and background noise from noise-contaminated speech signals. Through the powerful nonlinear mapping and representation learning capabilities of DNN, effective speech and noise distinction information can be extracted from the features of noisy speech, thereby achieving the separation of speech and noise. In the method of the present invention, a large number of pure speech and various noise samples are used in advance to train the speech / noise separation DNN, so that it learns to estimate the time-frequency region dominated by speech and noise from the BFCC features of noisy speech, and obtain a time-frequency mask (also called an ideal binary mask).

[0066] Specifically, the time-frequency mask is a binary matrix of the same size as the time-frequency spectrum of the speech signal, where an element with a value of 1 indicates that the time-frequency point is mainly dominated by speech, and an element with a value of 0 indicates that the time-frequency point is mainly dominated by noise. The time-frequency mask provides an ideal speech / noise separation standard, that is, by multiplying the time-frequency mask with the time-frequency spectrum of the speech signal, the ideal pure speech time-frequency spectrum can be obtained.

[0067] In the implementation process of this step, the extracted BFCC features are first input into the pre-trained speech / noise separation DNN. The DNN maps the BFCC features to a high-dimensional space through its internal multi-layer nonlinear transformation, and extracts the distinguishing features of speech and noise. Then, the output layer of the DNN converts these distinguishing features into probability values ​​between 0 and 1 through the sigmoid activation function, indicating the possibility that each time-frequency point belongs to speech or noise. Finally, the probability value of the output layer is binarized according to a certain threshold to obtain the final time-frequency mask.

[0068] By using a pre-trained speech / noise separation DNN to estimate the time-frequency mask, the distribution of speech and noise can be finely characterized in the time-frequency domain, providing important prior information for subsequent speech enhancement processing. The time-frequency mask not only indicates the time-frequency region dominated by speech and noise, but can also be used as a weighting coefficient to guide the signal synthesis and filtering operations in the speech enhancement process, so that the enhanced speech can retain the details and naturalness of the speech to the maximum extent while reducing noise.

[0069] Based on the above embodiment, as an optional embodiment, a pre-trained speech / noise separation DNN is used to process the BFCC feature to obtain a time-frequency mask, including: The BFCC features are input into the pre-trained speech / noise separation DNN, and the local features of the BFCC features are extracted through the convolutional neural network of the pre-trained speech / noise separation DNN; The recursive neural network of the pre-trained speech / noise separation DNN is used to model the temporal context information of the local features of the BFCC features to obtain the temporal context information; The local features of the BFCC features and the temporal context information are fused through the fully connected layer of the pre-trained speech / noise separation DNN to output a soft mask. The soft mask is binarized to obtain a time-frequency mask.

[0070] In a preferred embodiment of the present invention, after obtaining the BFCC features, the next step of the method of the present invention is to process the BFCC features using a pre-trained speech / noise separation DNN to obtain a time-frequency mask. Speech / noise separation is a core task of speech enhancement, which aims to separate clear speech components from noise-contaminated speech signals. Traditional speech / noise separation methods are usually based on statistical models or signal processing techniques, while in recent years, speech / noise separation methods based on deep neural networks (DNNs) have made significant progress.

[0071] The pre-trained speech / noise separation DNN is a speech enhancement model based on deep learning. It is trained with a large amount of speech and noise data to learn the distinguishing features of speech and noise, thereby achieving effective separation of speech and noise. Compared with traditional methods, the pre-trained speech / noise separation DNN can automatically learn and extract high-level features of speech signals, and has stronger generalization ability and robustness.

[0072] The purpose of using a pre-trained speech / noise separation DNN to process BFCC features is to use the powerful feature learning and mapping capabilities of DNN to extract the distinguishing information between speech and noise from the BFCC features, generate a time-frequency mask, and guide the subsequent speech enhancement process. The time-frequency mask is a binary matrix of the same size as the time-frequency representation of the speech signal, where 1 represents the time-frequency point dominated by speech and 0 represents the time-frequency point dominated by noise. Through the time-frequency mask, speech and noise can be selectively retained and suppressed in the time-frequency domain, thereby achieving speech enhancement.

[0073] The specific implementation process is as follows: First, the BFCC features are input into the pre-trained speech / noise separation DNN, and the local features of the BFCC features are extracted through the convolutional neural network (CNN) of the DNN. The convolutional neural network is a special neural network structure that can effectively extract local features of the input data through local connections and weight sharing. In the speech / noise separation task, the convolutional neural network can capture the local patterns and correlations of the BFCC features in the time-frequency domain and extract the distinguishing information between speech and noise.

[0074] Specifically, the convolutional neural network performs convolution operations on the BFCC features through sliding convolution kernels to generate a series of feature maps. Each convolution kernel can be regarded as a local feature extractor, which scans different regions of the BFCC features to extract local time-frequency patterns. The mathematical expression of the convolution operation is: F(i,j) = sum(W(m,n) * BFCC(im,jn)) + b Among them, F(i,j) is the feature map after convolution, W(m,n) is the weight coefficient of the convolution kernel, BFCC(im,jn) is the local area of ​​the BFCC feature, and b is the bias term.

[0075] Through layer-by-layer extraction of multi-layer convolutional neural networks, local feature representations of BFCC features at different scales and abstract levels can be obtained.

[0076] Then, the recurrent neural network (RNN) of the pre-trained speech / noise separation DNN is used to model the temporal context information of the local features of the BFCC features to obtain the temporal context information. The recurrent neural network is a neural network structure suitable for processing sequence data. It can capture the dependency and context information of the data in the time dimension through recurrent connections and the transfer of internal states. In the speech / noise separation task, the recurrent neural network can model the continuity and contextual relevance of the BFCC features in the time dimension and extract the temporal distinction information of speech and noise.

[0077] Specifically, the recurrent neural network gradually models the temporal context information of the BFCC feature by iteratively updating the internal state and output. The mathematical expression of the recurrent neural network is: h(t) = f(W(h)*h(t-1) + W(x)*x(t) + b) y(t) = g(W(y)*h(t) + c) Among them, h(t) is the internal state of the recurrent neural network at time t, x(t) is the input feature at time t, y(t) is the output at time t, W(h), W(x), W(y) are weight matrices, b and c are bias terms, and f and g are activation functions.

[0078] Through the iterative calculation of the recursive neural network, the contextual information of the BFCC feature in the time dimension can be obtained, capturing the time domain evolution characteristics of speech and noise.

[0079] Next, the local features of the BFCC features and the temporal context information are fused through the fully connected layer (FC) of the pre-trained speech / noise separation DNN to output a soft mask. The fully connected layer is a common neural network structure that maps input features to the output space through linear transformation and nonlinear activation. In the speech / noise separation task, the fully connected layer can integrate the local features extracted by the convolutional neural network and the temporal context information extracted by the recurrent neural network to generate a soft mask that represents the distinction between speech and noise.

[0080] Specifically, the fully connected layer fuses local features and temporal context information together through matrix multiplication and nonlinear transformation to generate a soft mask of the same size as the BFCC feature time-frequency representation. The mathematical expression of the fully connected layer is: M(i,j) = sigmoid(W(fc)*[F(i,j);H(i)] + b) Among them, M(i,j) is the time-frequency point value of the soft mask, W(fc) is the weight matrix of the fully connected layer, [F(i,j);H(i)] is the concatenation of local features and time domain context information, b is the bias term, and sigmoid is the activation function.

[0081] The soft mask value range is [0,1], which indicates the probability of each time-frequency point belonging to speech or noise. A value close to 1 indicates that the time-frequency point mainly contains speech components, and a value close to 0 indicates that the time-frequency point mainly contains noise components.

[0082] Finally, the soft mask is binarized to obtain a time-frequency mask. Binarization is the process of converting the probability value in the soft mask to 0 or 1 to obtain a binary time-frequency mask. This process can be achieved by setting a threshold. The time-frequency points greater than the threshold are set to 1, indicating speech dominance; the time-frequency points less than or equal to the threshold are set to 0, indicating noise dominance. The mathematical expression of binarization is: BM(i,j) = (M(i,j)>threshold) 1 : 0 Among them, BM(i,j) is the time-frequency mask after binarization, and threshold is the binarization threshold.

[0083] Through binarization, the soft mask can be converted into a binary matrix of the same size as the BFCC feature time-frequency representation, where 1 represents the time-frequency point dominated by speech and 0 represents the time-frequency point dominated by noise. The time-frequency mask provides clear guidance information for the subsequent speech enhancement process, telling which time-frequency points should retain the speech components and which time-frequency points should suppress the noise components.

[0084] S104, adaptively designing the parameters of a comb filter according to the fundamental frequency characteristics to obtain a target comb filter, and filtering the initial speech signal based on the target comb filter to obtain a speech signal after comb filtering; In a preferred embodiment of the present invention, a comb filter is a linear filter that can enhance the periodic components of a speech signal and suppress non-periodic noise. Its frequency response presents a series of evenly distributed peaks, similar to the shape of a comb, hence the name. The key parameter of the comb filter is the interval of its peak frequency, which should match the fundamental frequency of the speech signal to achieve selective enhancement of the speech fundamental frequency and its harmonics.

[0085] In the implementation process of this step, firstly, according to the fundamental frequency characteristics extracted in the previous step, the parameters of the comb filter are adaptively designed. Specifically, the reciprocal of the fundamental frequency is used as the delay interval parameter of the comb filter, and the peak frequency of its frequency response is aligned with the fundamental frequency and its harmonics by adjusting the feedback coefficient and feedforward coefficient of the filter. Since the fundamental frequency may change between different time frames, it is necessary to dynamically adjust the parameters of the filter according to the estimated value of the fundamental frequency of each frame to achieve adaptive filtering.

[0086] After obtaining the target comb filter, it is applied to the filtering process of the initial speech signal. The specific operation is to pass the initial speech signal through the designed comb filter and perform a time domain convolution operation on it. During the filtering process, the frequency components corresponding to the fundamental frequency and its harmonics will be selectively enhanced, while other frequency components will be relatively suppressed. In this way, the comb filter can effectively extract the periodic components in the speech signal, weaken the influence of non-periodic noise, and make the filtered speech signal clearer and more stable.

[0087] The speech signal after comb filtering not only retains the basic structure and rhythmic features of speech, but also has a higher harmonic-to-noise ratio (HNR), that is, the energy of the speech harmonic component is more prominent than the energy of the noise component. This periodic enhancement effect helps to improve the clarity and naturalness of speech, laying a good foundation for subsequent speech synthesis and enhancement processing.

[0088] It should be noted that the design and implementation of the comb filter is a key technical link. In order to achieve efficient and real-time adaptive filtering, a comb filter based on the IIR (infinite impulse response) structure is adopted, which has low computational complexity and good frequency selectivity. At the same time, a smooth parameter adjustment mechanism is designed to avoid the introduction of sharp changes in the filter response when the fundamental frequency estimation occurs suddenly, thereby improving the stability and robustness of the filtering process.

[0089] Based on the above embodiment, as an optional embodiment, the parameters of the comb filter are adaptively designed according to the fundamental frequency characteristics to obtain the target comb filter, including: The order of the comb filter is determined according to the fundamental frequency characteristics, and the order is inversely proportional to the fundamental frequency; Determine the passband center frequency of the comb filter according to the fundamental frequency characteristics; Determine the bandwidth of the comb filter according to the fundamental frequency characteristics; According to the order, passband center frequency and bandwidth, the filter coefficients of the comb filter are determined to obtain the target comb filter.

[0090] In a preferred embodiment of the present invention, after obtaining the fundamental frequency characteristics, the next step of the method of the present invention is to adaptively design the parameters of the comb filter according to the fundamental frequency characteristics to obtain the target comb filter. The comb filter is a commonly used speech enhancement and noise suppression filter, which achieves speech enhancement and purification by selectively filtering speech and noise in the frequency domain. Traditional comb filters usually use fixed parameter settings, such as order, passband center frequency and bandwidth, etc., while the method of the present invention proposes an adaptive comb filter design method, which dynamically adjusts the filter parameters according to the fundamental frequency characteristics of the speech signal to better match the frequency structure of the speech and improve the filtering effect.

[0091] The fundamental frequency is an important feature of speech signals, which reflects the vibration characteristics of the speaker's vocal tract and the pitch of the speech. The fundamental frequency of different speakers and different speech contents is different. Therefore, designing a comb filter based on the fundamental frequency characteristics can make the filter better adapt to the frequency structure of different speech and achieve more accurate speech enhancement.

[0092] The specific implementation process is as follows: First, determine the order of the comb filter according to the fundamental frequency characteristics. The order of the comb filter determines the periodicity and selectivity of the filter in the frequency domain. The higher the order, the denser the peaks on the frequency response curve of the filter, and the stronger the ability to distinguish between speech and noise. However, too high an order will also increase the computational complexity and reduce the filtering effect. Therefore, it is necessary to appropriately select the order of the comb filter according to the fundamental frequency characteristics.

[0093] The method of the present invention proposes that the order of the comb filter is inversely proportional to the fundamental frequency. The higher the fundamental frequency, the higher the pitch of the voice and the denser the frequency structure, and a lower order should be selected at this time; conversely, the lower the fundamental frequency, the lower the pitch of the voice and the sparser the frequency structure, and a higher order should be selected at this time. The calculation formula of the comb filter order is: N = round(fs / (k * F0)) Among them, N is the order of the comb filter, fs is the sampling frequency of the speech signal, F0 is the fundamental frequency, and k is a proportional coefficient used to adjust the size of the order.

[0094] Through the above formula, the order of the comb filter can be adaptively determined according to the fundamental frequency characteristics of the speech signal to match it with the frequency structure of the speech and improve the accuracy of filtering.

[0095] Then, the passband center frequency of the comb filter is determined according to the fundamental frequency characteristics. The passband center frequency is the center position of the comb filter in the frequency domain, which determines the filter's enhancement area for the speech frequency components. In order to better enhance the speech, the passband center frequency of the comb filter should be aligned with the main frequency components of the speech, and the main frequency components of the speech are usually concentrated near the fundamental frequency and its multiples.

[0096] The method of the present invention proposes to use the multiple frequency of the fundamental frequency as the passband center frequency of the comb filter. The multiple frequency of the fundamental frequency can be expressed as: Fc(m) = m * F0 Among them, Fc(m) is the center frequency of the mth passband, F0 is the fundamental frequency, and m is the frequency multiplication coefficient, which is a positive integer.

[0097] By using the multiple of the fundamental frequency as the passband center frequency, the comb filter can achieve gain on the main frequency components of speech and effectively enhance the speech signal.

[0098] Next, determine the bandwidth of the comb filter based on the fundamental frequency characteristics. The bandwidth is the width of the comb filter in the frequency domain, which determines the frequency selection range of the filter for speech and noise. The narrower the bandwidth, the stronger the filter's ability to distinguish between speech and noise, but it may also cause speech distortion; the wider the bandwidth, the weaker the filter's ability to distinguish between speech and noise, but it can also retain more speech details. Therefore, it is necessary to appropriately select the bandwidth of the comb filter based on the fundamental frequency characteristics.

[0099] The method of the present invention proposes to use a certain proportion of the fundamental frequency as the bandwidth of the comb filter. The bandwidth calculation formula is: BW = α * F0 Among them, BW is the bandwidth of the comb filter, F0 is the fundamental frequency, and α is a proportional coefficient used to adjust the size of the bandwidth.

[0100] By using a certain proportion of the fundamental frequency as the bandwidth, the comb filter can adaptively adjust the frequency selection range according to the pitch characteristics of the speech, thus achieving a balance between speech enhancement and detail preservation.

[0101] Finally, according to the order, passband center frequency and bandwidth, the filter coefficients of the comb filter are determined to obtain the target comb filter. The filter coefficients of the comb filter determine the specific response characteristics of the filter in the time domain and frequency domain, and are the key parameters for realizing comb filtering.

[0102] The method of the present invention proposes to design the filter coefficients of the comb filter by using the window function method. First, a comb function in the frequency domain is generated according to the order of the comb filter and the passband center frequency, and its mathematical expression is: H(k) = 1, if |k - mN| ≤ BW / 2= 0, otherwise Among them, H(k) is the value of the comb function in the frequency domain, k is the index of the frequency domain sample point, m is the index of the passband, N is the order of the comb filter, and BW is the bandwidth of the passband.

[0103] Then, a window function is applied to the comb function to smooth the frequency response of the filter and reduce the ringing effect. Commonly used window functions include Hamming window, Hanning window and Blackman window. The choice of window function should be weighed according to the specific application scenario and performance requirements.

[0104] Finally, the inverse Fourier transform is performed on the windowed comb function to obtain the filter coefficients of the comb filter in the time domain. The mathematical expression of the filter coefficients is: h(n) = IDFT[H(k) · W(k)] Among them, h(n) is the filter coefficient of the comb filter, H(k) is the windowed comb function, W(k) is the window function, and IDFT represents the inverse discrete Fourier transform.

[0105] By designing the filter coefficients of the comb filter using the window function method, a parameter-adaptive target comb filter can be obtained, which can dynamically adjust the filter order, passband center frequency and bandwidth according to the fundamental frequency characteristics of the speech signal to achieve more accurate and effective speech enhancement.

[0106] S105, converting the time-frequency mask into a speech time-frequency mask, multiplying the speech signal after comb filtering by the speech time-frequency mask as a weighting coefficient to obtain the energy of the weighted signal, and determining the target speech energy estimation value based on the energy of the weighted signal; In a preferred embodiment of the present invention, after the initial speech signal is subjected to adaptive comb filtering, the next step of the method of the present invention is to multiply the time-frequency mask as a weighting coefficient with the comb-filtered speech signal to obtain the energy of the weighted signal, and use the energy of the weighted signal as the target speech energy estimate.

[0107] The time-frequency mask is estimated by the speech / noise separation DNN in the previous step, which represents the dominant area of ​​the speech signal in the time-frequency domain. Using the time-frequency mask as a weighting coefficient can selectively enhance the signal energy in the speech-dominated area and suppress the energy in the noise-dominated area, thereby achieving accurate estimation of the speech signal energy.

[0108] In the implementation process of this step, firstly, the comb-filtered speech signal is subjected to short-time Fourier transform (STFT) to obtain its time-frequency spectrum representation. Then, the time-frequency mask is multiplied with the time-frequency spectrum of the speech signal to obtain the weighted time-frequency spectrum. During the point multiplication process, the time-frequency points corresponding to the elements with a value of 1 in the time-frequency mask remain unchanged, while the time-frequency points corresponding to the elements with a value of 0 are suppressed to 0. In this way, in the weighted time-frequency spectrum, the energy of the time-frequency points dominated by speech is retained, while the energy of the time-frequency points dominated by noise is effectively suppressed. Next, the energy of the weighted time-frequency spectrum is summed to obtain the total energy of the weighted signal. Specifically, the energy of all frequency points in each frame of the weighted time-frequency spectrum is added to obtain the energy value of the frame. The energy values ​​of all frames are accumulated to obtain the total energy of the entire weighted signal. This total energy reflects the main energy distribution of the speech signal in the time-frequency domain, eliminates most of the interference of noise energy, and can be used as a more accurate estimate of the target speech energy.

[0109] By multiplying the time-frequency mask as a weighting coefficient with the comb-filtered speech signal and calculating the energy of the weighted signal, an estimated value of the target speech energy is obtained. This estimated value comprehensively considers the time-domain periodicity characteristics and frequency-domain energy distribution characteristics of the speech signal, fully utilizes the fundamental frequency characteristics and time-frequency mask information extracted in the previous steps, and has high reliability and accuracy.

[0110] Based on the above embodiment, as an optional embodiment, the time-frequency mask is converted into a speech time-frequency mask, the speech time-frequency mask is used as a weighting coefficient and multiplied with the speech signal after comb filtering to obtain the energy of the weighted signal, and the target speech energy estimation value is determined based on the energy of the weighted signal, including: The value of the element representing the target speech in the time-frequency mask is set to 1, and the value of the element representing the noise is set to 0, to obtain the speech time-frequency mask; The speech time-frequency mask is used as a weighting coefficient and multiplied with the speech signal after comb filtering to obtain a weighted signal; Calculate the average energy of the weighted signal in the time dimension and use the average energy as the target speech energy estimate.

[0111] In a preferred embodiment of the present invention, after obtaining the target comb filter, the next step of the method of the present invention is to multiply the time-frequency mask as a weighting coefficient with the comb-filtered speech signal to obtain the energy of the weighted signal. The purpose of this step is to use the time-frequency mask to weight the comb-filtered speech signal, highlight the target speech component, suppress the noise component, and estimate the energy of the target speech.

[0112] The time-frequency mask is a matrix of the same size as the time-frequency representation of the speech signal, which represents the energy distribution of the speech signal on the time-frequency plane. The value range of the elements in the time-frequency mask is usually [0,1], where elements close to 1 indicate that the time-frequency point mainly contains the target speech component, and elements close to 0 indicate that the time-frequency point mainly contains the noise component. The time-frequency mask can be used to distinguish the target speech and noise in the speech signal, providing guidance for subsequent speech enhancement processing.

[0113] The specific implementation process is as follows: First, the element value representing the target speech in the time-frequency mask is set to 1, and the element value representing the noise is set to 0 to obtain the speech time-frequency mask. This step is to binarize the original time-frequency mask and convert the time-frequency mask into a matrix containing only 0 and 1. Among them, the element with a value of 1 indicates that the corresponding time-frequency point belongs to the target speech, and the element with a value of 0 indicates that the corresponding time-frequency point belongs to the noise. The generation of the speech time-frequency mask can be achieved by setting an energy threshold. The time-frequency points above the threshold are considered to be speech-dominated, and the time-frequency points below the threshold are considered to be noise-dominated. The selection of the threshold needs to be adjusted according to the specific signal-to-noise ratio conditions and speech characteristics to achieve the best speech / noise separation effect. Then, the speech time-frequency mask is used as a weighting coefficient and multiplied with the speech signal after comb filtering to obtain a weighted signal. The speech signal after comb filtering is a speech signal processed by an adaptive comb filter, which enhances the target speech and suppresses noise in the time-frequency domain. Multiplying the speech time-frequency mask with the speech signal after comb filtering is equivalent to performing a second weighting on the speech signal, further highlighting the target speech component and weakening the residual noise component. The weighting process can be described by the following mathematical expression: S_weighted(t,f) = S_comb(t,f) · M_speech(t,f) Among them, S_weighted(t,f) represents the weighted speech signal, S_comb(t,f) represents the speech signal after comb filtering, M_speech(t,f) represents the speech time-frequency mask, and t and f represent the indexes of the time and frequency dimensions respectively.

[0114] Through the weighted operation, a new time-frequency representation is obtained, in which the target speech component is amplified, the noise component is suppressed, and the clarity and purity of the speech are further improved.

[0115] Finally, the average energy of the weighted signal in the time dimension is calculated and used as the target speech energy estimate. Energy is an important indicator to measure signal strength, which reflects the amplitude and change of the signal. By calculating the average energy of the weighted signal in the time dimension, the energy level of the target speech in the entire speech segment can be estimated, providing a reference for subsequent signal-to-noise ratio estimation and speech quality assessment. The calculation formula for average energy is: E_speech = (1 / T) * sum(S_weighted(t,f)^2, t=1,2,...,T) Among them, E_speech represents the average energy of the target speech, T represents the total duration of the speech segment, and sum represents the summation of the time dimension.

[0116] Taking the average energy of the weighted signal as the target speech energy estimation value can reflect the proportion and intensity of the target speech component in the speech signal, provide an energy reference for the speech enhancement algorithm, and assist in adjusting the gain coefficient and suppression factor to achieve better speech enhancement effect.

[0117] S106, multiplying the time-frequency mask as a weighting coefficient with the initial speech signal to obtain a noise energy estimation value; In a preferred embodiment of the present invention, after obtaining the target speech energy estimation value, the next step of the method of the present invention is to multiply the time-frequency mask as a weighting coefficient with the initial speech signal to obtain the noise energy estimation value.

[0118] The purpose of this step is to estimate the energy of the background noise in the speech signal and provide the necessary information for the subsequent calculation of the signal-to-noise ratio and speech gain. Different from the previous step of using the time-frequency mask to estimate the target speech energy, the complement of the time-frequency mask (one minus the time-frequency mask) is used here to weight the initial speech signal to obtain an estimate of the noise energy.

[0119] The specific implementation process is as follows: First, the initial speech signal is subjected to a short-time Fourier transform (STFT) to obtain its time-frequency spectrum representation. Then, the complement of the time-frequency mask is calculated, that is, the time-frequency mask is subtracted from the all-1 matrix to obtain the indicator matrix of the noise-dominated area. Next, the indicator matrix of the noise-dominated area is point-multiplied with the time-frequency spectrum of the initial speech signal to obtain the weighted noise time-frequency spectrum. During the point multiplication process, the time-frequency points corresponding to the elements with a value of 1 in the noise-dominated area indicator matrix remain unchanged, while the time-frequency points corresponding to the elements with a value of 0 are suppressed to 0. In this way, in the weighted noise time-frequency spectrum, the energy of the noise-dominated time-frequency points is retained, while the energy of the speech-dominated time-frequency points is effectively suppressed.

[0120] Finally, the energy of the weighted noise time-frequency spectrum is summed to obtain the total energy estimate of the noise signal. Specifically, the energy of all frequency points in each frame of the weighted noise time-frequency spectrum is added to obtain the noise energy estimate of the frame. The noise energy estimates of all frames are accumulated to obtain the total energy estimate of the entire noise signal.

[0121] In this way, the complement of the time-frequency mask is used as the weighting coefficient to effectively extract the energy information of the background noise in the initial speech signal and suppress the interference in the speech-dominated area. Since the time-frequency mask is estimated by the speech / noise separation DNN in the previous step, it has a high accuracy in distinguishing speech from noise. Therefore, the weighted processing based on the complement of the time-frequency mask can obtain a reliable noise energy estimate.

[0122] Obtaining the noise energy estimate is crucial for speech enhancement algorithms. With the noise energy estimate, we can further calculate the energy ratio of the speech signal and the noise signal, namely the signal-to-noise ratio (SNR). The signal-to-noise ratio is an important indicator for measuring speech quality and noise pollution. Its calculation formula is: SNR = 10 * log10 (speech energy / noise energy). In the subsequent steps, the target speech energy estimate and the noise energy estimate will be used to calculate the signal-to-noise ratio, and based on this, the speech gain function will be designed to adaptively enhance the speech signal, ultimately achieving high-quality noise suppression and speech extraction.

[0123] S107, calculating a frequency-dependent Bark gain coefficient on a Bark frequency scale according to the target speech energy estimation value and the noise energy estimation value; In a preferred embodiment of the present invention, after obtaining the target speech energy estimation value and the noise energy estimation value, the next step of the method of the present invention is to calculate the frequency-dependent Bark gain coefficient on the Bark frequency scale based on the two energy estimation values.

[0124] The Bark frequency scale is a frequency scale based on the human hearing characteristics. It divides the frequency into several critical bands. The width of each critical band becomes wider as the frequency increases, which is more in line with the frequency resolution characteristics of the human ear. In the field of speech enhancement, using the Bark frequency scale for gain calculation can better match the human hearing perception and obtain a more natural and comfortable enhancement effect.

[0125] The frequency-dependent Bark gain coefficient is a frequency-dependent gain function defined on the Bark frequency scale. Its purpose is to adaptively adjust the gain coefficient of each frequency band according to the energy distribution of speech signals and noise signals in different Bark frequency bands, thereby achieving frequency-selective speech enhancement.

[0126] In the implementation process of this step, first, the energy of the speech signal and the noise signal in each Bark frequency band is calculated according to the target speech energy estimation value and the noise energy estimation value. Specifically, the time-frequency spectrum energy of the speech signal and the noise signal is accumulated according to the division of the Bark frequency band to obtain the speech energy and noise energy in each Bark frequency band. Then, according to the speech energy and noise energy in each Bark frequency band, the signal-to-noise ratio (SNR) of the frequency band is calculated. The calculation formula of the Bark frequency band signal-to-noise ratio is: Bark_SNR(i) = 10 * log10(Bark_Speech_Energy(i) / Bark_Noise_Energy(i)), where i represents the i-th Bark frequency band.

[0127] Next, the Bark band signal-to-noise ratio is used to calculate the frequency-dependent Bark gain coefficient. Commonly used gain calculation formulas include Wiener filter, MMSE estimator, log spectrum amplitude estimator, etc. Here, an improved Wiener filter formula is used, and additional gain adjustment factors and noise suppression factors are introduced to balance the trade-off between speech distortion and noise suppression. The specific Bark gain coefficient calculation formula is as follows: Bark_Gain(i) = max(min((Bark_SNR(i) / (Bark_SNR(i) + 1))^α * β, 1),0) Among them, α is the gain adjustment factor, β is the noise suppression factor, and the max and min functions are used to ensure that the gain coefficient is between 0 and 1.

[0128] Finally, the calculated Bark gain coefficient is mapped back to the time-frequency domain to obtain a frequency-dependent time-varying gain function. For each time-frequency point, the corresponding gain coefficient is equal to the gain coefficient of the Bark frequency band in which it is located. In this way, a frequency-dependent Bark gain function defined in the time-frequency domain is obtained, which can be used for subsequent speech enhancement processing.

[0129] Based on the above embodiment, as an optional embodiment, the frequency-related Bark gain coefficient is calculated on the Bark frequency scale according to the target speech energy estimation value and the noise energy estimation value, including: The noise energy estimation value and the target speech energy estimation value are smoothed on the Bark frequency scale to obtain a smoothed target speech energy estimation value and a smoothed noise energy estimation value; Calculate the ratio of the smoothed target speech energy estimate to the smoothed noise energy estimate to obtain the signal-to-noise ratio on the Bark frequency scale; The signal-to-noise ratio is mapped to the Bark gain coefficient on the Bark frequency scale through a preset exponential function.

[0130] In a preferred embodiment of the present invention, after obtaining the target speech energy estimation value and the noise energy estimation value, the next step of the method of the present invention is to calculate the frequency-dependent Bark gain coefficient on the Bark frequency scale based on the two energy estimation values. The Bark gain coefficient is a frequency-dependent gain adjustment factor, which reflects the relative size of speech and noise energy in different frequency regions and is used to adaptively enhance the speech signal in the frequency domain.

[0131] The Bark frequency scale is a nonlinear frequency scale based on the human hearing characteristics. It divides the frequency range into several critical bands, and the frequency components in each critical band contribute similarly to the human ear's perception. By processing on the Bark frequency scale, the human hearing characteristics can be better considered, and a speech enhancement effect that matches the human ear's perception can be achieved.

[0132] The specific implementation process is as follows: First, the noise energy estimation value and the target speech energy estimation value are smoothed on the Bark frequency scale to obtain the smoothed target speech energy estimation value and the smoothed noise energy estimation value. The purpose of smoothing is to reduce the short-term fluctuation and estimation error of the energy estimation value and improve the stability and reliability of the energy estimation. Smoothing can be achieved by applying a smoothing window function on the Bark frequency scale. Commonly used smoothing window functions include triangular window, Hanning window and Gaussian window. The selection of the smoothing window function should be weighed according to the specific frequency resolution and smoothness requirements.

[0133] The smoothing process can be described by the following mathematical expression: E_speech_smooth(b) = sum(E_speech(f) * W(b,f), f=1,2,...,F) E_noise_smooth(b) = sum(E_noise(f) * W(b,f), f=1,2,...,F) Among them, E_speech_smooth(b) and E_noise_smooth(b) represent the smoothed target speech energy estimate and the smoothed noise energy estimate, respectively, E_speech(f) and E_noise(f) represent the original target speech energy estimate and the noise energy estimate, respectively, W(b,f) represents the smoothing window function, b and f represent the indexes of the Bark frequency band and the linear frequency, respectively, and F represents the total number of frequency points.

[0134] Through smoothing, a more stable and reliable energy estimate is obtained on the Bark frequency scale, laying the foundation for subsequent signal-to-noise ratio calculation and gain estimation. Then, the ratio of the smoothed target speech energy estimate to the smoothed noise energy estimate is calculated to obtain the signal-to-noise ratio on the Bark frequency scale. The signal-to-noise ratio is an important indicator for measuring the quality of speech signals, which reflects the relative strength of speech components and noise components in speech signals. By calculating the signal-to-noise ratio on the Bark frequency scale, the relative strength of speech and noise in different frequency regions can be evaluated, providing a basis for adaptive gain control. The formula for calculating the signal-to-noise ratio on the Bark frequency scale is: SNR_bark(b) = E_speech_smooth(b) / E_noise_smooth(b) Wherein, SNR_bark(b) represents the signal-to-noise ratio of the b-th Bark frequency band.

[0135] According to the size of the signal-to-noise ratio, the dominance of speech and noise in different Bark frequency bands can be judged. When the signal-to-noise ratio is high, it indicates that the speech component is dominant and the gain of this frequency band should be amplified; when the signal-to-noise ratio is low, it indicates that the noise component is dominant and the gain of this frequency band should be suppressed.

[0136] Finally, the signal-to-noise ratio is mapped to the Bark gain coefficient on the Bark frequency scale through a preset exponential function. The exponential function is a commonly used nonlinear mapping function that can compress the variation range of the signal-to-noise ratio into a limited gain coefficient range to achieve smooth gain adjustment. The general form of the exponential function is: G_bark(b) = a * exp(b * SNR_bark(b)) Wherein, G_bark(b) represents the Bark gain coefficient of the b-th Bark frequency band, a and b are parameters for controlling the dynamic range of the gain coefficient, and can be set according to specific gain adjustment requirements.

[0137] Through exponential function mapping, a set of frequency-related Bark gain coefficients are obtained, which reflect the relative size of speech and noise energy in different Bark frequency bands and are used to adaptively enhance speech signals in the frequency domain. The value range of the Bark gain coefficient is usually between [0,1], where a value close to 1 indicates that the speech component of the corresponding frequency band is dominant and a larger gain amplification is required; a value close to 0 indicates that the noise component of the corresponding frequency band is dominant and a larger gain suppression is required.

[0138] S108, multiplying the speech signal after comb filtering by the Bark gain coefficient, enhancing the initial speech signal in the time-frequency domain to obtain an enhanced speech signal; In a preferred embodiment of the present invention, after calculating the frequency-dependent Bark gain coefficient, the next step of the method of the present invention is to multiply the comb-filtered speech signal by the Bark gain coefficient, enhance the initial speech signal in the time-frequency domain, and obtain an enhanced speech signal.

[0139] The purpose of this step is to use an adaptive frequency-selective gain function to adjust the time-varying gain of the comb-filtered speech signal to improve the clarity and naturalness of the speech while suppressing background noise. By multiplying the comb-filtered speech signal with the Bark gain coefficient, refined speech enhancement processing can be achieved in the time-frequency domain.

[0140] The specific implementation process is as follows: First, the comb-filtered speech signal is subjected to a short-time Fourier transform (STFT) to obtain its time-frequency spectrum representation. The comb-filtered speech signal here is obtained by processing the initial speech signal with the adaptive comb filter designed based on the fundamental frequency characteristics in the previous step, which contains the enhanced speech fundamental frequency and its harmonic components, as well as the suppressed noise components. Then, the Bark gain coefficient is mapped to the dimension of the time-frequency spectrum to obtain a gain matrix of the same size as the time-frequency spectrum. Specifically, for each time-frequency point in the time-frequency spectrum, the Bark frequency band to which it belongs is determined according to its frequency value, and the gain coefficient of the corresponding Bark frequency band is assigned to the time-frequency point to form a gain matrix. Next, the time-frequency spectrum of the comb-filtered speech signal is multiplied by the gain matrix to obtain the enhanced speech time-frequency spectrum. In the process of dot multiplication, the speech-dominated time-frequency points in the time-frequency spectrum will be amplified by the Bark gain coefficient, while the noise-dominated time-frequency points will be suppressed by the Bark gain coefficient. This time-varying frequency-selective gain adjustment can flexibly balance the effects of speech enhancement and noise suppression at different time and frequency positions.

[0141] Finally, the enhanced speech time-frequency spectrum is subjected to inverse short-time Fourier transform (ISTFT) to obtain the enhanced speech signal. The ISTFT process includes phase recovery, overlap-addition and other operations on the time-frequency spectrum, and finally reconstructs the speech waveform in the time domain.

[0142] By multiplying the speech signal after comb filtering with the Bark gain coefficient, adaptive speech enhancement processing is achieved in the time-frequency domain. Compared with the traditional time-domain filtering method, this time-frequency domain enhancement method can more flexibly and finely adjust the gain coefficient at different time and frequency positions, making full use of the time-varying and frequency-varying characteristics of the speech signal to obtain a more optimized enhancement effect.

[0143] S109, using an inverse BFCC transform to convert the enhanced speech signal from the Bark frequency scale into a time domain signal, to obtain a target enhanced speech pointing in the target direction.

[0144] In a preferred embodiment of the present invention, after the initial speech signal is enhanced in the time-frequency domain, the last step of the method of the present invention is to use an inverse BFCC transform to convert the enhanced speech signal from the Bark frequency scale into a time domain signal to obtain a target enhanced speech pointing in the target direction.

[0145] BFCC (Bark Frequency Cepstral Coefficients) is a speech feature representation method similar to MFCC (Mel Frequency Cepstral Coefficients). It uses the Bark frequency scale instead of the Mel frequency scale in the calculation process. The Bark frequency scale is more in line with the auditory characteristics of the human ear, so the BFCC feature is widely used in the fields of speech enhancement and speech recognition. In the previous step, the speech signal was enhanced on the Bark frequency scale to obtain an enhanced speech signal in the Bark frequency domain. In order to convert the enhanced speech signal back to the time domain, an inverse BFCC transform is required.

[0146] The specific implementation process is as follows: First, the logarithmic energy of the enhanced speech signal in the Bark frequency domain is calculated to obtain the logarithmic energy spectrum of the Bark frequency band. This step is to convert the energy value of the Bark frequency band into a logarithmic scale to make it more consistent with the loudness perception characteristics of the human ear. Then, the logarithmic energy spectrum of the Bark frequency band is subjected to discrete cosine transform (DCT) to obtain the BFCC characteristic coefficient. The DCT transform can compress the band energy spectrum into a small number of cepstral coefficients, and has good decorrelation and energy concentration. The BFCC characteristic coefficient represents the envelope characteristics of the speech signal on the Bark frequency scale. Next, the BFCC characteristic coefficient is subjected to inverse discrete cosine transform (IDCT) to convert it back to the logarithmic energy spectrum of the Bark frequency band. The IDCT transform is the inverse process of the DCT transform, which can restore the cepstral coefficients to the band energy spectrum. Then, the logarithmic energy spectrum of the Bark frequency band is subjected to exponential operation to convert the energy value of the logarithmic scale into a linear scale. This step is to restore the band energy spectrum to the original energy value for subsequent band synthesis.

[0147] Finally, the energy value of the Bark band is converted into a time domain signal using a Bark band synthesis filter to obtain the target enhanced speech pointing in the target direction. The Bark band synthesis filter is a set of bandpass filters corresponding to the Bark band division, and its frequency response matches the critical band characteristics of the human ear. By multiplying the energy value of each Bark band with the corresponding synthesis filter and adding the filtering results of all bands, the enhanced speech signal in the time domain is obtained.

[0148] See also Figure 2 , Figure 2 An architecture diagram of a directional sound pickup system provided in an embodiment of the present application, the directional sound pickup system may include: The data acquisition module 1 is used to perform MVDR beamforming processing on the initial speech signal input by the microphone array to obtain the initial speech signal after beamforming; The feature extraction module 2 is used to extract features from the initial speech signal to obtain BFCC features and fundamental frequency features; Special processing module 3, used to process BFCC features using pre-trained speech / noise separation DNN to obtain time-frequency mask; The filtering module 4 is used to adaptively design the parameters of the comb filter according to the fundamental frequency characteristics to obtain the target comb filter, and filter the initial speech signal based on the comb filter to obtain the speech signal after the comb filtering; The target speech energy estimation module 5 is used to convert the time-frequency mask into a speech time-frequency mask, multiply the speech time-frequency mask with the comb-filtered speech signal as a weighting coefficient to obtain the energy of the weighted signal, and determine the target speech energy estimation value based on the energy of the weighted signal; A noise energy estimation module 6, used for multiplying the time-frequency mask as a weighting coefficient with the initial speech signal to obtain a noise energy estimation value; A gain coefficient calculation module 7, used for calculating a frequency-dependent Bark gain coefficient on a Bark frequency scale according to a target speech energy estimate and a noise energy estimate; A first signal enhancement module 8, configured to multiply the comb-filtered speech signal by a Bark gain coefficient, enhance the initial speech signal in the time-frequency domain, and obtain an enhanced speech signal; The second signal enhancement module 9 is used to convert the enhanced speech signal from the Bark frequency scale into a time domain signal by using an inverse BFCC transform to obtain a target enhanced speech pointing in a target direction.

[0149] It should be noted that: when the system provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0150] Please refer to Figure 3 The application also discloses an electronic device. Figure 3 The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302 or end-to-end wireless communication.

[0151] The communication bus 302 is used to realize the connection and communication between these components.

[0152] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0153] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0154] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field~Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0155] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read~Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally be at least one storage system located away from the aforementioned processor 301. Refer to Figure 3, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a directional sound pickup method.

[0156] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program storing the directional sound pickup method in the memory 305. When executed by one or more processors 301, the electronic device 300 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application. In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0157] In the several implementations provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, such as the division of modules, which is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, indirect coupling or communication connection of systems or modules, which can be electrical or other forms.

[0158] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figure 1 The directional sound pickup method of the embodiment shown in the figure can be found in the specific implementation process. Figure 1 The specific description of the illustrated embodiment will not be repeated here.

[0160] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0161] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0162] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0163] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A directional sound pickup method, characterized in that: The method comprises: Performing MVDR beamforming processing on the initial speech signal input by the microphone array to obtain the initial speech signal after beamforming; Extracting features of the initial speech signal to obtain BFCC features and fundamental frequency features; Processing the BFCC features using a pre-trained speech / noise separation DNN to obtain a time-frequency mask; Adaptively designing the parameters of a comb filter according to the fundamental frequency characteristics to obtain a target comb filter, and filtering the initial speech signal based on the target comb filter to obtain a speech signal after comb filtering; Converting the time-frequency mask into a speech time-frequency mask, multiplying the speech time-frequency mask as a weighting coefficient with the speech signal after the comb filtering to obtain the energy of the weighted signal, and determining the target speech energy estimation value based on the energy of the weighted signal; Multiplying the time-frequency mask as a weighting coefficient with the initial speech signal to obtain a noise energy estimation value; Calculating a frequency-dependent Bark gain coefficient on a Bark frequency scale according to the target speech energy estimate and the noise energy estimate; Multiplying the comb-filtered speech signal by a Bark gain coefficient, enhancing the initial speech signal in the time-frequency domain to obtain an enhanced speech signal; The enhanced speech signal is converted from the Bark frequency scale to a time domain signal using an inverse BFCC transform to obtain a target enhanced speech pointing in a target direction.

2. The method according to claim 1, characterized in that The performing MVDR beamforming processing on the initial speech signal input by the microphone array to obtain the initial speech signal after beamforming includes: Determining an array manifold vector according to geometric arrangement information of the microphone array and azimuth information of the initial speech signal; Calculating the weight coefficient of the MVDR beamformer according to the spatial covariance matrix of the initial speech signal and the array manifold vector to obtain the MVDR weight coefficient; The MVDR weight coefficient is used to perform weighted summation on the initial speech signal to form a spatial filter pointing in the direction of the target speech, and the initial speech signal is subjected to MVDR beamforming processing based on the spatial filter pointing in the direction of the target speech to obtain the initial speech signal after beamforming.

3. The method according to claim 1, characterized in that Performing feature extraction on the initial speech signal to obtain BFCC features includes: Performing short-time Fourier transform on the initial speech signal to obtain a frequency spectrum; Mapping the spectrum to the Bark frequency scale to obtain a Bark spectrum; Taking the logarithm of the Bark spectrum to obtain a logarithmic Bark spectrum; Performing discrete cosine transform on the logarithmic Bark spectrum to obtain the BFCC feature.

4. The method according to claim 1, characterized in that: The method of using a pre-trained speech / noise separation DNN to process the BFCC feature to obtain a time-frequency mask includes: Inputting the BFCC feature into the pre-trained speech / noise separation DNN, and extracting local features of the BFCC feature through the convolutional neural network of the pre-trained speech / noise separation DNN; Modeling the time domain context information of the local features of the BFCC features using the recursive neural network of the pre-trained speech / noise separation DNN to obtain the time domain context information; The local features of the BFCC features and the time domain context information are fused through the fully connected layer of the pre-trained speech / noise separation DNN to output a soft mask; The soft mask is binarized to obtain the time-frequency mask.

5. The method according to claim 1, characterized in that Adaptively designing the parameters of the comb filter according to the fundamental frequency characteristics to obtain a target comb filter, including: Determining the order of the comb filter according to the fundamental frequency characteristic, wherein the order is inversely proportional to the fundamental frequency; Determine the passband center frequency of the comb filter according to the fundamental frequency characteristics; Determining the bandwidth of the comb filter according to the fundamental frequency characteristics; The parameters of the comb filter are determined according to the order, the passband center frequency and the bandwidth to obtain the target comb filter.

6. The method according to claim 1, characterized in that The step of converting the time-frequency mask into a speech time-frequency mask, multiplying the speech time-frequency mask as a weighting coefficient with the speech signal after comb filtering to obtain the energy of the weighted signal, and determining the target speech energy estimation value based on the energy of the weighted signal includes: The value of the element representing the target speech in the time-frequency mask is set to 1, and the value of the element representing the noise is set to 0, to obtain a speech time-frequency mask; The speech time-frequency mask is used as a weighting coefficient and multiplied with the speech signal after the comb filtering to obtain the energy of the weighted signal; The average energy of the weighted signal in the time dimension is calculated, and the average energy is used as the target speech energy estimation value.

7. The method according to claim 1, characterized in that Calculating a frequency-dependent Bark gain coefficient on a Bark frequency scale according to the target speech energy estimate and the noise energy estimate, including: Smoothing the noise energy estimation value and the target speech energy estimation value on a Bark frequency scale to obtain a smoothed target speech energy estimation value and a smoothed noise energy estimation value; Calculating the ratio of the smoothed target speech energy estimate to the smoothed noise energy estimate to obtain a signal-to-noise ratio on a Bark frequency scale; The signal-to-noise ratio is mapped to a Bark gain coefficient on a Bark frequency scale through a preset exponential function.

8. A directional sound pickup system, characterized in that: The system comprises: A data acquisition module is used to perform MVDR beamforming processing on the initial speech signal input by the microphone array to obtain the initial speech signal after beamforming; A feature extraction module is used to extract features from the initial speech signal to obtain BFCC features and fundamental frequency features; A special processing module, used to process the BFCC features using a pre-trained speech / noise separation DNN to obtain a time-frequency mask; A filtering module, configured to adaptively design the parameters of a comb filter according to the fundamental frequency characteristics to obtain a target comb filter, and filter the initial speech signal based on the target comb filter to obtain a speech signal after comb filtering; A target speech energy estimation module, used for converting the time-frequency mask into a speech time-frequency mask, multiplying the speech time-frequency mask as a weighting coefficient with the speech signal after the comb filtering to obtain the energy of the weighted signal, and determining the target speech energy estimation value based on the energy of the weighted signal; A noise energy estimation module, used for multiplying the time-frequency mask as a weighting coefficient with the initial speech signal to obtain a noise energy estimation value; A gain coefficient calculation module, used for calculating a frequency-dependent Bark gain coefficient on a Bark frequency scale according to the target speech energy estimate and the noise energy estimate; A first signal enhancement module, configured to multiply the comb-filtered speech signal by a Bark gain coefficient, and enhance the initial speech signal in the time-frequency domain to obtain an enhanced speech signal; The second signal enhancement module is used to convert the enhanced speech signal from the Bark frequency scale into a time domain signal by using an inverse BFCC transform to obtain a target enhanced speech pointing in a target direction.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

Images