A speech enhancement method, apparatus and system based on cross-domain feature fusion

Abstract

This application relates to the field of speech enhancement technology, specifically to a speech enhancement method, apparatus, and system based on cross-domain feature fusion. The method includes: extracting the speech signal to be enhanced, denoted as the full speech signal; segmenting the full speech signal into frames; determining the time-domain perturbation index of the full speech signal; acquiring the frequency domain signal of the full speech signal, detecting each peak in the frequency domain signal, and determining the abrupt change coefficient of each peak; acquiring the time-frequency interference degree of the full speech signal; converting the full speech signal to a high-dimensional phase space to obtain a high-dimensional denoised signal; determining a threshold in a denoising algorithm based on the time-frequency interference degree; using the denoising algorithm to obtain a low-dimensional denoised signal of the full speech signal; and combining the low-dimensional denoised signal with the high-dimensional denoised signal to obtain the enhanced fused speech signal. This improves the speech signal enhancement effect.

Description

Technical Field

[0001] The present application relates to the field of speech enhancement technology, and in particular to a speech enhancement method, device and system for cross-domain feature fusion. Background Art

[0002] In real life, voice signals are often affected by background noise, echo, environmental interference and other factors, making it difficult for listeners to clearly understand the speech content. Especially in noisy environments, voice signals are often drowned out or distorted by noise, reducing the intelligibility of speech. In multi-party video conferences or remote collaboration, voice signals are easily drowned out due to multiple people speaking at the same time or interference from environmental noise. Speech enhancement can effectively reduce noise and echo, strengthen the voice of the main speaker, and enhance the clarity of voice signals, thereby improving the audibility and intelligibility of speech.

[0003] However, existing speech enhancement technologies usually only use a single time-frequency domain for analysis, and have problems such as single extracted features and insufficient correlation between feature attributes. It is difficult to simultaneously meet the needs of noise reduction and maintaining speech quality. Although a certain noise reduction effect is achieved, the clarity of the speech is also lost, resulting in speech distortion and poor speech enhancement effect. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide a speech enhancement method, device and system for cross-domain feature fusion. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for speech enhancement by cross-domain feature fusion, the method comprising the following steps:

[0006] Extracting the speech signal to be enhanced, recording it as the full speech signal, and dividing the full speech signal into frames;

[0007] The dynamic range of the envelope of each frame of speech signal and the difference between the high-frequency component and the low-frequency component in each frame of speech signal are combined into two-dimensional data points, and the distribution of the two-dimensional data points of all frames of speech signals is analyzed to determine the time domain disturbance index of the whole speech signal;

[0008] Obtain the frequency domain signal of the full speech signal, detect each peak in the frequency domain signal, perform curve fitting on all peaks, analyze the distance relationship between each peak and the fitting curve, as well as the range and fluctuation of the amplitude in the local neighborhood of each peak, and determine the mutation coefficient of each peak;

[0009] Determine each signal period of the full speech signal based on the fundamental frequency of the full speech signal, record the signal period containing the second resonance peak as the detection period, and screen out the mutation peak among all peaks based on the mutation coefficient; obtain the time-frequency interference degree of the full speech signal based on the energy distribution in each detection period and the clarity of the sound in all detection periods, combined with the time domain disturbance index and the mutation coefficient of all mutation peaks;

[0010] The full speech signal is converted into a high-dimensional phase space, and each component is obtained after decomposition, and the difference between each component and the remaining component and the contribution rate of each component are analyzed to obtain the noise confidence of each component; all components are denoised and dimensionally reduced using the noise confidence to obtain a high-dimensional denoised signal;

[0011] The threshold in the denoising algorithm is determined by the time-frequency interference degree, and a low-dimensional denoised signal of the full speech signal is obtained by using the denoising algorithm. The enhanced fused speech signal is obtained by combining the high-dimensional denoised signal.

[0012] In one embodiment, determining the difference between the high frequency component and the low frequency component in each frame of speech signal includes:

[0013] Perform modal decomposition on each frame of speech signal to obtain the decomposed high-frequency components and low-frequency components, and calculate the mean of the sum of squares of all elements in all high-frequency components and the mean of the sum of squares of all elements in all low-frequency components, and record them as the high-frequency energy ratio and the low-frequency energy ratio respectively;

[0014] The ratio of the high-frequency energy proportion to the low-frequency energy proportion is calculated and recorded as a first ratio. The difference between the high-frequency component and the low-frequency component in each frame of the speech signal is the first ratio.

[0015] In one embodiment, determining the time domain disturbance index includes:

[0016] Clustering all the two-dimensional data points of the full speech signal, calculating the average value of the modules of all the two-dimensional data points in each cluster, recording the cluster with the largest average value as the detection cluster, calculating the average value of the difference between the average value of the detection cluster and the average values ​​of all remaining clusters, and recording it as the outlier index of the detection cluster;

[0017] Calculate the ratio of the number of elements in all clusters to the number of elements in the detection cluster, recorded as the second ratio; calculate the number of clusters, the outlier index of the detection cluster, and the fusion result of the mean of the modulus of all the two-dimensional data points in the detection cluster;

[0018] The time domain disturbance index is positively correlated with the fusion result, and negatively correlated with the second ratio.

[0019] In one embodiment, the determination of the mutation coefficient includes:

[0020] Calculate the shortest distance between each peak and the fitting curve, calculate the product of the range of amplitude change and the degree of fluctuation in the local neighborhood of each peak, and record it as the local fluctuation index of each peak; calculate the product of the local fluctuation index of each peak and the shortest distance, and record it as the first product;

[0021] The mutation coefficient is the sum of the normalized value of the first product of each peak value and the normalized value of the frequency corresponding to each peak value.

[0022] In one embodiment, determining the mutation peak value includes:

[0023] The segmentation threshold of the mutation coefficient of all peaks is obtained, and the peak whose mutation coefficient is greater than the segmentation threshold is determined as the mutation peak.

[0024] In one embodiment, the determination of the time-frequency interference degree includes:

[0025] The ratio of the center frequency of the second resonance peak in each detection cycle to the bandwidth is recorded as the voiceprint purity index of each detection cycle, and the average of the voiceprint purity indexes of all detection cycles is recorded as the overall voiceprint purity; the average of the short-term energy of the speech signal at all times in all detection cycles is recorded as the energy stability index; the average of the mutation coefficients of all mutation peaks is calculated and recorded as the average mutation degree of the spectrum;

[0026] The cumulative sum of the energy stability index and the overall voiceprint purity is calculated, and the product of the time domain disturbance index and the average mutation degree of the spectrum is calculated, which is recorded as the second product; the time-frequency interference degree is positively correlated with the second product and negatively correlated with the cumulative sum.

[0027] In one embodiment, the determination of the noise confidence level includes:

[0028] The mean of the metric distances between each component and all remaining components is calculated and recorded as the waveform difference index of each component. The noise confidence is the ratio of the waveform difference index of each component to the contribution rate.

[0029] In one embodiment, the threshold in the denoising algorithm is determined by the time-frequency interference degree, and the calculation method is:

[0030] ; In the formula, is the threshold in the denoising algorithm, K is the time-frequency interference of the full speech signal, n is the length of the full speech signal, and lg() is a logarithmic function with base 10.

[0031] In a second aspect, an embodiment of the present application further provides a speech enhancement device for cross-domain feature fusion, wherein a computer program is stored in the device, and when the computer program is executed by a processor, the speech enhancement method described in any one of the above is implemented.

[0032] In a third aspect, an embodiment of the present application also provides a speech enhancement system for cross-domain feature fusion, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0033] This application has at least the following beneficial effects:

[0034] The present application extracts the speech signal to be enhanced, records it as the whole speech signal, and divides the whole speech signal into frames; the dynamic range of the envelope of each frame speech signal and the difference between the high-frequency component and the low-frequency component of each frame speech signal are composed into two-dimensional data points, and the distribution of the two-dimensional data points of all frame speech signals is analyzed to determine the time domain disturbance index of the whole speech signal; the time domain disturbance index reflects the degree of noise interference of the whole speech signal that can be detected under the time domain signal, thereby improving the accuracy of subsequent noise extraction of the whole speech signal; the frequency domain signal of the whole speech signal is obtained, each peak value in the frequency domain signal is detected, and all peak values ​​are subjected to a curve. Line fitting is performed to analyze the distance relationship between each peak and the fitting curve, as well as the range of change and fluctuation of the amplitude in the local neighborhood of each peak, and determine the mutation coefficient of each peak; the mutation coefficient reflects the possibility that each peak is mutated due to noise interference, which improves the reliability of noise extraction; based on the fundamental frequency of the full speech signal, each signal period of the full speech signal is determined, and the signal period containing the second resonance peak is recorded as the detection period, and the mutation peaks among all peaks are screened out based on the mutation coefficient; based on the distribution of energy in each detection period and the clarity of the sound in all detection periods, the time domain disturbance index and the The mutation coefficient of all mutation peaks is used to obtain the time-frequency interference of the full speech signal; the time-frequency interference reflects the possibility that the speech signal will fluctuate violently in the time domain waveform, and significant mutation peaks will appear in the frequency domain, and the overall voiceprint will be blurred when the speaker is in a calm mood. This eliminates the phenomenon that the speech signal fluctuation caused by the speaker's unstable emotions causes errors in speech signal denoising, thereby improving the accuracy of full speech signal denoising; the full speech signal is converted into a high-dimensional phase space, and each component is obtained after decomposition, and the degree of difference between each component and the remaining components and the contribution rate of each component are analyzed to obtain the noise confidence of each component; After denoising and reducing the dimension of all components by using the noise confidence, a high-dimensional denoised signal is obtained, which can effectively separate potential noise components; the threshold in the denoising algorithm is determined by the time-frequency interference degree, and the low-dimensional denoised signal of the full speech signal is obtained by using the denoising algorithm, and the mutation noise in the time-frequency domain can be processed by the denoising algorithm in the low dimension; combined with the high-dimensional denoised signal, an enhanced fused speech signal is obtained; by cross-domain processing of the speech signal and fusion of the denoising results, useful speech features can be better retained, avoiding the speech distortion problem caused by excessive denoising, and improving the speech enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flowchart of a method for speech enhancement using cross-domain feature fusion provided by an embodiment of the present application;

[0037] Figure 2 Determine a flow chart for time-frequency interference;

[0038] Figure 3 Obtain a flow chart for fusing speech signals. DETAILED DESCRIPTION

[0039] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation methods, structures, features and effects of a speech enhancement method, device and system for cross-domain feature fusion proposed in the present application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0040] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0041] The following is a detailed description of a specific solution of a cross-domain feature fusion speech enhancement method, device and system provided by the present application in conjunction with the accompanying drawings.

[0042] See also Figure 1 , which shows a flowchart of a method for speech enhancement by cross-domain feature fusion provided by an embodiment of the present application, the method comprising the following steps:

[0043] S1, extract the speech signal to be enhanced and divide it into frames.

[0044] In this embodiment, in a conference scenario, the speaker's voice signal is collected through the speaker's microphone device, the audio collection frequency is 15kHz, and each collection time is 5 seconds. The collected voice signal is recorded as Y, and Y is called the full voice signal, that is, the voice signal to be enhanced. The collection frequency and collection time can be set by the implementer according to the actual situation, and this embodiment does not limit it here.

[0045] It should be noted that the speech signal to be enhanced analyzed in this embodiment is a discrete speech signal, which can be represented as a data sequence.

[0046] The acquired speech signal Y is subjected to framing and windowing processing, the frame length is set to 30ms, the overlap is 15ms, and the windowing mode is a Hamming window. Framing and windowing processing are well-known technologies and will not be described in detail herein. The frame length and overlap can be determined by the implementer, and this embodiment does not limit this.

[0047] S2, the dynamic range of the envelope of each frame of speech signal and the difference between the high-frequency component and the low-frequency component in each frame of speech signal are combined into two-dimensional data points, the distribution of the two-dimensional data points of all frame speech signals is analyzed, and the time domain disturbance index of the whole speech signal is determined.

[0048] In a meeting, if the speaker's environment is relatively quiet and there is no serious noise interference, the speaker's voice is very clear, the signal envelope of the voice signal Y fluctuates relatively smoothly with the voice, and the dynamic range of the voice signal envelope in the frame is small and there is no significant change. If there is a lot of noise interference in the speaker's environment, such as conversation sounds, paper and pen friction sounds, door opening and closing sounds and other non-stationary noises with high randomness, the overlap of the noise signal and the voice signal will cause a large mutation interference to the speaker's voice; since the amplitude of non-stationary instantaneous noise is significantly different from that of normal voice signals, signal overlap will cause significant fluctuations in the signal envelope, the dynamic range of the voice signal envelope in the frame will increase, and the envelope dynamic range in different frames will vary greatly.

[0049] Therefore, the speech signal in the i-th frame is taken as an example for analysis. The dynamic range of the envelope of the speech signal in the i-th frame is obtained and recorded as the envelope fluctuation index of the i-th frame speech signal. The acquisition of the speech signal envelope and the calculation of the dynamic range of the envelope are well-known technologies and are not repeated here. The larger the envelope fluctuation index, the greater the change amplitude of the envelope of the i-th frame speech signal.

[0050] Furthermore, due to the structure and function of the human vocal organ, the speech signal has higher energy in the low-frequency band. If the speech signal in the i-th frame is subject to greater noise interference, the noise will cause the high-frequency energy in the speech signal to increase. The greater the noise interference, the more high-frequency energy there is, which will seriously affect the energy distribution of the speech signal in the low-frequency and high-frequency bands.

[0051] In this embodiment, the speech signal in the i-th frame is used as the input of the variational mode decomposition algorithm, and the number of decomposition layers is 6. Since the decomposed imf components are arranged from high frequency to low frequency, the imf components of the 1-3 layers are all recorded as high frequency components, and the imf components of the 4-6 layers are all recorded as low frequency components. The mean of the sum of squares of all elements in all high-frequency components imf1, imf2, and imf3 is uniformly calculated and recorded as the high-frequency energy ratio, and the mean of the sum of squares of all elements in all low-frequency components imf4, imf5, and imf6 is uniformly calculated and recorded as the low-frequency energy ratio; the ratio of the high-frequency energy ratio to the low-frequency energy ratio is recorded as the high-frequency significance index of the i-th frame speech signal, and the high-frequency significance index is the first ratio. The larger the high-frequency significance index is, the more high-frequency energy there is compared to the low-frequency energy in the speech signal of the i-th frame, reflecting that the i-th frame speech signal is subject to greater noise interference.

[0052] For the speech signal Y, the envelope fluctuation index and high-frequency significant index of all frame speech signals are obtained. In order to eliminate the dimensional influence between parameters, all envelope fluctuation indexes and high-frequency significant indexes are subjected to Z-score standardization; then, the envelope fluctuation index of all frames is used as the horizontal coordinate, and the high-frequency significant index is used as the vertical coordinate to form each two-dimensional data point, and the envelope fluctuation index and high-frequency significant index corresponding to all frame speech signals are mapped to a two-dimensional rectangular coordinate system, and clustering is performed using the DPC density mean clustering algorithm, and the cross-validation method is used to obtain the truncation distance of the DPC clustering algorithm, and the output is each cluster cluster. The variational mode decomposition algorithm, Z-score standardization, DPC density clustering, and cross-validation method are all well-known technologies, which will not be repeated here, and the implementer can select other existing feasible mode decomposition algorithms, standardization algorithms, and clustering algorithms according to actual conditions, and this embodiment does not limit this.

[0053] If the speech signal Y is not seriously disturbed by noise, the envelope fluctuation index and high-frequency significance index between all frame speech signals should be relatively consistent, with small differences. Since non-stationary noise is random, the more times the overall speech signal is disturbed by noise and the more inconsistent the degree of interference, the more clusters will be obtained and the greater the difference between clusters.

[0054] For each cluster obtained, the average value of the modulus of all two-dimensional data points in each cluster is calculated, and the cluster with the largest average value is recorded as the detection cluster. The mean of the difference between the average value of the detection cluster and the average values ​​of all remaining clusters is calculated and recorded as the outlier index of the detection cluster.

[0055] It should be noted that, for example, if a two-dimensional data point is (x, y), then the modulus operation of the two-dimensional data point is The difference indicates the degree of difference between two variables, which can be calculated by using the absolute value of the difference, the square of the difference, the ratio, etc. In this embodiment, the average of the absolute values ​​of the differences between the average value of the detection cluster and the average values ​​of all the remaining clusters is recorded as the outlier index of the detection cluster.

[0056] Based on the above analysis, the time domain disturbance index of the speech signal Y is calculated, which is:

[0057] For the speech signal Y, the ratio of the number of elements in all clusters to the number of elements in the detection cluster is calculated, which is recorded as the second ratio; the number of clusters, the outlier index of the detection cluster, and the fusion result of the mean of the modulus of all the two-dimensional data points in the detection cluster are calculated;

[0058] The time domain disturbance index is positively correlated with the fusion result, and negatively correlated with the second ratio.

[0059] It should be noted that fusion means combining multiple variables, which can be calculated by addition, multiplication, addition and multiplication mixture, etc. In this embodiment, the calculation process of the time domain disturbance index of the speech signal Y is:

[0060] ; In the formula, A is the time domain disturbance index of the speech signal Y; J is the number of clusters; B is the mean value of the modulus of all the two-dimensional data points in the detection cluster; C is the outlier index of the detection cluster; D is the ratio of the number of elements in all clusters to the number of elements in the detection cluster, that is, the second ratio. is the fusion result.

[0061] It should be understood that in the process of analyzing and calculating the time domain disturbance index, all the clusters mentioned are all the clusters included in the clustering process of all the frame voice signals of the voice signal Y. If the number of clusters J is larger, the possibility of reflecting that the voice signal is disturbed by multiple inconsistent noises is greater; if B is larger, the envelope fluctuation index and high-frequency significant index of the voice signal in the corresponding frame of the element in the detection cluster are larger, and the degree of noise interference of the voice signal in the frame is greater; if the outlier index C is larger, the difference between the detection cluster and other clusters is larger, the significance of the detection cluster is larger, the difference between the voice signal disturbed by noise and the voice signal not disturbed is larger, and the noise interference is more serious. If D is smaller, it means that there are more elements in the detection cluster, and the frames reflecting that the voice signal is disturbed by noise are more. Therefore, if the time domain disturbance index A is larger, it means that under the time domain signal, the greater the noise interference of the voice signal Y can be detected.

[0062] S3, obtain the frequency domain signal of the full speech signal, detect each peak in the frequency domain signal, perform curve fitting on all peaks, analyze the distance relationship between each peak and the fitting curve, as well as the range and fluctuation of the amplitude in the local neighborhood of each peak, and determine the mutation coefficient of each peak.

[0063] Furthermore, if the speaker's emotions change, it will also affect the change of the time domain waveform of the voice signal Y, causing the envelope to fluctuate. At the same time, since non-stationary noise is random and irregular in the time domain, it may be difficult to capture the specific characteristics of the noise using only time domain analysis, which may easily cause false detection. Therefore, this embodiment converts the voice signal Y into the frequency domain for further analysis.

[0064] Perform fast Fourier transform on the speech signal Y and convert it to the frequency domain, which is recorded as the frequency domain signal P. Since the characteristics of non-stationary noise are time-varying and spectral randomness, non-stationary noise will cause sudden peaks in the spectrum shape. Although the speaker's tone changes will also show certain fluctuations in the spectrum, the spectrum fluctuations caused by tone changes are continuous.

[0065] Therefore, this embodiment obtains all the peaks in the frequency domain signal P through the AMPD (Automatic multiscale-based peak detection) peak detection algorithm. Take the jth peak as an example for analysis. The least squares method is used to perform curve fitting on all the peaks in the frequency domain signal P according to the position sequence in the frequency domain signal, and the shortest distance between the jth peak and the fitting curve is calculated, and recorded as the point-line fitting distance of the jth peak. The larger the point-line fitting distance, the more inconsistent the energy distribution of the frequency position corresponding to the jth peak is with the overall trend, and the greater the possibility that the signal at the frequency position corresponding to the jth peak is interfered by non-stationary noise. Among them, the fast Fourier transform, the AMPD peak detection algorithm, and the least squares method are all existing well-known technologies, and the implementer can choose other existing feasible frequency domain transformation algorithms, peak detection algorithms, and curve fitting algorithms.

[0066] Since the sudden peaks caused by non-stationary noise are instantaneous, short-lived and usually located at high frequencies, in this embodiment, a signal window of size 1×9 is constructed with each peak as the center in the frequency domain signal P, and the size of the signal window can be set by the implementer. It should be noted that if there is a phenomenon of vacant values ​​in the signal window of the peak, the mean filling method is used to fill it.

[0067] For each peak value, the variation range and fluctuation degree of all amplitudes in its signal window are calculated, and the product of the variation range and the fluctuation degree is recorded as the local fluctuation index of each peak value. In this embodiment, the variation range is the extreme difference of all amplitudes in the signal window, and the fluctuation degree is the coefficient of variation of all amplitudes in the signal window. In other embodiments, variance, standard deviation, etc. can be used as the calculation method of the fluctuation degree. The larger the local fluctuation index is, the more significant the signal fluctuation in the signal window where the peak value is located in the frequency domain signal P is.

[0068] Based on the above analysis, the mutation coefficient of each peak is calculated. The specific calculation method is:

[0069] ; In the formula, is the mutation coefficient of the j-th peak; G is the point-line fitting distance of the j-th peak; H is the local fluctuation index of the j-th peak; W is the frequency corresponding to the j-th peak; is the normalization function. Recorded as the first product.

[0070] If the point-line fitting distance G is larger, it means that the jth peak deviates significantly from the fitting curve, and the frequency position corresponding to the jth peak is more inconsistent with the energy distribution of the overall trend; if the local fluctuation index H is larger, it means that the fluctuation at the jth peak is more violent, the possibility of energy continuity is smaller, and the possibility of a mutation peak is greater; if the corresponding frequency W is larger, it means that the jth peak is more likely to be located at a high frequency, and the possibility of being a high-frequency noise is greater. Therefore, if the mutation coefficient The larger it is, the more likely it is that the j-th peak is caused by noise interference.

[0071] S4, determining each signal period of the full speech signal based on the fundamental frequency of the full speech signal, recording the signal period containing the second resonance peak as the detection period, and screening out the mutation peak among all peaks based on the mutation coefficient; based on the energy distribution in each detection period and the clarity of the sound in all detection periods, combining the time domain disturbance index and the mutation coefficient of all mutation peaks, obtaining the time-frequency interference degree of the full speech signal.

[0072] Obtain the mutation coefficients of all peaks in the frequency domain signal P, and obtain the segmentation thresholds of the mutation coefficients of all peaks by the Otsu threshold method, and determine the peaks whose mutation coefficients are greater than the segmentation threshold as mutation peaks. The Otsu threshold method is an existing well-known technology, and the implementer can choose other existing feasible threshold segmentation algorithms.

[0073] Furthermore, if the speaker's emotions change, not only will the waveform envelope change, but also the short-time energy in the speech signal will fluctuate. Obtain the fundamental frequency of the speech signal Y, and obtain the signal period of the speech signal Y through the fundamental frequency. Perform a second resonance peak detection on each signal period, and record all signal periods containing the second resonance peak as detection periods. According to the time domain waveform diagram of the speech signal in each detection period, obtain the mean of the short-time energy of the speech signal at all times in each detection period, and record it as the average energy of each detection period. The calculation of short-time energy is a well-known technology and will not be repeated here.

[0074] The ratio of the center frequency of the second resonance peak in each detection cycle to the bandwidth is recorded as the voiceprint purity index of each detection cycle, and the average of the voiceprint purity indexes of all detection cycles is recorded as the overall voiceprint purity. The greater the overall voiceprint purity, the sharper the second resonance peak in each detection cycle, the more concentrated the energy, and the higher the sound clarity.

[0075] Based on the above analysis, the time-frequency interference degree of the speech signal Y is calculated. The specific process is:

[0076] The mean of the average energy of all detection cycles is recorded as the energy stability index, and the mean of the mutation coefficients of all mutation peaks is calculated, which is recorded as the average spectrum mutation degree; the sum of the energy stability index and the overall voiceprint purity is calculated, and the product of the time domain disturbance index and the average spectrum mutation degree is calculated, which is recorded as the second product; the time-frequency interference degree is positively correlated with the second product and negatively correlated with the cumulative sum.

[0077] In this embodiment, the time-frequency interference degree is calculated as follows:

[0078] ; In the formula, K is the time-frequency interference of the full speech signal, and A is the time-domain interference index of the speech signal Y; is the average mutation degree of the spectrum of the speech signal Y, is the energy stability index of speech signal Y, and Z is the overall voiceprint purity of speech signal Y. is the second product. The flow chart for determining the time-frequency interference degree is as follows: Figure 2 shown.

[0079] If the time domain disturbance index A is larger, it means that the noise interference detected by the speech signal Y in the time domain state is greater; if the average mutation degree of the spectrum is The larger the value is, the greater the average intensity of the mutation phenomenon in the overall spectrum, the greater the mutation degree of the peak value, and the more serious the noise interference. The smaller it is, the smaller the average energy of each detection cycle is, which reflects that the rhythmic characteristics of the speech signal are more stable, and the possibility that the speaker is in a calm mood is greater; if the overall voiceprint purity Z is smaller, it means that the speech signal is interfered by noise, which makes the sound more likely to be blurred.

[0080] Therefore, if the time-frequency interference degree K is larger, it means that when the speaker is in a calm mood, the voice signal will fluctuate violently in the time domain waveform, and the possibility of significant mutation peaks in the frequency domain and overall voiceprint blurring is greater, reflecting that the voice signal is subject to greater noise interference.

[0081] S5, converting the full speech signal into a high-dimensional phase space, and decomposing it to obtain each component, analyzing the degree of difference between each component and the remaining components, and the contribution rate of each component, to obtain the noise confidence of each component; using the noise confidence to denoise and reduce the dimension of all components, to obtain a high-dimensional denoised signal.

[0082] Furthermore, since the speech signal is also a typical nonlinear and non-stationary signal, although the analysis of the time domain and frequency domain can provide relevant speech features, there are still certain limitations in processing nonlinear and non-stationary characteristics. Therefore, this embodiment converts the speech signal into a high-dimensional space domain to further analyze the speech signal. This embodiment converts the one-dimensional speech signal Y into a high-dimensional phase space for processing.

[0083] According to the FNN (False Nearest Neighbors) false nearest neighbor algorithm, the embedding dimension m of the conversion is obtained, and the delay time is determined by the autocorrelation function , and then the one-dimensional speech signal Y is converted to a high-dimensional phase space by the coordinate delay reconstruction method, which is recorded as a multidimensional signal The FNN algorithm, autocorrelation function, and coordinate delay reconstruction method are all well-known technologies and will not be described in detail here.

[0084] The independent component analysis (ICA) algorithm is used to analyze the multidimensional signal Decompose to obtain each component, that is, one-dimensional component. Calculate multidimensional signal The average of the measured distances between each component and all the remaining components is recorded as the waveform difference index of each component; the contribution rate of each component is calculated, and the contribution rate can reflect the degree of influence of each component on the overall signal. The greater the contribution rate, the greater the possibility of reflecting a useful characteristic signal. The independent component analysis algorithm and the calculation of the contribution rate are both well-known technologies and will not be repeated here. The average of the DTW (Dynamic Time Warping) distances between each component and all remaining components. There are many existing distance measurement calculation methods, such as Euclidean distance, Manhattan distance, etc., which are not limited in this embodiment.

[0085] The ratio of the waveform difference index and contribution rate of each component is recorded as the noise confidence of each component; the larger the noise confidence, the greater the possibility that the corresponding component is a noise component. The noise confidence of all components is used as the input of the Otsu threshold algorithm to obtain the segmentation threshold. All components with noise confidence less than the segmentation threshold are recorded as non-noise components. The reconstructed phase space is then dimensionalized and converted back to a one-dimensional time domain signal, which is recorded as a high-dimensional denoised signal Q.

[0086] S6, determining a threshold in a denoising algorithm by the time-frequency interference degree, using the denoising algorithm to obtain a low-dimensional denoised signal of the full speech signal, and combining the high-dimensional denoised signal to obtain an enhanced fused speech signal.

[0087] The wavelet denoising algorithm has the advantage of time-frequency joint analysis and is suitable for processing non-stationary signals. Therefore, this embodiment performs denoising processing using the wavelet algorithm.

[0088] In the threshold estimation method of wavelet denoising, the default threshold determination model determines the threshold according to the intensity of the noise. Therefore, this embodiment calculates the default threshold of wavelet denoising by calculating the time-frequency interference degree to obtain the threshold ; In the formula, is the threshold in the denoising algorithm, K is the time-frequency interference of the full speech signal, n is the length of the full speech signal, that is, the number of elements in the speech signal Y, and lg() is a logarithmic function with a base of 10. Then, according to the threshold thr, the collected speech signal Y is denoised using the wavelet denoising algorithm, which is recorded as the low-dimensional denoised signal X. Among them, the wavelet denoising algorithm is an existing well-known technology and will not be described in detail here.

[0089] The high-dimensional denoised signal Q and the low-dimensional denoised signal X are added and averaged to obtain the denoised fused speech signal. The fused speech signal can effectively separate the potential noise components by reconstructing it in the high-dimensional phase space, and can process the mutation noise in the time-frequency domain through wavelet denoising in the low dimension; by cross-domain processing of the speech signal and fusing the denoising results, useful speech features can be better retained, avoiding speech distortion caused by excessive denoising, and improving the speech enhancement effect. The flow chart for obtaining the fused speech signal is shown in the figure. Figure 3 shown.

[0090] An embodiment of the present application also provides a speech enhancement device for cross-domain feature fusion, wherein a computer program is stored in the device, and when the computer program is executed by a processor, any of the above-mentioned speech enhancement methods is implemented.

[0091] Based on the same inventive concept as the above method, an embodiment of the present application also provides a speech enhancement system with cross-domain feature fusion, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of any one of the above-mentioned speech enhancement methods with cross-domain feature fusion are implemented.

[0092] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0093] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0094] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A speech enhancement method based on cross-domain feature fusion, characterized in that: The method comprises the following steps: Extracting the speech signal to be enhanced, recording it as the full speech signal, and dividing the full speech signal into frames; The dynamic range of the envelope of each frame of speech signal and the difference between the high-frequency component and the low-frequency component in each frame of speech signal are combined into two-dimensional data points, and the distribution of the two-dimensional data points of all frames of speech signals is analyzed to determine the time domain disturbance index of the whole speech signal; Obtain the frequency domain signal of the full speech signal, detect each peak in the frequency domain signal, perform curve fitting on all peaks, analyze the distance relationship between each peak and the fitting curve, as well as the range and fluctuation of the amplitude in the local neighborhood of each peak, and determine the mutation coefficient of each peak; Determine each signal period of the full speech signal based on the fundamental frequency of the full speech signal, record the signal period containing the second resonance peak as the detection period, and screen out the mutation peak among all peaks based on the mutation coefficient; obtain the time-frequency interference degree of the full speech signal based on the energy distribution in each detection period and the clarity of the sound in all detection periods, combined with the time domain disturbance index and the mutation coefficient of all mutation peaks; The full speech signal is converted into a high-dimensional phase space, and each component is obtained after decomposition, and the difference between each component and the remaining component and the contribution rate of each component are analyzed to obtain the noise confidence of each component; all components are denoised and dimensionally reduced using the noise confidence to obtain a high-dimensional denoised signal; Determine the threshold in the denoising algorithm by the time-frequency interference degree, obtain a low-dimensional denoised signal of the full speech signal by using the denoising algorithm, and combine the high-dimensional denoised signal to obtain an enhanced fused speech signal; The determination of the time domain disturbance index includes: Clustering all the two-dimensional data points of the full speech signal, calculating the average value of the modules of all the two-dimensional data points in each cluster, recording the cluster with the largest average value as the detection cluster, calculating the average value of the difference between the average value of the detection cluster and the average values ​​of all remaining clusters, and recording it as the outlier index of the detection cluster; Calculate the ratio of the number of elements in all clusters to the number of elements in the detection cluster, recorded as the second ratio; calculate the number of clusters, the outlier index of the detection cluster, and the fusion result of the mean of the modulus of all the two-dimensional data points in the detection cluster; The time domain disturbance index is positively correlated with the fusion result, and negatively correlated with the second ratio; The determination of the mutation coefficient includes: Calculate the shortest distance between each peak and the fitting curve, calculate the product of the range of amplitude change and the degree of fluctuation in the local neighborhood of each peak, and record it as the local fluctuation index of each peak; calculate the product of the local fluctuation index of each peak and the shortest distance, and record it as the first product; The mutation coefficient is the sum of the normalized value of the first product of each peak value and the normalized value of the frequency corresponding to each peak value; The determination of the time-frequency interference degree includes: The ratio of the center frequency of the second resonance peak in each detection cycle to the bandwidth is recorded as the voiceprint purity index of each detection cycle, and the average of the voiceprint purity indexes of all detection cycles is recorded as the overall voiceprint purity; the average of the short-term energy of the speech signal at all times in all detection cycles is recorded as the energy stability index; the average of the mutation coefficients of all mutation peaks is calculated and recorded as the average mutation degree of the spectrum; Calculate the cumulative sum of the energy stability index and the overall voiceprint purity, calculate the product of the time domain disturbance index and the average frequency spectrum mutation degree, and record it as the second product; the time-frequency interference degree is positively correlated with the second product and negatively correlated with the cumulative sum; The determination of the noise confidence level includes: Calculate the mean of the metric distance between each component and all remaining components, record it as the waveform difference index of each component, and the noise confidence is the ratio of the waveform difference index of each component to the contribution rate; The threshold in the denoising algorithm is determined by the time-frequency interference degree, and the calculation method is: Where thr is the threshold in the denoising algorithm, K is the time-frequency interference of the full speech signal, n is the length of the full speech signal, and lg() is a logarithmic function with base 10.

2. The method for speech enhancement by cross-domain feature fusion according to claim 1, characterized in that: The determination of the difference between the high frequency component and the low frequency component in each frame of speech signal includes: Perform modal decomposition on each frame of speech signal to obtain the decomposed high-frequency components and low-frequency components, and calculate the mean of the sum of squares of all elements in all high-frequency components and the mean of the sum of squares of all elements in all low-frequency components, and record them as the high-frequency energy ratio and the low-frequency energy ratio respectively; The ratio of the high-frequency energy proportion to the low-frequency energy proportion is calculated and recorded as a first ratio. The difference between the high-frequency component and the low-frequency component in each frame of the speech signal is the first ratio.

3. The method for speech enhancement by cross-domain feature fusion according to claim 1, characterized in that: The determination of the mutation peak value includes: The segmentation threshold of the mutation coefficient of all peaks is obtained, and the peak whose mutation coefficient is greater than the segmentation threshold is determined as the mutation peak.

4. A speech enhancement device for cross-domain feature fusion, wherein a computer program is stored in the device, characterized in that: When the computer program is executed by a processor, the speech enhancement method according to any one of claims 1 to 3 is implemented.

5. A speech enhancement system for cross-domain feature fusion, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

Images