Audio processing method and system for speech spectrum reconstruction

Abstract

The invention relates to the technical field of audio processing, and discloses an audio processing method and system for reconstructing a speech spectrum. The method comprises: preprocessing an original speech signal collected by a vehicle-mounted microphone to obtain a speech signal time-frequency matrix; performing singular value decomposition on the speech signal time-frequency matrix to obtain a sparse matrix representing speech components and a low-rank matrix representing noise components, and establishing a frequency band feature association matrix by a mutual information entropy calculation method; inputting the sparse matrix into a selective forgetting extreme learning machine to perform speech feature prediction to obtain predicted speech feature data; based on the predicted speech feature data and the frequency band feature association matrix, using a logarithmic spectrum amplitude calculation method to perform speech probability calculation and feature interpolation on each time-frequency point to obtain reconstructed spectrum data. The invention improves the accuracy of speech recognition and meets the requirements of real-time processing in a vehicle-mounted environment.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio processing method and system for speech spectrum reconstruction. Background Art

[0002] With the widespread application of in-vehicle voice interaction systems, accurately recognizing the driver's voice commands in complex driving environments has become an important issue. However, there are many interference sources in the in-vehicle environment, such as engine noise, road noise, and wind noise. These noises highly overlap with voice signals in the time and frequency domains, seriously affecting the quality and recognizability of voice signals.

[0003] Traditional speech enhancement methods are mainly based on technologies such as spectral subtraction and Wiener filtering. Although they can suppress background noise to a certain extent, they often introduce problems such as music noise and speech distortion. Especially under low signal-to-noise ratio conditions, these methods are difficult to effectively distinguish between speech and noise components, resulting in a significant decrease in speech recognition accuracy. Summary of the invention

[0004] The present invention provides an audio processing method and system for speech spectrum reconstruction, which improves the accuracy of speech recognition and meets the requirements of real-time processing in a vehicle-mounted environment.

[0005] In a first aspect, the present invention provides an audio processing method for speech spectrum reconstruction, the audio processing method for speech spectrum reconstruction comprising:

[0006] Preprocess the original speech signal collected by the vehicle microphone to obtain the speech signal time-frequency matrix;

[0007] Performing singular value decomposition on the speech signal time-frequency matrix to obtain a sparse matrix representing speech components and a low-rank matrix representing noise components, and establishing a frequency band feature correlation matrix by a mutual information entropy calculation method;

[0008] Inputting the sparse matrix into a selective forgetting extreme learning machine to perform speech feature prediction to obtain predicted speech feature data;

[0009] Based on the predicted speech feature data and the frequency band feature association matrix, a logarithmic spectrum amplitude calculation method is used to perform speech probability calculation and feature interpolation on each time-frequency point to obtain reconstructed spectrum data.

[0010] In a second aspect, the present invention provides an audio processing system for speech spectrum reconstruction, the audio processing system for speech spectrum reconstruction comprising:

[0011] A preprocessing module is used to preprocess the original speech signal collected by the vehicle microphone to obtain a speech signal time-frequency matrix;

[0012] A singular value decomposition module is used to perform singular value decomposition on the time-frequency matrix of the speech signal to obtain a sparse matrix representing the speech component and a low-rank matrix representing the noise component, and to establish a frequency band feature correlation matrix by a mutual information entropy calculation method;

[0013] A prediction module, used for inputting the sparse matrix into a selective forgetting extreme learning machine to perform speech feature prediction to obtain predicted speech feature data;

[0014] The calculation module is used to perform speech probability calculation and feature interpolation on each time-frequency point based on the predicted speech feature data and the frequency band feature association matrix by using a logarithmic spectrum amplitude calculation method to obtain reconstructed spectrum data.

[0015] In the technical solution provided by the present invention, a selective forgetting extreme learning machine is introduced to predict speech features. Combined with the selection mechanism and the forgetting mechanism, the temporal dependency of speech signals can be effectively captured, thereby improving the prediction accuracy of speech features; a matrix decomposition method based on singular value decomposition is adopted to decompose the speech signal into a sparse matrix and a low-rank matrix, and a frequency band feature correlation matrix is ​​established through mutual information entropy calculation, thereby achieving effective separation of speech components and noise components; combined with logarithmic spectrum amplitude calculation and feature interpolation technology, the time-frequency points are accurately reconstructed, thereby effectively maintaining the continuity and naturalness of the speech signal; an adaptive Wiener filter and a dynamic gain control strategy are designed, and the filter parameters and gain coefficients are dynamically adjusted according to the signal-to-noise ratio, thereby significantly improving the speech enhancement effect; a deep neural network model is adopted for speech recognition, and the long-term and short-term dependencies of speech features are effectively extracted through a bidirectional long short-term memory network and a self-attention mechanism, thereby improving the accuracy of speech recognition and meeting the requirements of real-time processing in a vehicle environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 A schematic diagram of the steps of an audio processing method for reconstructing speech spectrum in an embodiment of the present invention;

[0018] Figure 2 It is a structural schematic diagram of an audio processing system for speech spectrum reconstruction in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Embodiments of the present invention provide an audio processing method and system for speech spectrum reconstruction. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the audio processing method for speech spectrum reconstruction in an embodiment of the present invention includes:

[0021] Step S1, preprocessing the original speech signal collected by the vehicle microphone to obtain a speech signal time-frequency matrix;

[0022] It is understandable that the execution subject of the present invention may be an audio processing system for speech spectrum reconstruction, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0023] Specifically, the original voice signal collected by the vehicle microphone exists in the form of an analog signal. The original voice signal is converted into a digital signal by analog-to-digital conversion to obtain voice signal data. The voice signal data is subjected to DC component elimination processing. By subtracting the mean of the signal, the signal is converted into a zero-mean voice signal, eliminating the influence of DC offset on subsequent processing. The zero-mean voice signal is segmented and intercepted to obtain overlapping voice signal segments. When segmenting, the overlapping interception method can effectively avoid spectrum leakage and ensure the continuity of frequency information. That is, each adjacent signal segment has a certain overlapping part, which ensures the smooth transition of each signal segment and improves the time-frequency resolution. The segmented voice signal segments are weighted by applying a preset window function. The Hamming window function is selected as the window function. By multiplying the signal segment by the weighting coefficient of the Hamming window function point by point, the signal boundary is smoothed, and the spectrum leakage caused by signal truncation in spectrum analysis is reduced. The Hamming window has good spectrum smoothing performance, taking into account the balance between the main lobe width and sidelobe suppression in the frequency domain. Fast Fourier transform is applied to the weighted voice signal segment. Fast Fourier transform is a discrete Fourier transform implementation method that converts time domain signals into frequency domain signals to reveal the frequency characteristics of speech signals. After fast Fourier transform, the signal is represented as a complex spectrum containing amplitude information and phase information, where the amplitude part reflects the energy distribution of the signal at each frequency, and the phase part describes the relative phase of the frequency component. The complex spectrum is processed using short-time Fourier transform. Short-time Fourier transform obtains the time-varying amplitude spectrum and time-varying phase spectrum of the speech signal by gradually moving the window function on the time axis and performing Fourier transform on the signal in each window. The time-varying amplitude spectrum reflects the distribution of the signal's energy over time, while the time-varying phase spectrum records the phase changes of the frequency components. The time-varying amplitude spectrum and the time-varying phase spectrum are resynthesized in the complex domain to obtain the complex spectrum features. The complex spectrum features of the speech signal are filled into a two-dimensional matrix according to the time series to obtain the speech signal time-frequency matrix, which consists of N rows and M columns, where N represents the frequency dimension and M represents the time dimension.

[0024] Step S2, performing singular value decomposition on the time-frequency matrix of the speech signal to obtain a sparse matrix representing the speech component and a low-rank matrix representing the noise component, and establishing a frequency band feature correlation matrix by a mutual information entropy calculation method;

[0025] Specifically, the time-frequency matrix of the speech signal is subjected to singular value decomposition, and the decomposition form of USV is adopted, where U and V are orthogonal matrices, respectively, and S is a diagonal matrix containing all singular values. This decomposition method decomposes the time-frequency matrix of the speech signal into the product of three matrices, thereby separating the information representing different characteristics of the signal. In the decomposition, the singular value sequence is sorted according to the amplitude to distinguish the speech component and the noise component. The singular values ​​with larger amplitudes mainly correspond to the significant features of the speech signal, while the singular values ​​with smaller amplitudes mainly reflect the noise or background components. These singular value sequences are divided based on the preset threshold to obtain the speech singular value sequence and the noise singular value sequence respectively. A new diagonal matrix is ​​constructed based on the speech singular value sequence, and a three-matrix multiplication operation is performed on it with the U matrix and the V matrix in the decomposition matrix group to generate a sparse matrix representing the speech component. The sparse matrix retains the significant features of the speech signal and achieves a higher concentration of speech components by reducing noise interference. At the same time, a diagonal matrix is ​​constructed based on the noise singular value sequence, and the diagonal matrix is ​​multiplied by three matrices with the U matrix and the V matrix in the decomposition matrix group to obtain a low-rank matrix that represents the noise component. The low-rank matrix mainly reflects the smooth distribution characteristics of the background noise and is a supplement to the speech sparse matrix. The frequency band characteristics of the speech sparse matrix are divided into sub-bands according to the Mel frequency scale. The Mel frequency scale is a nonlinear frequency scale that conforms to human auditory perception. Its division method can better reflect the perceptual characteristics of speech signals in different frequency bands. The sparse matrix is ​​decomposed into a sub-matrix sequence of K frequency bands, and each sub-matrix corresponds to a speech feature in a frequency band. The sub-matrix sequences of K frequency bands are paired in pairs, and the mutual information entropy value is calculated for each pair of sub-matrices. Mutual information entropy is an indicator used to measure the statistical correlation between two variables, and is used to quantify the degree of correlation between sub-matrices of different frequency bands, thereby obtaining a mutual information entropy matrix. The mutual information entropy matrix is ​​tested for symmetry and normalized. The symmetry test can verify whether the mutual information entropy matrix meets the symmetry requirements of the frequency band association, while the normalization process maps the matrix element values ​​to a standardized range, usually between [0,1], so that the degree of association between different frequency bands is more intuitive. The frequency band feature association matrix is ​​constructed based on the frequency band correlation data. The element values ​​of this matrix directly represent the degree of association between different frequency bands. For example, the larger the value, the stronger the association between the two frequency bands.

[0026] Step S3, inputting the sparse matrix into the selected forgetting extreme learning machine to perform speech feature prediction to obtain predicted speech feature data;

[0027] Specifically, the sparse matrix is ​​segmented according to the time axis, and each segment represents the speech feature data within a time window. Each segment of speech features is used as a training sample to form a speech feature training sequence. The segmented processing method can effectively capture the local characteristics of speech signals in time and reduce the computational complexity. The connection weights between the input layer and the hidden layer of the selected forgetting extreme learning machine are randomly generated based on the speech feature training sequence. The input weight matrix is ​​constructed by random generation, and each column of the matrix represents the weight distribution of a neuron in the hidden layer. The input weight matrix is ​​matrix multiplied with the speech feature training sequence, and a bias term is superimposed to map the linear calculation result to the hidden layer. In order to introduce nonlinear characteristics, the hyperbolic tangent function (tanh) is used to activate the input value of the hidden layer to generate the output features of the hidden layer. The Moore-Penrose generalized inverse matrix is ​​calculated for the output features of the hidden layer, and the weight matrix from the hidden layer to the output layer is solved analytically. The analytical weight matrix optimizes the weight distribution of the hidden layer by minimizing the output error to ensure the accuracy of speech feature prediction. The output features of the hidden layer are screened based on the selection mechanism. The selection mechanism measures the similarity of each feature by calculating the Euclidean distance between each feature and the central feature. The central feature represents the feature mean or weighted center within a time window, and the feature with higher similarity reflects the main speech characteristics within the current time window. Through this step, a speech feature subset is screened out to remove redundant or noise features. After the feature screening is completed, the features at each moment in the speech feature subset are subjected to time decay processing through the forgetting mechanism. The forgetting mechanism calculates its decay coefficient according to the time interval of the feature, and the coefficient decays in an exponential form, which can simulate the characteristic that the influence of historical features on current features in the speech signal gradually weakens. Through this process, the historical features are gradually forgotten, thereby dynamically updating the input feature set of the model to make it more in line with the actual situation of the current speech signal. The updated speech features are matrix multiplied with the analytical weight matrix to generate the speech feature prediction value at the next moment. By sequentially splicing the prediction results at each moment, the feature sequence of the entire speech signal is reconstructed to obtain the predicted speech feature data.

[0028] Step S4: Based on the predicted speech feature data and the frequency band feature correlation matrix, a logarithmic spectrum amplitude calculation method is used to perform speech probability calculation and feature interpolation on each time-frequency point to obtain reconstructed spectrum data.

[0029] Specifically, the predicted speech feature data is logarithmically calculated, and the amplitude information of the speech signal is re-expressed in logarithmic form, so that the data distribution is more consistent with the perception characteristics of the human ear. The logarithmic amplitude data is normalized by the frequency response function to ensure that the amplitude characteristics of different frequency ranges are on a unified numerical scale, and a standardized logarithmic spectrum sequence is obtained. The frequency band energy mapping relationship is constructed based on the frequency band feature association matrix. The frequency band feature association matrix describes the correlation between different frequency bands. Through this matrix, the energy contribution of each frequency band is determined, and the standardized logarithmic spectrum sequence is weighted and summed according to the frequency band using this mapping relationship to obtain the frequency band energy distribution matrix. The matrix takes time and frequency as dimensions and reflects the energy distribution characteristics of the speech signal at each time-frequency point. The Euclidean distance is calculated for each time-frequency point in the frequency band energy distribution matrix and its adjacent time-frequency point to measure the similarity between the time-frequency points. By comparing with a preset distance threshold, it is determined whether the time-frequency point belongs to the high probability area of ​​the speech signal, the speech probability is calculated, and a time-frequency probability distribution matrix is ​​generated. Cluster analysis is performed on the speech time-frequency points according to the time-frequency probability distribution matrix. Cluster analysis divides the time-frequency points into the main speech area and the transition area according to the speech probability, and generates a regional labeling matrix. The main speech area contains significant speech signal features, while the transition area contains the edge part of the speech signal or low-probability speech features. With the support of the regional labeling matrix, the contour of the main speech area is extracted and the optimal speech boundary trajectory is calculated. The speech boundary trajectory can accurately depict the main time-frequency area of ​​the speech signal and provide a reference for the boundary processing and interpolation calculation of the spectrum. Based on the speech boundary trajectory, cubic spline interpolation calculation is performed on the time-frequency points in the transition area to generate the spectrum interpolation data of the transition area. The spectrum data of the main speech area and the spectrum interpolation data of the transition area are weighted and fused to form a logarithmic spectrum matrix. The logarithmic spectrum matrix is ​​inversely logarithmically transformed to restore the original amplitude information of the signal. At the same time, phase reconstruction is performed to supplement the phase part of the spectrum. By combining the original phase information of the time-frequency matrix or generating new phase values ​​based on the consistency algorithm, the reconstructed spectrum data is finally obtained.

[0030] The power spectrum is calculated for each time-frequency point of the reconstructed spectrum data. The power spectrum is obtained by squaring the modulus value of the complex spectrum to reflect the energy distribution characteristics of the signal. In order to smooth the instantaneous fluctuations in the power spectrum, a sliding average is performed on it. The sliding average effectively reduces the influence of short-term random fluctuations by averaging the power spectrum values ​​at multiple consecutive moments on the time axis, and obtains a smoothed power spectrum matrix. Based on the smoothed power spectrum matrix, the power spectrum value of the background noise is calculated by frequency band partitioning, and the noise power spectrum matrix is ​​generated by using the minimum tracking algorithm or statistical model to reflect the characteristics of the background noise in each frequency band. According to the ratio relationship between the smoothed power spectrum matrix and the noise power spectrum matrix, the priori signal-to-noise ratio and the posterior signal-to-noise ratio of each frequency band are calculated. The priori signal-to-noise ratio estimates the predicted power of the speech signal through historical features, while the posterior signal-to-noise ratio is directly calculated based on the ratio of the power spectrum value at the current moment to the noise power spectrum value. The two are combined to form a signal-to-noise ratio parameter matrix. The signal-to-noise ratio parameter matrix is ​​weighted by the transfer function of the Wiener filter. The Wiener filter can optimize the frequency response of the filter according to the signal-to-noise ratio and generate an adaptive filter coefficient matrix. The matrix adaptively adjusts the gain of the signal in the frequency domain and effectively suppresses the noise component. The adaptive filter coefficient matrix is ​​used to perform complex matrix multiplication with the reconstructed spectrum data to obtain the filtered speech spectrum. The filtered speech spectrum is divided into subbands, the energy characteristics of each subband are analyzed, and the energy mean and variance of each subband are calculated to reflect the dynamic change characteristics of the spectrum signal, which is then used to generate dynamic gain control parameters. The dynamic gain control parameters perform nonlinear mapping operations and signal amplitude compensation according to the energy characteristics of each subband. Nonlinear mapping uses compression or expansion operations to ensure that the amplitude characteristics of the signal are more in line with the auditory perception of the human ear. Through signal amplitude compensation, the adjusted speech spectrum restores its original dynamic characteristics to obtain the gain-compensated speech spectrum. The gain-compensated speech spectrum is restored to the time domain signal through the inverse short-time Fourier transform. In the inverse transform process, windowing and overlap-add operations are performed simultaneously to ensure the smooth transition and continuity of the signal on the time axis to generate an enhanced speech signal. Mel-frequency cepstral coefficient (MFCC) features are extracted from the enhanced speech signal. MFCC feature is a feature representation method widely used in speech recognition. It maps the spectral characteristics of speech signals to the nonlinear frequency space of human auditory perception based on the Mel frequency scale, and extracts feature parameters containing speech information through cepstrum calculation. The extracted MFCC features are input into the deep neural network model for speech recognition. The deep neural network uses its representation ability to learn the semantic information of the signal from complex speech features and obtain speech recognition results.

[0031] The enhanced speech signal is preprocessed by applying a 512-point fast Fourier transform after framing the signal to convert the time domain signal to the frequency domain to reveal the frequency components of the signal. After obtaining the spectrum, the spectrum is filtered based on 26 triangular Mel filter banks to calculate the energy distribution of each frequency band and generate a Mel-band energy spectrum. These filters divide the frequency bands according to the Mel frequency scale, which conforms to the auditory perception characteristics of the human ear and makes the frequency distribution of the speech signal more auditory relevant. The Mel-band energy spectrum is logarithmically operated to compress its dynamic range and make the data more consistent with the energy perception characteristics of the speech signal. The logarithmic energy spectrum is mapped from the frequency domain to the cepstrum domain through discrete cosine transform to remove the correlation between frequency bands and extract representative Mel-frequency cepstrum coefficients. The first 13 cepstrum coefficients are selected as static features to capture the main spectral characteristics of the speech signal. In order to enhance the temporal dynamic information of the features, the first-order difference and second-order difference features are calculated for the static features, which represent the speed and acceleration of the feature change, respectively, to form a complete Mel-frequency cepstrum coefficient feature set. The extracted Mel frequency cepstral coefficient features are input into the Bidirectional Long Short-Term Memory (BiLSTM) layer of the deep neural network model for time series modeling. The bidirectional LSTM contains 256 memory units, which simultaneously models the feature sequence in both the forward and backward directions to capture the contextual information in the speech signal over a long period of time. Through bidirectional modeling, the generated context encoding features can simultaneously consider the features of the current speech frame and the dependencies between the previous and next frames. The context encoding features are input into the self-attention layer of the deep neural network model. The design of the self-attention layer contains 8 attention heads, each with a dimension of 32. Through the dot product attention mechanism, the self-attention layer dynamically assigns attention weights to capture the correlation between different time steps in the feature sequence. The attention mechanism generates weighted context features by weighting the features, so that the model can more effectively focus on the key information related to semantics in the speech signal. The weighted context features are input into the 3-layer feedforward neural network of the deep neural network for high-level feature extraction. Each layer of the feedforward neural network contains 512 neurons, and the ReLU activation function is used to introduce nonlinear expression capabilities. In order to alleviate the gradient vanishing problem and improve the training efficiency of the network, residual connections are used between each layer of the feedforward neural network so that the model can better capture complex speech features. High-level semantic features are generated through layer-by-layer processing of the multi-layer feedforward neural network. The high-level semantic features are sequence-labeled, and the state transition probability is calculated based on the dependency between the context information and the feature sequence to generate the phoneme recognition sequence. The phoneme recognition sequence represents the smallest language unit of the speech signal, but it does not yet have complete semantic information. In order to improve the accuracy of the recognition results, the phoneme recognition sequence is input into the language model for decoding. The decoding process uses pre-built dictionaries and grammar rules to impose language-level constraints on the phoneme sequence, correct the recognition errors, and finally generate the speech recognition results.

[0032] In the embodiment of the present invention, by introducing a selective forgetting extreme learning machine to predict speech features, combined with the selection mechanism and the forgetting mechanism, the temporal dependency of the speech signal can be effectively captured, thereby improving the prediction accuracy of the speech features; a matrix decomposition method based on singular value decomposition is adopted to decompose the speech signal into a sparse matrix and a low-rank matrix, and a frequency band feature correlation matrix is ​​established through mutual information entropy calculation, thereby achieving effective separation of speech components and noise components; by combining logarithmic spectrum amplitude calculation and feature interpolation technology, the time-frequency points are accurately reconstructed, effectively maintaining the continuity and naturalness of the speech signal; an adaptive Wiener filter and a dynamic gain control strategy are designed, and the filter parameters and gain coefficients are dynamically adjusted according to the signal-to-noise ratio, thereby significantly improving the speech enhancement effect; a deep neural network model is adopted for speech recognition, and the long-term and short-term dependencies of speech features are effectively extracted through a bidirectional long short-term memory network and a self-attention mechanism, thereby improving the accuracy of speech recognition and meeting the requirements of real-time processing in a vehicle environment.

[0033] In a specific embodiment, the process of executing step S1 may specifically include the following steps:

[0034] Perform analog-to-digital conversion on the original voice signal collected by the vehicle microphone to obtain voice signal data, and eliminate the DC component of the voice signal data to obtain a zero-mean voice signal;

[0035] Segmentally intercepting the zero-mean speech signal to obtain overlapping speech signal segments, and inputting the overlapping speech signal segments into a preset Hamming window function for weighted calculation to obtain speech signal data after windowing;

[0036] Performing fast Fourier transform on the windowed speech signal data to obtain a complex spectrum containing amplitude information and phase information;

[0037] Performing time-frequency analysis on the complex spectrum through short-time Fourier transform to obtain the time-varying amplitude spectrum and time-varying phase spectrum of the speech signal, and performing complex domain synthesis on the time-varying amplitude spectrum and time-varying phase spectrum of the speech signal to obtain the time-frequency feature data of the speech signal;

[0038] A matrix structure of N rows and M columns is constructed according to the time-frequency feature data of the speech signal, and data is filled in according to the time sequence relationship to obtain the speech signal time-frequency matrix, where N represents the frequency dimension and M represents the time dimension.

[0039] Specifically, the collected analog voice signal is converted into digital form. Indicates that is a continuous time variable, Is the signal at the moment Through sampling operation, it is converted into a discrete time signal ,in is a discrete time index, indicating the sampling points, is a discrete signal at index The sampling frequency Select a frequency greater than twice the maximum frequency of the speech signal to satisfy the Nyquist sampling theorem. The output of the analog-to-digital conversion is discrete speech signal data. Finally, in order to eliminate the DC component in the signal, the mean value of the signal is calculated , which is defined as:

[0040] ;

[0041] in, is the total number of sampling points of the signal; The signal is The amplitude of the sampling points is obtained by subtracting the mean value from the signal. , and we get a zero-mean signal Zero-mean processing can effectively remove the DC offset in the signal, ensuring that subsequent processing focuses on the dynamic components of the signal. The zero-mean signal is segmented to meet the needs of short-term analysis. The segmentation process is implemented through a fixed-length sliding window with a window length of , the sliding step size of the window is The result of signal segmentation is multiple overlapping segments, each of which is represented by Indicates that is the fragment index, indicating the fragments, is the sample index within the fragment, defined as:

[0042] ;

[0043] in, is a zero-mean signal; It is The starting position of the segment; is the window length, which indicates the number of sample points contained in each segment. The purpose of overlapping segments is to ensure the timing continuity of the signal and avoid information loss. Apply a preset Hamming window function for weighting. Hamming window function Defined as:

[0044] ;

[0045] in, is the sample point index, indicating the position of the currently calculated point in the window; is the window length, which controls the width of the window function. The main function of the Hamming window is to reduce the spectrum leakage effect of the signal. The signal after windowing The expression is:

[0046] ;

[0047] in, is the amplitude of the segmented signal; is the Hamming window coefficient, ranging from [0,1]. By weighting, the boundaries of the signal segments become smooth, thereby improving the accuracy of subsequent spectrum analysis. Perform fast Fourier transform to convert the time domain signal to the frequency domain and obtain the complex spectrum , and its calculation formula is:

[0048] ;

[0049] in, is the complex spectrum, indicating the time slice at frequency The spectrum value at ; is the signal after windowing; is the frequency index, ranging from ; is the window length, which indicates the number of samples per segment of the signal; is an imaginary unit, satisfying Complex spectrum Expressed in polar coordinate form ,in, : Amplitude spectrum, indicating frequency The amount of energy at : Phase spectrum, indicating frequency The complex spectrum is processed into a time-varying amplitude spectrum through short-time Fourier transform. and time-varying phase spectrum The result of the short-time Fourier transform is a set of time-varying spectra that can accurately capture the characteristics of the speech signal in the time and frequency domains. In order to reconstruct the complex domain characteristics of the signal, the time-varying amplitude spectrum and the time-varying phase spectrum are resynthesized as:

[0050] ;

[0051] Thus, the time-frequency characteristic data of the signal is generated. The matrix structure of represents the frequency dimension, which is equal to (determined by the number of frequency points of the fast Fourier transform), Represents the time dimension, which is determined by the total length of the signal and the segment step size. Each column of the time-frequency matrix corresponds to a time segment, and each row corresponds to a frequency component. The elements of the matrix are Defined as

[0052] ;

[0053] in, is an element in the time-frequency matrix, representing a time segment and frequency points The complex spectrum value of ; It is The time segment The amplitude value of each frequency point; It is The time segment The complete speech signal time-frequency matrix is ​​generated by filling each column of the matrix in time order.

[0054] In a specific embodiment, the process of executing step S2 may specifically include the following steps:

[0055] Perform USV singular value decomposition on the time-frequency matrix of the speech signal to obtain a decomposition matrix group containing the mixed characteristics of speech and noise;

[0056] Sorting the singular values ​​in the decomposition matrix group according to the magnitude of the amplitude, and performing separation processing on the speech component and the noise component based on a preset threshold, to obtain a speech singular value sequence and a noise singular value sequence;

[0057] A diagonal matrix is ​​constructed based on the speech singular value sequence, and a three-matrix multiplication operation is performed on the diagonal matrix with the U matrix and the V matrix in the decomposition matrix group to obtain a sparse matrix representing the speech components;

[0058] A diagonal matrix is ​​constructed based on the noise singular value sequence, and a three-matrix multiplication operation is performed on the diagonal matrix with the U matrix and the V matrix in the decomposition matrix group to obtain a low-rank matrix representing the noise component;

[0059] The sparse matrix is ​​divided into sub-bands according to the Mel frequency scale to obtain a sub-matrix sequence of K frequency bands, and the sub-matrix sequences of K frequency bands are paired with each other, and the mutual information entropy value is calculated for each pair of sub-matrices to obtain a mutual information entropy matrix;

[0060] The mutual information entropy matrix is ​​symmetry-checked and normalized to obtain frequency band correlation data, and a frequency band feature correlation matrix is ​​constructed based on the frequency band correlation data. The matrix element values ​​of the frequency band feature correlation matrix represent the degree of correlation between different frequency bands.

[0061] Specifically, the time-frequency matrix of the speech signal Apply singular value decomposition to decompose it into the product of three matrices, expressed as:

[0062] ;

[0063] in, Represents the time-frequency matrix of the speech signal, whose dimension is ,in is the frequency dimension, which represents the number of rows in the matrix, corresponding to the number of frequency components of the speech signal. is the time dimension, which represents the number of columns of the matrix, corresponding to the number of time frames of the speech signal; is a left singular matrix with dimension , each column is The left singular vector of is used to represent the characteristic information in the frequency domain; is a diagonal matrix with dimensions , the diagonal elements are singular values , these singular values ​​are sorted by magnitude and are used to measure the importance of the corresponding singular vectors; is a right singular matrix with dimension , each column is The right singular vector of is used to represent the characteristic information in the time domain. Through singular value decomposition, the time-frequency matrix is decomposed into a matrix group that characterizes the signal characteristics, where the singular values is a diagonal matrix The elements are used to reflect the importance of each component in the signal. According to the preset threshold , classify the singular values ​​and divide them into speech singular value sequences and noise singular value sequences. Speech singular value sequence Contains values ​​greater than The singular values ​​of the noise are Include values ​​less than or equal to The singular values ​​of . Its mathematical definition is:

[0064] ;

[0065] in and are the singular value diagonal matrices of speech and noise, respectively, and It is the row and column index of the matrix, indicating the location of the singular value. , construct a speech sparse matrix , and its calculation formula is:

[0066] ;

[0067] in is a speech sparse matrix, representing the main features of speech components, is the left singular matrix obtained by decomposition, is a diagonal matrix consisting of speech singular values, is the transpose of the right singular matrix. Similarly, based on the noise singular value sequence , construct a noisy low-rank matrix , the formula is:

[0068] ;

[0069] in is a noise low-rank matrix, used to characterize the characteristics of the noise component, is a diagonal matrix composed of noise singular values. The sub-band division is performed according to the Mel frequency scale. The Mel frequency scale is a nonlinear frequency division method, and its calculation formula is:

[0070] ;

[0071] in is the linear frequency, representing the frequency component of the signal, Mel is the corresponding Mel frequency, which is used to reflect the human ear's perception of frequency. Decompose into sub-matrices, each sub-matrix corresponds to a frequency band, denoted as Then pair these sub-matrices and calculate the mutual information entropy value , the formula is:

[0072] ;

[0073] in is a submatrix and The mutual information entropy value of is their joint probability distribution, and They are and The marginal probability distribution of is used to quantify the information correlation between frequency bands. After calculating the mutual information entropy values ​​of all pairs, the mutual information entropy matrix is ​​constructed ,in To ensure matrix symmetry and normalization, Symmetry check and normalization are performed, and the normalization formula is:

[0074] ;

[0075] in Is the largest element value in the mutual information entropy matrix. Based on the normalized mutual information entropy matrix , construct the frequency band feature correlation matrix , whose elements Characterizes the frequency band and frequency band Through this process, the speech and noise components are separated and the correlation information between the frequency features of the speech signal is extracted.

[0076] In a specific embodiment, the process of executing step S3 may specifically include the following steps:

[0077] The sparse matrix is ​​segmented according to the time axis, and each segment of speech features is used as a training sample to obtain a speech feature training sequence;

[0078] The connection weights between the input layer and the hidden layer of the forgetting extreme learning machine are randomly generated based on the speech feature training sequence to obtain an input weight matrix;

[0079] Multiply the input weight matrix with the speech feature training sequence and superimpose the bias term, perform activation operation through the hyperbolic tangent function, and obtain the hidden layer output feature;

[0080] Perform Moore-Penrose generalized inverse matrix calculation on the hidden layer output features to obtain the analytic weight matrix from the hidden layer to the output layer;

[0081] The hidden layer output features are screened based on the selection mechanism, and the similarity between each feature and the central feature is calculated by Euclidean distance to obtain the speech feature subset;

[0082] Based on the forgetting mechanism, the time decay coefficient of the features at each moment in the speech feature subset is calculated, and the historical features are forgotten to obtain the updated speech features.

[0083] The updated speech feature is multiplied by the parsing weight matrix to obtain the speech feature prediction value at the next moment, and the predicted speech feature prediction values ​​are sequentially spliced ​​to obtain the predicted speech feature data.

[0084] Specifically, for sparse matrices Segment processing is performed according to the time axis, dividing the matrix into multiple small segments, each of which represents the speech features within a period of time. The dimension is ,in represents the frequency dimension, Represents the time dimension. By choosing a fixed segment length and overlap step length , each feature is expressed as Indicates that is a segment index, It is After segmentation, each feature segment is flattened into a one-dimensional vector to form a speech feature training sample sequence, denoted as ,in It is The flattened feature vector of the segment, with dimension is the total number of samples after segmentation. The connection weights between the input layer and the hidden layer of the forgetting extreme learning machine are randomly generated based on the speech feature training sequence. Assume that the hidden layer contains neurons, then the input weight matrix The dimension is ,in is the number of neurons in the hidden layer, is the dimension of the input vector. The elements of the input weight matrix Randomly generated according to uniform distribution or Gaussian distribution, the formula is:

[0085] ;

[0086] in It is hidden layer neurons and the The connection weights between the input features, is the range of uniform distribution, and are the mean and variance of the Gaussian distribution respectively. The weight matrix will be input Training samples with speech features Multiply and add a bias vector , and then the nonlinear mapping is performed through the hyperbolic tangent activation function (tanh) to obtain the output features of the hidden layer The formula is:

[0087] ;

[0088] in is the hidden layer output feature matrix, with dimension , is the transpose of the input sample matrix, is the bias vector, whose dimension is , each element The hidden layer The output features of the hidden layer Perform Moore-Penrose generalized inverse matrix calculation to obtain the analytic weight matrix from the hidden layer to the output layer The calculation formula of the generalized inverse matrix is:

[0089] ;

[0090] in yes The Moore-Penrose generalized inverse of is the label matrix of the target output, which represents the true eigenvalue of the training sample. The generalized inverse matrix is ​​solved by the least squares method and used to optimize the output weights of the model. , and screen them through the selection mechanism. The selection mechanism calculates the Euclidean distance between the hidden layer output features and the central features to measure the similarity. Let the central feature vector be , then The Euclidean distance between a feature and the central feature is:

[0091] ;

[0092] in is the distance value, is the hidden layer output feature vectors, represents the two-norm. Select the distance less than the threshold The features of , which is used for subsequent forgetting mechanism processing. The forgetting mechanism calculates the time decay coefficient of each feature The historical features are weighted. The formula for the attenuation coefficient is:

[0093] ;

[0094] in is the time interval between the feature and the current moment, is a time constant that controls the decay speed. The updated speech features are:

[0095] ;

[0096] in is the updated feature matrix, It means that the matrix is ​​multiplied element by element. is a diagonal matrix of time decay coefficients. The updated speech features With the analytical weight matrix Multiply them together to get the predicted value of speech features at the next moment , the formula is:

[0097] ;

[0098] By sequentially concatenating the predicted values ​​at all times, complete predicted speech feature data is finally generated.

[0099] In a specific embodiment, the process of executing step S4 may specifically include the following steps:

[0100] The predicted speech feature data is logarithmically calculated, and the amplitude is normalized based on the frequency response function to obtain a standardized logarithmic spectrum sequence;

[0101] Based on the frequency band feature correlation matrix, the frequency band energy mapping relationship is constructed, and the weighted summation operation is performed on the standardized logarithmic spectrum sequence according to the frequency band to obtain the frequency band energy distribution matrix;

[0102] The Euclidean distance between each time-frequency point and the adjacent time-frequency point in the frequency band energy distribution matrix is ​​calculated, and the speech probability is calculated based on the distance threshold to obtain the time-frequency probability distribution matrix;

[0103] According to the time-frequency probability distribution matrix, the speech time-frequency points are clustered and divided into the speech main area and the transition area to obtain the area labeling matrix;

[0104] Perform contour extraction and optimal speech boundary calculation on the speech main body area in the regional labeling matrix to obtain the speech boundary trajectory, and perform cubic spline interpolation calculation on the time-frequency points of the transition area based on the speech boundary trajectory to obtain the spectrum interpolation data of the transition area;

[0105] The spectrum data of the main speech area and the spectrum interpolation data of the transition area are weightedly fused to obtain a logarithmic spectrum matrix, and the logarithmic spectrum matrix is ​​inversely logarithmically transformed and phase reconstructed to obtain reconstructed spectrum data.

[0106] Specifically, for predicting speech feature data , perform logarithmic amplitude calculation to compress its dynamic range and make the spectrum data more consistent with the perception characteristics of the human ear. The calculation formula for the logarithmic amplitude is:

[0107] ;

[0108] in is the frequency and time The amplitude value at It is a logarithmic amplitude spectrum, which is used to represent the spectral energy distribution on a logarithmic scale. Amplitude normalization is performed based on the frequency response function to eliminate the inhomogeneity of different frequency responses. The normalization formula is:

[0109] ;

[0110] in is the normalized logarithmic magnitude spectrum, is the frequency response function, which indicates the weight of different frequency components. Through this process, a standardized logarithmic spectrum sequence is generated. , whose amplitude values ​​are adjusted to a uniform scale in all frequency ranges. Based on the frequency band feature correlation matrix , construct the frequency band energy mapping relationship to extract the energy distribution of each frequency band. Frequency band feature correlation matrix yes The symmetric matrix, Indicates the number of frequency bands, the matrix elements Characterization frequency band and frequency band The degree of correlation between the normalized logarithmic spectrum sequence Perform weighted summation operation according to frequency band division to obtain the frequency band energy distribution matrix , and its calculation formula is:

[0111] ;

[0112] in Indicates frequency band In time The energy value, It is the frequency band The set of frequency components included. The energy distribution matrix of the frequency band The Euclidean distance between each time-frequency point and its adjacent time-frequency point is calculated to measure the degree of change in energy distribution. Suppose the frequency band energy of the current point is , the energy of the adjacent points is , then the Euclidean distance between two points The formula is:

[0113] ;

[0114] in, is the Euclidean distance, which is used to characterize the smoothness of the time-frequency point change. Compare and calculate the probability of speech , the formula is:

[0115] ;

[0116] Thus, the time-frequency probability distribution matrix is ​​generated , where 1 indicates a high probability speech area, and 0 indicates a non-speech area. Based on the time-frequency probability distribution matrix, cluster analysis is performed on the speech time-frequency points, and the time-frequency points are divided into the speech main area and the transition area. Cluster analysis uses the connected component method to classify adjacent high-probability points as the speech main area, and other points as the transition area, generating a regional labeling matrix , whose element value marks the region type to which the time-frequency point belongs. The contour of the speech main area in the image is extracted, the main boundaries of the speech are identified, and the optimal speech boundary trajectory is calculated through dynamic programming method. The formula for speech boundary trajectory is:

[0117] ;

[0118] in, It's time The corresponding frequency band index is used to describe the boundary of speech. Based on speech boundary trajectory , perform cubic spline interpolation calculation on the time-frequency points in the transition area to generate the spectrum interpolation data of the transition area The cubic spline interpolation formula is:

[0119] ;

[0120] in are the coefficients of the spline interpolation, calculated from the known boundary trajectory points, Indicates the frequency index. Interpolation data with spectrum in transition region Perform weighted fusion to obtain a complete logarithmic spectrum matrix The fusion formula is:

[0121] ;

[0122] in is the weighting coefficient used to balance the contribution of the two parts of data. Perform an inverse logarithmic transformation to restore the amplitude value and combine it with the original phase data Perform phase reconstruction and finally obtain reconstructed spectrum data , the formula is:

[0123] ;

[0124] in represents the reconstructed phase information, is the inverse logarithmic transform of the amplitude values.

[0125] In a specific embodiment, the audio processing method for performing speech spectrum reconstruction further includes the following steps:

[0126] The power spectrum is calculated and the sliding average is processed for each time-frequency point of the reconstructed spectrum data to obtain a smoothed power spectrum matrix, and the background noise power spectrum of each frequency band is calculated based on the smoothed power spectrum matrix to obtain a noise power spectrum matrix;

[0127] According to the ratio relationship between the smoothed power spectrum matrix and the noise power spectrum matrix, the a priori signal-to-noise ratio and the a posteriori signal-to-noise ratio are calculated for each frequency band to obtain a signal-to-noise ratio parameter matrix, and the signal-to-noise ratio parameter matrix is ​​weighted by the Wiener filter transfer function to obtain an adaptive filter coefficient matrix;

[0128] The adaptive filter coefficient matrix is ​​multiplied by a complex matrix with the reconstructed spectrum data to obtain a filtered speech spectrum, and the filtered speech spectrum is divided into sub-bands, and the energy mean and variance of each sub-band are calculated to obtain a dynamic gain control parameter;

[0129] The dynamic gain control parameter is subjected to nonlinear mapping operation and signal amplitude compensation on the filtered speech spectrum to obtain a gain-compensated speech spectrum, and the gain-compensated speech spectrum is subjected to inverse short-time Fourier transform and windowed overlap addition to obtain an enhanced speech signal;

[0130] The Mel-frequency cepstral coefficient features of the enhanced speech signal are extracted, and the Mel-frequency cepstral coefficient features are input into the deep neural network model for speech recognition to obtain the speech recognition result.

[0131] Specifically, to reconstruct the spectrum data The power spectrum is calculated at each time-frequency point. The expression is:

[0132] ;

[0133] in is the frequency and time The complex spectrum value at contains amplitude and phase information. is its amplitude, It is the power value of the corresponding time-frequency point, reflecting the energy distribution of the signal. Perform sliding average processing to smooth out short-term fluctuations and extract stable characteristics. The calculation formula for sliding average is:

[0134] ;

[0135] in is the smoothed power spectrum matrix, is the width of the sliding window, which controls the degree of smoothing. Represents the index of the adjacent frame on the time axis. Sliding average processing can eliminate high-frequency noise in the power spectrum and obtain a smoother energy distribution. Based on the smoothed power spectrum matrix , the background noise power spectrum of each frequency band is estimated by statistical methods. The minimum tracking method or recursive averaging method is used, where the background noise power spectrum The estimation formula is:

[0136] ;

[0137] in is the frequency The noise power value on the time axis is used to estimate the stable level of the background noise. According to the smoothed power spectrum matrix and background noise power spectrum The ratio of , calculate the prior signal-to-noise ratio and the posterior signal-to-noise ratio The calculation formula of the posterior signal-to-noise ratio is:

[0138] ;

[0139] in Indicates that the signal is at frequency and time The a priori signal-to-noise ratio is estimated by a recursive formula:

[0140] ;

[0141] in is the a priori signal-to-noise ratio, Is a smoothing factor that controls the update speed of the priori signal-to-noise ratio. Using the priori signal-to-noise ratio and the posterior signal-to-noise ratio, the adaptive filter coefficient matrix is ​​calculated through the transfer function of the Wiener filter. The transfer function formula of the Wiener filter is:

[0142] ;

[0143] in is the gain of the Wiener filter, which represents the frequency and time The degree of enhancement. The adaptive filter coefficient matrix and reconstructed spectrum data Perform complex matrix multiplication to obtain the filtered speech spectrum , the formula is:

[0144] ;

[0145] The filtered speech spectrum retains the speech components with high signal-to-noise ratio and suppresses the noise components with low signal-to-noise ratio. According to the sub-band division, the energy mean and variance of each sub-band are calculated. The frequency range of the sub-band is , then its energy mean and variance The formula is:

[0146] ;

[0147] in is the subband energy mean, is the variance of the subband energy, It is a subband The number of frequency points in . Using the energy mean and variance, the dynamic gain control parameters are calculated , the formula is:

[0148] ;

[0149] in is a small positive value used to avoid the denominator being zero. Applied to the filtered speech spectrum, nonlinear mapping operation and signal amplitude compensation are performed to obtain the speech spectrum after gain compensation. The formula for nonlinear mapping is:

[0150] ;

[0151] Speech spectrum after gain compensation Perform inverse short-time Fourier transform to restore to the time domain, and perform windowing overlap-addition operation to obtain enhanced speech signal By using the enhanced speech signal, the Mel-frequency cepstral coefficient features are extracted and input into the deep neural network model for speech recognition, and finally the speech recognition result is obtained.

[0152] This embodiment also includes: it also includes an adaptive speech enhancement step based on vehicle speed information: performing pitch tracking analysis on the enhanced speech signal, calculating the speech fundamental frequency trajectory, and performing harmonic decomposition on the signal based on the speech fundamental frequency trajectory to obtain a set of harmonic components; adaptively adjusting the bandwidth of the harmonic component set based on the vehicle speed, filtering each harmonic component through a speed-related bandpass filter group to obtain a speed-adaptive harmonic matrix; performing envelope detection on the speed-adaptive harmonic matrix, and establishing an envelope modulation function in combination with the vehicle speed information to obtain a dynamic modulation coefficient; constructing an adaptive gain control matrix based on the dynamic modulation coefficient, and adjusting the speed-adaptive harmonic matrix based on the dynamic modulation coefficient. The adaptive harmonic matrix is ​​used to perform nonlinear amplitude compensation to obtain an enhanced harmonic signal; a multi-resolution analysis method based on vehicle speed is used to adaptively adjust the time-frequency resolution of the enhanced harmonic signal to obtain a resampled speech signal; a noise prediction model is constructed based on vehicle speed and road condition information, and the resampled speech signal is pre-compensated to obtain a preprocessed speech signal; the preprocessed speech signal is scored by a speech quality evaluation function, and the gain control parameters are dynamically adjusted according to the evaluation results to obtain the optimized control parameters; the optimized control parameters are fed back to the speech enhancement module, and the speech signal is re-enhanced to obtain the final enhanced speech signal.

[0153] In a specific embodiment, the execution step extracts the Mel-frequency cepstral coefficient features of the enhanced speech signal, and inputs the Mel-frequency cepstral coefficient features into a deep neural network model for speech recognition. The process of obtaining the speech recognition result may specifically include the following steps:

[0154] Perform 512-point fast Fourier transform on the enhanced speech signal, and calculate the energy of each frequency band based on 26 triangular Mel filter banks to obtain the Mel band energy spectrum;

[0155] Perform logarithmic operation and discrete cosine transform on the Mel-band energy spectrum, extract the first 13 coefficients as static features, and calculate the first-order difference and second-order difference features to obtain the Mel-frequency cepstrum coefficient features;

[0156] The Mel frequency cepstral coefficient features are input into the bidirectional long short-term memory layer of the deep neural network model. The bidirectional long short-term memory layer contains 256 memory units, and the speech features are modeled in time series to obtain context encoding features.

[0157] The context encoding features are input into the self-attention layer of the deep neural network model. The self-attention layer contains 8 attention heads, each with a dimension of 32. The weighted context features are obtained through dot product attention calculation.

[0158] The weighted context features are input into the 3-layer feedforward neural network of the deep neural network model for processing. Each layer of the feedforward neural network contains 512 neurons and a ReLU activation function, and residual connections are added between layers to obtain high-level semantic features.

[0159] The high-level semantic features are sequence-labeled, and the state transition probability is calculated based on the dependency relationship between the previous and next frames to obtain the phoneme recognition sequence. The phoneme recognition sequence is decoded through the language model and constrained based on the dictionary and grammar rules to obtain the speech recognition result.

[0160] Specifically, to enhance the speech signal Perform short-time frame processing, each frame contains 512 sampling points. For each frame signal, apply fast Fourier transform to convert the time domain signal to the frequency domain to obtain the spectrum representation The calculation formula is:

[0161] ;

[0162] in is the frequency index, ranging from (corresponding to 256 positive frequency components), It is The time domain amplitude of the sampling points, is the frequency The complex spectrum value of . The amplitude spectrum calculated based on the fast Fourier transform , the energy distribution of each frequency band is calculated through 26 triangular Mel filter banks to generate the Mel band energy spectrum The Mel filter is divided according to the Mel frequency scale, and the calculation formula of the Mel frequency is:

[0163] ;

[0164] in is the linear frequency, is the corresponding Mel frequency. The frequency range of the filter is linearly and Decision. The energy calculation formula of a filter is:

[0165] ;

[0166] in is a Mel filter The energy value, It is The weight function of the filter is and is the frequency range of the filter. Logarithmic operations are performed to compress the dynamic range to make it more consistent with the perception characteristics of the human ear. The formula for the logarithmic energy spectrum is:

[0167] ;

[0168] in is the logarithmic energy value, Is a small constant that prevents zero from appearing in logarithmic operations. Apply discrete cosine transform to extract low-frequency components as speech features and generate Mel-frequency cepstral coefficients (MFCC). The formula for discrete cosine transform is:

[0169] ;

[0170] Among them, MFCC It is Cepstral coefficients, usually the first 13 coefficients are extracted as static features , these low-order coefficients contain the main speech information. In order to enhance the dynamic characteristics, the first-order difference and second-order difference features are calculated for the static features. The calculation formula for the first-order difference is:

[0171] ;

[0172] in It is The first-order difference features, is the length of the calculation window. The formula for the second-order difference feature is similar, using the first-order difference feature instead of the static feature for calculation. The complete MFCC feature sequence containing static features, first-order differences, and second-order differences is input into the bidirectional long short-term memory layer (BiLSTM) of the deep neural network model. The BiLSTM contains 256 memory cells, which can capture long-term contextual information from bidirectional modeling. The state update formula of the memory cell is:

[0173] ;

[0174] in It's time The hidden state of is the current input feature, is the hidden state of the previous moment. Bidirectional processing considers the information of past and future frames at the same time to generate contextual encoding features . Encode the context features Input to the self-attention layer, which contains 8 attention heads, each with a dimension of 32. The attention mechanism calculates the weight through dot product, and the formula is:

[0175] ;

[0176] in are query, key, and value matrices respectively, is the dimension of the key vector. The weighted context features are obtained through the attention mechanism to capture the important relationships in the sequence. The weighted context features are input into the 3-layer feedforward network of the deep neural network. Each layer contains 512 neurons and ReLU activation function. The residual connection of the feedforward network improves the training stability and model expression ability. The output of the feedforward network is high-level semantic features. The high-level semantic features are sequence-labeled, and the state transition probability is calculated through the dependency relationship between the previous and next frames to generate the phoneme recognition sequence. The phoneme sequence is decoded based on the language model, and the dictionary and grammar rules are used to constrain it to generate the speech recognition result.

[0177] This embodiment also includes adaptive learning of noise features and dynamic correction of speech features for speech recognition results, specifically including: performing subspace analysis on a low-rank matrix representing noise components, extracting the main direction of noise features using principal component analysis, and obtaining a noise feature subspace; arranging the noise feature subspace in descending order according to energy contribution rate, selecting feature vectors with cumulative contribution rates exceeding 95%, and obtaining a key noise feature base; constructing an online noise dictionary based on the key noise feature base, performing projection analysis on a newly input speech signal, and obtaining a noise pattern mapping matrix; sparsely encoding the noise pattern mapping matrix, and calculating the sparse representation coefficient using an orthogonal matching pursuit algorithm to obtain a noise feature code; performing time-frequency alignment on the noise feature code and the speech recognition result, establishing a noise-speech feature mapping relationship, and obtaining a feature correction template; adaptively adjusting the predicted speech feature data based on the feature correction template, updating the feature value by a weighted average method, and obtaining a corrected speech feature; re-performing spectrum reconstruction and speech recognition processing on the corrected speech feature to obtain an optimized speech recognition result; feeding the optimized speech recognition result back to the online noise dictionary, updating the noise feature base and the feature correction template, and forming an adaptive optimization cycle.

[0178] The above describes the audio processing method for speech spectrum reconstruction in the embodiment of the present invention. The following describes the audio processing system for speech spectrum reconstruction in the embodiment of the present invention. Figure 2 , an embodiment of the audio processing system for speech spectrum reconstruction in the embodiment of the present invention includes:

[0179] A preprocessing module is used to preprocess the original speech signal collected by the vehicle microphone to obtain a speech signal time-frequency matrix;

[0180] The singular value decomposition module is used to perform singular value decomposition on the time-frequency matrix of the speech signal to obtain a sparse matrix representing the speech component and a low-rank matrix representing the noise component, and to establish a frequency band feature correlation matrix by calculating the mutual information entropy.

[0181] A prediction module, used for inputting the sparse matrix into a selective forgetting extreme learning machine to predict speech features and obtain predicted speech feature data;

[0182] The calculation module is used to calculate the speech probability and feature interpolation of each time-frequency point based on the predicted speech feature data and the frequency band feature association matrix using the logarithmic spectrum amplitude calculation method to obtain the reconstructed spectrum data.

[0183] Through the coordinated cooperation of the above-mentioned components, the speech features are predicted by introducing a selective forgetting extreme learning machine. Combining the selection mechanism and the forgetting mechanism, the temporal dependencies of the speech signal can be effectively captured, thereby improving the prediction accuracy of the speech features. A matrix decomposition method based on singular value decomposition is adopted to decompose the speech signal into a sparse matrix and a low-rank matrix, and a frequency band feature correlation matrix is ​​established through mutual information entropy calculation, thereby achieving effective separation of speech components and noise components. Combined with logarithmic spectrum amplitude calculation and feature interpolation technology, the time-frequency points are accurately reconstructed, effectively maintaining the continuity and naturalness of the speech signal. An adaptive Wiener filter and dynamic gain control strategy are designed, and the filter parameters and gain coefficients are dynamically adjusted according to the signal-to-noise ratio, which significantly improves the speech enhancement effect. A deep neural network model is used for speech recognition, and the long-term and short-term dependencies of speech features are effectively extracted through a bidirectional long short-term memory network and a self-attention mechanism, thereby improving the accuracy of speech recognition and meeting the requirements of real-time processing in a vehicle environment.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0186] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio processing method for speech spectrum reconstruction, characterized in that: The method comprises: Preprocess the original speech signal collected by the vehicle microphone to obtain the speech signal time-frequency matrix; The speech signal time-frequency matrix is ​​subjected to singular value decomposition to obtain a sparse matrix representing the speech component and a low-rank matrix representing the noise component, and a frequency band feature correlation matrix is ​​established by a mutual information entropy calculation method; specifically, the method comprises: performing a USV singular value decomposition operation on the speech signal time-frequency matrix to obtain a decomposition matrix group containing mixed features of speech and noise; sorting the singular values ​​in the decomposition matrix group according to the amplitude, and performing separation processing on the speech component and the noise component based on a preset threshold to obtain a speech singular value sequence and a noise singular value sequence; constructing a diagonal matrix based on the speech singular value sequence, and performing a three-matrix multiplication operation on the diagonal matrix with the U matrix and the V matrix in the decomposition matrix group to obtain a sequence representing A sparse matrix of speech components; constructing a diagonal matrix based on the noise singular value sequence, and performing a three-matrix multiplication operation on the diagonal matrix and the U matrix and the V matrix in the decomposition matrix group to obtain a low-rank matrix representing the noise component; sub-dividing the sparse matrix into sub-bands according to the Mel frequency scale to obtain a sub-matrix sequence of K frequency bands, and pairing the sub-matrix sequences of the K frequency bands in pairs, and calculating the mutual information entropy value for each pair of sub-matrices to obtain a mutual information entropy matrix; performing a symmetry test and normalization processing on the mutual information entropy matrix to obtain frequency band correlation data, and constructing a frequency band feature correlation matrix based on the frequency band correlation data, wherein the matrix element values ​​of the frequency band feature correlation matrix represent the degree of correlation between different frequency bands; Inputting the sparse matrix into a selective forgetting extreme learning machine to perform speech feature prediction to obtain predicted speech feature data; Based on the predicted speech feature data and the frequency band feature association matrix, a logarithmic spectrum amplitude calculation method is used to perform speech probability calculation and feature interpolation on each time-frequency point to obtain reconstructed spectrum data.

2. The audio processing method for speech spectrum reconstruction according to claim 1, characterized in that: The preprocessing of the original speech signal collected by the vehicle-mounted microphone to obtain the speech signal time-frequency matrix includes: Performing analog-to-digital conversion on the original voice signal collected by the vehicle-mounted microphone to obtain voice signal data, and performing DC component elimination on the voice signal data to obtain a zero-mean voice signal; The zero-mean speech signal is segmented to obtain overlapping speech signal segments, and the overlapping speech signal segments are input into a preset Hamming window function for weighted calculation to obtain speech signal data after windowing; Performing a fast Fourier transform on the speech signal data after the windowing process to obtain a complex spectrum containing amplitude information and phase information; Performing time-frequency analysis on the complex spectrum by short-time Fourier transform to obtain a time-varying amplitude spectrum and a time-varying phase spectrum of the speech signal, and performing complex domain synthesis on the time-varying amplitude spectrum and the time-varying phase spectrum of the speech signal to obtain time-frequency feature data of the speech signal; A matrix structure of N rows and M columns is constructed according to the time-frequency feature data of the speech signal, and data is filled in according to the time sequence relationship to obtain a speech signal time-frequency matrix, wherein N represents the frequency dimension and M represents the time dimension.

3. The audio processing method for speech spectrum reconstruction according to claim 1, characterized in that: The step of inputting the sparse matrix into a selective forgetting extreme learning machine for speech feature prediction to obtain predicted speech feature data comprises: Segmenting the sparse matrix according to the time axis, taking each segment of speech features as a training sample, and obtaining a speech feature training sequence; Based on the speech feature training sequence, randomly generate and select the connection weights between the input layer and the hidden layer of the forgetting extreme learning machine to obtain an input weight matrix; The input weight matrix is ​​multiplied by the speech feature training sequence and a bias term is superimposed, and an activation operation is performed through a hyperbolic tangent function to obtain a hidden layer output feature; Performing Moore-Penrose generalized inverse matrix calculation on the hidden layer output features to obtain an analytical weight matrix from the hidden layer to the output layer; The hidden layer output features are screened based on a selection mechanism, and the similarity between each feature and the central feature is calculated by Euclidean distance to obtain a speech feature subset; Based on the forgetting mechanism, the time decay coefficient of the features at each moment in the speech feature subset is calculated, and the historical features are forgotten to obtain updated speech features; The updated speech feature is multiplied by the analytical weight matrix to obtain the speech feature prediction value at the next moment, and the predicted speech feature prediction values ​​are sequentially spliced ​​to obtain predicted speech feature data.

4. The audio processing method for speech spectrum reconstruction according to claim 3, characterized in that: The method of performing speech probability calculation and feature interpolation on each time-frequency point based on the predicted speech feature data and the frequency band feature association matrix using a logarithmic spectrum amplitude calculation method to obtain reconstructed spectrum data includes: Performing logarithmic amplitude calculation on the predicted speech feature data, and performing amplitude normalization processing based on a frequency response function to obtain a standardized logarithmic spectrum sequence; Constructing a frequency band energy mapping relationship based on the frequency band characteristic association matrix, performing a weighted sum operation on the standardized logarithmic spectrum sequence according to the frequency band to obtain a frequency band energy distribution matrix; Calculating the Euclidean distance between each time-frequency point and an adjacent time-frequency point in the frequency band energy distribution matrix, and performing speech probability calculation based on a distance threshold to obtain a time-frequency probability distribution matrix; Performing cluster analysis on the speech time-frequency points according to the time-frequency probability distribution matrix, dividing the time-frequency points into a speech main area and a transition area, and obtaining a regional labeling matrix; Performing contour extraction and optimal speech boundary calculation on the speech main body area in the area marking matrix to obtain a speech boundary trajectory, and performing cubic spline interpolation calculation on the time-frequency points of the transition area based on the speech boundary trajectory to obtain spectrum interpolation data of the transition area; The spectrum data of the main speech area and the spectrum interpolation data of the transition area are weightedly fused to obtain a logarithmic spectrum matrix, and the logarithmic spectrum matrix is ​​inversely logarithmically transformed and phase reconstructed to obtain reconstructed spectrum data.

5. The audio processing method for speech spectrum reconstruction according to claim 4, characterized in that: The audio processing method for speech spectrum reconstruction also includes: Performing power spectrum calculation and sliding average processing on each time-frequency point of the reconstructed spectrum data to obtain a smoothed power spectrum matrix, and calculating the background noise power spectrum of each frequency band based on the smoothed power spectrum matrix to obtain a noise power spectrum matrix; According to the ratio relationship between the smoothed power spectrum matrix and the noise power spectrum matrix, a priori signal-to-noise ratio and a posteriori signal-to-noise ratio are calculated for each frequency band to obtain a signal-to-noise ratio parameter matrix, and the signal-to-noise ratio parameter matrix is ​​weightedly calculated by a Wiener filter transfer function to obtain an adaptive filter coefficient matrix; Performing a complex matrix multiplication operation on the adaptive filter coefficient matrix and the reconstructed spectrum data to obtain a filtered speech spectrum, and dividing the filtered speech spectrum into sub-bands, calculating the energy mean and variance of each sub-band, and obtaining a dynamic gain control parameter; The dynamic gain control parameter is subjected to nonlinear mapping operation and signal amplitude compensation on the filtered speech spectrum to obtain a gain-compensated speech spectrum, and the gain-compensated speech spectrum is subjected to inverse short-time Fourier transform and windowed overlap-addition to obtain an enhanced speech signal; The Mel-frequency cepstral coefficient features of the enhanced speech signal are extracted, and the Mel-frequency cepstral coefficient features are input into a deep neural network model for speech recognition to obtain a speech recognition result.

6. The audio processing method for speech spectrum reconstruction according to claim 5, characterized in that: The step of extracting the Mel-frequency cepstral coefficient features of the enhanced speech signal and inputting the Mel-frequency cepstral coefficient features into a deep neural network model for speech recognition to obtain a speech recognition result includes: Performing a 512-point fast Fourier transform on the enhanced speech signal, and calculating the energy of each frequency band based on 26 triangular Mel filter banks to obtain a Mel band energy spectrum; Performing logarithmic operation and discrete cosine transform on the Mel-frequency band energy spectrum, extracting the first 13 coefficients as static features, and calculating first-order difference and second-order difference features to obtain Mel-frequency cepstrum coefficient features; Inputting the Mel frequency cepstral coefficient feature into a bidirectional long short-term memory layer of a deep neural network model, wherein the bidirectional long short-term memory layer includes 256 memory units, performing time series modeling on the speech feature, and obtaining a context coding feature; Input the context encoding feature into the self-attention layer of the deep neural network model, wherein the self-attention layer includes 8 attention heads, each of which has a dimension of 32, and obtains the weighted context feature through dot product attention calculation; Inputting the weighted context features into a 3-layer feedforward neural network of the deep neural network model for processing, each layer of the feedforward neural network includes 512 neurons and a ReLU activation function, and adding residual connections between layers to obtain high-level semantic features; The high-level semantic features are sequence-labeled, the state transition probability is calculated based on the dependency relationship between the previous and next frames, a phoneme recognition sequence is obtained, and the phoneme recognition sequence is decoded through a language model, constrained based on a dictionary and grammar rules, to obtain a speech recognition result.

7. An audio processing system for speech spectrum reconstruction, characterized in that: An audio processing method for performing speech spectrum reconstruction according to any one of claims 1 to 6, wherein the audio processing system for speech spectrum reconstruction comprises: A preprocessing module is used to preprocess the original speech signal collected by the vehicle microphone to obtain a speech signal time-frequency matrix; A singular value decomposition module is used to perform singular value decomposition on the time-frequency matrix of the speech signal to obtain a sparse matrix representing the speech component and a low-rank matrix representing the noise component, and to establish a frequency band feature correlation matrix by a mutual information entropy calculation method; A prediction module, used for inputting the sparse matrix into a selective forgetting extreme learning machine to perform speech feature prediction to obtain predicted speech feature data; The calculation module is used to perform speech probability calculation and feature interpolation on each time-frequency point based on the predicted speech feature data and the frequency band feature association matrix by using a logarithmic spectrum amplitude calculation method to obtain reconstructed spectrum data.

Images