Audio data enhancement method, device and storage medium

Abstract

The application discloses an audio data enhancement method and device and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: performing Fourier transform on an audio signal based on different window lengths to obtain frequency domain features, and encoding the frequency domain features to obtain an encoded feature sequence; determining an attention score matrix of the encoded feature sequence through an attention mechanism, multiplying a preset lower triangular matrix with the attention score matrix to obtain attention weights, weighting the encoded feature sequence with the attention weights, and obtaining a fusion feature sequence; determining a first adaptive weight according to a sampling rate of the audio signal, determining a second adaptive weight according to the encoded feature sequence corresponding to each window length, weighting the fusion feature sequence based on the first adaptive weight and the second adaptive weight, and obtaining an adaptive feature sequence; inputting the adaptive feature sequence into a one-way LSTM layer, mapping the hidden state output by the LSTM layer to a target dimension through linear transformation, obtaining an enhanced audio feature sequence, and realizing adaptive sampling rate.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to audio data enhancement methods, devices and storage media. Background Technology

[0002] In scenarios such as video conferencing systems and industrial monitoring, audio signals typically originate from various devices, including mobile phones, computers, and professional recording equipment, with significant differences in their sampling rates. Traditional recurrent neural networks and related time-series signal processing models have network parameters strongly tied to the sampling rate used during training. When the input signal sampling rate differs from the training sampling rate, the model's temporal modeling capability is compromised, leading to a substantial decrease in signal processing quality. For example, a speech denoising model trained at a 16kHz sampling rate may misidentify noise types when processing 8kHz signals due to a lack of high-frequency information, and it may fail to effectively suppress newly added high-frequency noise when processing 48kHz signals.

[0003] To address these issues, related technologies employ either resampling the input signal to the training sampling rate, processing it, and then restoring it to the original sampling rate, or training a separate model for each sampling rate, requiring reloading the model's multi-network training when switching sampling rates. However, these resampling and multi-network training schemes either sacrifice latency and fidelity due to resampling or sacrifice storage and compatibility due to multiple models, making it impossible to achieve multi-sampling rate signal processing with a single model.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide an audio data enhancement method, device, and storage medium, which aims to solve the technical problem of how to achieve real-time processing of multi-sample-rate time-series signals by a single model without resampling operations or relying on multi-model deployment.

[0006] To achieve the above objectives, this application proposes an audio data enhancement method, which includes:

[0007] After receiving an audio signal, performing a Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encoding the frequency domain features to obtain an encoded feature sequence;

[0008] The attention score matrix of the encoded feature sequence is determined by an attention mechanism. The attention weight is obtained by multiplying the attention score matrix by a preset lower triangular matrix. The encoded feature sequence is then weighted based on the attention weight to obtain the fused feature sequence.

[0009] A first adaptive weight is determined based on the sampling rate of the audio signal, a second adaptive weight is determined based on the encoded feature sequence corresponding to each window length, and the fused feature sequence is weighted based on the first adaptive weight and the second adaptive weight to obtain an adaptive feature sequence.

[0010] The adaptive feature sequence is input into a unidirectional LSTM layer, and the hidden state output by the LSTM layer is linearly transformed and mapped to the target dimension to obtain the enhanced audio feature sequence.

[0011] In one embodiment, the step of determining the attention score matrix of the encoded feature sequence through an attention mechanism, multiplying a preset lower triangular matrix by the attention score matrix to obtain attention weights, and weighting the encoded feature sequence based on the attention weights to obtain a fused feature sequence includes:

[0012] Determine the Q matrix, V matrix, and K matrix of the encoded feature sequence;

[0013] The Q matrix, the V matrix, and the K matrix are split into different attention heads;

[0014] The attention score matrix corresponding to each attention head is determined by the dot product of the Q matrix and the K matrix corresponding to each attention head.

[0015] The attention weight corresponding to each attention head is obtained by multiplying the preset lower triangular matrix element by element with the attention score matrix;

[0016] The attention weights and V matrix corresponding to each attention head are weighted and summed to obtain the fused features corresponding to each attention head.

[0017] The fused features corresponding to each attention head are concatenated to obtain the fused feature sequence.

[0018] In one embodiment, the steps of determining a first adaptive weight based on the sampling rate of the audio signal, determining a second adaptive weight based on the encoded feature sequence corresponding to each window length, and weighting the fused feature sequence based on the first adaptive weight and the second adaptive weight to obtain an adaptive feature sequence include:

[0019] Each sampling rate of the audio signal is converted into a feature vector of the target dimension, and the feature vector is passed through a first fully connected network to generate the first adaptive weight;

[0020] The first adaptive weight is multiplied element by element by the fused feature sequence to obtain the sampling rate adaptive feature sequence;

[0021] The encoded feature sequence corresponding to the shortest window length among all the window lengths is input into the second fully connected network, and the second adaptive weights corresponding to each window length are determined by the second fully connected network.

[0022] The second adaptive weight corresponding to each window length is weighted and summed with the encoded feature sequence corresponding to each window length to obtain the window length adaptive feature sequence.

[0023] The adaptive feature sequence is obtained by concatenating the sampling rate adaptive feature sequence and the window length adaptive feature sequence.

[0024] In one embodiment, the first fully connected network includes a first connection layer and a second connection layer. The step of converting each sampling rate of the audio signal into a feature vector of the target dimension, and generating the first adaptive weights from the feature vector via the first fully connected network, includes:

[0025] Convert each sampling rate of the audio signal into a feature vector of the target dimension;

[0026] The feature vector of the target dimension is input into the first connection layer, and the feature vector is reduced from the target dimension to the first dimension by the ReLU activation function of the first connection layer.

[0027] The feature vector of the first dimension is input into the second connection layer. After the feature vector is upgraded from the first dimension to the target dimension by the second connection layer, the first adaptive weight of the feature vector is determined by the sigmoid activation function of the second connection layer.

[0028] In one embodiment, the step of concatenating the sampling rate adaptive feature sequence and the window length adaptive feature sequence to obtain the adaptive feature sequence includes:

[0029] The sampling rate adaptive feature sequence and the window length adaptive feature sequence are concatenated along the feature dimension to obtain the concatenated feature sequence;

[0030] The concatenated feature sequence is input into a fully connected layer, which compresses the concatenated feature sequence to the target dimension to obtain the adaptive feature sequence.

[0031] In one embodiment, the step of performing Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encoding the frequency domain features to obtain an encoded feature sequence includes:

[0032] After obtaining frequency domain features by performing Fourier transform on the audio signal based on different window lengths, the feature representation of the frequency domain features is extracted by a convolutional neural network.

[0033] The feature representation extracted from each layer of the convolutional neural network is input into the pooling layer so that the feature representation is subjected to global average pooling in the frequency dimension.

[0034] The pooled feature representation is input into a fully connected layer, which compresses the pooled feature representation to the target dimension to obtain the encoded feature sequence.

[0035] In one embodiment, after the step of inputting the adaptive feature sequence into a unidirectional LSTM layer and mapping the hidden state output by the LSTM layer to the target dimension through a linear transformation to obtain the enhanced audio feature sequence, the method further includes:

[0036] The enhanced audio feature sequence is input into a pre-trained noise reduction network, which then identifies noise in the audio feature sequence.

[0037] Obtain the noise suppression mask generated by the noise reduction network based on the noise; or

[0038] Obtain the frequency domain features of the audio feature sequence after noise reduction based on the noise output by the noise reduction network.

[0039] In one embodiment, the denoised frequency domain features include the denoised amplitude spectrum. After the step of obtaining the frequency domain features obtained by denoising the audio feature sequence based on the noise from the output of the denoising network, the method further includes:

[0040] The amplitude spectrum after noise reduction is spliced ​​with the phase spectrum of the audio signal to form the complex spectrum after noise reduction;

[0041] The inverse Fourier transform is performed on the denoised complex spectrum to restore the frequency domain features to the time domain signal, thus obtaining the denoised time domain speech signal.

[0042] In addition, to achieve the above objectives, this application also proposes an audio data enhancement device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio data enhancement method as described above.

[0043] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the audio data enhancement method described above.

[0044] This application provides an audio data enhancement method. The method receives an audio signal, performs a Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encodes the frequency domain features to obtain an encoded feature sequence. An attention score matrix is ​​determined using an attention mechanism, and a preset lower triangular matrix is ​​multiplied by the attention score matrix to obtain attention weights. The encoded feature sequence is then weighted based on these attention weights to obtain a fused feature sequence. A first adaptive weight is determined based on the audio signal's sampling rate, and a second adaptive weight is determined based on the encoded feature sequence corresponding to each window length. The fused feature sequence is then weighted based on the first and second adaptive weights to obtain an adaptive feature sequence. The adaptive feature sequence is input into a unidirectional LSTM layer, and the hidden state output by the LSTM layer is linearly mapped to the target dimension to obtain the enhanced audio feature sequence.

[0045] The above method first generates a first adaptive weight based on the actual sampling rate of the input signal, dynamically adjusting the model's attention to features in different frequency bands to explicitly correct for frequency band differences caused by different sampling rates. Simultaneously, it generates a second adaptive weight based on the encoded feature sequences corresponding to multi-scale window lengths, dynamically optimizing the fusion strategy for features at different temporal resolutions. After the above frequency domain transformation, content-aware fusion, and dual adaptive correction, a causal LSTM is used to learn the general temporal patterns in the feature space. Since the features input to the LSTM have already been normalized by the sampling rate, the LSTM learns the temporal evolution of clean speech and noise in the feature space, which is largely independent of the original sampling rate. This ultimately enables a single model to dynamically adapt to input signals with different sampling rates without relying on resampling preprocessing or multi-model switching. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating an embodiment of the audio data enhancement method of this application.

[0049] Figure 2 This is a flowchart illustrating Embodiment 2 of the audio data enhancement method of this application;

[0050] Figure 3This is a flowchart illustrating Embodiment 3 of the audio data enhancement method of this application;

[0051] Figure 4 This is a simplified flowchart illustrating the audio data enhancement method in the embodiments of this application;

[0052] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the audio data enhancement method in the embodiments of this application.

[0053] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0055] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. It should be noted that all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located, and with authorization from the owner of the corresponding device.

[0056] In scenarios such as video conferencing systems and industrial monitoring, audio signals typically originate from various devices, including mobile phones, computers, and professional recording equipment, with significant differences in their sampling rates. Traditional recurrent neural networks and related time-series signal processing models have network parameters strongly tied to the sampling rate used during training. When the input signal sampling rate differs from the training sampling rate, the model's temporal modeling capability is compromised, leading to a substantial decrease in signal processing quality. For example, a speech denoising model trained at a 16kHz sampling rate may misidentify noise types when processing 8kHz signals due to a lack of high-frequency information, and it may fail to effectively suppress newly added high-frequency noise when processing 48kHz signals.

[0057] To address these issues, related technologies employ either resampling the input signal to the training sampling rate, processing it, and then restoring it to the original sampling rate, or training a separate model for each sampling rate, requiring reloading the model's multi-network training when switching sampling rates. However, these resampling and multi-network training schemes either sacrifice latency and fidelity due to resampling or sacrifice storage and compatibility due to multiple models, making it impossible to achieve multi-sampling rate signal processing with a single model.

[0058] In view of the above problems, this application proposes an audio data augmentation method. This method performs Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encodes these frequency domain features to obtain an encoded feature sequence. An attention score matrix is ​​determined using an attention mechanism, and a preset lower triangular matrix is ​​multiplied by the attention score matrix to obtain attention weights. The encoded feature sequence is then weighted based on these attention weights to obtain a fused feature sequence. A first adaptive weight is determined based on the audio signal's sampling rate, and a second adaptive weight is determined based on the encoded feature sequences corresponding to each window length. The fused feature sequence is then weighted based on the first and second adaptive weights to obtain an adaptive feature sequence. The adaptive feature sequence is input into a unidirectional LSTM layer, and the hidden states output by the LSTM layer are linearly mapped to the target dimension to obtain the augmented audio feature sequence. The above method first generates a first adaptive weight based on the actual sampling rate of the input signal, dynamically adjusting the model's attention to features in different frequency bands to explicitly correct for differences in frequency band range caused by different sampling rates. Simultaneously, a second adaptive weight is generated based on the encoded feature sequences corresponding to multi-scale window lengths to dynamically optimize the fusion strategy for features at different temporal resolutions. After the aforementioned frequency domain transformation, content-aware fusion, and dual adaptive correction, a causal LSTM is used to learn the general temporal patterns in the feature space. Since the features input to the LSTM have already been normalized by the sampling rate, what the LSTM needs to learn is the temporal evolution of clean speech and noise in the feature space. This pattern is basically independent of the original sampling rate, ultimately enabling a single model to dynamically adapt to input signals with different sampling rates without relying on resampling preprocessing or multi-model switching.

[0059] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer or personal computer, or an electronic device or audio data enhancement model capable of performing the above functions. The following description uses an audio data enhancement model as an example to illustrate this embodiment and the subsequent embodiments.

[0060] The audio data enhancement model includes a multi-scale feature extraction module, a frequency domain feature encoding module, a causal attention fusion module, a dual adaptive module, and a causal recurrent neural network module.

[0061] The system comprises several modules: a multi-scale feature extraction module for receiving audio signals and performing Fourier transforms on the signals based on different window lengths to obtain frequency domain features; a frequency domain feature encoding module for encoding these features to obtain an encoded feature sequence; a causal attention fusion module for determining the attention score matrix of the encoded feature sequence using an attention mechanism, multiplying a preset lower triangular matrix by the attention score matrix to obtain attention weights, and weighting the encoded feature sequence based on these attention weights to obtain a fused feature sequence; a dual adaptive module for determining a first adaptive weight based on the audio signal sampling rate and a second adaptive weight based on the encoded feature sequence corresponding to each window length, and weighting the fused feature sequence based on the first and second adaptive weights to obtain an adaptive feature sequence; and a causal recurrent neural network module for inputting the adaptive feature sequence into a unidirectional LSTM layer, mapping the hidden states output by the LSTM layer to the target dimension via a linear transformation to obtain an enhanced audio feature sequence.

[0062] Based on this, the first embodiment of this application provides an audio data enhancement method, referring to... Figure 1 In this embodiment, the audio data enhancement method includes steps S10 to S40:

[0063] Step S10: Receive an audio signal, perform a Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encode the frequency domain features to obtain an encoded feature sequence.

[0064] Optionally, the multi-scale feature extraction module employs multi-scale short-time Fourier transform, using different window lengths to extract frequency domain features of audio signals at different time scales, thus achieving multi-scale frequency domain information capture. Each window length corresponds to different temporal and frequency resolutions. Temporal resolution = window length / sampling rate, determining the accuracy of time localization; frequency resolution = sampling rate / window length, determining the accuracy of frequency analysis. Different window lengths have different feature capture capabilities. For example, short window lengths have high temporal resolution, suitable for capturing transient features; medium window lengths balance time and frequency resolution, suitable for capturing features at medium time scales; and long window lengths have high frequency resolution, suitable for capturing long-term features.

[0065] In this embodiment, the multi-scale feature extraction module receives the input time-domain audio signal. To ensure sufficient data for analyses of all window lengths, the audio data augmentation model maintains a data buffer. When the amount of data in the data buffer reaches the preset maximum window length requirement, subsequent multi-scale analysis is triggered. The multi-scale feature extraction module performs short-time Fourier transform on the prepared time-domain audio signal data block simultaneously using multiple analysis windows of different lengths, converting the time-domain signal into a time-frequency representation through the short-time Fourier transform.

[0066] For example, for each window length, the short-time Fourier transform can be expressed as: Where w is the window length, The input audio signal, The extracted frequency domain features.

[0067] Optionally, the aforementioned frequency domain features include amplitude spectrum and phase spectrum. That is, from each set of time-frequency representations The amplitude spectrum was extracted from the sample. and phase spectrum .

[0068] Optionally, step S10 includes steps S11 to S13:

[0069] Step S11: After performing Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, the feature representation of the frequency domain features is extracted through a convolutional neural network.

[0070] The frequency domain feature encoding module receives multiple sets of frequency domain features from the multi-scale feature extraction module, mainly the amplitude spectrum and phase spectrum mentioned above. For each window length, the frequency domain feature encoding module combines the amplitude spectrum and phase spectrum along the feature dimension to form a two-dimensional feature map corresponding to that window length. This two-dimensional feature map is in the form of a two-dimensional matrix with dimensions of [number of time frames, number of frequency channels]. The number of frequency channels is the number of frequency bins corresponding to each time frame after the short-time Fourier transform.

[0071] Optionally, the frequency domain feature encoding module includes convolutional layers, pooling layers, and fully connected layers. The frequency domain feature encoding module processes the input frequency domain features through a convolutional neural network containing multiple convolutional layers. Each convolutional layer uses a set of learnable convolutional kernels to perform sliding window computation on the input two-dimensional feature map. Each convolutional kernel extracts a local pattern from the input features, such as energy change patterns in a specific frequency region over a short period. Through multiple stacked convolutional layers, the convolutional neural network can combine low-level features layer by layer to form more complex and abstract high-level feature representations. For example, the first layer might capture basic frequency domain edge information, while deeper layers can capture complex features such as speech formant structures or harmonic patterns of specific noise. After multi-layer convolutional processing, a set of high-level feature maps is obtained. These high-level feature maps retain the temporal frame dimension but are enhanced and expanded in the feature channel dimension, containing richer semantic information.

[0072] Step S12: Input the feature representation extracted from each layer of the convolutional neural network into the pooling layer, so as to perform global average pooling on the feature representation in the frequency dimension through the pooling layer.

[0073] The high-level feature map output from the last layer of the convolutional neural network is input into the pooling layer, with dimensions of [number of time frames, number of feature channels]. The number of feature channels is the output channel number of the last convolutional layer of the convolutional neural network.

[0074] The pooling layer performs global average pooling on each feature channel, calculating the average value across all frequency points corresponding to that feature channel. This operation compresses each feature channel into a single scalar value along the frequency dimension. Its purpose is to aggregate global information across the entire frequency axis, eliminating sensitivity to frequency position and making the feature representation more robust to frequency shifts. After global average pooling, the original two-dimensional feature map—where each time frame corresponds to a multi-channel frequency vector—is compressed into a one-dimensional feature vector. Each element in this one-dimensional feature vector represents the average response intensity of a feature channel across the entire frequency range. The pooling layer output is then a two-dimensional matrix with dimensions [number of time frames, number of feature channels], where each time frame corresponds to a feature vector composed of the global average response values ​​of all feature channels.

[0075] Step S13: Input the pooled feature representation into a fully connected layer, and compress the pooled feature representation to the target dimension through the fully connected layer to obtain the encoded feature sequence.

[0076] The feature vectors of each time frame in the two-dimensional matrix obtained after global average pooling are independently input into the fully connected layer of the frequency domain feature encoding module. The fully connected layer performs a linear transformation on the feature vectors of each time frame through a weight matrix and a bias vector, mapping the feature vectors of each high-dimensional time frame to a preset target dimension, resulting in multiple sets of encoded feature sequences. Each set of encoded feature sequences corresponds to a window length with dimensions of [number of time frames, target dimension]. This encoded feature sequence will serve as the input to the subsequent causal attention fusion module. The target dimension vector at each time frame is a condensed representation of the audio signal at that moment.

[0077] Step S20: Determine the attention score matrix of the encoded feature sequence through an attention mechanism, multiply the preset lower triangular matrix with the attention score matrix to obtain the attention weight, and weight the encoded feature sequence based on the attention weight to obtain the fused feature sequence.

[0078] Understandably, standard attention mechanisms, such as self-attention in the Transformer encoder, are "non-causal," taking into account information from all future time frames when calculating the attention weights for the current time frame. However, this "non-causal" self-attention introduces latency in real-time audio streaming processing because it cannot capture future sounds in real time. Therefore, this embodiment uses a preset lower triangular matrix as a causal mask, where the elements below the diagonal are 1, and the elements above are 0 or negative infinity. When this lower triangular matrix is ​​multiplied by the attention score matrix, it is equivalent to forcibly setting all "current moment focusing on future moment" attention scores to zero or making them extremely small. In this way, when calculating the attention weights using softmax, the attention weights for the current frame can only be assigned to the current frame itself and past historical frames, using only current and past information for reasoning, thus meeting the requirements of real-time streaming processing.

[0079] Optionally, step S20 above includes steps S21 to S26:

[0080] Step S21: Determine the Q matrix, V matrix, and K matrix of the encoded feature sequence.

[0081] Optionally, the causal attention fusion module includes linear transformation layers. After receiving the encoded feature sequence e from the frequency domain feature encoding module, the causal attention fusion module inputs the encoded feature sequence e into three different linear transformation layers, each with an independent weight matrix. , and Through matrix multiplication, the three different linear transformation layers will output three new matrices: Q matrix (query matrix) = e V matrix (value matrix) = e K matrix (bond matrix) = e The dimensions of the Q, V, and K matrices are [time frame number, d_model], where d_model is the feature dimension after linear transformation.

[0082] Step S22: The Q matrix, the V matrix, and the K matrix are split into different attention heads.

[0083] The Q, V, and K matrices are partitioned along the feature dimensions. For example, assuming the number of attention heads h=4 and the linearly transformed feature dimension d_model=128, then each attention head is assigned a dimension dk=d_model / h=32, resulting in three sets of matrices, each containing h matrices: {Q1, Q2, ..., Qh}, {K1, K2, ..., Kh}, {V1, V2, ..., Vh}. The dimension of each Qi, Ki, Vi is [time frame number, d_k].

[0084] Step S23: Determine the attention score matrix corresponding to each attention head based on the dot product of the Q matrix and the K matrix corresponding to each attention head.

[0085] For example, for each attention head i, the matrix product of the Q matrix Qi and the K matrix Ki, calculated from the transposes of Qi and Ki, yields the attention score matrix. The matrix Si has dimensions [time frame number, time frame number], and each element Si[t, s] represents the dot product similarity between Qi[t] in time frame t and Ki[s] in time frame s. Optionally, the attention score matrix Si is divided by... Scaling is applied to prevent the dot product result from becoming too large, which could cause the gradient of the Softmax function to vanish.

[0086] Step S24: Multiply the preset lower triangular matrix element by element with the attention score matrix to obtain the attention weight corresponding to each attention head.

[0087] For example, first generate a preset lower triangular matrix M, where the elements at and below the diagonal are 0, and the elements above the diagonal are 0. ∞. Using the lower triangular matrix M as a causal mask ensures that each time step can only focus on historical information up to and including the current moment, and cannot use future information. The attention score matrix after masking. ,in This indicates element-wise multiplication. Ensure that future information is completely shielded. The attention score matrix obtained after the above calculations. The lower triangle will retain its original attention score, while the upper triangle will completely change. The attention score matrix after masking The attention score matrix is ​​used as the input parameter to the softmax function. The attention weight matrix Ai is obtained by calculating the exponent of each element in each row and normalizing it. Since the attention score matrix... The upper triangular portion, representing the fraction of the future time step, is negative infinity, and its exponent e is... Since ∞ = 0, the attention weights corresponding to these future time steps will be set to 0. That is, the sum of each row of the attention weight matrix Ai is 1, and for the t-th row, only the 1st to t-th columns have weights, while the weights of the (t+1)-th column and beyond are all 0.

[0088] The above steps, through masking operations, ensure that when the audio data augmentation model performs attention-weighted summation, its attention is focused only on the present moment and the past at any given time point. This simulates the causal constraints that humans face when listening to and processing audio streams in real time, enabling the entire audio data augmentation model to perform real-time processing with zero latency without relying on future information.

[0089] Step S25: The attention weights and V matrix corresponding to each attention head are weighted and summed to obtain the fused features corresponding to each attention head.

[0090] For each attention head i, the attention weight matrix Ai and the V matrix Vi are multiplied and summed using matrix multiplication to obtain the fused feature Oi = AiVi corresponding to attention head i. For each current time frame t, Oi[t] is the weighted sum of the value vectors Vi[s] of all historical and current time frames, with the weights determined by Ai[t, s]. The dimension of Oi is [number of time frames, d_k], which is the new feature sequence of attention head i after fusing causal historical information.

[0091] Step S26: Concatenate the fused features corresponding to each attention head to obtain the fused feature sequence.

[0092] The output matrices Oi of all attention heads are concatenated along their feature dimensions to obtain a fused feature sequence F, with dimensions of [time frame number, feature dimension]. For example, if each head outputs 32-dimensional features, concatenating the four heads yields 128-dimensional features. The concatenated result can also be passed through a trainable linear transformation layer to integrate information from different attention heads and potentially map the dimensions back to the desired output dimension.

[0093] Step S30: Determine a first adaptive weight based on the sampling rate of the audio signal, determine a second adaptive weight based on the encoded feature sequence corresponding to each window length, and weight the fused feature sequence based on the first adaptive weight and the second adaptive weight to obtain an adaptive feature sequence.

[0094] Understandably, in real-time speech noise reduction applications such as local sound reinforcement, video conferencing systems, voice call applications, and real-time speech recognition preprocessing, the system needs to process noisy speech signals from different devices and scenarios. Audio signals with different sampling rates have different effective frequency ranges. This means that the same type of noise will appear at different absolute frequency positions in signals with different sampling rates. For example, 8kHz high-frequency noise is at the Nyquist frequency limit in an 8kHz sampling rate signal, and its energy may be very weak; however, in the same recording at a 48kHz sampling rate, it is merely mid-to-low frequency noise with completely different energy characteristics.

[0095] Therefore, in this embodiment, the sampling rate metadata is encoded into a weight vector to scale the fused features and determine the first adaptive weight of the fused feature sequence. If the currently processed audio signal is predominantly low-frequency, the first adaptive weight will cause the model to focus on low-frequency features; if the currently processed audio signal is a full-band signal, the first adaptive weight will cause the model to process all frequencies equally. Through the above steps, the model can focus its attention on the features corresponding to each sampling rate according to the first adaptive weight, thus adapting to different sampling rates, without needing to resample the input signal to a fixed sampling rate for preprocessing.

[0096] Meanwhile, in this embodiment, multi-scale frequency domain features are extracted in step S10, and a multi-scale coded feature sequence is obtained through the frequency domain feature encoding module. Short-window-length features are good at capturing transient changes, but have low frequency resolution; long-window-length features have high frequency resolution and are good at analyzing steady-state components, but have slow time response. This embodiment determines the second adaptive weight based on the coded feature sequence corresponding to each window length, so that the model can intelligently integrate the complementary information provided by different window lengths according to the instantaneous characteristics of the current signal segment. When transient noise appears in the signal, the model will automatically assign higher weights to short-window-length features to achieve a fast response; when the signal is dominated by steady-state noise, the weight of long-window-length features will be increased to achieve more refined noise reduction.

[0097] Optionally, step S30 above includes steps S31 to S35:

[0098] Step S31: Convert each sampling rate of the audio signal into a feature vector of the target dimension, and generate the first adaptive weights from the feature vectors through a first fully connected network.

[0099] Optionally, the dual adaptive module has two parallel branches: a sampling rate adaptive submodule and a window length adaptive submodule. The sampling rate adaptive submodule converts the numerical sampling rate information into a weight vector that adaptively weights the features, and can be implemented using a small neural network. Its first fully connected network includes a first connected layer and a second connected layer.

[0100] During model training, the sampling rate adaptive submodule can set its training objective to minimize the difference between the denoised speech and the real clean speech. It continuously learns the mapping relationship between the feature vectors corresponding to different sampling rates and the frequency attention. The higher the frequency attention, the higher the adaptive weight of the corresponding frequency features.

[0101] Optionally, step S31 includes steps S311 to S313:

[0102] Step S311: Convert each sampling rate of the audio signal into a feature vector of the target dimension.

[0103] It is understandable that the sampling rate is essentially a discrete type of category information, and each sampling rate value can be mapped to a high-dimensional, learnable feature vector through a sampling rate embedding layer. Its dimension is the target dimension.

[0104] Step S312: Input the feature vector of the target dimension into the first connection layer, and reduce the feature vector from the target dimension to the first dimension through the ReLU activation function of the first connection layer.

[0105] The above feature vectors The first connection layer of the input sampling rate adaptive submodule. The weight matrix of this layer maps the 128-dimensional input to a dimension lower than the target dimension, i.e., the first dimension. Then, the ReLU activation function is used to perform a non-linear transformation on the output of this layer. Where h1 is the eigenvector after nonlinear transformation and dimensionality reduction, and w1 is the weight matrix of the first connection layer. This represents the bias matrix of the first connected layer. The ReLU function introduces non-linearity, enhancing the model's expressive power and helping to mitigate the vanishing gradient problem.

[0106] Step S313: Input the feature vector of the first dimension into the second connection layer. After the feature vector is upgraded from the first dimension to the target dimension through the second connection layer, the first adaptive weight of the feature vector is determined by the Sigmoid activation function of the second connection layer.

[0107] h1 is input to the second connected layer. The weight matrix of the second connected layer maps the first dimension back to the original target dimension. Then, the output of the second connected layer is transformed using the sigmoid activation function: . w1 is the first adaptive weight, and w2 is the weight matrix of the second connection layer. This is the bias matrix for the second connection layer.

[0108] Step S32: Multiply the first adaptive weight element-wise with the fused feature sequence to obtain the sampling rate adaptive feature sequence.

[0109] The above first adaptive weight The sample rate adaptive feature sequence Fsr is obtained by element-wise multiplication with the fused feature sequence F from the causal attention fusion module, with dimensions [time frame number, target dimension]. This sample rate adaptive feature sequence has been adjusted according to the sampling rate of the current input signal.

[0110] Step S33: Input the encoded feature sequence corresponding to the shortest window length among all the window lengths into the second fully connected network, and determine the second adaptive weights corresponding to each window length through the second fully connected network.

[0111] For example, the window length adaptive submodule receives a multi-scale encoded feature sequence from the frequency domain feature coding module. The encoded feature sequence corresponding to the shortest window length is selected. As a reference, the shortest window length is chosen because it has the highest temporal resolution, can capture instantaneous changes in the signal most quickly, and is most suitable as a reference for generating adaptive weights. For example, assuming the multi-scale feature extraction module uses window lengths of 512, 1024, 2048, and 4096, the window length adaptive submodule receives four sets of encoded feature sequences from the frequency domain feature coding module, corresponding to window lengths of 512, 1024, 2048, and 4096, respectively. For each time frame t, the corresponding shortest window length (512 points) encoded feature vector e512[t) is taken.

[0112] The encoded feature sequence corresponding to the shortest window length The input is a second fully connected network, which can be a multilayer perceptron (MLP). This second fully connected network learns a weight mapping from the current frame features to four window lengths, and determines the encoded feature sequence corresponding to the shortest window length based on this weight mapping. The corresponding second adaptive weights are assigned to each window length. A Softmax function is applied to the output of the second fully connected network to ensure that the sum of the weights for the four window lengths is 1. Each time frame t corresponds to a second adaptive weight vector. Each weight corresponds to a window length, and l is the total number of window lengths.

[0113] Step S34: The second adaptive weight corresponding to each window length is weighted and summed with the encoded feature sequence corresponding to each window length to obtain the window length adaptive feature sequence.

[0114] For each time frame t, the encoded feature sequences corresponding to each window length are summed according to their weights: , where e1[t], e2[t], ..., el[t] represent the feature vectors of the encoded feature sequences corresponding to each window length at the t-th time frame. Optionally, the encoded feature sequences corresponding to each window length at each time frame t can be concatenated into a long vector, and then mapped back to the target dimension through a linear layer.

[0115] Step S35: Concatenate the sampling rate adaptive feature sequence and the window length adaptive feature sequence to obtain the adaptive feature sequence.

[0116] Optionally, step S35 includes steps S351 to S352:

[0117] Step S351: The sampling rate adaptive feature sequence and the window length adaptive feature sequence are concatenated along the feature dimension to obtain the concatenated feature sequence.

[0118] Step S352: Input the spliced ​​feature sequence into a fully connected layer, and compress the spliced ​​feature sequence to the target dimension through the fully connected layer to obtain the adaptive feature sequence.

[0119] The aforementioned sampling rate adaptive feature sequence and window length adaptive feature sequence are concatenated. Specifically, for each time frame, the feature vectors corresponding to the two target dimensions are concatenated into a single long vector, the dimension of which is twice the target dimension. This concatenated long vector is then mapped back to the target dimension through a fusion network to obtain the adaptive feature sequence. The dimension of the adaptive feature sequence is [number of time frames, target dimension]. This adaptive feature sequence, a highly refined feature representation after dual adaptive adjustments to the sampling rate and window length, will be fed into a subsequent causal recurrent neural network for temporal modeling.

[0120] Step S40: Input the adaptive feature sequence into a unidirectional LSTM layer, and map the hidden state output by the LSTM layer to the target dimension through a linear transformation to obtain the enhanced audio feature sequence.

[0121] Preferably, after receiving the adaptive feature sequence output by the dual adaptive module, the causal recurrent neural network module inputs the adaptive feature sequence frame by frame into two layers of unidirectional LSTM (Long Short-Term Memory) for temporal modeling. The first layer of LSTM processes the following steps: For the first time frame t=1, the first layer of LSTM receives the feature vector F[1], combines it with the initial hidden state h0 and cell state c0, calculates and outputs the hidden state h1 and the updated cell state c1 for the first time step. For the second time frame t=2, the first layer of LSTM receives the feature vector F[2] and the hidden state h1 and cell state c1 of the previous time step, calculates and outputs the hidden state h2 and the updated cell state c2 for the second time step, and so on recursively until all time frames are processed. After processing each time frame, the first layer of LSTM outputs a corresponding hidden state h1t. The first layer of the LSTM uses gating mechanisms such as forget gates, input gates, and output gates to determine which historical information needs to be retained, which new information needs to be added, and which information needs to be output, thereby capturing the long-term dependencies of the signal. This hidden state is a high-dimensional vector that condenses all relevant historical context information from the beginning of the sequence to the current time t.

[0122] The steps of the second-layer LSTM processing include: taking the hidden state h1t output by the first-layer LSTM as input and inputting it into the second-layer LSTM for further processing. For the first time frame t=1, the second-layer LSTM receives the hidden state h1[1] of the first-layer LSTM, combines it with the initial hidden state h20 and the cell state c20, calculates and outputs the hidden state h2[1] of the first time step and the updated cell state c2[1]. For the second time frame t=2, the second-layer LSTM receives the hidden state h1[2] of the first-layer LSTM as well as the hidden state h2[1] and the cell state c2[1] of the previous time step, calculates and outputs the hidden state h2[2] of the second time step and the updated cell state c2[2], and so on recursively until all time frames are processed. After processing each time frame, the second-layer LSTM outputs a corresponding hidden state h2t. The second-layer LSTM also performs temporal modeling through a gating mechanism, which can capture more complex temporal dependencies.

[0123] Output layer processing: The hidden state h2t output by the second LSTM layer is linearly transformed using a trainable weight matrix and bias vector, mapping the hidden state to the target dimension to obtain the enhanced audio feature sequence. The target dimension needs to match the input dimension of subsequent modules such as the noise reduction module. The enhanced audio feature sequence is consistent with the input adaptive feature sequence in the time dimension, but the feature vector of each time frame has incorporated rich temporal context information.

[0124] It should be noted that in other specific embodiments, the number of unidirectional LSTM layers can also be determined according to the specific application.

[0125] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 After step S40, the audio data enhancement method further includes steps S50-S60:

[0126] Step S50: The enhanced audio feature sequence is input into a pre-trained noise reduction network, and the noise in the audio feature sequence is identified by the noise reduction network.

[0127] Noise reduction networks are typically deep neural networks, such as fully connected networks or convolutional networks. They can analyze and identify noise from the input audio feature sequence based on the clean speech and various noise feature patterns they learn during training. In other words, they can determine what proportion of the current signal is the desired speech and what proportion is noise that needs to be suppressed at each time point and each frequency unit.

[0128] For example, suppose the denoising network receives an enhanced audio feature sequence H from the output of a causal recurrent neural network module, with dimensions [T, D], where T is the number of time frames and D is the target dimension. H is passed through a first hidden layer, such as a fully connected layer or a one-dimensional convolutional layer, which performs a nonlinear transformation on the D-dimensional feature vector of each time frame t, mapping it to a new, possibly higher-dimensional feature space. This step aims to combine primary features to form more complex pattern representations. For example, it might combine basic features such as high-frequency energy spikes and periodic harmonic structures into the abstract concept of keyboard tapping. The output of the first hidden layer is then passed through multiple subsequent hidden layers. Each layer performs a similar nonlinear transformation, extracting and combining higher-level, more global feature representations layer by layer, resulting in a deeply abstracted feature sequence Z. For each time frame t, the corresponding abstract feature vector Z[t] is input to a dedicated output layer. The goal of this output layer is to generate an estimate for each frequency unit in the frequency domain features of the original audio signal.

[0129] In this network, the number of neurons in the output layer equals the number of frequency bins corresponding to the Fourier transform. Each neuron's task is to estimate a value for its corresponding frequency bin. The specific meaning of this value depends on the design goal of the denoising network: if the goal is to generate a mask, the output layer uses the Sigmoid activation function, with each neuron outputting a value between 0 and 1. This value represents the probability that the time-frequency unit (t, f) is dominated by speech, also known as the speech presence probability. The closer the value is to 1, the greater the likelihood that the time-frequency unit is clean speech; the closer it is to 0, the greater the likelihood that it is noise. If the goal is to directly estimate the spectrum of clean speech, the output layer uses activation functions such as ReLU or Sigmoid to directly estimate the amplitude or energy value of clean speech in that time-frequency unit.

[0130] Step S60: Obtain the noise suppression mask generated by the noise reduction network based on the noise.

[0131] If the output objective of the denoising network is to generate a noise suppression mask, then for each time-frequency unit (t, f), the mask value M(t, f) = S(t, f) / (S(t, f) + N(t, f)), where S is the energy of the clean speech and N is the energy of the noise, with a value between 0 and 1. In subsequent processing, the noise suppression mask is multiplied point-by-point by the time-frequency representation of the noisy speech, such as the amplitude spectrum. A mask value close to 1 indicates that the point is mainly speech and is preserved; a mask value close to 0 indicates that the point is mainly noise and is suppressed.

[0132] Optionally, step S70 may be included after step S50:

[0133] Step S70: Obtain the frequency domain features output by the noise reduction network after denoising the audio feature sequence based on the noise.

[0134] Noise reduction networks can also directly estimate the denoised frequency domain features instead of outputting a mask, such as directly generating the amplitude spectrum of clean speech or a more complex complex spectrum. If the noise reduction network first generates a noise suppression mask, then the denoised frequency domain features can be obtained by applying the noise suppression mask to the frequency domain features of the original audio signal. For example, the denoised amplitude spectrum might be Md(t,f)=M(t,f)⊙ .

[0135] Based on the second embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S70, the audio data enhancement method further includes steps S80-S90:

[0136] Step S80: The amplitude spectrum after noise reduction is spliced ​​with the phase spectrum of the audio signal to form the complex spectrum after noise reduction.

[0137] Step S90: Perform an inverse Fourier transform on the denoised complex spectrum to restore the frequency domain features to a time domain signal, thereby obtaining a denoised time domain speech signal.

[0138] For example, according to Euler's formula, the magnitude and phase in polar coordinates are transformed back into complex spectrum in Cartesian coordinates. ,in This is the amplitude spectrum after noise reduction. This is the phase spectrum of the original audio signal. Complex spectrum. Dimension and Md(t,f) and Consistent, containing complete frequency domain information of the denoised signal. An inverse short-time Fourier transform (STFT) is performed on the denoised complex spectrum to restore the frequency domain characteristics to the time domain signal. The inverse STFT can be expressed as... For each time frame, the corresponding time-domain segment is calculated using inverse FFT, achieving the conversion from the frequency domain to the time domain. Then, the overlap-addition method is used to concatenate the time-domain segments from different time frames into a complete time-domain speech signal. Where w(t) is the window function and hop is the jump length. Since the multi-scale feature extraction module uses overlapping windows during analysis, overlapping addition is also required during restoration to ensure the concatenated signal is continuous and smooth, without breaks or abrupt changes. Noise-reduced time-domain speech signal It can be played directly or used for subsequent processing.

[0139] For example, to help understand the implementation flow of the audio data enhancement method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 4 , Figure 4 A simplified flowchart of an audio data enhancement method is provided, specifically:

[0140] The multi-scale feature extraction module receives the original audio signal and performs time-frequency transformation on the same audio signal in parallel using four different window lengths of STFT. Each window length generates a corresponding amplitude spectrum and phase spectrum.

[0141] The frequency domain feature encoding module encodes the amplitude spectrum and phase spectrum corresponding to each window length. It extracts local time-frequency patterns by performing convolution on the time-frequency two-dimensional plane through convolutional layers and upscaling them to a high-dimensional feature space. It then compresses the global frequency information along the frequency dimension through pooling layers to form a compact representation of each frame. Finally, it maps the features to a unified target dimension through fully connected layers to form the encoded feature sequence corresponding to the window length.

[0142] The causal attention fusion module receives the encoded feature sequence from the frequency domain coding module, maps the input encoded feature sequence into Q matrix, K matrix and V matrix respectively, calculates the similarity between Q and K to obtain the attention score, and multiplies the attention score matrix by the preset lower triangular matrix to obtain the attention weight, ensuring that each time point can only focus on current and past information, preventing the leakage of future information and meeting the real-time requirements. The V matrix is ​​weighted and summed using the masked attention weights to obtain the fused feature sequence.

[0143] The sampling rate embedding layer of the dual adaptive module receives sampling rate metadata and maps the sampling rate values ​​to a learnable feature vector. The sampling rate adaptive submodule generates a first adaptive weight based on the feature vector output by the sampling rate embedding layer to guide the model's attention to different frequency components. The first adaptive weight is multiplied element-wise with the fused feature sequence from the causal attention fusion module to obtain the sampling rate adaptive feature sequence. The window length adaptive submodule takes the shortest window length encoded feature sequence as input, analyzes the characteristics of the current frame signal through a multilayer perceptron mechanism, and calculates a second adaptive weight. This weight determines the degree of importance given to each window length encoded feature sequence during fusion. The second adaptive weight is weighted and summed with the encoded feature sequences corresponding to each window length to obtain the window length adaptive feature sequence. The sampling rate adaptive feature sequence and the window length adaptive feature sequence are concatenated to obtain the final adaptive feature sequence.

[0144] The causal recurrent neural network module uses a unidirectional LSTM to process the adaptive feature sequence output by the dual adaptive module frame by frame, and outputs an enhanced audio feature sequence.

[0145] The noise reduction module receives the enhanced audio feature sequence and generates a noise suppression mask, or directly generates the noise-reduced frequency domain features such as the amplitude spectrum. This noise suppression mask indicates the ratio of speech to noise in each time-frequency unit.

[0146] The time-frequency conversion and recovery module combines the denoised amplitude spectrum with the phase spectrum of the original noisy speech to form a denoised complex spectrum. It performs an inverse Fourier transform on each frame of the complex spectrum to obtain a short-time time-domain signal frame. All time-domain signal frames are superimposed according to the overlapping method used during analysis to synthesize a complete denoised time-domain speech signal.

[0147] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the audio data enhancement method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0148] This application provides an audio data enhancement device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the audio data enhancement method in Embodiment 1 above.

[0149] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an audio data enhancement device suitable for implementing embodiments of this application. The audio data enhancement device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), and portable media players (PMPs), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The audio data enhancement device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0150] like Figure 5As shown, the audio data enhancement device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the audio data enhancement device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the audio data enhancement device to communicate wirelessly or wiredly with other devices to exchange data. Although audio data enhancement devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0151] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0152] The audio data enhancement device provided in this application, employing the audio data enhancement method described in the above embodiments, solves the technical problem of how to achieve real-time processing of multi-sample-rate time-series signals using a single model without resampling operations or relying on multi-model deployment. Compared with the prior art, the beneficial effects of the audio data enhancement device provided in this application are the same as those of the audio data enhancement method provided in the above embodiments, and other technical features of this audio data enhancement device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0155] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the audio data enhancement method in the above embodiments.

[0156] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0157] The aforementioned computer-readable storage medium may be included in the audio data enhancement device; or it may exist independently and not assembled into the audio data enhancement device.

[0158] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an audio data enhancement device, enable the audio data enhancement device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0161] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described audio data enhancement method. This solves the technical problem of how to achieve real-time processing of multi-sample-rate time-series signals using a single model without resampling operations or relying on multi-model deployment. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the audio data enhancement method provided in the above embodiments, and will not be repeated here.

[0162] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the audio data enhancement method described above.

[0163] The computer program product provided in this application solves the technical problem of how to achieve real-time processing of multi-sample-rate time-series signals using a single model without resampling operations or relying on multi-model deployment. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the audio data enhancement method provided in the above embodiments, and will not be repeated here.

[0164] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An audio data enhancement method, characterized in that, The audio data enhancement method includes: After receiving an audio signal, performing a Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encoding the frequency domain features to obtain an encoded feature sequence; The attention score matrix of the encoded feature sequence is determined by an attention mechanism. The attention weight is obtained by multiplying the attention score matrix by a preset lower triangular matrix. The encoded feature sequence is then weighted based on the attention weight to obtain the fused feature sequence. Each sampling rate of the audio signal is converted into a feature vector of the target dimension, and the feature vector is passed through a first fully connected network to generate a first adaptive weight. The first adaptive weight is multiplied element by element by the fused feature sequence to obtain the sampling rate adaptive feature sequence; The encoded feature sequence corresponding to the shortest window length among all the window lengths is input into the second fully connected network, and the second adaptive weights corresponding to each window length are determined by the second fully connected network. The second adaptive weight corresponding to each window length is weighted and summed with the encoded feature sequence corresponding to each window length to obtain the window length adaptive feature sequence. The sampling rate adaptive feature sequence and the window length adaptive feature sequence are concatenated to obtain the adaptive feature sequence; The adaptive feature sequence is input into a unidirectional LSTM layer, and the hidden state output by the LSTM layer is linearly transformed and mapped to the target dimension to obtain the enhanced audio feature sequence.

2. The audio data enhancement method as described in claim 1, characterized in that, The steps of determining the attention score matrix of the encoded feature sequence through an attention mechanism, multiplying a preset lower triangular matrix by the attention score matrix to obtain attention weights, and weighting the encoded feature sequence based on the attention weights to obtain the fused feature sequence include: Determine the Q matrix, V matrix, and K matrix of the encoded feature sequence; The Q matrix, the V matrix, and the K matrix are split into different attention heads; The attention score matrix corresponding to each attention head is determined by the dot product of the Q matrix and the K matrix corresponding to each attention head. The attention weight corresponding to each attention head is obtained by multiplying the preset lower triangular matrix element by element with the attention score matrix; The attention weights and V matrix corresponding to each attention head are weighted and summed to obtain the fused features corresponding to each attention head. The fused features corresponding to each attention head are concatenated to obtain the fused feature sequence.

3. The audio data enhancement method as described in claim 1, characterized in that, The first fully connected network includes a first connection layer and a second connection layer. The step of converting each sampling rate of the audio signal into a feature vector of the target dimension, and generating first adaptive weights from the feature vector via the first fully connected network includes: Convert each sampling rate of the audio signal into a feature vector of the target dimension; The feature vector of the target dimension is input into the first connection layer, and the feature vector is reduced from the target dimension to the first dimension by the ReLU activation function of the first connection layer. The feature vector of the first dimension is input into the second connection layer. After the feature vector is upgraded from the first dimension to the target dimension by the second connection layer, the first adaptive weight of the feature vector is determined by the sigmoid activation function of the second connection layer.

4. The audio data enhancement method as described in claim 1, characterized in that, The step of concatenating the sampling rate adaptive feature sequence and the window length adaptive feature sequence to obtain the adaptive feature sequence includes: The sampling rate adaptive feature sequence and the window length adaptive feature sequence are concatenated along the feature dimension to obtain the concatenated feature sequence; The concatenated feature sequence is input into a fully connected layer, which compresses the concatenated feature sequence to the target dimension to obtain the adaptive feature sequence.

5. The audio data enhancement method as described in claim 1, characterized in that, The step of performing Fourier transform on the audio signal based on different window lengths to obtain frequency domain features, and then encoding the frequency domain features to obtain an encoded feature sequence includes: After obtaining frequency domain features by performing Fourier transform on the audio signal based on different window lengths, the feature representation of the frequency domain features is extracted by a convolutional neural network. The feature representation extracted from each layer of the convolutional neural network is input into the pooling layer so that the feature representation is subjected to global average pooling in the frequency dimension. The pooled feature representation is input into a fully connected layer, which compresses the pooled feature representation to the target dimension to obtain the encoded feature sequence.

6. The audio data enhancement method as described in claim 1, characterized in that, After the steps of inputting the adaptive feature sequence into a unidirectional LSTM layer and mapping the hidden state output by the LSTM layer to the target dimension through a linear transformation to obtain the enhanced audio feature sequence, the method further includes: The enhanced audio feature sequence is input into a pre-trained noise reduction network, which then identifies noise in the audio feature sequence. Obtain the noise suppression mask generated by the noise reduction network based on the noise; or Obtain the frequency domain features of the audio feature sequence after noise reduction based on the noise output by the noise reduction network.

7. The audio data enhancement method as described in claim 6, characterized in that, The denoised frequency domain features include the denoised amplitude spectrum. After the step of obtaining the frequency domain features of the audio feature sequence after denoising based on the noise output by the denoising network, the method further includes: The amplitude spectrum after noise reduction is spliced ​​with the phase spectrum of the audio signal to form the complex spectrum after noise reduction; The inverse Fourier transform is performed on the denoised complex spectrum to restore the frequency domain features to the time domain signal, thus obtaining the denoised time domain speech signal.

8. An audio data enhancement device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio data enhancement method as claimed in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the audio data enhancement method as described in any one of claims 1 to 7.

Images