Voice instruction recognition method and system with noise robustness

Abstract

The invention discloses a voice instruction recognition method and system with noise robustness. The method comprises the following steps: S1, constructing a training data set; s2, constructing a noise robustness voice instruction recognition model and training the model; and S3, processing actual noisy voice by using the noise robustness voice instruction recognition model trained in the step S2 so as to execute instruction recognition operation. Wherein the constructed noise robustness voice instruction recognition model comprises a voice enhancement model used for carrying out noise reduction processing on input voice with noise and outputting enhanced voice; the voice distortion sensor is used for outputting a distortion probability according to the voice enhancement model; the voice recognition model is used for processing the fused enhanced voice and outputting a voice instruction; according to the method, the voice distortion perceptron is arranged in the model to obtain the distortion probability in real time, difference mixing is carried out on the audio, voice distortion is reduced, and the robustness and accuracy of the whole system are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of voice command recognition, and in particular to a noise-robust voice command recognition method and system. Background Art

[0002] With the rapid development of artificial intelligence (AI), speech recognition and speech enhancement technologies have become crucial components of human-computer interaction. Speech recognition enables users to interact naturally with devices through voice commands, while speech enhancement improves the quality of speech signals through noise reduction and dereverberation, thereby enhancing speech recognition accuracy.

[0003] Speech enhancement technology is a hot research area in the field of artificial intelligence. In recent years, the domestic intelligent voice market has experienced rapid growth, with domestic companies such as iFlytek and Baidu holding a significant market share. Domestic research institutions, such as the Institute of Acoustics at the Chinese Academy of Sciences, are conducting in-depth research on speech enhancement technology. Internationally, companies such as Nuance, Google, and Apple hold a significant share of the intelligent voice market.

[0004] Speech recognition technology, a key branch of artificial intelligence, has been widely adopted in a variety of fields, including smart homes, smart speakers, and in-car navigation systems. For example, in 2016, Microsoft announced that its conversational speech recognition technology had achieved a 5.9% word error rate, comparable to human performance. In China, intelligent speech technology is entering a period of rapid development, and the intelligent speech market is experiencing sustained rapid growth, with a projected market size of 145.2 billion yuan by 2030.

[0005] Early speech enhancement technologies relied on traditional signal processing methods, followed by the development of statistical model-based approaches, such as Gaussian mixture models and hidden Markov models. Currently, the technical foundation of the voice command recognition industry relies primarily on deep learning algorithms and big data processing capabilities. In recent years, deep learning technologies, particularly neural network models, have provided powerful tools for speech enhancement and recognition. These technologies can handle complex speech features and achieve better recognition performance in noisy environments. The first generation of speech recognition systems emerged in 1970, followed by the emergence of neural networks, and the second and third generations of speech recognition systems. In recent years, the introduction of deep learning models, leveraging multi-layer nonlinear transformations, end-to-end models, and multimodal interaction, has enabled more accurate acoustic modeling and speech interaction.

[0006] With the advancements in both speech enhancement and speech recognition technologies, researchers are exploring how to combine them to achieve a more efficient and accurate voice interaction experience. Integrated technology aims to handle both speech enhancement and recognition tasks simultaneously through a unified model, significantly improving the efficiency and accuracy of voice interaction, providing a better user experience, reducing latency, and improving overall performance.

[0007] To achieve these goals, existing technologies in the field of speech enhancement and recognition primarily focus on improving speech signal quality and recognition accuracy through deep learning models. The following are two existing technologies similar to the present invention, based on the Grouped Temporal Convolutional Recurrent Network (GTCRN) and Feedforward Sequential Memory Network (FSMN) models, respectively.

[0008] (1) Speech enhancement technology based on GTCRN

[0009] GTCRN is an ultra-lightweight speech enhancement model designed for resource-constrained edge devices. By combining grouped temporal convolutions and recurrent networks, it achieves efficient speech enhancement at a very low computational cost. The model has a reduced parameter count of approximately 23.7K and a computational overhead of 39.6 MMACs per second.

[0010] (2) FSMN-based speech recognition technology

[0011] FSMN is a feedforward sequence memory network designed specifically for speech recognition tasks. By introducing memory blocks, FSMN can capture long-term dependencies in speech signals while maintaining low computational complexity.

[0012] However, the above technical solution still has significant deficiencies, including at least the following defects:

[0013] (1) Speech enhancement technology

[0014] While GTCRN excels at lightweighting, its denoising effect is limited when processing large noisy datasets. Due to its simple model structure and small number of parameters, its generalization ability in complex noise environments is insufficient, making it difficult to cope with diverse noise scenarios. Furthermore, while GTCRN performs well with stationary noise (such as white noise), its denoising effect is limited when dealing with non-stationary noise (such as industrial and traffic noise). This limits its application in complex acoustic environments.

[0015] (2) Speech recognition technology

[0016] End-to-end speech recognition systems based on deep learning use models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformers, combined with attention mechanisms and language models, to efficiently convert speech signals into text. While this approach performs well under ideal conditions, it still suffers from significant disadvantages: First, the model relies on large amounts of labeled data, which is expensive to acquire and label. Second, training and inference require high-performance hardware, which consumes a lot of resources. Furthermore, recognition accuracy decreases significantly in noisy environments or when dealing with non-standard accents or dialects. Regarding real-time performance, despite optimizations, latency issues remain unresolved. Finally, the processing of speech data can raise privacy concerns, and debugging and optimizing complex models is challenging. These challenges limit the widespread application of this technology in real-world scenarios.

[0017] Therefore, in view of the defects of the existing technology, it is necessary to propose a technical solution to solve the technical problems existing in the existing technology. Summary of the Invention

[0018] In view of this, it is necessary to provide a noise-robust voice command recognition method and system, which can obtain the distortion probability ω in real time by setting a voice distortion sensor in the model. SE ,Difference mixing of audio is performed to reduce speech distortion, further improving the robustness and accuracy of the entire system.

[0019] In order to solve the technical problems existing in the prior art, the technical solutions of the present invention are as follows:

[0020] A noise-robust voice command recognition method comprises the following steps:

[0021] Step S1: constructing a training data set, wherein a large number of paired training sets of noisy speech and clean speech are obtained and data preprocessed for integrated model training;

[0022] Step S2: constructing a noise-robust voice command recognition model and training the model, wherein the constructed noise-robust voice command recognition model includes:

[0023] Build a speech enhancement model to reduce the noise of the input noisy speech and output enhanced speech;

[0024] Constructing a speech distortion sensor for outputting a distortion probability according to the speech enhancement model;

[0025] Constructing a speech recognition model for processing the fused enhanced speech to output a speech command; wherein the fused enhanced speech is obtained by adding the original noisy speech to the enhanced speech according to a certain proportional weight based on its distortion probability;

[0026] Step S3: Using the noise-robust speech command recognition model trained in step S2, the actual noisy speech is processed to perform a command recognition operation.

[0027] As a further improvement, in step S2, the noise robustness voice command recognition model is trained using an integrated model, and the training process is as follows:

[0028] Fixed the speech recognition model and trained the speech enhancement model;

[0029] A fixed speech enhancement model is used to train a speech recognition model. During the training of the speech recognition model, a speech distortion sensor is jointly optimized.

[0030] Repeat the above training process until the training is completed.

[0031] As a further improvement, the speech enhancement model adopts a Dual Net model.

[0032] As a further improvement, the speech recognition model adopts the FSMN model.

[0033] As a further improvement, the speech recognition model is trained using a CTC loss function.

[0034] As a further improvement, step S1 includes the following steps:

[0035] Step S11: Use Hey_snips, VCTK dataset and noise dataset DEMAND to mix and simulate a noisy speech dataset;

[0036] Step S12: Transcribe the clean audio in the Hey_snips and VCTK datasets into a text file;

[0037] Step S13: Obtain all words in the text file to generate a dictionary file, and convert the transcribed text into a numerical list according to the dictionary file;

[0038] Step S14: Match the noisy speech with the corresponding clean speech to form a training set and a data set to be denoised, write the audio name, audio path, sampling rate, and digital list into a CSV file and generate an hdf5 file, match the noisy speech with the corresponding clean speech to form a training set and a data set to be denoised.

[0039] The present invention also discloses a noise-robust voice command recognition system, which sets a trained noise-robust voice command recognition model for processing actual noisy speech to perform command recognition operations; the model includes:

[0040] Speech enhancement model, used to reduce noise in noisy input speech and output enhanced speech;

[0041] a speech distortion sensor, configured to output a distortion probability according to the speech enhancement model;

[0042] A speech recognition model is used to process the fused enhanced speech and output a speech instruction; wherein the fused enhanced speech is obtained by adding the original noisy speech to the enhanced speech according to its distortion probability in a certain proportion.

[0043] As a further improvement, the noise robustness voice command recognition model is trained using an integrated model, and the training process is as follows:

[0044] Fixed the speech recognition model and trained the speech enhancement model;

[0045] A fixed speech enhancement model is used to train a speech recognition model. During the training of the speech recognition model, a speech distortion sensor is jointly optimized.

[0046] Repeat the above training process until the training is completed.

[0047] As a further improvement, the speech enhancement model adopts a Dual Net model.

[0048] As a further improvement, the speech recognition model adopts the FSMN model.

[0049] Compared with the prior art, the technical solution of the present invention has the following technical effects:

[0050] 1. Incorporating a speech distortion sensor into the model effectively reduces distortion that may be introduced during speech enhancement processing and enhances the robustness of back-end recognition. Even in noisy streets, the model can accurately capture and understand voice commands, making interactions smoother and more natural, and improving work efficiency.

[0051] 2. By using a fixed front-end training method and a fixed back-end training method, alternating between front-end and back-end training, and simultaneously optimizing the speech distortion sensor during the integrated model training process, training on large amounts of diverse data allows for better generalization to different speakers, accents, and environmental noise conditions, enhancing recognition accuracy and enabling low-latency, high-accuracy speech recognition on resource-constrained devices.

[0052] 3. This model uses the advanced lightweight Dual Net algorithm, combined with the FSMN algorithm, to achieve lightweight processing on both the front-end and back-end of the model. This design enables the model to run smoothly even in resource-constrained hardware environments, making it particularly suitable for deployment on cost-effective devices.

[0053] Using the Dual Net model to improve noise reduction: Dual Net uses a multi-branch neural network architecture that can simultaneously process the complex spectrum and amplitude spectrum of speech signals. It further improves the speech enhancement effect through spectral masking technology, can better capture the noise components in speech signals, and achieve more efficient noise reduction in complex noisy environments.

[0054] Using the FSMN model improves robustness and reduces latency: FSMN is an efficient sequence modeling model that employs a feedforward structure and memory modules, avoiding the recursive computations of RNNs. It supports parallel processing, offers fast training, and reduces latency. Its memory modules can capture long-range dependencies, making it suitable for speech recognition and natural language processing tasks. FSMN is characterized by strong robustness, ease of optimization, and high flexibility, particularly excelling in latency-sensitive scenarios such as real-time speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of a noise-robust voice command recognition method of the present invention.

[0056] Figure 2 Schematic diagram of the algorithm framework of the noise-robust voice command recognition method of the present invention.

[0057] Figure 3 This is a flowchart of the Dual Net speech enhancement model in the present invention.

[0058] Figure 4 This is a flow chart of the speech distortion sensor in the present invention.

[0059] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0060] The technical solution provided by the present invention will be further described below with reference to the accompanying drawings.

[0061] In order to make up for the shortcomings of traditional technology, the technical solution of the present invention improves the performance of the speech recognition system in a noisy environment through the introduction of deep learning technology and speech enhancement and recognition models, realizes the integration and lightweight of the model, improves the practicality of the system, helps edge devices to carry applications, and improves user experience.

[0062] See also Figure 1 , shown is a flowchart of a noise-robust voice command recognition method provided by the present invention, comprising the following steps:

[0063] Step S1: constructing a training data set, wherein a large number of paired training sets of noisy speech and clean speech are obtained and data preprocessed for integrated model training;

[0064] Step S2: constructing a noise-robust voice command recognition model and training the model;

[0065] Step S3: Using the noise-robust speech command recognition model trained in step S2, the actual noisy speech is processed to perform a command recognition operation.

[0066] See also Figure 2 , which is a schematic diagram of the algorithm framework of the noise-robust voice command recognition model constructed by the present invention, including:

[0067] Speech enhancement model, used to reduce noise in noisy input speech and output enhanced speech;

[0068] a speech distortion sensor, configured to output a distortion probability according to the speech enhancement model;

[0069] A speech recognition model is used to process the fused enhanced speech and output a speech instruction; wherein the fused enhanced speech is obtained by adding the original noisy speech to the enhanced speech according to its distortion probability in a certain proportion.

[0070] In the above step S2, the noise robust voice command recognition model is trained using an integrated model, and the training process is as follows:

[0071] Fixed the speech recognition model and trained the speech enhancement model;

[0072] A fixed speech enhancement model is used to train a speech recognition model. During the training of the speech recognition model, a speech distortion sensor is jointly optimized.

[0073] Repeat the above training process until the training is completed.

[0074] In the above step S1, constructing the training data set further includes the following steps:

[0075] Step S11: Use Hey_snips, VCTK dataset and noise dataset DEMAND to mix and simulate a noisy speech dataset;

[0076] Step S12: Transcribe the clean audio in the Hey_snips and VCTK datasets into a text file;

[0077] Step S13: Obtain all words in the text file to generate a dictionary file, and convert the transcribed text into a numerical list according to the dictionary file;

[0078] Step S14: Match the noisy speech with the corresponding clean speech to form a training set and a data set to be denoised, write the audio name, audio path, sampling rate, and digital list into a CSV file and generate an hdf5 file, match the noisy speech with the corresponding clean speech to form a training set and a data set to be denoised.

[0079] Furthermore, the training process of the speech enhancement model is as follows:

[0080] Step S21: The speech enhancement model uses the Dual Net model to perform noise reduction on the input noisy speech and output enhanced speech. Figure 3 , shown is a flowchart of the Dual Net speech enhancement model in the present invention;

[0081] Step S21 is specifically as follows:

[0082] Step S211: The noisy speech is processed and decomposed by a shared encoder, and a feature tensor of a latent space is output;

[0083] Step S212: Input the feature dimension output by the shared encoder into the dual-path block (Dual-Path Block), and output the feature tensor after frequency and time path processing;

[0084] Step S213: The output feature tensor is passed through the amplitude and complex decoders to generate respective masks, and the masks are applied to the noisy speech spectrum to synthesize enhanced speech;

[0085] Step S214: merging the enhanced magnitude spectrum and complex spectrum to output the enhanced speech;

[0086] Step S211 is specifically as follows:

[0087] Step S2111: The noisy speech is passed through a complex cepstrum system to extract complex spectrum and amplitude spectrum features;

[0088] Step S2112: Map the complex spectrum and the amplitude spectrum to a latent space to extract high-level feature representations.

[0089] Wherein step S2111:

[0090] S21111: Divide the noisy speech y(n) into frames with a frame length of 400 samples (25ms). Use the Hamming window ω(n) and set an overlap of 100 samples (6.25ms). The framed signal is represented as y m (n)=y(n+mt)ω(n), where m is the frame index and t is the frame shift.

[0091] S21112: For each frame of speech y m(n) Perform a 512-point short-time Fourier transform (STFT) calculation to obtain the frequency domain representation Y(k,l), where k is the frequency index and l is the time frame index.

[0092] S21113: Take the logarithm of the complex amplitude spectrum |Y(k,l)| to obtain ln(|Y(k,l)|), and restore the complex cepstrum through the inverse short-time Fourier transform (ISTFT) Where FFT represents discrete-time Fourier transform processing, ln represents logarithmic amplitude processing, and IFFT represents discrete-time inverse Fourier transform processing.

[0093] The specific steps in step S212 are as follows:

[0094] S2121: Modeling the frequency dimension of the shared encoder output in the frequency path;

[0095] S2122: Modeling the time dimension of the shared encoder output in the time path;

[0096] S2123: Merge and normalize the outputs obtained from the frequency path and the time path;

[0097] The specific steps in step S2121 are as follows:

[0098] S21211: Reshape the feature tensor into the frequency dimension;

[0099] S21212: Use RNN blocks to model frequency and output feature tensors processed by the frequency path;

[0100] The specific steps in step S2122 are as follows:

[0101] S21221: Reshape the feature tensor into the time dimension;

[0102] S21222: Use RNN blocks to model time and output feature tensors processed by the time path;

[0103] The specific steps in step S2123 are as follows:

[0104] S21231: Merge the outputs of the frequency path and the time path;

[0105] S21232: Perform residual connection and normalization on the merged feature vectors;

[0106] The specific steps in step S213 are as follows:

[0107] S2131: Generate an amplitude spectrum mask by passing the amplitude spectrum through an amplitude spectrum decoder;

[0108] S2132: generating a complex spectrum mask by passing the complex spectrum through a complex spectrum decoder;

[0109] The specific steps in step S2131 are as follows:

[0110] S21311: Mask the amplitude spectrum D m Perform a dot multiplication with the noisy amplitude spectrum |Y| to obtain the enhanced amplitude spectrum, the formula is as follows:

[0111]

[0112] Among them D m is the amplitude spectrum mask (value range [0,1]), |Y| is the amplitude spectrum of the noisy speech.

[0113] S21312: synthesize the processed amplitude spectrum and the noisy phase spectrum;

[0114] The specific steps in step S2132 are as follows:

[0115] S21321: Mask the complex spectrum D c Perform a dot multiplication with the noisy spectrum Y to obtain the enhanced complex spectrum. The formula is as follows:

[0116]

[0117] Among them D c is the complex spectrum mask (value range [0,1]), and Y is the spectrum of the noisy speech.

[0118] The specific steps in step S214 are as follows:

[0119] S2141: Apply the two outputs of the decoder mask to the 3-layer Conv2D learning weight D m , get the enhanced spectrum as follows:

[0120]

[0121] Among them, D m is the dynamically adjusted fusion weight, is the enhanced result of the magnitude spectrum and complex spectrum branches.

[0122] S2142: Apply the inverse STFT to the enhanced spectrum and obtain the enhanced speech in the time domain by inverse FFT:

[0123]

[0124] in is the enhanced amplitude spectrum, argY is the phase information of the original noisy speech, and IFFT represents the discrete-time inverse Fourier transform processing.

[0125] The loss function L used for the enhanced result of merging the neural homomorphism and complex spectrum masking branches in step S214 is as follows:

[0126]

[0127]

[0128] in is the complex spectrum matrix of the enhanced speech signal in the i-th layer STFT (short time Fourier transform), X i is the complex spectrum matrix of the clean speech signal in the i-th layer STFT, |·| is the amplitude (i.e., absolute value) of the complex spectrum, log is the logarithmic transformation of the amplitude spectrum, ||·|| F is the Frobenius norm, and the square root of the sum of the squares of the matrix elements is calculated, which is ||·||1 is the L1 norm, and the sum of the absolute values ​​of the matrix elements is calculated, which is ||A||1=∑ k,l |A(k,l)|.

[0129] Furthermore, the training process of the speech recognition model is as follows:

[0130] Step S22: Construct a speech distortion sensor and output the distortion probability according to the speech enhancement model. Figure 4 , which is a flow chart of the speech distortion sensor in the present invention: wherein the input of the speech distortion sensor comes from the speech enhancement model, that is, the Dual Net model, that is, the features of the noisy speech after being processed by the shared encoder and the dual path block;

[0131] The specific steps of step S22 are as follows:

[0132] S221: The model weight ω of the perceptron is obtained through joint optimization of the integrated model SE ;

[0133] S222: Dynamically sense the distortion probability of the speech after noise reduction through the speech noise reduction module, and add the original noisy speech with a certain proportion of weight to reduce speech distortion;

[0134] S223: By calculating the joint loss function, the model weight ω of the perceptron SE For joint optimization, the joint loss function formula is as follows:

[0135]

[0136] Among them, α and β are weight coefficients, which need to be adjusted according to different training strategies.

[0137] The specific steps of step S221 are as follows:

[0138] S2211: Receive model configuration parameters, determine basic feature dimensions, and obtain the output of the dual-path block, which has a shape of [B, C, F, T] (batch size, channel, frequency, time frame);

[0139] S2212: Select the appropriate processing method according to different input data. If the global mode is selected, the final ω SE The output value is a value in the range of [0, 1];

[0140] S2213: Select the appropriate processing method according to different input data. If the frame mode is used, the final ω SE The output value is a two-dimensional vector in the range [0, 1];

[0141] S2214: Normalize the unnormalized value through the Sigmoid function to obtain ω SE ;

[0142] The specific steps of S2212 are as follows:

[0143] S22121: A configurable pooling method (average pooling or maximum pooling) will be used to compress global features;

[0144] S22122: Use feature flattening to compress the pooled 4D tensor into a 2D matrix [B, C], removing redundant single-element dimensions.

[0145] S223123: The features after pooling and flattening are passed through the fully connected layer to obtain the unnormalized ω SE ;

[0146] The specific steps of S2213 are as follows:

[0147] S22131: Flatten the encoder's output feature channel (C) and frequency (F) dimensions into one dimension;

[0148] S22132: Perform dimensionality conversion, placing the time frame dimension T in the middle to adapt to the input format of the fully connected layer;

[0149] S22133: Through the fully connected layer, the C×F dimensional features of each time frame are linearly transformed to obtain the unnormalized two-dimensional matrix ω SE ;

[0150] The specific steps of S222 are as follows:

[0151] S2221: Use the ω obtained in step S221 SE , the original noisy speech and the enhanced speech are compared with ω SE Perform interpolation to obtain enhanced signal Improve the performance and robustness of the integrated model. The formula is as follows:

[0152]

[0153] represents the output of the speech enhancement module, represents the original noisy speech, Represents the fused enhanced signal obtained after processing.

[0154] Step S23: The speech recognition model uses the FSMN model to process the fused enhanced speech and output voice commands.

[0155] Step S23 is specifically summarized as follows:

[0156] S231: Affine transformation, input processing of the fused enhanced speech;

[0157] S232: Use ReLU layer activation to solve the gradient vanishing problem and improve training stability;

[0158] S233: The input is sent to the first hidden layer of the FSMN network. The output is calculated based on the memory blocks in each hidden layer and the memory blocks are used.

[0159] S234: Send the output of the current hidden layer and the output of the memory block together to the next hidden layer, repeat the operation until the last hidden layer, and send it to the output layer for calculation.

[0160] S234: The high-level features output by the FSMN module undergo a final affine transformation and are mapped to the keyword category space using a sigmoid function. The posterior probability of the features is then predicted using an independent binary classifier.

[0161] S335: During the training phase, the posterior probability is used to calculate the error between the neural network output value and the labeled sequence using CTCLoss. CTCLoss effectively calculates the sum of the probabilities of all possible alignments between the input sequence and the output sequence by introducing a dynamic programming algorithm, thereby optimizing the sequence labeling problem. The mathematical definition of CTC Loss is as follows:

[0162]

[0163] in:

[0164]

[0165] π is a possible alignment, X=(x1,x2,…,x T ) is the input sequence, Y=(y1,y2,…,y U ) is the output sequence, is the set of all possible alignment paths, and P(π|X) is the probability of alignment path π, which is given by the product of the probabilities of each label in the alignment at each time step:

[0166]

[0167] in, is the model output label π at time step t t probability.

[0168] S235: Use the backpropagation algorithm to calculate the gradient of the loss function for the output layer, hidden layer, and fast memory, and use the optimization algorithm to update the gradient;

[0169] The above step S3: the inference process of the trained noise-robust voice command recognition model includes the following steps:

[0170] Step S31: The tested noisy speech is input into the front-end speech enhancement module, and the amplitude spectrum and complex spectrum of the noisy speech are processed respectively through the shared encoder, dual path block and decoder mask application to obtain a high-quality enhanced speech signal.

[0171] Step S32: Using the distortion probability ω obtained by the speech distortion sensor SE , the tested noisy speech and the enhanced speech are operated to obtain the fused enhanced speech, thus reducing speech distortion;

[0172] Step S33: performing voice command recognition processing on the fused enhanced voice and outputting the voice command;

[0173] Step S34: A joint loss function is used in the inference process of the noise robust voice command recognition model: the formula of the joint loss function is as follows:

[0174]

[0175] Among them, α and β are weight coefficients.

[0176] Step S31 is specifically as follows:

[0177] Step S311: The noisy speech is processed and decomposed by a shared encoder, and a feature tensor of a latent space is output;

[0178] Step S312: Input the feature dimension output by the shared encoder into the dual-path block (Dual-Path Block), and output the feature tensor after frequency and time path processing;

[0179] Step S313: The output feature tensor is passed through the amplitude and complex decoders to generate respective masks, and the masks are applied to the noisy speech spectrum to synthesize enhanced speech;

[0180] Step S314: merging the enhanced magnitude spectrum and complex spectrum to output the enhanced speech;

[0181] Step S311 is specifically as follows:

[0182] Step S3111: The noisy speech is passed through a complex cepstrum system to extract complex spectrum and amplitude spectrum features;

[0183] Step S3112: Map the complex spectrum and the amplitude spectrum into a latent space to extract high-level feature representations.

[0184] Wherein step S3111:

[0185] S31111: Divide the noisy speech y(n) into frames with a frame length of 400 samples (25ms). Use the Hamming window ω(n) and set an overlap of 100 samples (6.25ms). The framed signal is represented as y m (n)=y(n+mt)ω(n), where m is the frame index and t is the frame shift.

[0186] S31112: For each frame of speech y m (n) Perform a 512-point short-time Fourier transform (STFT) calculation to obtain the frequency domain representation Y(k,l), where k is the frequency index and l is the time frame index.

[0187] S31113: Take the logarithm of the complex amplitude spectrum |Y(k,l)| to obtain ln(|Y(k,l)|), and restore the complex cepstrum through the inverse short-time Fourier transform (ISTFT) Where FFT represents discrete-time Fourier transform processing, ln represents logarithmic amplitude processing, and IFFT represents discrete-time inverse Fourier transform processing.

[0188] The specific steps in step S312 are as follows:

[0189] S3121: Modeling the frequency dimension of the shared encoder output in the frequency path;

[0190] S3122: Modeling the time dimension of the shared encoder output in the time path;

[0191] S3123: Merge and normalize the outputs obtained from the frequency path and the time path;

[0192] The specific steps in step S3121 are as follows:

[0193] S31211: Reshape the feature tensor into the frequency dimension;

[0194] S31212: Use RNN blocks to model frequency and output feature tensors processed by the frequency path;

[0195] The specific steps in step S3122 are as follows:

[0196] S31221: Reshape feature tensor into time dimension;

[0197] S31222: Use RNN blocks to model time and output feature tensors processed by the time path;

[0198] The specific steps in step S3123 are as follows:

[0199] S31231: Merge the outputs of the frequency path and the time path;

[0200] S31232: Perform residual connection and normalization on the merged feature vectors;

[0201] The specific steps in step S313 are as follows:

[0202] S3131: Generate an amplitude spectrum mask by passing the amplitude spectrum through an amplitude spectrum decoder;

[0203] S3132: generating a complex spectrum mask by passing the complex spectrum through a complex spectrum decoder;

[0204] The specific steps in step S3131 are as follows:

[0205] S31311: Mask the amplitude spectrum D m Perform a dot multiplication with the noisy amplitude spectrum |Y| to obtain the enhanced amplitude spectrum, the formula is as follows:

[0206]

[0207] Among them D m is the amplitude spectrum mask (value range [0,1]), |Y| is the amplitude spectrum of the noisy speech.

[0208] S31312: synthesize the processed amplitude spectrum and the noisy phase spectrum;

[0209] The specific steps in step S3132 are as follows:

[0210] S31321: Mask the complex spectrum D c Perform a dot multiplication with the noisy spectrum Y to obtain the enhanced complex spectrum. The formula is as follows:

[0211]

[0212] Among them D cis the complex spectrum mask (value range [0,1]), and Y is the spectrum of the noisy speech.

[0213] The specific steps in step S314 are as follows:

[0214] S3141: Apply the two outputs of the decoder mask to the 3-layer Conv2D learning weight D m , get the enhanced spectrum as follows:

[0215]

[0216] Among them D m is the dynamically adjusted fusion weight, is the enhanced result of the magnitude spectrum and complex spectrum branches.

[0217] S3142: Apply inverse STFT to the enhanced spectrum to obtain enhanced speech in the time domain through inverse FFT.

[0218]

[0219] in is the enhanced amplitude spectrum, argY is the phase information of the original noisy speech, and IFFT represents the discrete-time inverse Fourier transform processing.

[0220] Step S32 is specifically as follows:

[0221] Step S321: Process the output of the dual path, and obtain ω through operations such as pooling, flattening, linear layer and Sigmoid function. SE ;

[0222] Step S322: Use the ω obtained in step S321 SE , the original noisy speech and the enhanced speech are compared with ω SE Perform interpolation to obtain enhanced signal Improve the performance and robustness of the integrated model. The formula is as follows:

[0223]

[0224] in, represents the output of the speech enhancement module, represents the original noisy speech, Represents the fused enhanced signal obtained after processing.

[0225] Step S33 is specifically as follows:

[0226] Step S331: inputting the fused enhanced speech;

[0227] Step S332: Send the input to the first hidden layer of the FSMN network, and calculate the output based on the memory blocks in each hidden layer according to the type of the memory blocks.

[0228] Step S333: Send the output of the current hidden layer and the output of the memory block together to the next hidden layer, and repeat the operation until the last hidden layer;

[0229] Step S334: Send the output of the last hidden layer to the output layer for calculation.

[0230] Step S335: The high-level features output by the FSMN module undergo a final affine transformation and are mapped to the keyword category space using a sigmoid function. The posterior probability is predicted using an independent binary classifier.

[0231] Step S336: Perform performance evaluation on the final output and calculate the word error rate (WER).

[0232] The present invention also discloses a noise robust voice command recognition system, which sets a trained noise robust voice command recognition model for processing actual noisy speech to perform command recognition operations; its algorithm framework is as follows: Figure 2 As shown, the model includes:

[0233] Speech enhancement model, used to reduce noise in noisy input speech and output enhanced speech;

[0234] a speech distortion sensor, configured to output a distortion probability according to the speech enhancement model;

[0235] A speech recognition model is used to process the fused enhanced speech and output a speech instruction; wherein the fused enhanced speech is obtained by adding the original noisy speech to the enhanced speech according to its distortion probability in a certain proportion.

[0236] The noise robust voice command recognition model is trained using an integrated model, and the training process is as follows:

[0237] Fixed the speech recognition model and trained the speech enhancement model;

[0238] A fixed speech enhancement model is used to train a speech recognition model. During the training of the speech recognition model, a speech distortion sensor is jointly optimized.

[0239] Repeat the above training process until the training is completed.

[0240] The specific process of training the speech enhancement model, speech distortion sensor, and speech recognition model is described above and will not be repeated here.

[0241] As a further improvement, the speech enhancement model and speech recognition model adopt the Dual Net speech enhancement model and the FSMN (Feedforward Sequential Memory Networks) back-end instruction recognition model respectively, and add a speech distortion sensor to solve the problems existing in the traditional speech enhancement and recognition optimization model. The introduction of the Dual Net model adopts a multi-branch neural network architecture that can simultaneously process the amplitude spectrum and complex spectrum of the speech signal, and further improve the speech enhancement effect through spectral masking technology. At the same time, the speech recognition model FSMN is used to reduce parameters, reduce latency, increase training speed, and simplify the model structure. By adding a speech distortion sensor to the integrated model, ω SE ,Difference mixing of audio is performed to reduce speech distortion, further improving the robustness and accuracy of the entire system.

[0242] The above embodiments are only intended to help understand the method and core concept of the present invention. It should be noted that, without departing from the principles of the present invention, a number of improvements and modifications may be made to the present invention by those skilled in the art, and such improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0243] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A noise-robust voice command recognition method, characterized in that: The following steps are involved: Step S1: constructing a training data set, wherein a large number of paired training sets of noisy speech and clean speech are obtained and data preprocessed for integrated model training; Step S2: constructing a noise-robust voice command recognition model and training the model, wherein the constructed noise-robust voice command recognition model includes: Build a speech enhancement model to reduce the noise of the input noisy speech and output enhanced speech; Constructing a speech distortion sensor for outputting a distortion probability according to the speech enhancement model; Constructing a speech recognition model for processing the fused enhanced speech to output a speech command; wherein the fused enhanced speech is obtained by adding the original noisy speech to the enhanced speech according to a certain proportional weight based on its distortion probability; Step S3: Using the noise-robust speech command recognition model trained in step S2, the actual noisy speech is processed to perform a command recognition operation.

2. The noise-robust voice command recognition method according to claim 1, characterized in that: In step S2, the noise robust voice command recognition model is trained using an integrated model, and the training process is as follows: Fixed the speech recognition model and trained the speech enhancement model; A fixed speech enhancement model is used to train a speech recognition model. During the training of the speech recognition model, a speech distortion sensor is jointly optimized. Repeat the above training process until the training is completed.

3. The noise-robust voice command recognition method according to claim 1, characterized in that: The speech enhancement model adopts a Dual Net model.

4. The noise-robust voice command recognition method according to claim 1, characterized in that: The speech recognition model adopts the FSMN model.

5. The noise-robust voice command recognition method according to claim 4, characterized in that: The speech recognition model is trained using the CTC loss function.

6. The noise-robust voice command recognition method according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: Use Hey_snips, VCTK dataset and noise dataset DEMAND to mix and simulate a noisy speech dataset; Step S12: Transcribe the clean audio in the Hey_snips and VCTK datasets into a text file; Step S13: Obtain all words in the text file to generate a dictionary file, and convert the transcribed text into a numerical list according to the dictionary file; Step S14: Match the noisy speech with the corresponding clean speech to form a training set and a data set to be denoised, write the audio name, audio path, sampling rate, and digital list into a CSV file and generate an hdf5 file, match the noisy speech with the corresponding clean speech to form a training set and a data set to be denoised.

7. A noise-robust voice command recognition system, characterized in that: Set up a trained noise-robust voice command recognition model to process actual noisy speech to perform command recognition operations; the model includes: Speech enhancement model, used to reduce noise in noisy input speech and output enhanced speech; a speech distortion sensor, configured to output a distortion probability according to the speech enhancement model; A speech recognition model is used to process the fused enhanced speech and output a speech instruction; wherein the fused enhanced speech is obtained by adding the original noisy speech to the enhanced speech according to its distortion probability in a certain proportion.

8. The noise-robust voice command recognition system according to claim 7, characterized in that: The noise robust voice command recognition model is trained using an integrated model, and the training process is as follows: Fixed the speech recognition model and trained the speech enhancement model; A fixed speech enhancement model is used to train a speech recognition model. During the training of the speech recognition model, a speech distortion sensor is jointly optimized. Repeat the above training process until the training is completed.

9. The noise-robust voice command recognition system according to claim 7, characterized in that: The speech enhancement model adopts a Dual Net model.

10. The noise-robust voice command recognition system according to claim 7, characterized in that: The speech recognition model adopts the FSMN model.

Images