Robust speech recognition method and system based on multi-stage feature fusion

Abstract

The invention discloses a robust speech recognition method and system based on multi-stage feature fusion, and relates to the technical field of speech recognition. The invention provides a speech recognition model for robust speech recognition, which comprises the following steps: firstly, extracting a magnitude spectrum | Y | from noisy speech Y through a speech coding part, coding the magnitude spectrum | Y | into a preliminary feature Ybasic, then, processing the Ybasic through a speech enhancement part to obtain a hidden feature Yhidden, a masking feature Ymask and a mapping feature Ymap, and finally, carrying out speech recognition on the hidden feature Yhidden, the masking feature Ymask and the mapping feature Ymap. Then, a fusion feature Ffuse is obtained through three-stage feature fusion, and finally, the Ffuse is decoded through a voice decoder to obtain Result. According to the model, feature complementation, feature interaction and high-low layer semantic alignment in the process are enhanced, the problem of voice distortion introduced by voice enhancement is systematically relieved, information loss in different stages is solved, and therefore the final voice recognition effect is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and more specifically to: 1. a robust speech recognition method based on multi-stage feature fusion; 2. a robust speech recognition system based on multi-stage feature fusion. Background Art

[0002] Currently, robust speech recognition aims to identify the target speaker in a noisy environment. The mainstream research approach currently uses a speech enhancement (SE) module as the front-end module of the system, preprocessing the input audio to remove noise and then feeding the enhanced speech into an automatic speech recognition (ASR) system for recognition.

[0003] However, the speech enhancement process inevitably leads to speech distortion and information loss. For example, the speaker's voice information may be mistakenly removed as noise, which can affect subsequent recognition results. This distortion occurs because the training objectives of speech enhancement and speech recognition are inconsistent: speech enhancement aims to remove noise, while speech recognition aims to obtain transcribed text. Although some researchers have proposed joint training methods that optimize SE and ASR as a unified task, ASR recognition performance still has room for improvement due to shortcomings in the model structure itself. Summary of the Invention

[0004] Based on this, it is necessary to provide a robust speech recognition method and system based on multi-stage feature fusion to address the problem that SE in the existing technology introduces speech distortion and information loss, resulting in a decrease in ASR.

[0005] The present invention is achieved by adopting the following technical solutions:

[0006] In a first aspect, the present invention discloses a robust speech recognition method based on multi-stage feature fusion, comprising:

[0007] Step 1: Obtain noisy speech Y;

[0008] Step 2: Input Y into the trained speech recognition model for processing to obtain the recognition result Result.

[0009] The speech recognition model includes: speech encoding unit, speech enhancement unit, attention fusion unit, cross-information fusion unit, adaptive feature fusion unit, and speech decoder.

[0010] The speech coding unit is used to extract the amplitude spectrum |Y| from Y and encode it into the preliminary feature Y basic .

[0011] The speech enhancement unit is used to basic Processing to obtain hidden features Y hidden , masking feature Ymask , mapping feature Y map .

[0012] The attention fusion unit is used to mask 、Y map Fusion is performed to obtain the fusion feature Y MCF .

[0013] The cross information fusion unit is used to combine |Y|, Y MCF Fusion is performed to obtain the fusion feature Y CIF .

[0014] The adaptive feature fusion unit is used to hidden 、Y CIF Fusion is performed to obtain the fusion feature F fused .

[0015] Speech decoder, which is used to fused Decode to get Result.

[0016] This robust speech recognition method based on multi-stage feature fusion implements the method or process according to the embodiment of the present disclosure.

[0017] In a second aspect, the present invention discloses a robust speech recognition system based on multi-stage feature fusion, which uses the robust speech recognition method based on multi-stage feature fusion disclosed in the first aspect.

[0018] The robust speech recognition system based on multi-stage feature fusion includes: a speech acquisition module and a speech recognition module.

[0019] The speech acquisition module is used to acquire noisy speech Y.

[0020] The speech recognition module is used to input Y into the trained speech recognition model for processing to obtain the recognition result Result.

[0021] This robust speech recognition system based on multi-stage feature fusion implements the method or process according to the embodiment of the present disclosure.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention provides a speech recognition model with a novel structural design for robust speech recognition. First, the speech encoding unit extracts the amplitude spectrum |Y| from the noisy speech Y and encodes it into a preliminary feature Y basic Then, the speech enhancement unit basic Processing to obtain hidden features Y hidden , masking feature Y mask , mapping feature Y map , and then Y is fused by attentionmask 、Y map Fusion is performed to obtain the fusion feature Y MCF , and then through the cross information fusion part |Y|, Y MCF Fusion is performed to obtain the fusion feature Y CIF , and then the adaptive feature fusion unit is used to fusion the Y hidden 、Y CIF Fusion is performed to obtain the fusion feature F fused , and finally the speech decoder is used to decode F fused Decoding is performed to obtain the result. This invention enhances feature complementarity, feature interaction, and high- and low-level semantic alignment within the process through three-stage feature fusion. This systematically alleviates the speech distortion problem introduced by speech enhancement and resolves information loss at different stages, thereby ensuring the ultimate speech recognition effect.

[0024] 2. The present invention uses a design based on the S-TCM layer in the speech enhancement unit and integrates the E-Branchformer architecture into the encoding and decoding of the entire network, thereby improving the noise suppression efficiency and semantic modeling capabilities.

[0025] 3. Simulation verification shows that the present invention has achieved the current best level in terms of performance indicators compared with existing methods, which illustrates the robustness and effectiveness of the present invention and demonstrates its cross-language versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A data flow diagram of the robust speech recognition method based on multi-stage feature fusion provided in Example 1 of the present invention;

[0028] Figure 2 for Figure 1 The structure diagram of the speech coding unit;

[0029] Figure 3 for Figure 1 The structure diagram of the speech enhancement unit;

[0030] Figure 4 for Figure 1 Structural diagram of the attention fusion unit;

[0031] Figure 5 for Figure 1 The structural diagram of the cross-information fusion unit;

[0032] Figure 6 for Figure 5 The structural diagram of the dual-branch network layer;

[0033] Figure 7 for Figure 6 Structural diagram of the middle C-STFA layer;

[0034] Figure 8 for Figure 6 The structural diagram of the DFIM layer;

[0035] Figure 9 for Figure 5 The structural diagram of the merged network department;

[0036] Figure 10 for Figure 1 Structural diagram of the adaptive feature fusion unit. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component. When a component is considered to be "fixed to" another component, it may be directly fixed to the other component or there may be a central component.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0040] The present invention aims to effectively integrate speech enhancement processing and overcome the distortion introduced by it, while making full use of the feature information of each stage, thereby improving the robust speech recognition performance.

[0041] Example 1

[0042] See Figure 1 , Figure 1The data flow diagram of a robust speech recognition method based on multi-stage feature fusion is shown. In fact, the flowchart of the robust speech recognition method based on multi-stage feature fusion is also shown, which includes:

[0043] Step 1: Obtain noisy speech Y.

[0044] As described in the background art, the noisy speech Y is the speech of the target speaker in a noisy environment, which can be collected and acquired by relevant sound collection equipment.

[0045] Step 2: Input Y into the trained speech recognition model for processing to obtain the recognition result Result.

[0046] The core of the present invention is to provide a speech recognition model with a new structure, which can be divided into six parts according to its functions: ① speech encoding part, ② speech enhancement part, ③ attention fusion part, ④ cross-information fusion part, ⑤ adaptive feature fusion part, and ⑥ speech decoder.

[0047] It should be noted that the present invention focuses on the design of three-stage fusion - namely ③, ④, and ⑤.

[0048] The following describes the above six parts one by one:

[0049] ① The speech coding unit is used to extract the amplitude spectrum |Y| from Y and encode it into the preliminary feature Y basic .

[0050] like Figure 2 As shown, the speech coding unit can be designed to include: 1 conversion and extraction layer, 1 speech encoder;

[0051] In the speech coding section:

[0052] The transformation and extraction layer uses a short-time Fourier transform to transform Y into the frequency domain to extract |Y|. It should be noted that transforming Y into the frequency domain yields the noisy speech spectrum Y(k,f), which contains the information about |Y|. k represents the time frame index, and f represents the frequency index.

[0053] The speech encoder is used to encode |Y| into Y basic In this embodiment 1, the speech encoder recommends using a GatedConv2D layer, which encodes |Y| through gated two-dimensional convolution.

[0054] Of course, the processing process of the above-mentioned speech coding unit can also be expressed as follows:

[0055] |Y|=STFT(Y);

[0056] Y basic=Encoder(|Y|);

[0057] Where STFT(.) represents the transformation extraction layer; Encoder(.) represents the speech encoder.

[0058] ②The voice enhancement part is used to basic Processing to obtain hidden features Y hidden , masking feature Y mask , mapping feature Y map .

[0059] like Figure 3 As shown, the speech enhancement unit can be designed to include: 1 S-TCM layer, 2 Linear layers, 2 ReLU layers, and 1 product layer;

[0060] In the Speech Enhancement section:

[0061] The S-TCM layer is used to basic Perform speech enhancement processing to obtain Y hidden . It should be noted that the S-TCM layer uses the Squeezed-Time Convolutional Module, which is a compressed version of the TCM (Time Convolutional Module). S-TCM differs from TCM mainly in two aspects: one is that in order to improve long-range modeling, the DD-Conv (depth dilated convolution) layer in TCM is replaced by the Dilated-Conv (traditional dilated convolution) layer; the other is that a gating branch is added and the Sigmoid activation function is used to control the output in the range of [0, 1]. S-TCM has about 72% fewer parameters than TCM while still providing comparable performance.

[0062] The first Linear layer is used to hidden Perform linear transformation;

[0063] The first ReLU layer is used to process the output of the first Linear layer through the ReLU activation function;

[0064] The product layer is used to multiply |Y| by the output of the first ReLU layer to obtain Y mask ;

[0065] The second Linear layer is used to hidden Perform linear transformation;

[0066] The second ReLU layer is used to process the output of the first Linear layer through the ReLU activation function to obtain Y map .

[0067] Of course, the processing process of the above-mentioned speech enhancement unit can also be expressed by the formula:

[0068] Y hidden =S-TCM(Y basic );

[0069] Y mask =|Y|⊙ReLU(Linear(Y hidden ));

[0070] Y map =ReLU(Linear(Y hidden ));

[0071] Where S-TCM(.) represents the S-TCM layer; Linear(.) represents the Linear layer; ReLU(.) represents the ReLU layer.

[0072] ③Attention fusion unit is used to mask 、Y map Fusion is performed to obtain the fusion feature Y MCF .

[0073] like Figure 4 As shown in the figure, the attention fusion unit can be designed to include: 6 Conv2D layers, 3 BN layers, 2 splicing layers, 1 LN layer, 2 MHSA layers, 1 residual connection layer, and 2 GELU layers.

[0074] In the attention fusion section:

[0075] The first Conv2D layer is used to mask Perform two-dimensional convolution processing;

[0076] The first BN layer is used to perform batch normalization on the output of the first Conv2D layer;

[0077] The second Conv2D layer is used to map Perform two-dimensional convolution processing;

[0078] The second BN layer is used to perform batch normalization on the output of the second Conv2D layer;

[0079] The first concatenation layer is used to concatenate the output of the first BN layer and the output of the second BN layer to obtain the concatenated feature Y concat1 ;

[0080] The LN layer is used for Y concat1 Perform layer normalization;

[0081] The third Conv2D layer is used to perform two-dimensional convolution on the output of the LN layer;

[0082] The first MHSA layer is used to use the multi-head self-attention mechanism to mask to process;

[0083] The second MHSA layer is used to use the multi-head self-attention mechanism to map to process;

[0084] The second concatenation layer is used to concatenate the output of the third Conv2D layer, the output of the first MHSA layer, and the output of the second MHSA layer to obtain the concatenated feature Y concat2 ;

[0085] The fourth Conv2D layer is used to concat2 Perform two-dimensional convolution processing;

[0086] The first GELU layer is used to process the output of the fourth Conv2D layer through the GELU activation function;

[0087] The residual connection layer is used to transform Y concat2 Superimpose with the output of the first GELU layer to obtain the fusion feature Y fusion ;

[0088] The third BN layer is used to fusion Perform batch normalization;

[0089] The fifth Conv2D layer is used to perform two-dimensional convolution on the output of the third BN layer;

[0090] The second GELU layer is used to process the output of the fifth Conv2D layer through the GELU activation function;

[0091] The sixth Conv2D layer is used to perform two-dimensional convolution on the output of the second GELU layer to obtain Y MCF .

[0092] Of course, the processing of the attention fusion part can also be expressed as follows:

[0093] Y MCF =MCF_Net(Y mask ,Y map );

[0094] Y concat1 =Conv2D(LN(Concat[BN(Conv2D(Y mask )),BN(Conv2D(Y map ))]));

[0095] Y concat2 =Concat[MHSA(Y mask),Y concat1 ,MHSA(Y map )];

[0096] Y fusion =GELU(Conv2D(Y concat2 ))+Y concat2 ;

[0097] Y MCF =Conv2D(GELU(Conv2D(BN(Y fusion ))));

[0098] Where MCF_Net(.) represents the attention fusion unit; Conv2D(.) represents the Conv2D layer; BN(.) represents the BN layer; Concat(.) represents the concatenation layer; LN(.) represents the LN layer; MHSA(.) represents the MHSA layer; GELU(.) represents the GELU layer.

[0099] In summary, Y is realized through the attention fusion part mask With Y map The first stage fusion of the proposed method enhances feature complementarity.

[0100] ④ The cross information fusion unit is used to combine |Y|, Y MCF Fusion is performed to obtain the fusion feature Y CIF .

[0101] like Figure 5 As shown in the figure, the cross-information fusion unit can be designed to include: 2 Fbank layers, 2 UpConv layers, 1 dual-branch network layer, 2 DwConv layers, and 1 merged network layer.

[0102] In the cross-information fusion section:

[0103] The first Fbank layer is used to MCF Perform Fbank feature extraction to obtain the intermediate feature F MCF ;

[0104] The first UpConv layer is used to MCF Perform up-sampling two-dimensional convolution to obtain the intermediate feature F enh ;

[0105] The second Fbank layer is used to extract Fbank features from |Y| to obtain the intermediate features F org ;

[0106] The second UpConv layer is used to org Perform up-sampling two-dimensional convolution to obtain the intermediate feature F nosiy ;

[0107] The dual-branch network layer is used to enh 、F nosiy Perform interactive information fusion to obtain the intermediate feature F enh_output 、F nosiy_output ;

[0108] The second DwConv layer is used to enh_output Perform down-sampled two-dimensional convolution to obtain enhanced features E output ;

[0109] The second DwConv layer is used to nosiy_output Perform downsampled two-dimensional convolution to obtain the noise feature N output ;

[0110] Merge network layers for F enh 、F nosiy 、E output 、N output Merge to get Y CIF .

[0111] Of course, the processing of the above cross-information fusion part can also be expressed by the formula:

[0112] Y CIF =CIF_Net(|Y|,Y MCF );

[0113] F MCF =Fbank(Y MCF );

[0114] F org =Fbank(|Y|);

[0115] F enh =UpConv(F MCF );

[0116] F noisy =UpConv(F org );

[0117] (F enh_output ,F nosiy_output )=Cross[F enh ,F nosiy ];

[0118] E output =DwConv(F enh_output );

[0119] N output =DwConv(F noisy_output );

[0120] Y CIF=Combine[F enh ,F nosiy ,E output ,N output ];

[0121] In the formula, CIF_Net(.) represents the cross information fusion part; Fbank(.) represents the Fbank layer; UpConv(.) represents the UpConv layer; Cross[.] represents the dual-branch network layer; DwConv(.) represents the DwConv layer; Combine[.] represents the merged network layer.

[0122] It should be noted that the above-mentioned dual-branch network layer and merged network layer are not simple designs:

[0123] 401, see Figure 6 The dual-branch network layer can be designed to include: 6 C-STFA (CondConv Sequential Time-Frequency Attention) layers and 6 DFIM (Dynamic Feature Interaction Module) layers.

[0124] In the dual-branch network layer:

[0125] The first C-STFA layer is used to enh Perform feature encoding and information modeling to obtain feature representation E1;

[0126] The second C-STFA layer is used to nosiy Perform feature encoding and information modeling to obtain feature representation N1;

[0127] The first DFIM layer is used to perform interactive feature fusion with E1 as the main input and N1 as the secondary input to obtain the feature representation E2;

[0128] The second DFIM layer is used to perform interactive feature fusion with N1 as the main input and E1 as the secondary input to obtain the feature representation N2;

[0129] The third C-STFA layer is used to perform feature encoding and information modeling on E2 to obtain feature representation E3;

[0130] The fourth C-STFA layer is used to perform feature encoding and information modeling on N2 to obtain feature representation N3;

[0131] The third DFIM layer is used to perform interactive feature fusion with E3 as the main input and N3 as the secondary input to obtain the feature representation E4;

[0132] The fourth DFIM layer is used to perform interactive feature fusion with N3 as the main input and E3 as the secondary input to obtain the feature representation N4;

[0133] The fifth C-STFA layer is used to perform feature encoding and information modeling on E4 to obtain feature representation E5;

[0134] The sixth C-STFA layer is used to perform feature encoding and information modeling on N4 to obtain feature representation N5;

[0135] The fifth DFIM layer is used to perform interactive feature fusion with E5 as the main input and N5 as the secondary input to obtain F enh_output ;

[0136] The sixth DFIM layer is used to perform interactive feature fusion with N5 as the main input and E5 as the secondary input to obtain F nosiy_output .

[0137] The above process of the dual-branch network layer can also be expressed as:

[0138] E1=C-STFA(F enh );

[0139] N1=C-STFA(F nosiy );

[0140] E2=DFIM[E1,N1];

[0141] N2=DFIM[N1,E1];

[0142] E3 = C-STFA (E2);

[0143] N3 = C-STFA (N2);

[0144] E4=DFIM[E3,N3];

[0145] N4=DFIM[N3,E3];

[0146] E5 = C-STFA (E4);

[0147] N5 = C-STFA (N4);

[0148] F enh_output =DFIM[E5,N5];

[0149] F nosiy_output =DFIM[N5,E5];

[0150] Where C-STFA(.) represents the C-STFA layer; DFIM[.] represents the DFIM layer.

[0151] Furthermore, the C-STFA layer and DFIM layer are also specially designed:

[0152] I, see Figure 7 ,The C-STFA layer is designed to include: 2 CondConv layers, 1 TMHSA layer, 1 FMHSA layer, and 2 Reshape layers.

[0153] Suppose the input of a C-STFA layer is in and the output is out. Then in the C-STFA layer: the first CondConv layer is used to perform conditional convolution on in; the second CondConv layer is used to perform conditional convolution on the output of the first CondConv layer; the TMHSA layer is used to process the output of the second CondConv layer through the temporal multi-head self-attention mechanism; the first Reshape layer is used to reshape the output of the TMHSA layer; the FMHSA layer is used to process the output of the first Reshape layer through the frequency multi-head self-attention mechanism; the second Reshape layer is used to reshape the output of the FMHSA layer to obtain out.

[0154] Of course, the above process of the C-STFA layer can also be expressed as:

[0155] out=Reshape(FMHSA(Reshape(TMHSA(CondConv(CondConv(in))))));

[0156] Where CondConv(.) represents the CondConv layer; TMHSA(.) represents the TMHSA layer; FMHSA(.) represents the FMHSA layer; and Reshape(.) represents the Reshape layer.

[0157] II, see Figure 8 ,The DFIM layer is designed to include: 1 splicing layer, 3 Conv layers, 1 GAP layer, 1 ReLU layer, and 2 Sigmoid layers.

[0158] Assume that the main input of a DFIM layer is X1, the secondary input is X2, and the output is Output. Then in any DFIM layer: the concatenation layer is used to concatenate X1 and X2 to obtain the concatenation feature cat; the first Conv layer is used to perform one-dimensional convolution on cat; the first Sigmoid layer is used to process the output of the first Conv layer through the Sigmoid activation function to obtain the spatial weight W s; The first GAP layer is used to perform global average pooling on cat; the second Conv layer is used to perform one-dimensional convolution on the output of the first GAP layer; the ReLU layer is used to process the output of the second Conv layer through the ReLU activation function; the third Conv layer is used to perform one-dimensional convolution on the output of the ReLU layer; the second Sigmoid layer is used to process the output of the third Conv layer through the Sigmoid activation function to obtain the channel weight W c ; The fusion layer is used to combine W s 、W c Fusion processing is performed on X1 and X2 to obtain Output.

[0159] The above process of the DFIM layer can also be expressed as:

[0160] cat=Concat[X1,X2];

[0161] W c =σ(Conv(ReLU(Conv(GAP(cat))))));

[0162] W s =σ(Conv(cat));

[0163] Output=fusion[W s ,W c ,X1,X2];

[0164] In the formula, Concat(.) represents the concatenation layer; GAP(.) represents the GAP layer; Conv(.) represents the Conv layer; ReLU(.) represents the ReLU layer; σ(.) represents the Sigmoid layer; fusion(.) represents the fusion layer.

[0165] 402, see Figure 9 , the merged network layer can be designed to include: 1 splicing layer, 2 Conv layers, 2 BN layers, 2 PReLU layers, 1 MHSA layer, 1 Sigmoid layer, and 1 weighted layer.

[0166] In the merge network layer:

[0167] The splicing layer is used to enh 、F nosiy 、E output 、N output Perform splicing to obtain splicing feature x merged ;

[0168] The first Conv layer is used to merged Perform one-dimensional convolution processing;

[0169] The first BN layer is used to perform batch normalization on the output of the first Conv layer;

[0170] The first PReLU layer is used to process the output of the first BN layer through the PReLU activation function;

[0171] The first MHSA layer is used to process the output of the first PReLU layer through the multi-head self-attention mechanism to obtain the intermediate feature Logits;

[0172] The second Conv layer is used to perform one-dimensional convolution on Logits;

[0173] The second BN layer is used to perform batch normalization on the output of the second Conv layer;

[0174] The second PReLU layer is used to process the output of the second BN layer through the PReLU activation function;

[0175] The Sigmoid layer is used to process the output of the second PReLU layer through the Sigmoid activation function to obtain the feature weight M;

[0176] The weighted layer is used to combine M to F enh 、F nosiy Perform weighted processing to obtain Y CIF ;

[0177] Among them, Y CIF =M⊙F enh +(1-M)⊙F noisy .

[0178] Of course, the above process of merging network layers can also be expressed as:

[0179] x merged =Concat[E output ,N output ,F enh ,F noisy ];

[0180] logits=MHSA(PReLU(BN(Conv(x merged ))));

[0181] M=σ(PReLU(BN(Conv(logits))));

[0182] Y CIF =Weighted[M,F enh ,F noisy ];

[0183] In the formula, Concat(.) represents the concatenation layer; Conv(.) represents the Conv layer; BN(.) represents the BN layer; PReLU(.) represents the PReLU layer; MHSA(.) represents the MHSA layer; σ(.) represents the Sigmoid layer; Weighted(.) represents the weighted layer.

[0184] In summary, the cross-information fusion department has realized Y mask With Y map The second stage fusion enhances feature interaction.

[0185] ⑤ The adaptive feature fusion unit is used to hidden 、Y CIF Fusion is performed to obtain the fusion feature F fused .

[0186] like Figure 10 As shown, the adaptive feature fusion unit can be designed to include:

[0187] 1 ASR encoder, 2 Linear layers, 1 ReLU layer, 1 Sigmoid layer, 1 addition layer, and 1 weighting layer;

[0188] In the adaptive feature fusion section:

[0189] The ASR encoder is used to convert Y CIF The coding layer abstract representation H ASR ; It should be noted that, in this embodiment 1, the ASR encoder recommends using an encoder of the E-Branchformer network.

[0190] The addition layer is used to add H ASR With Y hidden Add together to get the fusion feature F initial ;

[0191] The first Linear layer is used to transform H ASR Perform linear transformation;

[0192] The ReLU layer is used to process the output of the first Linear layer through the ReLU activation function;

[0193] The second Linear layer is used to linearly transform the output of the ReLU layer;

[0194] The Sigmoid layer is used to process the output of the second Linear layer through the Sigmoid activation function to obtain the confidence α;

[0195] The weighted layer is used to combine α with F initial 、H ASR Perform weighted processing to obtain F fused ;

[0196] Among them, F fused =α⊙F initial +(1-α)⊙H ASR .

[0197] Of course, the above process of the adaptive feature fusion part can also be expressed as follows:

[0198] F fused =AFF_Net(Y hidden ,Y CIF );

[0199] H ASR =ASR_Encoder(Y CIF );

[0200] F initial =H ASR +Y hidden ;

[0201] α=σ(ReLU(Linear(ReLU(Linear(H ASR )))));

[0202] F fused =Weighted(α,F initial ,H ASR );

[0203] Where ASR_Encoder(.) represents the ASR encoder; Linear(.) represents the Linear layer; ReLU(.) represents the ReLU layer; σ(.) represents the Sigmoid layer; Weighted(.) represents the weighted layer.

[0204] In summary, the adaptive feature fusion unit realizes Y hidden With Y CIF The third stage fusion enhances the semantic alignment of high and low layers.

[0205] ⑤ Voice decoder is used to fused Decode to get Result.

[0206] It should be noted that, in this embodiment 1, the speech decoder is recommended to adopt an encoder of the E-Branchformer network.

[0207] Of course, the processing of the speech decoder can be expressed as:

[0208] Result=Decoder(F fused );

[0209] Where Decoder(.) represents the speech decoder.

[0210] At this point, a complete speech recognition model is built.

[0211] It should be noted that the present invention requires the use of a trained speech recognition model with optimal network parameters and the best speech recognition effect.

[0212] It should be noted that when training the above speech recognition model, the joint loss function is used in, represents the speech recognition loss, which is a linear combination of CTC loss and cross entropy loss; The speech enhancement loss uses the mean squared error loss. This end-to-end joint optimization mechanism allows the entire model to be optimized under a unified objective, effectively avoiding performance bottlenecks caused by inconsistent front-end and back-end training objectives in traditional cascade systems.

[0213] Simulation Verification

[0214] This Example 1 simulates and verifies the above-mentioned method (abbreviated as Ours):

[0215] 1. Based on the AISHELL-1 dataset (an existing Chinese dataset), we designed different noise scenarios and introduced existing joint training methods (including Transformer-Joint-GRF, Transformer-Joint-MMF-GRF, and Conformer-Joint-MMF-IFF) for comparison to examine the CER performance of the model. The results are shown in Table 1.

[0216] CER stands for character error rate, which is a commonly used indicator in Chinese speech recognition. The lower the value, the better.

[0217] Table 1 Comparison of Chinese recognition results

[0218]

[0219] As shown in Table 1, under noise conditions ranging from 0dB to 20dB, Ours performs exceptionally well, achieving an average character error rate of only 8.46%. This result is a relative improvement of 21.88% over the previous best model (Conformer-Joint-MMF-IFF).

[0220] Under the challenging 0dB SNR, Ours achieves a CER reduction of 22.45% from 15.90% to 12.33% compared to Conformer-Joint-MMF-IFF.

[0221] It's worth noting that our CER at 20dB is lower than that of the clean condition. This is because "clean" test data (no artificial noise added) may contain natural variations in the recording environment (such as microphone noise floor, room reverberation, speaker breathing, etc.). This "implicit noise" not included in the training data may cause slight deviations in model recognition.

[0222] Ours also performs well under clean conditions, reducing the CER from 7.94% to 6.46% compared to Conformer-Joint-MMF-IFF, a reduction of 18.64%. This result demonstrates that Ours not only enhances noise immunity but also optimizes universal feature representation, leading to improved performance in ideal acoustic environments.

[0223] 2. Based on the Librispeech 100h dataset (an existing English dataset), different scenarios were designed, including both clean (test-clean) and noisy (test-other) environments.

[0224] Existing joint training methods (including Transformer-Joint-GRF, Transformer-Joint-MMF-GRF, and Conformer-Joint-MMF-IFF) are introduced for comparison to examine the WER performance of the model. The results are shown in Table 2.

[0225] WER stands for word error rate, which is a commonly used indicator in English speech recognition. The lower the value, the better.

[0226] Table 2 Comparison of English recognition results

[0227]

[0228] Under noise conditions ranging from 0dB to 20dB, our average word error rate is 15.39%, which is a relative improvement of 22.12% over the previous best model (Conformer-Joint-MMF-IFF).

[0229] In the test-clean environment, our approach achieved a WER of 12.54%, significantly lower than the 16.94% achieved by Conformer-Joint-MMF-IFF, representing a relative improvement of 25.97%. This superior performance in noise-free conditions demonstrates that the fusion architecture not only enhances noise immunity but also significantly improves general feature learning. By effectively capturing language structure and semantic dependencies, the model demonstrates enhanced speech understanding and recognition capabilities even in the absence of noise.

[0230] For the test-other environment, which contains inherent background noise and low speech volume, Ours achieves a WER reduction from 34.06% to 28.87% compared to Conformer-Joint-MMF-IFF. This significant reduction demonstrates the superior adaptability of Ours in real-world noisy scenarios.

[0231] In summary, simulation verification shows that this method has reached the current best level in terms of performance indicators, illustrates the robustness and effectiveness of the invention, and demonstrates its cross-language versatility.

[0232] Example 2

[0233] This embodiment 2 provides a robust speech recognition system based on multi-stage feature fusion, which uses the robust speech recognition method based on multi-stage feature fusion disclosed in embodiment 1.

[0234] The robust speech recognition system based on multi-stage feature fusion includes: a speech acquisition module and a speech recognition module.

[0235] The speech acquisition module is used to acquire noisy speech Y.

[0236] The speech recognition module is used to input Y into the trained speech recognition model for processing to obtain the recognition result Result.

[0237] Since this system uses the robust speech recognition method based on multi-stage feature fusion in Example 1, it also has the same effect and will not be repeated here.

[0238] Example 3

[0239] This embodiment 3 discloses a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the robust speech recognition method based on multi-stage feature fusion disclosed in embodiment 1.

[0240] Computer devices may include: mobile terminals and fixed terminals. Examples of the former include mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (such as in-vehicle navigation terminals); examples of the latter include digital TVs and desktop computers.

[0241] This embodiment 3 also discloses a readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps of the robust speech recognition method based on multi-stage feature fusion disclosed in embodiment 1 are executed.

[0242] Among them, the readable storage medium may include, but is not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0243] This embodiment 3 further discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the robust speech recognition method based on multi-stage feature fusion disclosed in embodiment 1 are implemented.

[0244] It should be noted that the computer program for executing the above-mentioned program can be written in one or more programming languages ​​or a combination thereof. Among them, the programming language includes object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages ​​such as "C" language or similar programming languages. The above-mentioned computer program can be executed completely on the user's computer, or partially on the user's computer, or partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0245] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A robust speech recognition method based on multi-stage feature fusion, comprising: Step 1: Acquire noisy speech Y; characterized by further comprising: Step 2: Input Y into the trained speech recognition model for processing to obtain the recognition result Result; wherein the speech recognition model includes: The speech coding unit is used to extract the amplitude spectrum |Y| from Y and encode it into the preliminary feature Y basic ; The speech enhancement unit is used to basic Processing to obtain hidden features Y hidden , masking feature Y mask , mapping feature Y map ; Attention fusion unit, which is used to mask 、Y map Fusion is performed to obtain the fusion feature Y MCF ; Cross-information fusion unit, which is used to combine |Y|, Y MCF Fusion is performed to obtain the fusion feature Y CIF ; Adaptive feature fusion unit, which is used to hidden 、Y CIF Fusion is performed to obtain the fusion feature F fused ;as well as Speech decoder, which is used to fused Decode to get Result.

2. The robust speech recognition method based on multi-stage feature fusion according to claim 1, characterized in that: The attention fusion unit includes: 6 Conv2D layers, 3 BN layers, 2 splicing layers, 1 LN layer, 2 MHSA layers, 1 residual connection layer, and 2 GELU layers; In the attention fusion section: The first Conv2D layer is used to mask Perform two-dimensional convolution processing; The first BN layer is used to perform batch normalization on the output of the first Conv2D layer; The second Conv2D layer is used to map Perform two-dimensional convolution processing; The second BN layer is used to perform batch normalization on the output of the second Conv2D layer; The first concatenation layer is used to concatenate the output of the first BN layer and the output of the second BN layer to obtain the concatenated feature Y concat1 ; The LN layer is used for Y concat1 Perform layer normalization; The third Conv2D layer is used to perform two-dimensional convolution on the output of the LN layer; The first MHSA layer is used to use the multi-head self-attention mechanism to mask to process; The second MHSA layer is used to use the multi-head self-attention mechanism to map to process; The second concatenation layer is used to concatenate the output of the third Conv2D layer, the output of the first MHSA layer, and the output of the second MHSA layer to obtain the concatenated feature Y concat2 ; The fourth Conv2D layer is used to concat2 Perform two-dimensional convolution processing; The first GELU layer is used to process the output of the fourth Conv2D layer through the GELU activation function; The residual connection layer is used to transform Y concat2 Superimpose with the output of the first GELU layer to obtain the fusion feature Y fusion ; The third BN layer is used to fusion Perform batch normalization; The fifth Conv2D layer is used to perform two-dimensional convolution on the output of the third BN layer; The second GELU layer is used to process the output of the fifth Conv2D layer through the GELU activation function; The sixth Conv2D layer is used to perform two-dimensional convolution on the output of the second GELU layer to obtain Y MCF .

3. The robust speech recognition method based on multi-stage feature fusion according to claim 1, characterized in that: The cross-information fusion unit includes: 2 Fbank layers, 2 UpConv layers, 1 dual-branch network layer, 2 DwConv layers, and 1 merge network layer; In the cross-information fusion section: The first Fbank layer is used to MCF Perform Fbank feature extraction to obtain the intermediate feature F MCF ; The first UpConv layer is used to MCF Perform up-sampling two-dimensional convolution to obtain the intermediate feature F enh ; The second Fbank layer is used to extract Fbank features from |Y| to obtain the intermediate features F org ; The second UpConv layer is used to org Perform up-sampling two-dimensional convolution to obtain the intermediate feature F nosiy ; The dual-branch network layer is used to enh 、F nosiy Perform interactive information fusion to obtain the intermediate feature F enh_output 、F nosiy_output ; The second DwConv layer is used to enh_output Perform down-sampled two-dimensional convolution to obtain enhanced features E output ; The second DwConv layer is used to nosiy_output Perform downsampled two-dimensional convolution to obtain the noise feature N output ; Merge network layers for F enh 、F nosiy 、E output 、N output Merge to get Y CIF .

4. The robust speech recognition method based on multi-stage feature fusion according to claim 3, characterized in that: The dual-branch network layer includes: 6 C-STFA layers and 6 DFIM layers; In the dual-branch network layer: The first C-STFA layer is used to enh Perform feature encoding and information modeling to obtain feature representation E1; The second C-STFA layer is used to nosiy Perform feature encoding and information modeling to obtain feature representation N1; The first DFIM layer is used to perform interactive feature fusion with E1 as the main input and N1 as the secondary input to obtain the feature representation E2; The second DFIM layer is used to perform interactive feature fusion with N1 as the main input and E1 as the secondary input to obtain the feature representation N2; The third C-STFA layer is used to perform feature encoding and information modeling on E2 to obtain feature representation E3; The fourth C-STFA layer is used to perform feature encoding and information modeling on N2 to obtain feature representation N3; The third DFIM layer is used to perform interactive feature fusion with E3 as the main input and N3 as the secondary input to obtain the feature representation E4; The fourth DFIM layer is used to perform interactive feature fusion with N3 as the main input and E3 as the secondary input to obtain the feature representation N4; The fifth C-STFA layer is used to perform feature encoding and information modeling on E4 to obtain feature representation E5; The sixth C-STFA layer is used to perform feature encoding and information modeling on N4 to obtain feature representation N5; The fifth DFIM layer is used to perform interactive feature fusion with E5 as the main input and N5 as the secondary input to obtain F enh_output ; The sixth DFIM layer is used to perform interactive feature fusion with N5 as the main input and E5 as the secondary input to obtain F nosiy_output .

5. The robust speech recognition method based on multi-stage feature fusion according to claim 4, characterized in that: The C-STFA layer includes: 2 CondConv layers, 1 TMHSA layer, 1 FMHSA layer, and 2 Reshape layers; In any C-STFA layer: the first CondConv layer is used to perform conditional convolution on the input in of the C-STFA layer; the second CondConv layer is used to perform conditional convolution on the output of the first CondConv layer; the TMHSA layer is used to process the output of the second CondConv layer through the temporal multi-head self-attention mechanism; the first Reshape layer is used to reshape the output of the TMHSA layer; the FMHSA layer is used to process the output of the first Reshape layer through the frequency multi-head self-attention mechanism; the second Reshape layer is used to reshape the output of the FMHSA layer to obtain the output out of the C-STFA layer.

6. The robust speech recognition method based on multi-stage feature fusion according to claim 4, characterized in that: The DFIM layer includes: 1 splicing layer, 3 Conv layers, 1 GAP layer, 1 ReLU layer, and 2 Sigmoid layers; In any DFIM layer: the concatenation layer is used to concatenate the main input X1 and the secondary input X2 of the DFIM layer to obtain the concatenated feature cat; the first Conv layer is used to perform one-dimensional convolution on cat; the first Sigmoid layer is used to process the output of the first Conv layer through the Sigmoid activation function to obtain the spatial weight W s ; The first GAP layer is used to perform global average pooling on cat; the second Conv layer is used to perform one-dimensional convolution on the output of the first GAP layer; the ReLU layer is used to process the output of the second Conv layer through the ReLU activation function; the third Conv layer is used to perform one-dimensional convolution on the output of the ReLU layer; the second Sigmoid layer is used to process the output of the third Conv layer through the Sigmoid activation function to obtain the channel weight W c ; The fusion layer is used to combine W s 、W c Fuse X1 and X2 to obtain the output of the DFIM layer; in, 7. The robust speech recognition method based on multi-stage feature fusion according to claim 3, characterized in that: The merged network layers include: 1 concatenation layer, 2 Conv layers, 2 BN layers, 2 PReLU layers, 1 MHSA layer, 1 Sigmoid layer, and 1 weighted layer; In the merged network layer: The splicing layer is used to enh 、F nosiy 、E output 、N output Perform splicing to obtain splicing feature x merged ; The first Conv layer is used to merged Perform one-dimensional convolution processing; The first BN layer is used to perform batch normalization on the output of the first Conv layer; The first PReLU layer is used to process the output of the first BN layer through the PReLU activation function; The first MHSA layer is used to process the output of the first PReLU layer through the multi-head self-attention mechanism to obtain the intermediate feature Logits; The second Conv layer is used to perform one-dimensional convolution on Logits; The second BN layer is used to perform batch normalization on the output of the second Conv layer; The second PReLU layer is used to process the output of the second BN layer through the PReLU activation function; The Sigmoid layer is used to process the output of the second PReLU layer through the Sigmoid activation function to obtain the feature weight M; The weighted layer is used to combine M to F enh 、F nosiy Perform weighted processing to obtain Y CIF ; Among them, Y CIF =M⊙F enh +(1-M)⊙F noisy .

8. The robust speech recognition method based on multi-stage feature fusion according to claim 1, characterized in that: The adaptive feature fusion unit includes: 1 ASR encoder, 2 Linear layers, 1 ReLU layer, 1 Sigmoid layer, 1 addition layer, and 1 weighting layer; In the adaptive feature fusion section: The ASR encoder is used to convert Y CIF The coding layer abstract representation H ASR ; The addition layer is used to add H ASR With Y hidden Add together to get the fusion feature F initial ; The first Linear layer is used to transform H ASR Perform linear transformation; The ReLU layer is used to process the output of the first Linear layer through the ReLU activation function; The second Linear layer is used to linearly transform the output of the ReLU layer; The Sigmoid layer is used to process the output of the second Linear layer through the Sigmoid activation function to obtain the confidence α; The weighted layer is used to combine α with F initial 、H ASR Perform weighted processing to obtain F fused ; Among them, F fused =α⊙F initial +(1-α)⊙H ASR .

9. A robust speech recognition system based on multi-stage feature fusion, characterized in that: It uses the robust speech recognition method based on multi-stage feature fusion as described in any one of claims 1 to 8; The robust speech recognition system based on multi-stage feature fusion includes: A speech acquisition module, which is used to acquire noisy speech Y; as well as The speech recognition module is used to input Y into the trained speech recognition model for processing to obtain the recognition result Result.

10. A computer program product, characterized in that The method comprises a computer program; when the computer program is executed by a processor, the method implements the steps of the robust speech recognition method based on multi-stage feature fusion as described in any one of claims 1 to 8.

Images