Noise-robust streaming speech recognition method and model integrated with speech enhancement front end

Abstract

The present application provides a noise-robust streaming speech recognition method and model integrated with a speech enhancement front end. The training method of the model includes obtaining a speech recognition training set; using a causal speech enhancement network to process speech training samples to obtain target filter group features containing global information; using a filter group-based feature extraction method to process speech training samples to obtain original filter group features; using an attention mechanism network to perform dynamic weighted fusion processing on the target filter group features and the original filter group features to obtain target fusion features containing supplementary information in the speech training samples; using an initial speech recognition network to process the target fusion features to obtain training text recognition information; generating a target loss result based on the target filter group features, the training text recognition information and the recognition text label; iteratively adjusting the network parameters of the initial speech recognition model based on the target loss result to obtain a speech enhancement recognition model.

Description

Technical Field

[0001] The present application relates to the field of speech recognition and neural network technology, and more specifically, to a training method for a speech enhancement recognition model, a speech enhancement recognition method, a training device for a speech enhancement recognition model, a speech enhancement recognition device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Streaming Automatic Speech Recognition (ASR) is a technology that can transcribe speech signals into text in real time while they are being input. It is widely used in scenarios such as intelligent voice assistants, real-time translation, and conference records. In recent years, with the rapid progress of acoustic models based on neural networks and large-scale training technologies, streaming ASR technology has made significant progress. Compared with the traditional GMM-HMM method, the end-to-end model based on neural networks not only simplifies the training process, but also has obvious advantages in recognition performance and can better handle the time series characteristics of speech. However, although streaming ASR has shown ideal recognition accuracy under clean speech conditions, its recognition accuracy still drops significantly under complex noise backgrounds or low signal-to-noise ratio (SNR) conditions. Real voice interaction scenarios usually contain complex and variable noise sources. How to improve the robustness of streaming ASR under noise conditions has become a difficult problem that needs to be solved in this field. Summary of the invention

[0003] In view of this, the present application provides a training method for a speech enhancement recognition model, a speech enhancement recognition method, a training device for a speech enhancement recognition model, a speech enhancement recognition device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] One aspect of the present application provides a method for training a speech enhancement recognition model, comprising:

[0005] In response to the training instruction, a speech recognition training set is obtained, wherein the speech recognition training set includes a plurality of speech training samples and a recognition text label corresponding to each of the speech training samples;

[0006] For each of the speech training samples, the speech training samples are processed using a causal speech enhancement network to obtain target filter group features containing global information;

[0007] The speech training samples are processed by using a filter bank-based feature extraction method to obtain original filter bank features;

[0008] The attention mechanism network is used to dynamically weight and fuse the target filter group features and the original filter group features to obtain the target fusion features containing the supplementary information in the speech training samples.

[0009] Using an initial speech recognition network to process the target fusion features to obtain training text recognition information, wherein the initial speech recognition model includes the causal speech enhancement network, the attention mechanism network and the initial speech recognition network;

[0010] Generate a target loss result according to the target filter group features, the training text recognition information and the recognition text label;

[0011] The network parameters of the initial speech recognition model are iteratively adjusted according to the target loss result to obtain a trained speech enhancement recognition model.

[0012] Another aspect of the present application provides a speech enhancement recognition method, comprising:

[0013] In response to the voice recognition instruction, obtaining voice information to be recognized;

[0014] The above-mentioned speech information to be recognized is input into the speech enhancement recognition model, and the target text recognition information is output.

[0015] Another aspect of the present application provides a training device for a speech enhancement recognition model, comprising:

[0016] A first acquisition module, configured to acquire a speech recognition training set in response to a training instruction, wherein the speech recognition training set includes a plurality of speech training samples and a recognition text label corresponding to each of the speech training samples;

[0017] A causal enhancement module is used to process each of the speech training samples using a causal speech enhancement network to obtain a target filter group feature containing global information;

[0018] Attention fusion module, used to:

[0019] Processing the speech training sample using a filter bank-based feature extraction method to obtain the original filter bank features;

[0020] Using an attention mechanism network to perform dynamic weighted fusion processing on the target filter group features and the original filter group features to obtain target fusion features containing supplementary information in the speech training sample;

[0021] A speech recognition module, used to process the target fusion features using an initial speech recognition network to obtain training text recognition information, wherein the initial speech recognition model includes the causal speech enhancement network, the attention mechanism network and the initial speech recognition network;

[0022] A loss calculation module, used to generate a target loss result according to the target filter group characteristics, the training text recognition information and the recognition text label;

[0023] The iterative training module is used to iteratively adjust the network parameters of the initial speech recognition model according to the target loss result to obtain a trained speech enhancement recognition model.

[0024] Another aspect of the present application provides a speech enhancement recognition device, comprising:

[0025] A second acquisition module, used for acquiring speech information to be recognized in response to a training instruction;

[0026] The enhanced recognition module is used to input the above-mentioned voice information to be recognized into the voice enhancement recognition model and output the target text recognition information.

[0027] Another aspect of the present application provides an electronic device, comprising:

[0028] one or more processors;

[0029] a memory for storing one or more programs,

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0031] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0032] Another aspect of the present application provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the method as described above when executed.

[0033] According to an embodiment of the present application, based on the target filter group features containing global information output by the causal speech enhancement network, the original filter group features containing rich background information and speech details are fused with the target filter group features using a noise fusion network, and finally the target fusion features are recognized by the initial speech recognition network to obtain training text recognition information. This can effectively suppress noise under real-time requirements and avoid speech distortion problems caused by related technologies. At the same time, the use of the noise fusion network further reduces the distortion effect caused by enhancement, effectively improving the recognition accuracy. In addition, the speech enhancement recognition model has excellent noise robustness, making it more applicable in complex and changeable real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0035] Figure 1 An exemplary system architecture to which a training method for a speech enhancement recognition model or a speech enhancement recognition method can be applied according to an embodiment of the present application is shown;

[0036] Figure 2 A flow chart showing a method for training a speech enhancement recognition model according to an embodiment of the present application is shown;

[0037] Figure 3 A data processing flow chart of a causal speech enhancement network according to an embodiment of the present application is shown;

[0038] Figure 4 A flow chart showing a method for training a speech enhancement recognition model according to another embodiment of the present application is shown;

[0039] Figure 5 A data processing flow chart of an attention convolution encoder according to an embodiment of the present application is shown;

[0040] Figure 6 A flow chart of a speech enhancement recognition method according to an embodiment of the present application is shown;

[0041] Figure 7 A block diagram of a training device for a speech enhancement recognition model according to an embodiment of the present application is shown;

[0042] Figure 8 A block diagram of a speech enhancement recognition device according to an embodiment of the present application is shown;

[0043] Fig. 9 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.

[0045] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0046] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0047] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0048] To address the noise robustness problem of streaming ASR, the Speech Enhancement (SE) module is usually used as a front-end processing unit to suppress noise. The SE module can process the noise in the input speech signal before the ASR module processes the speech, thereby improving the input signal quality of the ASR. Common SE methods include time domain and frequency domain processing techniques. The time domain method directly reconstructs the waveform of the target speech to achieve the purpose of noise reduction, while the frequency domain method relies on the spectrum mask matrix to calculate the spectrum of the target speech, thereby effectively suppressing the noise. Although the SE module can significantly improve speech clarity in some scenarios, when it is trained separately from the ASR module, since the training objectives of SE are usually related to speech quality (such as signal-to-noise ratio, mean square error, etc.), this independent training may introduce speech distortion, making the enhanced speech not completely consistent with the real speech features, resulting in an increase in the word error rate (WER) of ASR, limiting the overall performance of the system.

[0049] In recent years, in order to solve the performance mismatch problem between SE and ASR modules, scholars have proposed a joint training framework, which puts SE and ASR modules under the same model for integrated optimization to reduce the distortion introduced in the enhancement process and improve the recognition effect of ASR. For example, through joint training, the mask-based enhancement network and the attention-based ASR encoder-decoder can adapt to each other and obtain better recognition effects in noisy environments. However, in the existing joint training methods, most SE modules use non-causal designs, which makes them unable to meet the real-time requirements of streaming ASR. Non-causal SE modules usually rely on future frame information for noise reduction, which is not practical in real-time applications.

[0050] In addition, traditional joint models still have limitations when facing complex and diverse noise scenarios. For example, some enhancement modules may introduce additional speech distortion when suppressing specific noise, and this distortion is often not completely compensated by the subsequent ASR module. In addition, most joint models fail to effectively utilize the original noise information, resulting in the enhanced speech features, although the quality is improved, lacking the environmental context information retained in the original signal, thus affecting the final recognition effect. Therefore, how to reduce the distortion introduced by enhancement while maintaining real-time performance and make full use of noise information to improve recognition results has become a key challenge in the current development of streaming ASR.

[0051] In view of this, an embodiment of the present application provides a noise-robust streaming speech recognition method and model integrated with a speech enhancement front end, wherein the training method of the model includes obtaining a speech recognition training set in response to a training instruction, wherein the speech recognition training set includes multiple speech training samples and a recognition text label corresponding to each speech training sample; for each speech training sample, using a causal speech enhancement network to process the speech training sample to obtain a target filter group feature containing global information; using a filter group-based feature extraction method to process the speech training sample to obtain an original filter group feature; using an attention mechanism network to dynamically weighted fuse the target filter group feature and the original filter group feature to obtain a target fused feature containing supplementary information in the speech training sample; using an initial speech recognition network to process the target fused feature to obtain training text recognition information, wherein the initial speech recognition model includes a causal speech enhancement network, an attention mechanism network and an initial speech recognition network; generating a target loss result according to the target filter group feature, the training text recognition information and the recognition text label; iteratively adjusting the network parameters of the initial speech recognition model according to the target loss result to obtain a trained speech enhancement recognition model.

[0052] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0053] In the embodiments of the present application, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0054] Figure 1 An exemplary system architecture 100 to which a training method for a speech enhancement recognition model or a speech enhancement recognition method according to an embodiment of the present application can be applied is shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0055] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0056] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive voice information or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0057] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0058] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0059] It should be noted that the training method or speech enhancement recognition method of the speech enhancement recognition model provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the training device or speech enhancement recognition device of the speech enhancement recognition model provided in the embodiment of the present application can generally be arranged in the server 105. The training method or speech enhancement recognition method of the speech enhancement recognition model provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the training device or speech enhancement recognition device of the speech enhancement recognition model provided in the embodiment of the present application can also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the training method of the speech enhancement recognition model or the speech enhancement recognition method provided in the embodiment of the present application may also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the training device of the speech enhancement recognition model or the speech enhancement recognition device provided in the embodiment of the present application may also be provided in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may be provided in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0060] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0061] Figure 2 A flowchart of a method for training a speech enhancement recognition model according to an embodiment of the present application is shown.

[0062] like Figure 2 As shown, the training method of the speech enhancement recognition model includes operations S201~S207.

[0063] In operation S201, in response to a training instruction, a speech recognition training set is obtained, wherein the speech recognition training set includes a plurality of speech training samples and a recognition text label corresponding to each speech training sample;

[0064] In operation S202, for each speech training sample, the speech training sample is processed using a causal speech enhancement network to obtain a target filter bank feature containing global information;

[0065] In operation S203, the speech training sample is processed using a filter bank-based feature extraction method to obtain original filter bank features;

[0066] In operation S204, the target filter group features and the original filter group features are dynamically weighted and fused using the attention mechanism network to obtain target fusion features containing supplementary information in the speech training sample;

[0067] In operation S205, the target fusion feature is processed by using the initial speech recognition network to obtain training text recognition information, wherein the initial speech recognition model includes a causal speech enhancement network, an attention mechanism network and an initial speech recognition network;

[0068] In operation S206, a target loss result is generated according to the target filter group feature, the training text recognition information and the recognition text label;

[0069] In operation S207, the network parameters of the initial speech recognition model are iteratively adjusted according to the target loss result to obtain a trained speech enhancement recognition model.

[0070] According to an embodiment of the present application, the training instruction may be a corresponding operation input by a user on an electronic device such as a mobile phone or a computer, and the electronic device automatically generates a training instruction in response to the operation.

[0071] According to an embodiment of the present application, the speech training sample may be a segment captured from the streaming speech, and the recognition text label may be the text content corresponding to the speech training sample.

[0072] According to an embodiment of the present application, the causal speech enhancement network achieves real-time noise suppression through causal design, and generates high-quality features adapted to streaming speech recognition, namely, target filter group features containing global information. The causal speech enhancement network only relies on the current frame and historical frame information for processing, ensuring low latency while significantly improving noise reduction performance. The target filter group features output by the causal speech enhancement network have high speech clarity.

[0073] According to an embodiment of the present application, the target filter group features output by the causal speech enhancement network are input into the noise fusion network, and the noise fusion network is intended to utilize the attention fusion mechanism to combine the original noise features (i.e., the original filter group features corresponding to the speech training samples) with the enhanced FBank features (i.e., the target filter group features) to reduce the distortion that may be introduced in the speech enhancement process and further improve the speech recognition performance, thereby obtaining the target fusion features. Specifically, the original filter group features retain the rich background information and speech details in the original noise (speech training samples), and the target fusion features outputted thereby can provide more possible contextual support for subsequent processing. By combining the target filter group features and the original filter group features, the enhanced clear speech features (i.e., the target filter group features) and the supplementary information that may be carried in the original noise (i.e., the speech training samples) can be effectively utilized to compensate for some speech distortion problems that may be caused during the enhancement process.

[0074] According to an embodiment of the present application, the noise fusion network performs operation S203 and operation S204, and the specific processing flow is as follows:

[0075] The specific process of the filter bank feature extraction method is: pre-emphasize the speech training samples to extract the frequency features, segment, window, and fast Fourier transform the frequency features in sequence to obtain the spectrum features, calculate the Mel eigenvalues ​​of the spectrum features, and take the logarithm of the Mel eigenvalues ​​to obtain the original filter bank features.

[0076] In the process of feature fusion of the attention mechanism network, the attention mechanism network uses the Attention mechanism to dynamically weight the target filter group features and the original filter group features. The Attention mechanism dynamically assigns weights according to the importance of each frame feature, captures the correlation between features, highlights the weight ratio of key features, and suppresses irrelevant noise or redundant information to obtain the target fusion features. Specifically, the attention mechanism network calculates the degree of match between each frame feature and the context through self-attention, thereby generating an attention distribution for weighted fusion. Enhanced features are usually given higher weights in the main speech area, while the original noise features provide supplementary information in complex scenarios to achieve a balance between speech detail preservation and noise suppression.

[0077] The final output fusion feature contains both enhanced clarity and a portion of the original noise background information. This fusion method not only reduces the distortion that may be introduced by the speech enhancement module, but also enhances the model's ability to understand the background noise context, enabling the streaming speech recognition module to adapt more effectively to complex noise environments. Experiments show that this fusion feature can improve speech recognition accuracy and robustness while performing well in low signal-to-noise ratio scenarios, providing higher reliability and stability for real-time speech recognition tasks.

[0078] The original filter bank features required by the noise fusion network are generated using a filter bank-based feature extraction method, that is, the original filter bank features are generated through the spectrogram. The specific parameter settings include 80 Mel frequency bandwidth, 10 ms frame shift, and 25 ms frame length, which can effectively capture the time-frequency characteristics of the speech signal. In addition, speed perturbation and spectrum enhancement (SpecAugment) are enabled for data preprocessing. The data enhancement parameters include two time masks, two frequency masks, a maximum time mask of 50 frames, and a maximum frequency mask of 10 bands. These configurations retain the key information of the speech while improving the robustness of the model, and enhance the system's ability to cope with different noise and speed-changing environments.

[0079] According to an embodiment of the present application, the target fusion features output by the noise fusion network are input into the initial speech recognition network for recognition, thereby obtaining the training text recognition information. According to the target filter group features, the training text recognition information and the recognition text label, the target loss result is calculated, and then the network parameters of the initial speech recognition model are iteratively adjusted based on the target loss result, so as to obtain the trained speech enhancement recognition model.

[0080] According to an embodiment of the present application, based on the target filter group features containing global information output by the causal speech enhancement network, the original filter group features containing rich background information and speech details are fused with the target filter group features using a noise fusion network, and finally the target fusion features are recognized by the initial speech recognition network to obtain training text recognition information. This can effectively suppress noise under real-time requirements and avoid speech distortion problems caused by related technologies. At the same time, the use of the noise fusion network further reduces the distortion effect caused by enhancement, effectively improving the recognition accuracy. In addition, the speech enhancement recognition model has excellent noise robustness, making it more applicable in complex and changeable real-world scenarios.

[0081] Figure 3 A data processing flow chart of a causal speech enhancement network according to an embodiment of the present application is shown.

[0082] According to the embodiments of the present application, Figure 3As shown, the causal speech enhancement network is used to process speech training samples to obtain target filter bank features containing global information, including:

[0083] Performing Fourier transform processing on the speech training sample to obtain spectrum features, wherein the spectrum features include amplitude information, real phase information and imaginary phase information;

[0084] For multiple feature information in the spectrum features, a channel extraction network is used to process the multiple feature information to obtain a channel feature set;

[0085] Processing the channel feature set using the first subband model to obtain a real feature vector and an imaginary feature vector, wherein the first subband model is constructed based on a long short-term memory network and a fully connected network;

[0086] Generate target filter bank features based on the real eigenvector and the imaginary eigenvector.

[0087] According to an embodiment of the present application, the input speech signal is subjected to a short-time Fourier transform (STFT) to extract spectral features, including amplitude X m , phase real part information X r and the imaginary phase information X i , laying the foundation for subsequent noise reduction processing.

[0088] According to the embodiment of the present application, for multiple feature information in the spectral features, a channel extraction network is used to process the multiple feature information to obtain a context-related channel feature set, and then the first sub-band model is used in each frequency band to perform sub-band modeling on the channel feature set to capture local spectral details and improve performance in different noise environments, thereby obtaining a real feature vector M r and the imaginary eigenvector M i .

[0089] According to an embodiment of the present application, the enhanced real eigenvector M r and the imaginary eigenvector M i Converted into the target filter bank feature FBank to be compatible with the initial speech recognition network. Through this causal speech enhancement network, efficient and low-latency real-time noise suppression can be achieved, providing high-quality feature input for subsequent speech recognition.

[0090] According to the embodiments of the present application, Figure 3 As shown, the channel extraction network is used to process multiple feature information to obtain a channel feature set, including:

[0091] For any feature information in the spectral features, the channel attention mechanism layer is used to process the feature information to obtain the first weighted feature vector;

[0092] For any first weighted feature vector, use a temporal convolutional network to process the weighted feature vector to obtain a second weighted feature vector containing context information;

[0093] Processing the first weighted feature vector corresponding to the amplitude information using a second subband model to obtain a frequency prediction vector, wherein the second subband model is constructed based on a long short-term memory network and a fully connected network;

[0094] A channel feature set is constructed according to the multiple second weighted feature vectors and the frequency prediction vector.

[0095] According to an embodiment of the present application, the channel attention mechanism layer can be a lightweight channel attention mechanism (MulCA) layer, such as a multi-scale time-sensitive channel attention layer. It can weight different frequency channels, dynamically adjust the model's attention to specific frequency bands, and improve the noise suppression effect, thereby obtaining the first weighted feature vector, that is, Figure 3 In , , .

[0096] According to an embodiment of the present application, the temporal convolutional network may be a full-band extractor (TCN), which captures the global characteristics of the spectrum through a convolutional network and identifies the long-range dependencies between frequencies to adapt to complex noise environments, thereby obtaining a second weighted feature vector corresponding to amplitude information, real phase information, and imaginary phase information, namely Figure 3 In , , .

[0097] According to an embodiment of the present application, for the first weighted feature vector corresponding to the amplitude information, the frequency prediction vector can be obtained by processing with the second subband model. , and then construct a channel feature set based on multiple second weighted feature vectors and frequency prediction vectors.

[0098] In a specific embodiment, the first sub-band model and the second sub-band model may each include two long short-term memory networks connected in sequence and one fully connected network.

[0099] In a specific embodiment, a 32 ms (512 points) Hanning window and a frame shift of 16 ms are used to convert the signal to the short-time Fourier transform (STFT) domain. The causal speech enhancement network used in the experiment has 257 channels in the channel attention mechanism (MulCA) layer, and the convolution kernel sizes of the parallel 1-D depth convolution are {3, 5, 10}. Each full-band feature extractor contains 2 groups of TCN modules, each group contains 4 TCN modules, the convolution kernel size is 3, and the expansion rates are {1, 2, 5, 9}. Each LSTM layer in the sub-band model contains 384 hidden units. The Adam optimizer is used for training the speech enhancement recognition model, and the initial learning rate is 1e-3. For the time delay parameter τ, it is set to 2, which is equivalent to using the information of the next two frames (32 ms in total) when enhancing the current frame. In the sub-band model, the parameter n is set to 15. The causal speech enhancement network can effectively improve the clarity of the speech signal in a complex noise environment, and provide optimized input for the subsequent feature extraction and recognition modules.

[0100] Figure 4 A flowchart of a method for training a speech enhancement recognition model according to another embodiment of the present application is shown.

[0101] like Figure 4 As shown, the initial speech recognition network is used to process the target fusion features to obtain the training text recognition information, including:

[0102] The attention convolution encoder is used to process the target fusion features to obtain the attention encoding features containing time information and frequency information;

[0103] Use the temporal decoder to process the attention encoding features and obtain the temporal decoded text sequence;

[0104] The attention encoding features and the temporal decoding text sequence are rescored and decoded using an attention decoder to obtain an attention decoding text sequence, wherein the training text recognition information includes the temporal decoding text sequence and the attention decoding text sequence.

[0105] According to an embodiment of the present application, the attention convolution encoder may be a branchformer encoder, an E-branchformer encoder or other encoders capable of achieving the same function, and the embodiment of the present application preferably uses an E-branchformer encoder. The temporal decoder may be a CTC (Connectionist Temporal Classification) decoder.

[0106] According to an embodiment of the present application, the initial speech recognition network is composed of a shared E-branchformer encoder, a CTC decoder and an attention decoder, which is a core component for realizing efficient and real-time speech recognition. The shared E-branchformer encoder undertakes the key task of feature extraction. By multi-level encoding of the input fusion features, the time and frequency information in the speech signal is captured to obtain the attention coding features. The E-branchformer encoder can utilize the deep structure of 17-layer coding blocks, combined with relative position encoding and multi-head attention mechanism, to ensure the complete retention of key speech context and provide high-quality feature representation for the decoder. This shared design not only reduces the amount of model parameters, but also improves computational efficiency, which is suitable for real-time application scenarios.

[0107] In a specific embodiment, the E-branchformer encoder consists of 17 encoder blocks with an output dimension of 256 and 8 Attention heads to fully capture the timing and frequency information of the speech signal. Each encoder block contains 1024 linear units, and the convolution kernel size of the convolution layer is set to 31, which enhances the depth of feature extraction. In addition, the E-branchformer encoder is equipped with relative position encoding and self-attention mechanism to ensure position awareness when processing variable-length sequences. The input layer selects the Conv2D convolution layer to optimize the capture of the time-frequency structure. The dropout rate, position dropout rate, and attention dropout rate in the module are all set to 0.1 to avoid overfitting.

[0108] According to an embodiment of the present application, the CTC decoder processes the features frame by frame through a linear layer and a log softmax layer, and its training target is optimized by the CTC loss function to align the time series of input and output. A major advantage of the CTC decoder is that it can generate intermediate results frame by frame without the need for complete input, which is very suitable for real-time needs in low-latency scenarios. During operation, the CTC decoder quickly generates a time-series decoding text sequence by quickly decoding the time features generated by the encoder, thereby achieving efficient transcription of the speech signal. Due to the characteristics of its streaming mode operation, the CTC decoder can provide stable and timely preliminary recognition results for complex scenarios.

[0109] According to an embodiment of the present application, the attention decoder further optimizes the recognition accuracy through multiple Transformer decoding layers. The attention decoder can capture the long-range dependencies of the input sequence (i.e., the time-sequential decoded text sequence) and make full use of the global context information of the speech. In the second stage of the model operation, the Attention decoder combines the high-quality representation of the shared encoder (i.e., the attention encoding features) and the preliminary output of the CTC decoder (i.e., the time-sequential decoded text sequence) to generate a more accurate recognition result, i.e., the attention decoded text sequence, by rescoring the comprehensive information. The multi-layer structure of the Attention decoder shows excellent robustness and adaptability in noise interference or complex speech signals.

[0110] In a specific embodiment, the decoder module (i.e., the time series decoder and the attention decoder) is based on the Transformer structure and consists of 6 layers of decoding blocks, each layer containing 4 Attention heads and 2048 linear units. To improve the accuracy of speech-to-text conversion, the various dropout rates (dropout rate, position dropout rate, self-attention dropout rate, and source attention dropout rate) in the decoder are set to 0.1. The decoder adopts a CTC and Attention hybrid decoding mechanism, in which the weight of the CTC loss is set to 0.3 to maintain real-time performance, and the Attention loss is used to further optimize the recognition accuracy. The label smoothing weight (Label Smoothing) is 0.1 to improve the stability of the output.

[0111] According to the embodiments of the present application, in actual operation, the process of the initial speech recognition network is divided into two passes: in the first pass, the shared encoder (i.e., the attention convolutional encoder) processes limited context information, and the CTC decoder quickly outputs intermediate results to meet the low latency requirements of streaming scenarios; in the second pass, the Attention decoder integrates the outputs of the shared encoder and the CTC decoder to further improve recognition accuracy. This design combines the deep feature extraction capability of the E-branchformer encoder, the real-time performance of the CTC decoder, and the global optimization capability of the Attention decoder to achieve robust and efficient real-time speech transcription in complex scenarios. Through the collaborative work of multiple modules, the module can maintain high accuracy in a variety of application environments and adapt to various challenges from low signal-to-noise ratio speech to multi-speaker speech.

[0112] Figure 5 A data processing flow chart of an attention convolutional encoder according to an embodiment of the present application is shown.

[0113] According to the embodiments of the present application, Figure 5As shown in the figure, the attention convolution encoder is used to process the target fusion features to obtain the attention encoding features containing time information and frequency information, including:

[0114] Using a first feedforward neural network to process the target fusion feature, obtaining a first weighted feature;

[0115] Generate a first weighted fusion feature according to the target fusion feature and the first weighted feature;

[0116] Using the attention-aware network to process the first weighted fusion feature to obtain the attention-aware feature;

[0117] Using a second feedforward neural network to process the attention perception feature to obtain a second weighted feature;

[0118] Generate a second weighted fusion feature according to the second weighted feature and the attention-aware feature;

[0119] The second weighted fusion feature is layer-normalized to obtain the attention encoding feature.

[0120] According to the embodiments of the present application, Figure 5 As shown, the first weighted fusion feature is processed by the attention-aware network to obtain the attention-aware feature, including:

[0121] Use the multi-head attention mechanism layer to process the first weighted fusion feature to obtain the attention feature;

[0122] Processing the first weighted fusion feature using the conditional generative perception layer to obtain a conditional perception feature;

[0123] The attention feature and the conditional perception feature are concatenated to obtain a concatenated feature;

[0124] Attention-aware features are generated based on the concatenated features and the first weighted fusion features.

[0125] According to an embodiment of the present application, when the attention convolution encoder is an E-branchformer encoder, the structure of the E-branchformer encoder is as follows: Figure 5As shown in Figure 1, it is an enhanced version of Branchformer. Similar to Conformer, E-Branchformer contains two feedforward neural networks (FFN). Its notable feature is the introduction of a dual-branch structure: two parallel branches are used to capture contextual information of different ranges. One branch uses a multi-head attention mechanism (MHA) to focus on extracting global contextual information; the other branch uses a convolution-gated conditional generative perception layer (cgMLP) to focus on modeling local contextual features. The two branches use a combination of concatenation, one-dimensional deep convolution, and linear projection for feature fusion. Compared with the simple concatenation plus linear projection merging method in Branchformer, this design can more efficiently integrate global and local information and significantly improve the expression of contextual information. The improvements in E-Branchformer further optimize the performance of the model in speech processing tasks, especially showing stronger robustness and adaptability in complex scenarios.

[0126] According to an embodiment of the present application, a target loss result is generated according to the target filter group characteristics, the training text recognition information and the recognition text label, including:

[0127] The target loss result is generated based on the target filter group features, the temporal decoded text sequence, the attention decoded text sequence and the recognized text label.

[0128] According to an embodiment of the present application, a target loss result is generated according to the target filter group features, the temporal decoded text sequence, the attention decoded text sequence and the recognized text label, including:

[0129] According to the target filter bank characteristics, a speech enhancement loss result corresponding to the causal speech enhancement network is generated;

[0130] Generate a temporal loss result of a temporal decoder according to the temporal decoding text sequence and the recognition text label;

[0131] Generate the attention loss result of the attention decoder based on the attention decoded text sequence and the recognized text label;

[0132] A target loss result is generated according to the speech enhancement loss result, the timing loss result and the attention loss result.

[0133] According to the embodiment of the present application, the Adam optimizer is used in the training process, the initial learning rate is set to 0.001, the weight decay rate is 1e-6, and the linear warmup (Warmup LR) strategy is used to schedule the learning rate, and linear warmup is performed within the first 35,000 steps to ensure that the model converges stably in the early stage of training. The maximum number of training rounds is 240 rounds, and the gradient clipping is set to 5 to prevent the risk of gradient explosion. The batch processing method uses static batch processing, and the batch size is 16.

[0134] According to an embodiment of the present application, the speech enhancement loss result is based on the mean square error (MSE) between the enhanced FBank features (target filter bank features) and the target FBank features (original filter bank features), which is used to optimize the output clarity and quality of the causal speech enhancement network. The recognition loss adopts a mixed loss of the CTC decoder and the Attention decoder. The CTC loss weight can be 0.3 to achieve real-time requirements, while the Attention loss further optimizes the conversion accuracy of speech to text. In addition, the label smoothing weight is set to 0.1 to ensure the stability of the output, which overall improves the performance of the speech enhancement recognition model in speech recognition tasks.

[0135] According to an embodiment of the present application, the loss function of the speech enhancement recognition model includes two parts: speech enhancement loss and recognition loss, so as to jointly optimize the speech enhancement quality and recognition accuracy. The speech enhancement loss result is based on the mean square error (MSE) between the enhanced FBank features and the target FBank features, as shown in formula (1):

[0136] (1)

[0137] in, is the target filter bank feature, is the original filter bank feature, N is the total number of frames of speech training samples, is the mean square error between the enhanced FBank feature and the target FBank feature, that is, Figure 4 Speech enhancement loss in This part of the loss aims to optimize the output clarity and quality of the causal speech enhancement network, ensuring that the speech signal has better intelligibility.

[0138] According to the embodiment of the present application, the recognition loss combines the hybrid mechanism of CTC and Attention to balance the requirements of real-time performance and recognition accuracy in a weighted manner. The recognition loss is shown in formula (2):

[0139] (2)

[0140] in, is the CTC loss, that is Figure 4 Timing loss results in , weight It can be set to 0.3 or any value between (0.2, 0.4). It is mainly used to align the time series of input and output to meet the real-time requirements in streaming scenarios. is the Attention loss, that is Figure 4 Attention loss results in , Attention loss is optimized through cross entropy to further improve the accuracy of speech-to-text conversion, In addition, in order to ensure the stability of the output, a label smoothing mechanism is added to the recognition loss, and the weight is set to 0.1 to reduce the risk of overfitting.

[0141] According to an embodiment of the present application, the speech enhancement loss and the recognition loss are combined into a target loss result, which is used to optimize the overall model performance, as shown in formula (3):

[0142] (3)

[0143] in, is the target loss result, The weight representing the loss of speech enhancement may be within a range of no more than 0.5. In the present application, 0.5 is used as an example for illustration. Used to control the balance between speech enhancement and recognition performance. Through the design of the above loss function, the speech enhancement recognition model significantly optimizes the recognition accuracy while improving speech clarity, meeting the multiple requirements of real-time speech recognition.

[0144] In a specific embodiment, training and experimental verification were performed on an NVIDIA GeForce RTX 4090 GPU. The speech enhancement recognition model was built based on the Wenet framework, using the Python language and the PyTorch framework. In terms of hyperparameter configuration, all experiments used the Adam optimizer, with the initial learning rate set to 0.001 and the weight decay rate set to 10. -6 The learning rate scheduling strategy uses linear warmup (Warmup LR), which gradually increases the learning rate in a linear manner within the first 35,000 steps of training, and then adjusts it according to the preset learning rate decay strategy.

[0145] According to an embodiment of the present application, in terms of speech recognition training sets, the Aishell-1 dataset and the Noise-X92 noise set were used in the experiment to verify the effectiveness of the speech enhancement recognition model. The Aishell-1 dataset is an open Chinese speech recognition dataset that contains about 150 hours of Mandarin speech data with a sampling rate of 16 kHz, covering recordings of speakers from different age groups, genders and regions. These recordings include a variety of daily sentences with high voice quality and language diversity, and are important benchmark datasets for verifying the performance of speech recognition models. The sampling rate of noise data is 16 kHz, and the noise types are divided into two categories: relatively stable noise (including "traffic", "subway" and "car" noise) and relatively non-stationary noise (including "yiyi sound", "airport", "vacuum" and "café" noise). Each noise type contains 10 different audio streams, 8 audio streams are included in the test set, and the total noise data duration is about 2 hours.

[0146] In the experiment, the Noise-X92 noise set was used to dynamically add noise to the training set and test set of the Aishell-1 dataset to generate noisy speech data. Through this mixing operation, a noisy dataset with a random signal-to-noise ratio (SNR) in the range of -5 to 10 dB, namely, speech training samples, was constructed to more realistically simulate complex real-world environmental noise conditions.

[0147] According to the embodiments of the present application, the word error rate (CER) and the real-time factor (RTF) are used as core evaluation indicators to measure the performance of the speech enhancement recognition model. CER is used to measure the difference between the recognition output and the reference text. The calculation includes three types of errors: insertion, deletion, and substitution. It is a key indicator for evaluating the accuracy of the speech recognition model; the lower the CER, the higher the recognition accuracy of the model. RTF represents the ratio of time required to process one second of audio, which is used to evaluate the real-time performance of the model; the lower the RTF, the faster the processing speed of the model, which is suitable for real-time application scenarios. These two indicators provide a comprehensive performance evaluation for the experiment from the two core dimensions of accuracy and real-time.

[0148] The Aishell-1 dataset and Noise-X92 noise dataset were used in the experiment, covering different noise environments and test scenarios with multiple signal-to-noise ratios (SNR, -5 to 10 dB). The speech enhancement module compares the causal speech enhancement Fullsubnet+ and non-causal speech enhancement Frcrn methods. The CER performance was tested in different signal-to-noise ratio ranges, and the real-time performance of the model was evaluated in combination with RTF. The experimental results are shown in Table 1, showing the detailed performance of the model of the present invention on the noisy speech dataset and the clean speech dataset. In the experiment, in order to verify the effectiveness of the model, we conducted a comparative study on the speech enhancement module and the streaming speech recognition module in two ways: separate training and joint training. Separate training refers to training the speech enhancement module and the streaming speech recognition module separately, and then connecting the two in series; joint training refers to first training the streaming speech recognition module, initializing its parameters in the joint framework and freezing it, and then jointly optimizing it with the speech enhancement module. By comparing the two training methods, we can clearly analyze the advantages of joint optimization in speech enhancement and speech recognition performance. The present invention compares the experimental effects of two speech enhancement front-ends, non-causal speech enhancement (Frcrn) and causal speech enhancement (Fullsubnet+), in separate training, and compares the experimental effects of causal speech enhancement (Fullsubnet+) without noise fusion and with noise fusion in joint training.

[0149] Table 1 Performance of streaming speech recognition model on noisy dataset (clean dataset WER=5.16%)

[0150]

[0151] Experimental results show that compared with the baseline model trained separately, the speech enhancement recognition model of this application significantly improves the accuracy and robustness of speech recognition. On the noisy speech dataset, the CER was reduced from 46.61% of the baseline to 32.81%, achieving a relative reduction of about 29.6%. In high-noise scenarios (such as the "cafe" scenario with "non-stationary noise"), the advantages of the speech enhancement recognition model jointly trained in this application are particularly obvious, with the CER reduced from 52.74% of the baseline to 42.02%. This shows that the speech enhancement recognition model of this application can effectively alleviate the impact of complex noise environments on recognition performance.

[0152] From the comparison of speech enhancement effects, non-causal speech enhancement (Frcrn) is slightly better than causal speech enhancement (Fullsubnet+) in terms of CER indicators, but because Frcrn needs to rely on future frame information, the computational complexity is high, resulting in a significant increase in RTF, which is not suitable for real-time application scenarios. In contrast, although Fullsubnet+ has a slight disadvantage in recognition rate, thanks to its causal design, RTF is only 0.025, which fully meets the low latency requirements of streaming speech recognition.

[0153] In addition, the speech enhancement recognition model jointly trained in this application also showed a significant improvement in the overall robustness of the model. Under various noise types (such as the "stationary noise" "subway" or "non-stationary noise" "airport" scenes), the CER has achieved a significant decrease, indicating that the speech enhancement recognition model jointly trained in this application can better coordinate the performance of speech enhancement and recognition modules, and reduce the negative impact of speech distortion on recognition results.

[0154] In summary, the speech enhancement recognition model jointly trained in this application not only solves the performance bottleneck problem caused by the decoupling of speech enhancement and streaming speech recognition in related technologies, but also significantly improves the recognition accuracy and robustness in complex noise environments through the coordinated optimization of the causal speech enhancement network and the initial speech recognition network. At the same time, the speech enhancement recognition model jointly trained in this application maintains a low RTF value in terms of computational efficiency, has high real-time performance, is suitable for low-latency real-time speech recognition tasks, and has broad practical application potential.

[0155] Figure 6 A flow chart of a speech enhancement recognition method according to an embodiment of the present application is shown.

[0156] like Figure 6 As shown, the speech enhancement recognition method includes operations S601 to S602.

[0157] In operation S601, in response to a training instruction, voice information to be recognized is acquired;

[0158] In operation S602, the speech information to be recognized is input into a speech enhancement recognition model, and target text recognition information is output.

[0159] According to an embodiment of the present application, any segment of voice information to be recognized of the target user, which may also be voice information to be recognized obtained in real time, is obtained and input into a speech enhancement recognition model, and text recognition can be performed on the voice information to be recognized, thereby obtaining target text recognition information.

[0160] According to an embodiment of the present application, based on the target filter group features containing global information output by the causal speech enhancement network, the original filter group features containing rich background information and speech details are fused with the target filter group features using a noise fusion network, and finally the target fusion features are recognized by the initial speech recognition network to obtain training text recognition information. This can effectively suppress noise under real-time requirements and avoid speech distortion problems caused by related technologies. At the same time, the use of the noise fusion network further reduces the distortion effect caused by enhancement, effectively improving the recognition accuracy. In addition, the speech enhancement recognition model has excellent noise robustness, making it more applicable in complex and changeable real-world scenarios.

[0161] Figure 7 A block diagram of a training device for a speech enhancement recognition model according to an embodiment of the present application is shown.

[0162] like Figure 7 As shown, the training device 700 for the speech enhancement recognition model includes a first acquisition module 710, a causal enhancement module 720, an attention fusion module 730, a speech recognition module 740, a loss calculation module 750, and an iterative training module 760.

[0163] A first acquisition module 710 is used to acquire a speech recognition training set in response to a training instruction, wherein the speech recognition training set includes a plurality of speech training samples and a recognition text label corresponding to each speech training sample;

[0164] A causal enhancement module 720 is used to process each speech training sample using a causal speech enhancement network to obtain a target filter bank feature containing global information;

[0165] The attention fusion module 730 is used to:

[0166] The speech training samples are processed by using a filter bank-based feature extraction method to obtain original filter bank features;

[0167] The attention mechanism network is used to dynamically weight and fuse the target filter group features and the original filter group features to obtain the target fusion features containing the supplementary information in the speech training samples.

[0168] A speech recognition module 740 is used to process the target fusion features using an initial speech recognition network to obtain training text recognition information, wherein the initial speech recognition model includes a causal speech enhancement network, an attention mechanism network and an initial speech recognition network;

[0169] A loss calculation module 750 is used to generate a target loss result according to the target filter group characteristics, the training text recognition information and the recognition text label;

[0170] The iterative training module 760 is used to iteratively adjust the network parameters of the initial speech recognition model according to the target loss result to obtain a trained speech enhancement recognition model.

[0171] Figure 8 A block diagram of a speech enhancement and recognition device according to an embodiment of the present application is shown.

[0172] like Figure 8 As shown, the speech enhancement recognition device 800 includes a second acquisition module 810 and an enhancement recognition module 820 .

[0173] The second acquisition module 810 is used to obtain the voice information to be recognized in response to the voice recognition instruction;

[0174] The enhanced recognition module 820 is used to input the above-mentioned speech information to be recognized into the speech enhancement recognition model and output the target text recognition information.

[0175] According to an embodiment of the present application, based on the target filter group features containing global information output by the causal speech enhancement network, the original filter group features containing rich background information and speech details are fused with the target filter group features using a noise fusion network, and finally the target fusion features are recognized by the initial speech recognition network to obtain training text recognition information. This can effectively suppress noise under real-time requirements and avoid speech distortion problems caused by related technologies. At the same time, the use of the noise fusion network further reduces the distortion effect caused by enhancement, effectively improving the recognition accuracy. In addition, the speech enhancement recognition model has excellent noise robustness, making it more applicable in complex and changeable real-world scenarios.

[0176] According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0177] It should be noted that the training device or the speech enhancement recognition device part of the speech enhancement recognition model in the embodiments of the present application corresponds to the training method or the speech enhancement recognition method part of the speech enhancement recognition model in the embodiments of the present application. The description of the training device or the speech enhancement recognition device part of the speech enhancement recognition model specifically refers to the training method or the speech enhancement recognition method part of the speech enhancement recognition model, which will not be repeated here.

[0178] Fig. 9 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0179] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0180] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present application by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present application by executing the program stored in the one or more memories.

[0181] According to an embodiment of the present application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0182] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0183] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0184] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0185] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0186] When the computer program is executed by the processor 901, the above functions defined in the system / device of the embodiment of the present application are executed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0187] The embodiments of the present application are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present application. Although each embodiment is described above, this does not mean that the measures in each embodiment cannot be used in combination advantageously. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A method for training a speech enhancement recognition model, characterized in that: include: In response to the training instruction, a speech recognition training set is acquired, wherein the speech recognition training set includes a plurality of speech training samples and a recognition text label corresponding to each of the speech training samples; For each of the speech training samples, the speech training samples are processed using a causal speech enhancement network to obtain target filter group features containing global information; Processing the speech training sample using a filter bank-based feature extraction method to obtain original filter bank features; Using an attention mechanism network to perform dynamic weighted fusion processing on the target filter group features and the original filter group features to obtain target fusion features containing supplementary information in the speech training sample; Processing the target fusion features using an initial speech recognition network to obtain training text recognition information, wherein the initial speech recognition model includes the causal speech enhancement network, the attention mechanism network and the initial speech recognition network; Generate a target loss result according to the target filter group feature, the training text recognition information and the recognition text label; The network parameters of the initial speech recognition model are iteratively adjusted according to the target loss result to obtain a trained speech enhancement recognition model.

2. The method according to claim 1, characterized in that The speech training samples are processed using a causal speech enhancement network to obtain target filter bank features containing global information, including: Performing Fourier transform processing on the speech training sample to obtain spectrum features, wherein the spectrum features include amplitude information, real phase information, and imaginary phase information; For the plurality of feature information in the spectrum features, a channel extraction network is used to process the plurality of feature information to obtain a channel feature set; Processing the channel feature set using a first subband model to obtain a real feature vector and an imaginary feature vector, wherein the first subband model is constructed based on a long short-term memory network and a fully connected network; The target filter bank feature is generated according to the real feature vector and the imaginary feature vector.

3. The method according to claim 2, characterized in that The channel extraction network is used to process the plurality of feature information to obtain a channel feature set, including: For any feature information in the spectral features, the feature information is processed by using a channel attention mechanism layer to obtain a first weighted feature vector; For any of the first weighted feature vectors, use a temporal convolutional network to process the weighted feature vector to obtain a second weighted feature vector containing context information; Processing the first weighted feature vector corresponding to the amplitude information using a second subband model to obtain a frequency prediction vector, wherein the second subband model is constructed based on a long short-term memory network and a fully connected network; The channel feature set is constructed according to a plurality of the second weighted feature vectors and the frequency prediction vector.

4. The method according to claim 1, characterized in that The target fusion features are processed using the initial speech recognition network to obtain training text recognition information, including: Processing the target fusion features using an attention convolution encoder to obtain attention encoding features containing time information and frequency information; Processing the attention encoding features using a temporal decoder to obtain a temporal decoded text sequence; The attention encoding features and the temporal decoding text sequence are rescored and decoded using an attention decoder to obtain an attention decoding text sequence, wherein the training text recognition information includes the temporal decoding text sequence and the attention decoding text sequence.

5. The method according to claim 4, characterized in that The target fusion features are processed using an attention convolution encoder to obtain attention encoding features containing time information and frequency information, including: Processing the target fusion feature using a first feedforward neural network to obtain a first weighted feature; Generate a first weighted fusion feature according to the target fusion feature and the first weighted feature; Processing the first weighted fusion feature using an attention-aware network to obtain an attention-aware feature; Processing the attention perception feature using a second feedforward neural network to obtain a second weighted feature; Generating a second weighted fusion feature according to the second weighted feature and the attention-aware feature; The second weighted fusion feature is layer-normalized to obtain the attention encoding feature.

6. The method according to claim 5, characterized in that The first weighted fusion feature is processed by the attention-aware network to obtain an attention-aware feature, including: Processing the first weighted fusion feature using a multi-head attention mechanism layer to obtain an attention feature; Processing the first weighted fusion feature using a conditional generative perception layer to obtain a conditional perception feature; Performing splicing processing on the attention feature and the conditional perception feature to obtain a spliced ​​feature; The attention-aware feature is generated according to the concatenated feature and the first weighted fusion feature.

7. The method according to any one of claims 4 to 6, characterized in that Generating a target loss result according to the target filter group feature, the training text recognition information and the recognition text label, including: The target loss result is generated according to the target filter group features, the temporal decoded text sequence, the attention decoded text sequence and the recognized text label.

8. The method according to claim 7, characterized in that Generating the target loss result according to the target filter group feature, the temporal decoded text sequence, the attention decoded text sequence and the recognized text label, including: Generating a speech enhancement loss result corresponding to the causal speech enhancement network according to the target filter bank characteristics; Generating a timing loss result of the timing decoder according to the timing decoded text sequence and the recognition text label; Generating an attention loss result of the attention decoder according to the attention decoding text sequence and the recognition text label; The target loss result is generated according to the speech enhancement loss result, the timing loss result and the attention loss result.

9. A speech enhancement recognition method, characterized in that: include: In response to the voice recognition instruction, obtaining voice information to be recognized; Inputting the speech information to be recognized into a speech enhancement recognition model, and outputting target text recognition information; Wherein, the speech enhancement recognition model is trained by the method described in any one of claims 1 to 8.

Images