End-to-end dialect audio recognition method and system based on multi-layer information fusion

Abstract

An end-to-end dialect audio recognition method (10) based on multi-layer information fusion. The method (10) comprises: performing audio preprocessing on dialect audio to generate an acoustic feature (S102); inputting the acoustic feature into a coder, and then performing, by the coder, a progressive downsampling operation on the acoustic feature to generate multi-layer fine-grained acoustic features (S104); performing, by means of a layer adaptation module, multi-layer information fusion on the multi-layer fine-grained acoustic features to generate a fused acoustic feature (S106); performing, by means of a cross-attention mechanism, cross-fusion on the fused acoustic feature to generate a corrected acoustic feature (S108); and inputting the corrected acoustic feature into an end-to-end dialect recognition model to generate a dialect audio recognition result (S110). In the method, complex speech signals and multi-accent features can be efficiently captured and processed, and dialect audio can also be classified and decoded online in real time, thereby improving the accuracy and robustness of speech recognition.

Description

A dialect audio recognition method and system based on multi-layer information fusion in an end-to-end manner TECHNICAL FIELD

[0001] The present application relates to the field of computer information processing, in particular to a dialect audio recognition method and system based on multi-layer information fusion in an end-to-end manner, an electronic device, a computer readable medium and a computer program product. BACKGROUND

[0002] Speech recognition technology has made significant progress in the past few decades, especially with the promotion of large-scale data and deep learning models. However, due to the influence of individual physiological structure, region and mother tongue, etc. on pronunciation, the existence of dialects and accents poses a great challenge to speech recognition systems. Deviation of standard pronunciation will cause deviation in modeling of pronunciation units (such as phonemes), which will seriously affect the accuracy of recognition. In practical applications, such as business outbound scenarios, the pronunciation of a large number of customers has accents, and the recognition of accents in Mandarin has become one of the main challenges faced by current speech recognition systems.

[0003] Currently, the mainstream solution for dialect audio recognition mainly includes transfer learning and multi-task learning. Transfer learning is based on a pre-trained general speech recognition model, and fine-tunes it on a specific dialect dataset. The advantage of transfer learning is that the method is direct, and good results can be achieved for a single accent dialect with a large amount of training data; the disadvantage is that for a scene where multiple languages are mixed and each language has less training resources, transfer learning will have large recognition differences for different languages. Multi-task learning is to perform speech recognition and accent recognition simultaneously during training, and to enhance the adaptability of the model to multiple tasks by sharing feature representations. The advantage of multi-task learning is that it can better utilize the complementary information of multiple tasks to improve the generalization ability and robustness of the model; the disadvantage is that a complex multi-task network architecture needs to be designed and optimized simultaneously, which is difficult to train.

[0004] Therefore, a new dialect audio recognition method, system, electronic device and computer readable medium based on multi-layer information fusion in an end-to-end manner are needed.

[0005] The above information disclosed in the background section is only used to strengthen the understanding of the background of the present application, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art.

[0006] SUMMARY

[0007] Therefore, the application provides a dialect audio recognition method and system based on multi-layer information fusion in an end-to-end manner, an electronic device and a computer readable medium, which can efficiently capture and process complex speech signals and multi-accent features, and can classify and decode dialect audio in real time online, thereby improving the accuracy and robustness of speech recognition.

[0008] Other characteristics and advantages of the application will become apparent from the detailed description that follows, or can be learned by practice of the application.

[0009] According to an aspect of the application, a dialect audio recognition method based on multi-layer information fusion in an end-to-end manner is provided, which comprises: performing audio preprocessing on dialect audio to generate acoustic features; inputting the acoustic features into an encoder, which performs progressive downsampling operation thereon to generate multi-layer fine-grained acoustic features; performing multi-layer information fusion on the multi-layer fine-grained acoustic features by a layer adaptation module to generate fused acoustic features; performing cross fusion on the fused acoustic features by a cross attention mechanism to generate modified acoustic features; inputting the modified acoustic features into an end-to-end dialect recognition model to generate a dialect audio recognition result.

[0010] Optionally, the method further comprises: performing training preprocessing on dialect audio with dialect labels to generate training fused acoustic features; performing cross fusion on the training fused acoustic features by a cross attention mechanism to generate training modified acoustic features; extracting accent features of the training fused acoustic features; training an initial end-to-end dialect recognition model based on the training modified acoustic features and the accent features by a multi-task learning mode to generate a trained end-to-end dialect recognition model, wherein the end-to-end dialect recognition model comprises a speech recognition model and an accent recognition model.

[0011] Optionally, the audio preprocessing on the dialect audio to generate acoustic features comprises: performing feature extraction on the dialect audio to generate mel-frequency cepstral coefficient features of the dialect audio; and performing convolution downsampling operation on the mel-frequency cepstral coefficient features to generate the acoustic features of the dialect audio.

[0012] Optionally, the inputting of the acoustic features into an encoder, which performs progressive downsampling operation thereon to generate multi-layer fine-grained acoustic features, comprises: inputting the acoustic features into an encoder, which comprises a plurality of conformer encoders; the plurality of conformer encoders performing progressive downsampling operation on the acoustic features; and the multi-layer fine-grained acoustic features being generated by the progressive downsampling operation to reduce the redundancy of the time dimension layer by layer.

[0013] Optionally, the multi-layer fine-grained acoustic features are subjected to multi-layer information fusion by a layer adaptation module to generate fused acoustic features, including: the layer adaptation module extracts fine-grained acoustic features of each layer in the multi-layer fine-grained acoustic features; each layer of fine-grained acoustic features is subjected to multi-layer information fusion respectively to generate the fused acoustic features.

[0014] Optionally, the fused acoustic features are subjected to cross fusion by a cross attention mechanism to generate modified acoustic features, including: the multi-layer fine-grained acoustic features are taken as key-value vectors and the fused acoustic features are taken as query vectors; the cross fusion is performed by the cross attention mechanism to generate the modified acoustic features.

[0015] Optionally, the modified acoustic features are input into an end-to-end dialect recognition model to generate a dialect audio recognition result, including: the modified acoustic features are input into the end-to-end dialect recognition model; a speech recognition model and an accent recognition model in the end-to-end dialect recognition model recognize the modified acoustic features respectively to generate a text recognition result and an accent classification label.

[0016] Optionally, dialect audio with a dialect label is subjected to training preprocessing to generate training fused acoustic features, including: the dialect audio with the dialect label is subjected to audio preprocessing to generate training acoustic features; the training acoustic features are input into an encoder, and the encoder performs progressive downsampling operation to generate multi-layer training fine-grained acoustic features; the training multi-layer fine-grained acoustic features are subjected to multi-layer information fusion by a layer adaptation module to generate the training fused acoustic features.

[0017] Optionally, an initial end-to-end dialect recognition model is trained based on the training modified acoustic features and the accent features by a multi-task learning mode to generate a trained end-to-end dialect recognition model, the end-to-end dialect recognition model including a speech recognition model and an accent recognition model, including: a multi-task learning framework is determined, the multi-task learning framework including a cross information fusion mechanism, the speech recognition model and the accent recognition model; the speech recognition model is trained based on the multi-task learning framework and the training modified acoustic features; the accent recognition model is trained based on the multi-task learning framework and the accent features; when a whole loss function meets a set index during the training process, the trained end-to-end dialect recognition model is generated.

[0018] Optionally, the accent recognition model is trained based on the multi-task learning framework and the accent features, including: the accent features are input into the accent recognition model based on the multi-task learning framework, the accent recognition model including two layers of causal convolution structures and a linear discriminator.

[0019] Optionally, the end-to-end dialect recognition model is generated when the overall loss function meets the set index during the training process, including: generating an overall loss function through a speech recognition classification loss, a decoder attention loss, and a cross-entropy loss of accent recognition; in each training, the current speech recognition classification loss, the decoder attention loss, and the cross-entropy loss of accent recognition are calculated; when the speech recognition classification loss, the decoder attention loss, and the cross-entropy loss of accent recognition meet the set index respectively, the end-to-end dialect recognition model is generated.

[0020] According to an aspect of the present application, an end-to-end dialect audio recognition system based on multi-layer information fusion is provided, which comprises: a processing module for performing audio preprocessing on dialect audio to generate acoustic features; a sampling module for inputting the acoustic features into an encoder, and the encoder performs progressive downsampling operation to generate multi-layer fine-grained acoustic features; a fusion module for performing multi-layer information fusion on the multi-layer fine-grained acoustic features through a layer adaptation module to generate fused acoustic features; a correction module for performing cross fusion on the fused acoustic features through a cross-attention mechanism to generate corrected acoustic features; and a recognition module for inputting the corrected acoustic features into an end-to-end dialect recognition model to generate a dialect audio recognition result.

[0021] Optionally, it further comprises: a preprocessing module for performing training preprocessing on dialect audio with dialect labels to generate training fused acoustic features; a cross module for performing cross fusion on the training fused acoustic features through a cross-attention mechanism to generate training corrected acoustic features; a feature module for extracting accent features of the training fused acoustic features; and a training module for training an initial end-to-end dialect recognition model based on the training corrected acoustic features and the accent features through a multi-task learning mode to generate a trained end-to-end dialect recognition model, wherein the end-to-end dialect recognition model comprises a speech recognition model and an accent recognition model.

[0022] According to an aspect of the present application, an electronic device is provided, which comprises: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0023] According to an aspect of the present application, a computer readable medium having a computer program stored thereon is provided, and the program is executed by a processor to implement the method as described above.

[0024] According to an aspect of the present application, a computer program product is provided, which comprises: computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the method as described above.

[0025] The dialect audio recognition method and system based on multi-layer information fusion end to end according to the present application, by audio preprocessing of the dialect audio, acoustic features are generated; the acoustic features are input into the encoder, the encoder performs progressive downsampling operation, and multi-layer fine-grained acoustic features are generated; the multi-layer information fusion of the multi-layer fine-grained acoustic features is performed through the layer adaptation module, and the fusion acoustic features are generated; the cross fusion of the fusion acoustic features is performed through the cross attention mechanism, and the modified acoustic features are generated; the modified acoustic features are input into the end-to-end dialect recognition model, and the dialect audio recognition result is generated. The way can efficiently capture and process complex speech signals and multi-accent features, and can also classify and decode dialect audio online in real time, improving the accuracy and robustness of speech recognition.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0027] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. The drawings described below are only some embodiments of the present application, and other drawings can be obtained according to these drawings without creative labor for those skilled in the art.

[0028] Fig. 1 is a flowchart of a dialect audio recognition method based on multi-layer information fusion end to end according to an exemplary embodiment.

[0029] Fig. 2 is a schematic diagram of a dialect audio recognition method based on multi-layer information fusion end to end according to an exemplary embodiment.

[0030] Fig. 3 is a schematic diagram of a dialect audio recognition method based on multi-layer information fusion end to end according to another exemplary embodiment.

[0031] Fig. 4 is a schematic diagram of a dialect audio recognition method based on multi-layer information fusion end to end according to another exemplary embodiment.

[0032] Fig. 5 is a flowchart of a dialect audio recognition method based on multi-layer information fusion end to end according to another exemplary embodiment.

[0033] Fig. 6 is a schematic diagram of a dialect audio recognition method based on multi-layer information fusion end to end according to an exemplary embodiment.

[0034] Fig. 7 is a block diagram of a dialect audio recognition system based on multi-layer information fusion end to end according to an exemplary embodiment.

[0035] FIG. 8 is a block diagram of a dialect audio recognition system based on multi-layer information fusion end-to-end, according to an example embodiment.

[0036] FIG. 9 is a block diagram of an electronic device, according to an example embodiment. DETAILED DESCRIPTION

[0037] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the several views and the description of the figures.

[0038] FIG. 1 is a flowchart of a dialect audio recognition method based on multi-layer information fusion end-to-end, according to an example embodiment. The dialect audio recognition method based on multi-layer information fusion end-to-end 10 includes at least steps S102-S108.

[0039] As shown in FIG. 1, in S102, the dialect audio is preprocessed for audio to generate acoustic features. For example, feature extraction can be performed on the dialect audio to generate Mel-Frequency Cepstral Coefficient (MFCC) features of the dialect audio; and a convolutional down-sampling operation can be performed on the MFCC features to generate the acoustic features of the dialect audio.

[0040] More specifically, the dialect audio is preprocessed. This can include removing background noise (e.g., using a filter or a noise reduction algorithm), and normalizing the audio signal to a uniform range (e.g., [-1, 1]) to ensure the quality and consistency of the audio data. Then, Mel-Frequency Cepstral Coefficient (MFCC) features are extracted from the preprocessed audio.

[0041] More specifically, a short-time Fourier transform (STFT) can be performed to convert the audio signal in time domain to frequency domain. A Mel filter bank can also be applied to convert the frequency spectrum to a Mel scale that is more consistent with human auditory perception. Then, a cepstrum can be calculated by taking the log of the Mel spectrum and performing a Discrete Cosine Transform (DCT) to generate a series of MFCC features. Finally, a convolutional down-sampling operation can be performed on the extracted MFCC features to reduce the feature dimension and temporal redundancy through convolutional layers, thereby generating more compact acoustic features.

[0042] In S104, the acoustic feature is input into an encoder, and the encoder performs progressive down-sampling operation thereon to generate multi-layer fine-grained acoustic features. The acoustic feature can be input into an encoder, and the encoder includes a plurality of conformer encoders; the plurality of conformer encoders perform progressive down-sampling operation on the acoustic feature; and the multi-layer fine-grained acoustic features are generated by progressively reducing the redundancy of the time dimension layer by layer through the progressive down-sampling operation.

[0043] More specifically, in the present application, the acoustic encoder structure adopts a Conformer module, which integrates convolutional layers into the transformer module to enhance the local modeling capability of the signal sequence. As shown in FIG. 2, in the present application, the conformer structure in the prior art is optimized, and the present application inserts a progressive down-sampling operation into the stacked conformer encoder. This progressive down-sampling operation can compress the features of the dialect audio in the time dimension, which is beneficial to the effective extraction of dialect features by the layer-adaptive module.

[0044] More specifically, the mathematical process of progressive down-sampling can be defined as follows:

[0045] wherein X t may be the speech feature after preprocessing, is the feature after the first FFN processing, is the feature after the MHSA processing, is the feature after the convolutional down-sampling processing, is the feature after the second FFN processing, is the finally extracted frame-level feature.

[0046] More specifically, the acoustic feature generated by preprocessing can be input into an encoder. The encoder is usually composed of a plurality of Conformer encoder layers. Each Conformer encoder layer processes the input acoustic feature and progressively reduces the redundant information in the time dimension layer by layer through the progressive down-sampling, thereby generating multi-layer fine-grained acoustic features. This process ensures efficient encoding and feature extraction of the speech signal.

[0047] In S106, the multi-layer information fusion is performed on the multi-layer fine-grained acoustic features by a layer adaptation module to generate fused acoustic features. For example, the layer adaptation module extracts fine-grained acoustic features of each layer in the multi-layer fine-grained acoustic features, and performs multi-layer information fusion on the fine-grained acoustic features of each layer respectively to generate the fused acoustic features.

[0048] Current solutions for multi-accent recognition tasks usually train an accent identifier (AID) model using acoustic features of a general ASR model as input. However, such methods lack information sharing between the two models, leading to performance degradation in multi-task learning of ASR and AID. As shown in FIG. 3, the present application proposes a layer adaptation module that extracts fine-grained accent information in different layers of the acoustic encoder in the previous step, while promoting frame-by-frame correction of ASR results through a cross-attention module.

[0049] Furthermore, after extracting the fused accent features from the adaptation layer in the acoustic encoder, an AID can be constructed using a two-layer causal convolution structure and a linear-based discriminator. This module can effectively extract accent information and classify the input multi-accent speech into different accent categories frame by frame.

[0050] After generating the multi-layer fine-grained acoustic features, the layer adaptation module is used to perform multi-layer information fusion on these features. Specifically, the layer adaptation module extracts fine-grained acoustic features of each layer and fuses these features. In this way, the layer adaptation module can comprehensively utilize the information of each layer feature to generate more comprehensive and accurate fused acoustic features.

[0051] In S108, the fused acoustic features are cross-fused by a cross-attention mechanism to generate corrected acoustic features. More specifically, as shown in FIG. 4, the multi-layer fine-grained acoustic features can be used as key-value vectors, and the fused acoustic features can be used as a query vector. The cross-attention mechanism is used to cross-fuse the features to generate the corrected acoustic features.

[0052] To further improve the expression ability of the features, the cross-attention mechanism is used to cross-fuse the fused acoustic features. Specifically, the multi-layer fine-grained acoustic features are used as key-value vectors (key), and the fused acoustic features are used as a query vector (Query). The cross-attention mechanism is used to cross-fuse the features to generate corrected acoustic features. The cross-attention mechanism can effectively align and fuse features of different levels, thereby improving the expression accuracy and robustness of the features.

[0053] The cross-information fusion process based on the attention mechanism can be represented as follows:

[0054] wherein, V t is a weight matrix, is a fused acoustic feature, is a multi-layer fine-grained acoustic feature, and other parameters are parameters determined in training.

[0055] In S110, the corrected acoustic feature is input into an end-to-end dialect recognition model to generate a dialect audio recognition result. For example, the corrected acoustic feature can be input into an end-to-end dialect recognition model; a speech recognition model and an accent recognition model in the end-to-end dialect recognition model respectively recognize the corrected acoustic feature to generate a text recognition result and an accent classification label.

[0056] The corrected acoustic feature can be input into an end-to-end dialect recognition model to generate a dialect audio recognition result. The end-to-end dialect recognition model usually includes two sub-models: one is a speech recognition model responsible for converting the corrected acoustic feature into a text recognition result; the other is an accent recognition model responsible for generating an accent classification label. The two sub-models work together to provide accurate text and accent recognition results.

[0057] It is worth mentioning that the dialect recognition model in the present application can be streaming or non-streaming, more specifically, the end-to-end dialect recognition model can adopt two different working modes: streaming and non-streaming. In the streaming model, the speech input is processed in real time, that is, when the speech input is received, the dialect recognition model will immediately start processing and continuously update the output result, which is usually used in applications that require real-time response, such as voice assistants or real-time subtitles. The non-streaming model processes the entire speech input as a whole and generates the final recognition result after the entire speech input is completed. This mode is suitable for scenarios that do not require real-time processing, such as speech transcription or speech analysis, but the present application is not limited thereto.

[0058] According to the end-to-end dialect audio recognition method based on multi-layer information fusion of the present application, the dialect audio is preprocessed to generate acoustic features; the acoustic features are input into an encoder, which performs progressive downsampling operation to generate multi-layer fine-grained acoustic features; the multi-layer information fusion is performed on the multi-layer fine-grained acoustic features through a layer adaptation module to generate a fused acoustic feature; the cross-attention mechanism is used to cross-fuse the fused acoustic feature to generate a corrected acoustic feature; and the corrected acoustic feature is input into an end-to-end dialect recognition model to generate a dialect audio recognition result. This method can efficiently capture and process complex speech signals and multi-accent features, and can also classify and decode dialect audio in real time online, thereby improving the accuracy and robustness of speech recognition.

[0059] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teachings of the present disclosure, these principles can be applied to many other embodiments.

[0060] FIG. 5 is a flowchart illustrating a dialect audio recognition method based on multi-layer information fusion end-to-end according to another exemplary embodiment. The flow 50 shown in FIG. 5 is a supplementary description of the flow shown in FIG. 1.

[0061] As shown in FIG. 5, in S502, the dialect audio with a dialect label is preprocessed for training to generate training fusion acoustic features. For example, the dialect audio with a dialect label can be preprocessed for audio to generate training acoustic features; the training acoustic features are input into an encoder, which performs progressive downsampling operations thereon to generate multi-layer training fine-grained acoustic features; the training multi-layer fine-grained acoustic features are subjected to multi-layer information fusion through a layer adaptation module to generate the training fusion acoustic features.

[0062] In S504, the training fusion acoustic features are subjected to cross fusion through a cross-attention mechanism to generate training corrected acoustic features. The training fusion acoustic features are subjected to cross fusion according to the steps as shown above to generate training corrected acoustic features.

[0063] In S506, an accent feature of the training fusion acoustic features is extracted. The accent feature can be extracted through a neural network model.

[0064] In S508, an initial end-to-end dialect recognition model is trained based on the training corrected acoustic features and the accent feature through a multi-task learning mode to generate a trained end-to-end dialect recognition model, the end-to-end dialect recognition model including a speech recognition model and an accent recognition model.

[0065] For example, a multi-task learning framework can be determined, the multi-task learning framework including a cross information fusion mechanism, a speech recognition model and an accent recognition model; the speech recognition model is trained based on the multi-task learning framework through the training corrected acoustic features; the accent recognition model is trained based on the multi-task learning framework through the accent feature; when the overall loss function meets a set index during the training process, the trained end-to-end dialect recognition model is generated.

[0066] More specifically, the accent feature is input into the accent recognition model based on the multi-task learning framework, the accent recognition model including two layers of causal convolution structure and a linear discriminator.

[0067] More specifically, the overall loss function is generated by the speech recognition classification loss, the decoder attention loss and the accent recognition cross-entropy loss; in each training, the current speech recognition classification loss, the decoder attention loss and the accent recognition cross-entropy loss are calculated; when the speech recognition classification loss, the decoder attention loss and the accent recognition cross-entropy loss meet the set indicators respectively, the end-to-end dialect recognition model is generated.

[0068] In the multi-task learning framework, the mathematical expression of the loss function is as follows: L all = L att + λ ctc + L ctc + λ aid L aid

[0069] In the multi-task learning framework, the mathematical expression of the loss function is as follows: L all is the overall loss, L att is the decoder attention loss, L ctc is the speech recognition classification loss, λ ctc is the corresponding weight, L aid is the accent recognition cross-entropy loss, and λ aid is the corresponding weight.

[0070] In the present application, the dialect information is attributed to higher-level features, which is different from the phoneme pronunciation feature. Directly using the encoding output for dialect feature representation has deviation. Therefore, in the present application, frame-level dialect features are used. Different dialects have different deviations from standard pronunciation for different modeling units. Therefore, the present application uses phoneme-level modeling to eliminate the deviation caused by the accent in the phoneme angle.

[0071] FIG. 6 is a schematic diagram of an end-to-end dialect audio recognition method based on multi-layer information fusion according to another exemplary embodiment. FIG. 6 describes the processing flow of an end-to-end speech recognition system, which involves steps such as audio processing, feature extraction, acoustic coding, dialect information extraction, feature fusion and decoding.

[0072] First, the dialect audio is preprocessed. The speech input can be received, and the speech audio can be, for example, an audio file in.wav format. Then, MFCC (Mel-frequency cepstral coefficients) technology can be used to extract features from the audio. After extracting the MFCC features, convolution operation and down-sampling operation are performed on these features. Convolution operation can help extract spatial information between features, while down-sampling can help reduce data volume and computational complexity.

[0073] After that, acoustic feature extraction is performed. After processing the MFCC features, they are input into a general acoustic encoder that converts the MFCC features into acoustic features. In this process, the system retains the output of each layer of the acoustic encoder, which will be used for subsequent processing. On the output of the acoustic encoder, a LayerAdapted module is used to fuse the output of each layer of the acoustic encoder. This step is used to interact and fuse information between features at different levels.

[0074] After the fusion of features, a dialect information extraction process is performed to extract information about the dialect or accent used by the speaker from the acoustic features. After extracting the dialect information, it is cross-fused with the output of the general acoustic encoder. This step can help to correct the acoustic features to better adapt to different speaking styles and dialects.

[0075] Finally, the corrected acoustic features are input into a decoder that converts the acoustic features into text, realizing end-to-end speech recognition. The decoder is a neural network model, and its output is the recognized text. Such a system design allows direct conversion from audio input to text output, and by considering dialect information and general acoustic features in the acoustic feature extraction and fusion process, the recognition accuracy for different speakers and different language styles can be improved.

[0076] In this application, a new layer-adaptive fusion (LAF) model is proposed, which does not require prior knowledge of the target accent, uses a dynamic block strategy, realizes streaming and non-streaming decoding, and improves recognition accuracy through frame-level acoustic feature extraction and fine-grained information fusion. On multiple test datasets, the end-to-end dialect recognition model proposed in this application can significantly reduce the character error rate in speech recognition.

[0077] Those skilled in the art can understand that all or part of the steps of the above embodiments are implemented as computer programs executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present application are executed. The program can be stored in a computer readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0078] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limiting purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0079] The following is an embodiment of the device of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0080] FIG. 7 is a block diagram of a dialect audio recognition system based on multi-layer information fusion end-to-end, according to an exemplary embodiment. As shown in FIG. 7, the dialect audio recognition system based on multi-layer information fusion end-to-end 70 includes a processing module 702, a sampling module 704, a fusion module 706, a correction module 708, and a recognition module 710.

[0081] The processing module 702 is configured to perform audio preprocessing on the dialect audio to generate acoustic features. The processing module 702 is also configured to perform feature extraction on the dialect audio to generate mel-frequency cepstral coefficient features of the dialect audio. The processing module 702 is further configured to perform convolutional downsampling on the mel-frequency cepstral coefficient features to generate the acoustic features of the dialect audio.

[0082] The sampling module 704 is configured to input the acoustic features into an encoder, which performs progressive downsampling on the acoustic features to generate multi-layer fine-grained acoustic features. The sampling module 704 is also configured to input the acoustic features into an encoder, which includes multiple conformer encoders. The multiple conformer encoders perform progressive downsampling on the acoustic features. The multiple conformer encoders generate the multi-layer fine-grained acoustic features by progressively reducing the redundancy of the time dimension layer by layer.

[0083] The fusion module 706 is configured to perform multi-layer information fusion on the multi-layer fine-grained acoustic features by a layer adaptation module to generate fused acoustic features. The fusion module 706 is also configured to extract fine-grained acoustic features of each layer of the multi-layer fine-grained acoustic features by the layer adaptation module. The fusion module 706 is further configured to perform multi-layer information fusion on each layer of fine-grained acoustic features to generate the fused acoustic features.

[0084] The correction module 708 is configured to perform cross-fusion on the fused acoustic features by a cross-attention mechanism to generate corrected acoustic features. The correction module 708 is also configured to input the multi-layer fine-grained acoustic features as key-value vectors and the fused acoustic features as query vectors. The correction module 708 is further configured to perform cross-fusion on the key-value vectors and the query vectors by the cross-attention mechanism to generate the corrected acoustic features.

[0085] The recognition module 710 is configured to input the corrected acoustic features into an end-to-end dialect recognition model to generate a dialect audio recognition result. The recognition module 710 is also configured to input the corrected acoustic features into the end-to-end dialect recognition model. The speech recognition model and the accent recognition model in the end-to-end dialect recognition model perform recognition on the corrected acoustic features to generate a text recognition result and an accent classification label, respectively.

[0086] FIG. 8 is a block diagram of a dialect audio recognition system based on multi-layer information fusion end-to-end, according to another exemplary embodiment. As shown in FIG. 8, the dialect audio recognition system based on multi-layer information fusion end-to-end 80 includes a preprocessing module 802, a cross module 804, a feature module 806, and a training module 808.

[0087] The preprocessing module 802 is configured to perform training preprocessing on the dialect audio with a dialect label to generate training fusion acoustic features. The preprocessing module 802 is also configured to perform audio preprocessing on the dialect audio with a dialect label to generate training acoustic features. The training acoustic features are input into an encoder, which performs progressive downsampling operations to generate multi-layer training fine-grained acoustic features. The training multi-layer fine-grained acoustic features are fused by a layer adaptation module to generate the training fusion acoustic features.

[0088] The cross module 804 is configured to perform cross fusion on the training fusion acoustic features by a cross attention mechanism to generate training correction acoustic features.

[0089] The feature module 806 is configured to extract accent features of the training fusion acoustic features.

[0090] The training module 808 is configured to train an initial end-to-end dialect recognition model based on the training correction acoustic features and the accent features by a multi-task learning mode to generate a trained end-to-end dialect recognition model, the end-to-end dialect recognition model including a speech recognition model and an accent recognition model. The training module 808 is also configured to determine a multi-task learning framework including a cross information fusion mechanism, the speech recognition model, and the accent recognition model. The speech recognition model is trained based on the multi-task learning framework and the training correction acoustic features. The accent recognition model is trained based on the multi-task learning framework and the accent features. When a whole loss function in the training process meets a set index, the trained end-to-end dialect recognition model is generated.

[0091] The dialect audio recognition system based on multi-layer information fusion in an end-to-end manner according to the present application can efficiently capture and process complex speech signals and multi-accent features, and can also classify and decode dialect audio in real time online, thereby improving the accuracy and robustness of speech recognition.

[0092] As shown in FIG. 9, the electronic device provided by the embodiment of the present application includes a processor 910, a memory 920 and a bus, wherein the processor 910 and the memory 920 complete mutual communication through the bus 940.

[0093] The memory 920 is used to store a computer program.

[0094] The processor 910 is used to execute the program stored in the memory 920, and implement the dialect audio recognition method based on multi-layer information fusion in an end-to-end manner of any of the above-mentioned embodiments.

[0095] The communication interface 920 is used for communication between the above-mentioned electronic device and other devices.

[0096] The memory 920 can include a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory 920 can also be at least one storage device located away from the processor 910.

[0097] The above-mentioned method in the present application, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc.

[0098] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the dialect audio recognition method based on end-to-end multi-layer information fusion of any one of the above-mentioned embodiments. For example, the dialect audio can be preprocessed to generate acoustic features; the acoustic features are input into an encoder, the encoder performs a progressive downsampling operation on the acoustic features to generate multi-layer fine-grained acoustic features; the multi-layer fine-grained acoustic features are subjected to multi-layer information fusion through a layer adaptation module to generate fused acoustic features; the fused acoustic features are subjected to cross fusion through a cross attention mechanism to generate corrected acoustic features; and the corrected acoustic features are input into an end-to-end dialect recognition model to generate a dialect audio recognition result.

[0099] According to an aspect of the present application, a computer program product is provided, comprising: computer programs / instructions which, when executed by a processor, implement the method steps of any of the above embodiments.

[0100] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A dialect audio recognition method based on end-to-end multi-layer information fusion, characterized in that, The method comprises the following steps: audio preprocessing of the dialect audio to generate acoustic features; inputting the acoustic features into an encoder, which performs progressive downsampling operations thereon to generate multi-layer fine-grained acoustic features; performing multi-layer information fusion on the multi-layer fine-grained acoustic features through a layer adaptation module to generate fused acoustic features; cross-fusing the fused acoustic features through a cross-attention mechanism to generate corrected acoustic features; inputting the corrected acoustic features into an end-to-end dialect recognition model to generate a dialect audio recognition result.

2. The method of claim 1, wherein, audio preprocessing of the dialect audio to generate acoustic features, comprising: feature extraction on the dialect audio to generate mel-frequency cepstral coefficient features of the dialect audio; convolutional downsampling operation on the mel-frequency cepstral coefficient features to generate the acoustic features of the dialect audio.

3. The method of claim 1, wherein, inputting the acoustic features into an encoder, which performs progressive downsampling operations thereon to generate multi-layer fine-grained acoustic features, comprising: inputting the acoustic features into an encoder, which comprises a plurality of conformer encoders; the plurality of conformer encoders performing progressive downsampling operations on the acoustic features; generating the multi-layer fine-grained acoustic features through the progressive downsampling operations to reduce the redundancy of the time dimension layer by layer.

4. The method of claim 1, wherein, performing multi-layer information fusion on the multi-layer fine-grained acoustic features through a layer adaptation module to generate fused acoustic features, comprising: the layer adaptation module extracting fine-grained acoustic features of each layer in the multi-layer fine-grained acoustic features; performing multi-layer information fusion on each layer of fine-grained acoustic features respectively to generate the fused acoustic features.

5. The method of claim 1, wherein, cross-fusing the fused acoustic features through a cross-attention mechanism to generate corrected acoustic features, comprising: taking the multi-layer fine-grained acoustic features as key-value vectors and taking the fused acoustic features as query vectors; cross-fusing through the cross-attention mechanism to generate the corrected acoustic features.

6. The method of claim 1, wherein, inputting the corrected acoustic features into an end-to-end dialect recognition model to generate a dialect audio recognition result, comprising: inputting the corrected acoustic features into an end-to-end dialect recognition model; the speech recognition model and the accent recognition model in the end-to-end dialect recognition model respectively recognizing the corrected acoustic features to generate a text recognition result and an accent classification label.

7. The method of claim 1, wherein, Further comprising: training preprocessing of dialect audio with dialect labels to generate training fused acoustic features; cross-fusing the training fused acoustic features through a cross-attention mechanism to generate training corrected acoustic features; extracting accent features of the training fused acoustic features; training an initial end-to-end dialect recognition model based on the training corrected acoustic features and the accent features through a multi-task learning mode to generate a trained end-to-end dialect recognition model, the end-to-end dialect recognition model comprising a speech recognition model and an accent recognition model.

8. The method of claim 7, wherein, training preprocessing of dialect audio with dialect labels to generate training fused acoustic features, comprising: audio preprocessing of dialect audio with dialect labels to generate training acoustic features; input the training acoustic feature into an encoder, the encoder performs a progressive downsampling operation thereon to generate multi-layer training fine-grained acoustic features; perform multi-layer information fusion on the training multi-layer fine-grained acoustic features through a layer adaptation module, generate the training fused acoustic features.

9. The method of claim 7, wherein, train an initial end-to-end dialect recognition model based on the training corrected acoustic features and the accent features through a multi-task learning mode to generate a trained end-to-end dialect recognition model, the end-to-end dialect recognition model including a speech recognition model and an accent recognition model, comprising: determine a multi-task learning framework, the multi-task learning framework including a cross information fusion mechanism, a speech recognition model and an accent recognition model; train the speech recognition model based on the training corrected acoustic features through the multi-task learning framework; train the accent recognition model based on the accent features through the multi-task learning framework; generate the trained end-to-end dialect recognition model when the overall loss function meets the set index during the training process.

10. A dialect audio recognition system based on end-to-end multi-layer information fusion, characterized in that, comprising: a processing module configured to perform audio preprocessing on dialect audio to generate acoustic features; a sampling module configured to input the acoustic features into an encoder, the encoder performing a progressive downsampling operation thereon to generate multi-layer fine-grained acoustic features; a fusion module configured to perform multi-layer information fusion on the multi-layer fine-grained acoustic features through a layer adaptation module to generate fused acoustic features; a correction module configured to perform cross fusion on the fused acoustic features through a cross attention mechanism to generate corrected acoustic features; an identification module configured to input the corrected acoustic features into an end-to-end dialect recognition model to generate a dialect audio recognition result.

Images