Multi-dialect speech recognition method and device, computer device and storage medium

Abstract

The application relates to a multi-dialect speech recognition method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining audio data to be recognized; performing deep frame-level feature extraction on the audio data to be recognized to obtain sentence-level dialect feature information, and encoding the audio data to be recognized to obtain encoded feature information; splicing the sentence-level dialect feature information and the encoded feature information to obtain to-be-recognized feature information; and decoding the to-be-recognized feature information to obtain a speech recognition result. The method can accurately recognize speech and obtain an accurate speech recognition result.

Description

[0001] This application claims priority to Chinese Patent Application No. 202111554070.8, filed on December 17, 2021, entitled “Multi-dialect speech recognition method, apparatus, computer equipment and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for multi-dialect speech recognition. Background Technology

[0003] With the development of artificial intelligence technology, speech recognition technology has emerged. Speech recognition technology is mainly used to convert speech signals into corresponding text representations. Speech recognition technology can be applied to regional dialect recognition.

[0004] In traditional technologies, when applying speech recognition to regional dialect recognition, the main processing method involves first extracting frame-level features from the dialect speech data, then feeding the acoustic features into a decoder. The decoder, through the combined action of an acoustic model, a language model, and a pronunciation dictionary, scores the acoustic features to obtain a decoded sequence, and finally, the corresponding text representation is derived from the decoded sequence. To simultaneously recognize multiple regional dialects, the above processing method is typically used to directly model a mixture of the various regional dialects.

[0005] However, traditional methods suffer from poor recognition performance when recognizing multiple regional dialects because they directly perform mixed modeling of multiple regional dialects. Summary of the Invention

[0006] Therefore, it is necessary to provide a method, apparatus, computer device, storage medium, and program product for multi-dialect speech recognition that can achieve accurate recognition, addressing the aforementioned technical problems.

[0007] A multi-dialect speech recognition method, the method comprising:

[0008] Obtain the audio data to be recognized;

[0009] Deep frame-level feature extraction is performed on the audio data to be recognized to obtain sentence-level dialect feature information, and the audio data to be recognized is encoded to obtain encoded feature information;

[0010] By concatenating sentence-level dialect feature information and encoded feature information, the feature information to be identified is obtained;

[0011] The feature information to be recognized is decoded to obtain the speech recognition result.

[0012] A multi-dialect speech recognition device, the device comprising:

[0013] The acquisition module is used to acquire the audio data to be recognized;

[0014] The feature extraction module is used to perform deep frame-level feature extraction on the audio data to be recognized, obtain sentence-level dialect feature information, and encode the audio data to be recognized to obtain encoded feature information.

[0015] The splicing module is used to splice sentence-level dialect feature information and encoded feature information to obtain the feature information to be identified.

[0016] The decoding module is used to decode the feature information to be recognized and obtain the speech recognition result.

[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0018] Obtain the audio data to be recognized;

[0019] Deep frame-level feature extraction is performed on the audio data to be recognized to obtain sentence-level dialect feature information, and the audio data to be recognized is encoded to obtain encoded feature information;

[0020] By concatenating sentence-level dialect feature information and encoded feature information, the feature information to be identified is obtained;

[0021] The feature information to be recognized is decoded to obtain the speech recognition result.

[0022] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0023] Obtain the audio data to be recognized;

[0024] Deep frame-level feature extraction is performed on the audio data to be recognized to obtain sentence-level dialect feature information, and the audio data to be recognized is encoded to obtain encoded feature information;

[0025] By concatenating sentence-level dialect feature information and encoded feature information, the feature information to be identified is obtained;

[0026] The feature information to be recognized is decoded to obtain the speech recognition result.

[0027] A computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0028] Obtain the audio data to be recognized;

[0029] Deep frame-level feature extraction is performed on the audio data to be recognized to obtain sentence-level dialect feature information, and the audio data to be recognized is encoded to obtain encoded feature information;

[0030] By concatenating sentence-level dialect feature information and encoded feature information, the feature information to be identified is obtained;

[0031] The feature information to be recognized is decoded to obtain the speech recognition result.

[0032] The aforementioned multi-dialect speech recognition methods, devices, computer equipment, storage media, and program products acquire audio data to be recognized, perform deep frame-level feature extraction on the audio data to be recognized to obtain sentence-level dialect feature information, encode the audio data to be recognized to obtain encoded feature information, concatenate the sentence-level dialect feature information and the encoded feature information to obtain the feature information to be recognized, and decode the feature information to be recognized to obtain the speech recognition result. By additionally concatenating sentence-level dialect feature information before decoding, additional dialect information is provided for decoding, enabling accurate speech recognition to be achieved during decoding using the concatenated feature information to be recognized, thus obtaining an accurate speech recognition result. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a multi-dialect speech recognition method in one embodiment;

[0034] Figure 2 This is a schematic diagram of the encoding block of a Conformer (convolution-enhanced Transformer) Encoder model in one embodiment;

[0035] Figure 3 This is a schematic diagram of frame-level depth feature extraction in one embodiment;

[0036] Figure 4 This is a schematic diagram of the identification process of a CTC (Connectionist Temporal Classification) network in one embodiment;

[0037] Figure 5 This is a schematic diagram of the model structure of a multi-dialect speech recognition model in one embodiment;

[0038] Figure 6 This is a schematic diagram of the model structure of a multi-dialect speech recognition model in another embodiment;

[0039] Figure 7 This is a schematic diagram of the model structure of a multi-dialect speech recognition model in another embodiment;

[0040] Figure 8This is a schematic diagram illustrating the application of a multi-dialect speech recognition model to multi-dialect speech recognition within the same language in one embodiment;

[0041] Figure 9 This is a structural block diagram of a multi-dialect speech recognition device in one embodiment;

[0042] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] This application relates to the field of artificial intelligence (AI). AI is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0044] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. This application mainly focuses on speech processing technology, with key technologies including automatic speech recognition, speech synthesis, and voiceprint recognition. Enabling computers to hear, see, speak, and feel represents the future direction of human-computer interaction, with speech being one of the most promising methods for future interaction.

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] In one embodiment, such as Figure 1As shown, a multi-dialect speech recognition method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, and vehicle terminals. The server can be a standalone server or a server cluster composed of multiple servers, or it can be a node on a blockchain. In this embodiment, the method includes the following steps:

[0047] Step 102: Obtain the audio data to be recognized.

[0048] The audio data to be recognized refers to the data corresponding to dialect speech. For example, the audio data to be recognized can specifically refer to the acoustic features corresponding to the dialect speech. For instance, acoustic features can specifically be MFCC (Mel-Frequency Cepstral Coefficients) features, Fbank (Filter Bank) features, etc.

[0049] Specifically, when dialect speech recognition is required, the server will acquire the dialect speech, perform operations such as spectrum framing, time-frequency conversion, and filtering on the dialect speech, and obtain the corresponding audio data to be recognized.

[0050] Step 104: Perform deep frame-level feature extraction on the audio data to be identified to obtain sentence-level dialect feature information, and encode the audio data to be identified to obtain encoded feature information.

[0051] Among them, deep frame-level feature extraction is used to extract deeper information from the frame-level audio data to be recognized, so as to obtain sentence-level dialect feature information. Encoding is used to model the audio data to be recognized, encoding the audio data to be recognized into higher-dimensional features, and obtaining encoded feature information.

[0052] Specifically, the server extracts features from the audio data to be recognized based on a deep neural network to extract frame-level deep feature information. This frame-level deep feature information is then pooled to convert it into fixed-dimensional sentence-level dialect feature information. Furthermore, in this embodiment, the process of obtaining sentence-level dialect feature information can be implemented using a pre-trained dialect recognition model. This model consists of three modules: a deep neural network, a temporal pooling module, and a classification recognition model. The deep neural network is used to extract deeper information from the frame-level acoustic features, thus obtaining the frame-level deep feature information. The temporal pooling module integrates and statistically analyzes the temporally related frame-level deep feature information, ultimately outputting a fixed-dimensional sentence-level feature information. The temporal pooling layer can be selected using global average pooling, max pooling, or min pooling. The classification and recognition module is used to map sentence-level feature information to target dialect categories. Specifically, the classification and recognition module can be a fully connected layer classifier. Through two fully connected network layers, it maps sentence-level feature information to target dialect categories and outputs the probability of the corresponding dialect after the final activation function.

[0053] Furthermore, deep neural networks can specifically be time-delay neural networks. Compared to traditional statistical methods such as Gaussian mixture models, time-delay neural networks can learn information from data more effectively. After the dialect recognition model is pre-trained, the server can input the audio data to be recognized into the dialect recognition model, and then extract sentence-level dialect feature information with a fixed dimension for predicting the classification result through the penultimate layer of the dialect recognition model.

[0054] Specifically, the server can encode the audio data to be recognized using a pre-trained encoding network to obtain encoded feature information. This pre-trained encoding network can be a Conformer Encoder model, an improvement on the Transformer model. By introducing convolutional neural networks, it enhances speech recognition performance, taking into account both local and full-sequence information. The Transformer model's encoder contains six encoder blocks, each consisting of a self-attention module and a feed-forward module. The Conformer Encoder model improves upon the Transformer's encoder by adding a convolution module to each block, adding feed-forward modules at both ends of each block, introducing residuals into each module, and incorporating relative position encoding into the self-attention module. For example... Figure 2 As shown, the main structure of the encoding blocks (i.e., Conformer Blocks) of the Conformer Encoder model can include four sub-modules: Feed Forward Module, Multi-Head Self attention Module, Convolution Module, and LayerNorm.

[0055] The mathematical expression for the Encoder block in the Conformer Encoder model is as follows:

[0056]

[0057]

[0058] x Conv =x MHSA +Conv(x MHSA )

[0059]

[0060] Where x represents the input data, x FFN1 For the feature information obtained through the first feedforward module, x MHSA x represents the feature information obtained after processing by the multi-head self-attention module.Conv x represents the feature information obtained through the convolution module. FFN2 This represents the feature information after residual processing. This represents the feature information obtained through the second feedforward module, and LayerNorm() represents the residual function.

[0061] Step 106: Concatenate sentence-level dialect feature information and encoded feature information to obtain the feature information to be identified.

[0062] Specifically, the server concatenates sentence-level dialect feature information and encoded feature information to obtain richer feature information for recognition, enabling accurate speech recognition. The concatenation method involves appending sentence-level dialect feature information to the corresponding feature information of each frame in the encoded feature information to obtain the feature information to be recognized. For example, assuming the encoded feature information includes 10 frames of feature information, each frame having a dimension of 40 (i.e., the encoded feature information is 10*40 dimensions), and the sentence-level dialect feature information also has a dimension of 40, the concatenation involves appending this 40-dimensional feature to each of these 10 frames, ultimately resulting in 10*80 dimensional feature information for recognition.

[0063] Step 108: Decode the feature information to be recognized to obtain the speech recognition result.

[0064] Speech recognition results refer to the results obtained by recognizing the audio data to be recognized. For example, a speech recognition result can specifically be a target text recognition result, that is, the text expression corresponding to the audio data to be recognized. Another example is an audio classification result, that is, the dialect category corresponding to the audio data to be recognized.

[0065] Specifically, the server decodes the feature information to be recognized using a trained decoding network to obtain the speech recognition result. This trained decoding network includes a classification decoding network and a text decoding network. The classification decoding network classifies the feature information to determine the category of the audio data to be recognized (the audio classification result), while the text decoding network decodes the feature information to determine the corresponding text representation (the target text recognition result). Further, the text decoding network may include an attention mechanism decoding network and a temporal classification decoding network. The attention mechanism decoding network decodes the feature information based on an attention mechanism to obtain the text decoding result, while the temporal classification decoding network decodes the feature information based on temporal classification to obtain the text decoding result. The server can then combine the two text decoding results to obtain the target text recognition result.

[0066] Furthermore, when combining the two text decoding results to obtain the target text recognition result, the server can use either text decoding result as the target text recognition result, or it can input the text decoding result output by the temporal classification decoding network into the attention mechanism decoding network, so that the attention mechanism decoding network combines the text decoding result output by the temporal classification decoding network and the feature information to be recognized to decode and obtain the text decoding result, and use the text decoding result as the target text recognition result.

[0067] Among them, the classification decoding network refers to a neural network used to determine the dialect category. This network only needs to be able to identify the dialect category of the audio data to be identified based on the feature information to be identified; this embodiment does not limit the specific network. For example, the classification decoding network can be VGGNet, ResNet, etc., which processes the feature information to be identified through hidden layers, fully connected layers, etc., to obtain the audio classification result. The attention mechanism decoding network refers to a network built based on the attention mechanism, which can output the text sequence corresponding to the audio data to be identified by decoding the feature information to be identified. For example, the attention mechanism decoding network can specifically refer to the Transformer Decoder module. The temporal classification decoding network is used to solve the classification problem of temporal data. By decoding the feature information to be identified, it can also output the text sequence corresponding to the audio data to be identified. For example, the temporal classification decoding network can specifically be the CTC (Connectionist temporal classification) network.

[0068] The aforementioned multi-dialect speech recognition method acquires the audio data to be recognized, performs deep frame-level feature extraction on the audio data to be recognized to obtain sentence-level dialect feature information, encodes the audio data to be recognized to obtain encoded feature information, concatenates the sentence-level dialect feature information and encoded feature information to obtain the feature information to be recognized, and decodes the feature information to be recognized to obtain the speech recognition result. By additionally concatenating sentence-level dialect feature information before decoding, additional dialect information is provided to the decoding process, enabling accurate speech recognition to be achieved by utilizing the concatenated feature information to be recognized during decoding, thus obtaining an accurate speech recognition result.

[0069] In one embodiment, deep frame-level feature extraction is performed on the audio data to be recognized to obtain sentence-level dialect feature information, including:

[0070] Based on deep neural networks, feature extraction is performed on the audio data to be recognized to obtain frame-level deep feature information;

[0071] Pooling is performed on frame-level deep feature information to obtain sentence-level dialect feature information.

[0072] Specifically, the audio data to be identified consists of multi-frame acoustic features. The server will use a deep neural network to simultaneously consider the acoustic features of consecutive frames to extract deep frame-level features from the audio data to be identified. The extracted deep frame-level feature information is then integrated by pooling the deep frame-level feature information, which is then converted into fixed-dimensional sentence-level dialect feature information.

[0073] The deep frame-level feature extraction based on the acoustic features of consecutive frames, considering deep neural networks simultaneously, refers to extracting acoustic features from multiple consecutive frames simultaneously using deep neural networks. For example, with a latency of 2, acoustic features from three consecutive frames will be extracted simultaneously. The pooling operation can employ various methods such as global average pooling, max pooling, and min pooling; this embodiment does not specify a particular pooling method. The fixed dimension during transformation can be set as needed. Furthermore, the dimensionality of the sentence-level dialect feature information obtained after pooling may be high. Therefore, the server needs to reduce the dimensionality of the obtained sentence-level dialect feature information to obtain the final sentence-level dialect feature information. For example, the server can use LDA (linear discriminant analysis) to reduce the dimensionality and enhance the discriminative power of the sentence-level dialect feature information.

[0074] For example, in this embodiment, the process of extracting deep frame-level features from the audio data to be recognized to obtain sentence-level dialect feature information can be implemented based on a pre-trained dialect recognition model. This pre-trained dialect recognition model can be a dialect recognition model based on a time-delay neural network. Compared with traditional statistical methods such as Gaussian mixture models, time-delay neural networks can learn information from the data more effectively. The dialect recognition model in this embodiment mainly consists of three modules: a frame-level deep feature extraction module based on a time-delay neural network, a temporal pooling module, and a classification and recognition module. The frame-level deep feature extraction module is used to extract deeper information from the frame-level acoustic features, thereby obtaining frame-level deep feature information. The temporal pooling module is used to integrate and statistically analyze the temporally related frame-level deep feature information, and finally output a sentence-level feature information of a fixed dimension. The selection of the temporal pooling layer mainly includes global average pooling, max pooling, and min pooling. The classification and recognition module is used to map sentence-level feature information to target dialect categories. Specifically, the classification and recognition module can be a fully connected layer classifier. Through two fully connected network layers, it maps sentence-level feature information to target dialect categories and outputs the probability of the corresponding dialect after the final activation function.

[0075] To further illustrate, the frame-level deep feature extraction process can be as follows: Figure 3 As shown, assuming a delay of 2, three consecutive frames (frame 1, frame 2, and frame 3) will be considered. The hidden layers serve as feature extraction layers. Each rectangle in the input layer contains 13 small black dots, representing the 13-dimensional MFCC features of that frame, i.e., acoustic features. Assuming there are 10 hidden layers, the number of connection weights is 3 * 13 * 10 = 390.

[0076] Specifically, by inputting the audio data to be recognized into the dialect recognition model, the server can extract a sentence-level dialect feature information with a fixed dimension for predicting the classification result through the penultimate layer of the dialect recognition model.

[0077] It should be noted that when training the pre-trained dialect recognition model, the training data used can be training audio data with category labels. The category labels are used to identify dialect categories, and the training audio data refers to the training data corresponding to the dialect speech. For example, the training audio data can specifically refer to the acoustic features corresponding to the dialect speech used for training. It should be noted that, to conserve resources, the training audio data in this embodiment can be the same as the training audio data used in training the initial decoding network mentioned below. The training loss function can be a function suitable for multi-class classification, such as cross-entropy; no specific loss function is limited in this embodiment.

[0078] In this embodiment, feature extraction is performed on the audio data to be recognized based on a deep neural network to obtain frame-level deep feature information. Pooling is then performed on the frame-level deep feature information to obtain sentence-level dialect feature information. This results in rich sentence-level dialect feature information, which can then be used to achieve accurate speech recognition.

[0079] In one embodiment, decoding the feature information to be recognized to obtain the speech recognition result includes:

[0080] The trained decoding network decodes the feature information to be recognized to obtain the speech recognition result. The trained decoding network is constructed in the following way:

[0081] Acquire training audio data carrying recognition labels and an initial decoding network, which includes a classification decoding network and a text decoding network;

[0082] Feature extraction is performed on the training audio data to obtain the feature information to be decoded;

[0083] The feature information to be decoded is input into the classification decoding network to obtain the classification decoding result, and the feature information to be decoded is input into the text decoding network to obtain the text decoding result;

[0084] Based on the classification decoding results, text decoding results, and recognition labels, the initial decoding network is adjusted to obtain the trained decoding network.

[0085] The identification labels are used to identify different training audio data. Specifically, identification labels can include text labels and category labels. Text labels represent the correct text representation of the training audio data, while category labels identify the dialect category. Training audio data refers to training data corresponding to dialect speech. For example, training audio data can specifically refer to the acoustic features corresponding to the dialect speech used for training. The initial decoding network refers to the decoding network before parameter training. The initial decoding network includes a classification decoding network and a text decoding network. The classification decoding network is used to classify and decode the training audio data to obtain the corresponding predicted dialect category, while the text decoding network is used to decode the training audio data to obtain the corresponding predicted text representation. The classification decoding result refers to the probability that the training audio data belongs to each dialect category. The text decoding result includes the text sequence corresponding to the training audio data and the probability of obtaining that text sequence.

[0086] Specifically, the server decodes the feature information to be recognized using a trained decoding network to obtain the speech recognition result. This trained decoding network includes a classification decoding network and a text decoding network. The classification decoding network classifies and decodes the feature information to determine the category of the audio data to be recognized, i.e., the audio classification result. The text decoding network decodes the feature information to determine the corresponding text expression of the audio data to be recognized, i.e., the target text recognition result. After obtaining the audio classification result and the target text recognition result, the server uses these two results as the speech recognition result.

[0087] Specifically, when constructing the trained decoding network, the server acquires training audio data carrying category labels and an initial decoding network. It performs deep frame-level feature extraction on the training audio data to obtain sentence-level dialect feature information corresponding to the training audio data. The server then encodes the training audio data to obtain encoded feature information corresponding to the training audio data. By concatenating the sentence-level dialect feature information and the encoded feature information corresponding to the training audio data, the server obtains the feature information to be decoded. This feature information is then input into a classification decoding network, which classifies the feature information to be decoded, obtaining a classification decoding result. Finally, the feature information is input into a text decoding network for text recognition, obtaining a text decoding result. Based on the classification decoding result, the text decoding result, and the recognition labels, the server calculates the network loss function. The parameters of the initial decoding network are then adjusted based on the network loss function to obtain the trained decoding network.

[0088] In this embodiment, by simultaneously employing a classification decoding network and a text decoding network for multi-task learning during the training of the decoding network, the speech recognition performance of the trained decoding network in multi-dialect scenarios can be improved, thereby enabling accurate speech recognition using the trained decoding network.

[0089] In one embodiment, the initial decoding network is adjusted based on the classification decoding result, the text decoding result, and the recognition label to obtain a trained decoding network, including:

[0090] Based on the classification decoding results and the category labels in the recognition tags, calculate the first loss function, and based on the text decoding results and the text labels in the recognition tags, calculate the second loss function.

[0091] By weighting the first and second loss functions, the network loss function is obtained.

[0092] Based on the network loss function, the parameters of the initial decoding network are adjusted. The process of inputting the feature information to be decoded into the classification decoding network to obtain the classification decoding result, and inputting the feature information to be decoded into the text decoding network to obtain the text decoding result continues until the network loss function meets the preset stopping iteration condition, thus obtaining the trained decoding network.

[0093] Specifically, since the classification decoding result represents the probability of the training audio data belonging to each dialect category, the server can determine the target category probability from the classification decoding result based on the category labels in the recognition tags. The target category probability refers to the probability predicted by the classification decoding network that the training audio data belongs to the category corresponding to the category label. After obtaining the target category probability, the server can calculate the first loss function of the classification decoding network when making predictions. For example, the true category probability should be 1, but the target category probability obtained in actual prediction is usually a value less than 1. By calculating the difference between the true category probability and the target category probability, the first loss function can be obtained. For a single prediction, the first loss function can be the square of the difference between the true category probability and the target category probability.

[0094] Specifically, since the text decoding result includes the text sequence corresponding to the training audio data and the probability of obtaining that text sequence, and the text label in the recognition tag is the real text sequence corresponding to the training audio data, the server can determine the target text sequence probability from the text decoding result based on the text label. The target text sequence probability refers to the probability of obtaining the text label predicted by the text decoding network. After obtaining the target text sequence probability, the server can calculate the second loss function of the text decoding network when performing text decoding.

[0095] It should be noted that when there are at least two sub-text decoding networks in the text decoding network, the server will calculate the loss function of each sub-text decoding network separately, and then weight the loss functions of the sub-text decoding networks to obtain the second loss function. For example, the text decoding network may include an attention mechanism decoding network and a temporal classification decoding network. The server will calculate the loss functions of the attention mechanism decoding network and the temporal classification decoding network separately, and then obtain the second loss function based on the calculated loss functions.

[0096] Specifically, after obtaining the first and second loss functions, the server weights the first and second loss functions to obtain the network loss function. Based on the network loss function, the parameters of the initial decoding network are adjusted using methods such as gradient descent. The process then returns to inputting the feature information to be decoded into the classification decoding network to obtain the classification decoding result, and inputting the feature information to be decoded into the text decoding network to obtain the text decoding result. This process yields new classification decoding results and new text decoding results. These new results are then used to recalculate the new network loss function until the network loss function meets a preset stopping iteration condition, resulting in a trained decoding network. The weighting coefficients can be set as needed during the weighting calculation. The preset stopping iteration condition can also be set as needed; for example, it could be that the network loss function converges, or it could be that the network loss function is less than a preset loss function threshold. The preset loss function threshold can also be set as needed.

[0097] In this embodiment, by using the classification decoding result and category label to calculate the first loss function, and using the text decoding result and text label to calculate the second loss function, an accurate network loss function can be obtained by comprehensively considering the classification decoding loss and the text decoding loss. This network loss function can then be used to adjust the parameters of the initial decoding network to optimize it. By returning to the steps of inputting the feature information to be decoded into the classification decoding network to obtain the classification decoding result, and inputting the feature information to be decoded into the text decoding network to obtain the text decoding result, iterative training of the initial decoding network can be further realized. Thus, when the network loss function meets the preset stopping iteration condition, a trained decoding network capable of accurate speech recognition can be obtained.

[0098] In one embodiment, the speech recognition result is obtained by decoding the feature information to be recognized using a trained decoding network, including:

[0099] The audio classification result is obtained by decoding the feature information to be identified through the classification decoding network in the trained decoding network, and the target text recognition result is obtained by decoding the feature information to be identified through the text decoding network in the trained decoding network.

[0100] Based on the audio classification results and the target text recognition results, the speech recognition results are obtained.

[0101] Specifically, the server uses a classification decoding network to classify and decode the feature information to be identified, determine the probability that the audio data to be identified belongs to each dialect category, identify the dialect category corresponding to the highest probability, and use it as the audio classification result. Then, it uses a text decoding network to decode the feature information to be identified, determine the probability that the feature information to be identified belongs to a text sequence, identify the text sequence corresponding to the highest probability, and use it as the target text recognition result. The audio classification result and the target text recognition result are used as the speech recognition result.

[0102] In this embodiment, an audio classification result is obtained by using a classification decoding network, and a target text recognition result is obtained by using a text decoding network. Based on the audio classification result and the target text recognition result, a more comprehensive speech recognition result can be obtained.

[0103] In one embodiment, the text decoding network includes an attention mechanism decoding network and a temporal classification decoding network; the text decoding network in the trained decoding network decodes the feature information to be recognized, and the target text recognition result is obtained, including:

[0104] The first text recognition result is obtained by decoding the feature information to be recognized through an attention mechanism decoding network, and the second text recognition result is obtained by decoding the feature information to be recognized through a temporal classification decoding network.

[0105] Based on the first text recognition result and the second text recognition result, the target text recognition result is obtained.

[0106] Among them, the temporal classification decoding network refers to the network used to solve the classification problem of temporal data.

[0107] Specifically, the server uses an attention-based decoding network to decode the feature information to be recognized, obtaining a text sequence corresponding to the feature information, i.e., the first text recognition result. The decoded text sequence is a sequence composed of target text characters. Through decoding, the attention-based decoding network can predict the probability that each target text character in the decoded text sequence is a preset text character, and selects the preset text character with the highest probability as the target text character. Here, preset text characters refer to characters existing in a pre-built dialect dictionary, which can be set as needed. For example, the attention-based decoding network can specifically refer to the Transformer Decoder module.

[0108] Specifically, the server decodes the feature information to be recognized using a temporal classification decoding network to obtain a text sequence corresponding to the audio data to be recognized. This text sequence can specifically be a syllable sequence. For example, the temporal classification decoding network can be a CTC network. The CTC network is an improved RNN model that mainly addresses the problems in temporal models where the number of inputs exceeds the number of outputs and how to align inputs and outputs. After the feature information to be recognized is input into the CTC network, the CTC network can perform syllable matching on the feature information to obtain a syllable sequence corresponding to the feature information. By deduplicating and filtering the syllable sequence, the second text recognition result can be obtained.

[0109] Specifically, after combining the two text decoding results to obtain the target text recognition result, the server can use either text decoding result as the target text recognition result, or it can input the text decoding result output by the temporal classification decoding network into the attention mechanism decoding network, so that the attention mechanism decoding network combines the text decoding result output by the temporal classification decoding network and the feature information to be recognized to decode and obtain the text decoding result, which is then used as the target text recognition result.

[0110] In this embodiment, the first text recognition result is obtained by using an attention mechanism decoding network, and the second text recognition result is obtained by using a temporal classification decoding network. The first and second text recognition results can be combined to obtain a more accurate target text recognition result.

[0111] In one embodiment, the second text recognition result is obtained by decoding the feature information to be recognized using a temporal classification decoding network, including:

[0112] A temporal classification decoding network is used to match syllables of the feature information to be identified, thereby obtaining a syllable sequence corresponding to the feature information to be identified.

[0113] The syllables in the syllable sequence are filtered and deduplicated to obtain the second text recognition result.

[0114] Specifically, the temporal classification decoding network divides the feature information to be identified into several segments, performs syllable matching on each segment, and determines the probability that each segment belongs to each preset syllable. Based on the probability of each segment belonging to each preset syllable, a syllable sequence corresponding to the feature information to be identified can be obtained. Here, preset syllables refer to syllables existing in dialects. It should be noted that, in order to achieve alignment, there may be repeated syllables and blank syllables in the syllable sequence. Therefore, after obtaining the syllable sequence, the server needs to filter and remove duplicate syllables in the syllable sequence to obtain the second text recognition result. Deduplication refers to merging identical syllables that are not separated by blank syllables into a single syllable, and filtering refers to removing blank syllables from the syllable sequence after merging.

[0115] For example, such as Figure 4 As shown, the recognition process of the CTC network is illustrated using the decoding of "world!" as an example. It divides the feature information to be recognized into several segments, performs syllable matching on each segment, and predicts the syllable sequence corresponding to the feature information to be recognized. Figure 4 The first line of the syllable sequence shows repeated syllables and blank syllables (denoted by the symbol ∈). Therefore, it is necessary to merge identical syllables that are not separated by blank syllables and remove blank syllables from the syllable sequence to obtain a deduplicated and filtered syllable sequence. Figure 4 After deduplication and filtering, the second line of the syllable sequence needs to be normalized to the same fixed length to obtain the final syllable sequence. Figure 4 The third line of syllables in the sequence.

[0116] In this embodiment, by using a temporal classification decoding network to perform syllable matching on the feature information to be identified, a syllable sequence corresponding to the feature information to be identified is obtained. By filtering and deduplicating the syllables in the syllable sequence, the second text recognition result can be obtained.

[0117] This application also provides an application scenario in which the above-described multi-dialect speech recognition method is applied. Specifically, the multi-dialect speech recognition method is applied in this scenario as follows:

[0118] The multi-dialect speech recognition method in this application can be implemented based on a pre-constructed multi-dialect speech recognition model. The specific structure of this multi-dialect speech recognition model can be as follows: Figure 5As shown, the system includes a dialect recognition model 502, an encoding network 504, and a decoding network 506. The dialect recognition model 504 performs deep frame-level feature extraction on the audio data to be recognized, obtaining sentence-level dialect feature information. The encoding network 506 encodes the audio data to be recognized, obtaining encoded feature information. The decoding network 508 encodes the feature information to be recognized obtained by concatenating the sentence-level dialect feature information and the encoded feature information, resulting in a speech recognition result. By additionally concatenating sentence-level dialect feature information before decoding, providing additional dialect information to the decoding network, accurate speech recognition can be achieved during decoding using the concatenated feature information, resulting in an accurate speech recognition result.

[0119] Furthermore, such as Figure 6 As shown, the decoding network 506 can specifically include a classification decoding network 602 and a text decoding network 604. The classification decoding network decodes the feature information to be recognized to obtain an audio classification result, while the text decoding network decodes the feature information to be recognized to obtain a target text recognition result. The server uses the audio classification result and the target text recognition result as the speech recognition result. Furthermore, as... Figure 7 As shown, the text decoding network 604 may further include an attention mechanism decoding network 702 and a temporal classification decoding network 704. The attention mechanism decoding network 702 can decode the feature information to be identified based on the attention mechanism to obtain a first text recognition result. The temporal classification decoding network 704 can decode the feature information to be identified based on temporal classification to obtain a second text recognition result. The server will combine the first text recognition result and the second text recognition result to obtain the target text recognition result.

[0120] In this embodiment, the dialect recognition model can be a dialect recognition model based on a time-delay neural network, the encoding network can be the Encoder module in the sequence-to-sequence Conformer end-to-end speech recognition model, the attention mechanism decoding network can be the Decoder module in the sequence-to-sequence Conformer end-to-end speech recognition model, and the temporal classification decoding network can be a CTC network.

[0121] The following is based on Figure 7 Taking the model structure in the example, the construction process of the multi-dialect speech recognition model in this embodiment will be explained. Specifically, the multi-dialect speech recognition model is constructed in the following way:

[0122] The server acquires training audio data carrying recognition labels and simultaneously inputs this data into a dialect recognition model and an encoding network. The dialect recognition model obtains sentence-level dialect feature information, while the encoding network obtains encoded feature information. Specifically, the dialect recognition model is used to extract features from the audio data to be recognized based on a deep neural network, obtaining frame-level deep feature information. Pooling is then performed on the frame-level deep feature information to obtain sentence-level dialect feature information.

[0123] After obtaining sentence-level dialect feature information and encoded feature information, the server concatenates these two information to obtain the feature information to be decoded. This feature information is simultaneously input into a classification decoding network, an attention mechanism decoding network, and a temporal classification decoding network. The classification decoding network yields the classification decoding result, the attention mechanism decoding network yields the first text decoding result, and the temporal classification decoding network yields the second text decoding result. Based on the classification decoding result and the category label in the recognition tags, a first loss function is calculated. Based on the first and second text decoding results and the text labels in the recognition tags, a second loss function is calculated. Based on the first and second loss functions, the model loss function is obtained. Based on the model loss function, the parameters of all models and networks in the multi-dialect speech recognition model are adjusted until the model loss function meets a preset stopping iteration condition, thus completing the construction of the multi-dialect speech recognition model. The formula for calculating the model loss function can be:

[0124]

[0125] Where X represents the training audio data, and Y represents the output text decoding result. The loss corresponding to the decoding result of the first text, The loss corresponding to the second text decoding result, Let Z be the first loss function, Z be the output classification and decoding result, and λ1 and λ2 be the weighting coefficients. The first half of the equation is... The latter half represents the loss related to speech-to-text recognition. The contribution of both speech classification and recognition loss is controlled by λ². Specifically, when the time-order classification decoding network is a CTC network, This is the corresponding CTC loss function.

[0126] The multi-dialect speech recognition model constructed in this embodiment extracts sentence-level dialect feature information, concatenates sentence-level dialect feature information and encoded feature information to obtain the feature information to be decoded. This provides additional dialect information to the multi-dialect speech recognition model, improving its speech recognition performance across multiple dialects. By adding a classification decoding network in addition to the two text decoding networks during decoding, the multi-dialect speech recognition model can distinguish dialect categories to a certain extent, further improving its speech recognition performance across multiple dialects. At the same time, the multi-dialect speech recognition model can output dialect categories, thus providing dialect categories for subsequent services such as speech synthesis while performing speech recognition, which is beneficial for service integration.

[0127] This application also provides another application scenario where the aforementioned multi-dialect speech recognition method is applied. The multi-dialect speech recognition method in this application can be applied to the simultaneous recognition of multiple dialects within the same language. Specifically, the application of this multi-dialect speech recognition method in this scenario is as follows:

[0128] Suppose a language can be categorized into dialects A, B, and C. Existing speech recognition methods typically model only a single dialect, such as dialect A. Furthermore, due to limitations in the construction of pronunciation dictionaries, the performance of mixed dialect modeling suffers significantly compared to single-dialect modeling. For languages ​​with multiple dialect regions, a model for dialect A is difficult to apply in dialect regions B and C. Therefore, it is usually necessary to build three separate speech recognition models for each dialect. Moreover, since dialects A, B, and C typically differ considerably, directly mixing and modeling these three dialects leads to a significant decrease in recognition performance across different dialects.

[0129] The multi-dialect speech recognition method provided in this application, by using three dialect data as training audio data and based on the construction process of the multi-dialect speech recognition model involved in the above embodiments, can construct a multi-dialect speech recognition model that simultaneously supports the speech recognition of three dialects. Wherein, if the language is a phonetic script without spaces for word segmentation, single characters can be segmented using syllables. In this application, single characters of the language are used as modeling units, and the corresponding single-character dictionary is statistically derived from the text data through syllable segmentation.

[0130] Specifically, with Figure 7 Based on the multi-dialect speech recognition model in the present invention, this paper provides an application of the multi-dialect speech recognition method in the same language for multi-dialect speech recognition, and the corresponding application diagram is shown below. Figure 8 As shown, this multi-dialect speech recognition method specifically includes the following steps:

[0131] The server acquires speech data from multiple dialects within the same language, inputs the speech data into a dialect recognition model and an encoding module respectively, obtains sentence-level dialect feature information through the dialect recognition model, and obtains encoded feature information through the encoding network. Specifically, the dialect recognition model is used to extract features from the speech data based on a deep neural network to obtain frame-level deep feature information, and then performs pooling operations on the frame-level deep feature information to obtain sentence-level dialect feature information.

[0132] After obtaining sentence-level dialect feature information and encoded feature information, the server concatenates these two information to obtain the feature information to be recognized. This feature information is then simultaneously input into a classification decoding network, an attention mechanism decoding network, and a temporal classification decoding network. The classification decoding network obtains the audio classification result, indicating whether the speech data belongs to dialect A, dialect B, or dialect C. The attention mechanism decoding network yields the first text recognition result, and the temporal classification decoding network yields the second text recognition result. Based on the first and second text recognition results, the target text recognition result is obtained. Furthermore, the server can select either the first or second text recognition result as the final target text recognition result. It should be noted that because the temporal classification decoding network has a faster decoding speed, the server can directly use the second text recognition result as the target text recognition result.

[0133] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0134] In one embodiment, such as Figure 9 As shown, a multi-dialect speech recognition device is provided. This device can be a software module, a hardware module, or a combination of both integrated into a computer device. Specifically, the device includes: an acquisition module 902, a feature extraction module 904, a splicing module 906, and a decoding module 908, wherein:

[0135] The acquisition module 902 is used to acquire the audio data to be recognized;

[0136] The feature extraction module 904 is used to perform deep frame-level feature extraction on the audio data to be recognized to obtain sentence-level dialect feature information, and to encode the audio data to be recognized to obtain encoded feature information.

[0137] The splicing module 906 is used to splice sentence-level dialect feature information and encoded feature information to obtain the feature information to be identified.

[0138] The decoding module 908 is used to decode the feature information to be recognized and obtain the speech recognition result.

[0139] The aforementioned multi-dialect speech recognition device acquires the audio data to be recognized, performs deep frame-level feature extraction on the audio data to be recognized to obtain sentence-level dialect feature information, encodes the audio data to be recognized to obtain encoded feature information, concatenates the sentence-level dialect feature information and the encoded feature information to obtain the feature information to be recognized, decodes the feature information to be recognized to obtain the speech recognition result. It can provide additional dialect information for decoding by concatenating sentence-level dialect feature information before decoding, so that the concatenated feature information to be recognized can be used to achieve accurate speech recognition and obtain accurate speech recognition results.

[0140] In one embodiment, the feature extraction module is further configured to extract features from the audio data to be recognized based on a deep neural network to obtain frame-level deep feature information. Pooling is then performed on the frame-level deep feature information to obtain sentence-level dialect feature information.

[0141] In one embodiment, the decoding module is further configured to decode the feature information to be recognized through a trained decoding network to obtain a speech recognition result. The trained decoding network is constructed in the following manner: acquiring training audio data carrying recognition labels and an initial decoding network, the initial decoding network including a classification decoding network and a text decoding network; extracting features from the training audio data to obtain the feature information to be decoded; inputting the feature information to be decoded into the classification decoding network to obtain a classification decoding result; inputting the feature information to be decoded into the text decoding network to obtain a text decoding result; and adjusting the initial decoding network according to the classification decoding result, the text decoding result, and the recognition labels to obtain the trained decoding network.

[0142] In one embodiment, the decoding module is further configured to calculate a first loss function based on the classification decoding result and the category label in the recognition label, and calculate a second loss function based on the text decoding result and the text label in the recognition label. The first loss function and the second loss function are weighted to obtain a network loss function. Based on the network loss function, the parameters of the initial decoding network are adjusted. The process of inputting the feature information to be decoded into the classification decoding network to obtain the classification decoding result and inputting the feature information to be decoded into the text decoding network to obtain the text decoding result continues until the network loss function meets the preset stopping iteration condition, thus obtaining the trained decoding network.

[0143] In one embodiment, the decoding module is further configured to decode the feature information to be identified through the classification decoding network in the trained decoding network to obtain the audio classification result, and decode the feature information to be identified through the text decoding network in the trained decoding network to obtain the target text recognition result, and obtain the speech recognition result based on the audio classification result and the target text recognition result.

[0144] In one embodiment, the text decoding network includes an attention mechanism decoding network and a temporal classification decoding network; the decoding module is further configured to decode the feature information to be identified through the attention mechanism decoding network to obtain a first text recognition result, and decode the feature information to be identified through the temporal classification decoding network to obtain a second text recognition result, and obtain a target text recognition result based on the first text recognition result and the second text recognition result.

[0145] In one embodiment, the decoding module is further configured to perform syllable matching on the feature information to be identified through a temporal classification decoding network to obtain a syllable sequence corresponding to the feature information to be identified, and to filter and remove duplicate syllables in the syllable sequence to obtain a second text recognition result.

[0146] Specific limitations regarding the multi-dialect speech recognition device can be found in the limitations of the multi-dialect speech recognition method above, and will not be repeated here. Each module in the aforementioned multi-dialect speech recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0147] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as audio data to be recognized. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-dialect speech recognition method.

[0148] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0150] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0151] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-dialect speech recognition method, characterized in that, The method includes: Obtain the audio data to be recognized; Deep frame-level feature extraction is performed on the audio data to be identified to obtain sentence-level dialect feature information, and the audio data to be identified is encoded to obtain encoded feature information; By concatenating the sentence-level dialect feature information and the encoded feature information, the feature information to be identified is obtained; The audio classification result is obtained by decoding the feature information to be identified through the classification decoding network in the trained decoding network, and the target text recognition result is obtained by decoding the feature information to be identified through the text decoding network in the trained decoding network. Based on the audio classification result and the target text recognition result, a speech recognition result is obtained; wherein, the trained decoding network is constructed in the following manner: Acquire training audio data carrying recognition labels and an initial decoding network, wherein the initial decoding network includes a classification decoding network and a text decoding network; the recognition labels include category labels and text labels; Feature extraction is performed on the training audio data to obtain the feature information to be decoded; The feature information to be decoded is input into the classification decoding network to obtain the classification decoding result, and the feature information to be decoded is input into the text decoding network to obtain the text decoding result; Based on the classification decoding result, the text decoding result, the category label, and the text label, the initial decoding network is adjusted to obtain the trained decoding network.

2. The method according to claim 1, characterized in that, The step of performing deep frame-level feature extraction on the audio data to be identified to obtain sentence-level dialect feature information includes: The audio data to be identified is extracted using a deep neural network to obtain frame-level deep feature information. Pooling is performed on the frame-level deep feature information to obtain sentence-level dialect feature information.

3. The method according to claim 1, characterized in that, The step of adjusting the initial decoding network based on the classification decoding result, the text decoding result, the category label, and the text label to obtain the trained decoding network includes: Based on the classification decoding result and the category label, a first loss function is calculated, and based on the text decoding result and the text label, a second loss function is calculated. By weighting the first loss function and the second loss function, the network loss function is obtained; Based on the network loss function, the parameters of the initial decoding network are adjusted, and the process of inputting the feature information to be decoded into the classification decoding network to obtain the classification decoding result, and inputting the feature information to be decoded into the text decoding network to obtain the text decoding result is repeated until the network loss function meets the preset stopping iteration condition, thus obtaining the trained decoding network.

4. The method according to claim 1, characterized in that, The text decoding network includes an attention mechanism decoding network and a temporal classification decoding network; the process of decoding the target text recognition result by the text decoding network in the trained decoding network includes: The attention mechanism decoding network decodes the feature information to be identified to obtain a first text recognition result, and the temporal classification decoding network decodes the feature information to be identified to obtain a second text recognition result. Based on the first text recognition result and the second text recognition result, the target text recognition result is obtained.

5. The method according to claim 4, characterized in that, The step of decoding the feature information to be identified through the temporal classification decoding network to obtain the second text recognition result includes: The temporal classification decoding network is used to perform syllable matching on the feature information to be identified, thereby obtaining a syllable sequence corresponding to the feature information to be identified. The syllables in the syllable sequence are filtered and deduplicated to obtain the second text recognition result.

6. A multi-dialect speech recognition device, characterized in that, The device includes: The acquisition module is used to acquire the audio data to be recognized; The feature extraction module is used to perform deep frame-level feature extraction on the audio data to be identified to obtain sentence-level dialect feature information, and to encode the audio data to be identified to obtain encoded feature information; The splicing module is used to splice the sentence-level dialect feature information and the encoded feature information to obtain the feature information to be identified; A decoding module is used to acquire training audio data carrying recognition labels and an initial decoding network. The initial decoding network includes a classification decoding network and a text decoding network. The recognition labels include category labels and text labels. Feature extraction is performed on the training audio data to obtain feature information to be decoded. The feature information to be decoded is input into the classification decoding network to obtain a classification decoding result. The feature information to be decoded is input into the text decoding network to obtain a text decoding result. Based on the classification decoding result, the text decoding result, the category label, and the text label, the initial decoding network is adjusted to obtain a trained decoding network. The feature information to be recognized is decoded through the classification decoding network in the trained decoding network to obtain an audio classification result. The feature information to be recognized is decoded through the text decoding network in the trained decoding network to obtain a target text recognition result. Based on the audio classification result and the target text recognition result, a speech recognition result is obtained.

7. The apparatus according to claim 6, characterized in that, The feature extraction module is also used to extract features from the audio data to be identified based on a deep neural network to obtain frame-level deep feature information, and to perform pooling operations on the frame-level deep feature information to obtain sentence-level dialect feature information.

8. The apparatus according to claim 6, characterized in that, The decoding module is further configured to calculate a first loss function based on the classification decoding result and the category label, and calculate a second loss function based on the text decoding result and the text label; weight the first loss function and the second loss function to obtain a network loss function; adjust the parameters of the initial decoding network based on the network loss function; return to the steps of inputting the feature information to be decoded into the classification decoding network to obtain the classification decoding result, and inputting the feature information to be decoded into the text decoding network to obtain the text decoding result, until the network loss function meets a preset stopping iteration condition, thus obtaining a trained decoding network.

9. The apparatus according to claim 6, characterized in that, The text decoding network includes an attention mechanism decoding network and a temporal classification decoding network; the decoding module is further configured to decode the feature information to be identified through the attention mechanism decoding network to obtain a first text recognition result, and decode the feature information to be identified through the temporal classification decoding network to obtain a second text recognition result, and obtain a target text recognition result based on the first text recognition result and the second text recognition result.

10. The apparatus according to claim 9, characterized in that, The decoding module is further configured to perform syllable matching on the feature information to be identified through the temporal classification decoding network to obtain a syllable sequence corresponding to the feature information to be identified, and to filter and remove duplicate syllables in the syllable sequence to obtain a second text recognition result.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images