End-to-end mandarin and low-resource Cantonese unified identification method

Abstract

The invention relates to an end-to-end mandarin and low-resource Cantonese unified recognition method, and belongs to the field of speech recognition. The method is applied to the scene of simultaneous recognition of mandarin and Cantonese. According to the method, an end-to-end CTC-Attention combined training method is adopted, a mandarin-Cantonese unified recognition dictionary is constructed, and the problem of confusion of context recognition in multi-language recognition is relieved in combination with a language recognition LID module. According to the invention, unified recognition of mandarin and low-resource Cantonese is realized by constructing an end-to-end speech recognition neural network model architecture, a modeling unit construction mode is provided, and the convergence performance of the model is improved. According to the end-to-end voice recognition method, the requirement of voice recognition can be met in the scene that mandarin and Cantonese appear at the same time, and good performance is achieved.

Description

Technical Field

[0001] The invention belongs to the field of speech recognition, and in particular relates to an end-to-end Mandarin and low-resource Cantonese unified recognition method. Background Art

[0002] The rapid development of Automatic Speech Recognition (ASR) technology, especially the significant improvement in recognition performance of high-resource languages ​​such as Mandarin, English, and Spanish, has injected new vitality into applications in many fields. In recent years, the demand for recognition of low-resource languages ​​has also been increasing, but the recognition effect of ASR systems covering low-resource languages ​​is poor and cannot achieve similar recognition performance as high-resource languages. In addition, multilingual recognition systems with better performance are mostly concentrated in high-resource languages, with less attention paid to low-resource languages.

[0003] Cantonese is one of the most widely used dialects in China, with more than 100 million users. However, Cantonese is still a low-resource language, and the data shortage problem is one of the main reasons affecting the performance of the Cantonese recognition system. In addition, although Cantonese is a dialect of Chinese, there are significant differences in grammar and pronunciation between Mandarin and Cantonese, resulting in the inability to understand each other between the two languages. Even in areas where Cantonese is widely used, scenarios where Cantonese and Mandarin need to be recognized at the same time are also very common, such as in-vehicle voice interaction systems, customer service systems, and mobile phone voice assistants. Therefore, there is a high demand for building a unified Mandarin-Cantonese recognition system. At the same time, due to the imbalance of training data, the speech recognition model has better performance on high-resource languages, but performs poorly on low-resource languages. The method of the present invention is committed to providing a valuable and efficient Mandarin-Cantonese recognition system in this complex scenario.

[0004] Previous research on Cantonese and other dialects has focused on single-language model recognition. By pre-training recognition models for various dialects, and then using language classification models or additional methods such as manual selection and switching to select the corresponding model, an ASR system that supports unified recognition of Mandarin and other dialects is implemented. In addition, research on dialect or accent recognition mostly uses methods such as ensemble learning, adapters, and adversarial learning. These methods require the use of multiple models or modules to improve the recognition performance of dialects or accents, thereby increasing the computational cost and the complexity of the entire ASR model training process. Research on Mandarin and Cantonese recognition focuses on exploring Mandarin recognition with a Cantonese accent, or using transfer learning methods to use pre-trained Mandarin models to enhance the performance of Cantonese recognition. These methods are not suitable for building a simple and efficient Mandarin-Cantonese unified recognition system.

[0005] In recent years, the development of end-to-end speech recognition systems has achieved remarkable results. It is more convenient to use a single network to directly convert the input speech sequence into an output text sequence. Therefore, the method in the present invention adopts an end-to-end model construction method and uses a single network to achieve unified recognition of Mandarin and Cantonese. Currently commonly used end-to-end speech recognition frameworks include methods based on connectionist temporal classification (CTC) and attention. The objective function of the CTC-based method solves the frame-level alignment problem by inserting blank labels. The attention-based method maps acoustic data to high-level feature encoding through the attention mechanism. During decoding, an autoregressive method is used to combine the prediction results of the previous step to obtain the recognition label. The method of the present invention adopts a joint CTC-Attention multi-task learning method, which combines the advantages of CTC and Attention, thereby improving the robustness and convergence speed of the model.

[0006] The selection of acoustic modeling units is one of the keys to an end-to-end speech recognition system. For Chinese ASR, the modeling units have undergone a transformation from phonemes, context-dependent phonemes, syllables to characters. In recent years, characters have become the most commonly used modeling units in Attention-based models. Both Mandarin and Cantonese are tonal languages ​​and use Chinese characters, but they have significant differences in pronunciation, dictionary, and grammar. The syllable structure of Mandarin is relatively simple, with four tones, while the syllable structure of Cantonese is more complex, with six to nine tones. At the same time, there are differences in the dictionaries of the two languages. The same character may exist in the dictionaries of both languages, but the pronunciation and meaning are not exactly the same. In addition, there are differences in grammar and usage between Mandarin and Cantonese, and the word order of Cantonese is often more flexible than that of Mandarin. In the method of the present invention, we select characters as acoustic modeling units, merge the Mandarin dictionary and the Cantonese dictionary, and establish a unified modeling dictionary. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] The technical problem to be solved by the present invention is how to provide an end-to-end unified recognition method for Mandarin and low-resource Cantonese to solve the problem of unified recognition of Mandarin and low-resource Cantonese.

[0009] (II) Technical solution

[0010] In order to solve the above technical problems, the present invention proposes an end-to-end unified recognition method for Mandarin and low-resource Cantonese, which comprises the following steps:

[0011] S1. Prepare training datasets D for Mandarin and Cantonese 普 and D 粤, clean the data and organize the text to generate the training data set file;

[0012] S2. Copy the sorted Cantonese data twice and put it into the Mandarin and Cantonese data sets to generate the total training data set D 总 ;

[0013] S3. Design an end-to-end unified Mandarin and low-resource Cantonese recognition model, including a shared encoder, an attention decoder, a CTC decoder, and a LID module. The whole process is expressed by the following formula:

[0014] h enc = SharedEncoder(x) (1)

[0015] P Att (y m |x)=AttentionDecoder(h enc ,y 1:m-1 ) (2)

[0016] P CTC (y,x)=CTCDecoder(h enc ) (3)

[0017] The shared encoder is used to map the input speech signal x into a hidden feature representation h enc ;

[0018] P represents the probability, y represents the result of speech recognition, that is, the annotated text, and m represents the mth recognition unit decoded when the decoding time step is m;

[0019] The attention decoder generates decoding outputs one by one according to the time step. Through the self-attention mechanism, each newly generated output result will become the input information of the subsequent output. The decoder can use the previous prediction results to decode the new recognition unit. As shown in formula (2), when decoding the mth recognition unit, the recognition result y generated by the previous 1:m-1 decoding time steps will be used. 1:m-1 ;

[0020] The CTC decoder is used to promote the convergence of the attention decoder;

[0021] The LID module assists in selecting the correct language corresponding to the acoustic features during the decoding process, which is expressed as follows:

[0022] P LID (y LID ,x)=LIDModule(h enc ) (4)

[0023] where y LIDIndicates the language category to which the speech data belongs, that is, Mandarin or Cantonese.

[0024] (III) Beneficial effects

[0025] The present invention proposes an end-to-end unified recognition method for Mandarin and low-resource Cantonese. The present invention provides a simple and efficient end-to-end unified recognition method for Mandarin and Cantonese, which is suitable for scenarios where high-resource Mandarin and low-resource Cantonese coexist. The method can use a unified model to simultaneously recognize Mandarin and Cantonese, and at the same time can utilize the similarity between Mandarin and Cantonese, so that the convergence between Mandarin and Cantonese in the unified recognition model training can promote each other. In addition, the present invention proposes a method for constructing a Mandarin and Cantonese modeling dictionary, which solves the modeling unit problem of the unified recognition model, and adds a LID module to alleviate the context recognition confusion problem caused by multi-language training. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the model structure of the present invention;

[0027] Figure 2 A schematic diagram for constructing a dictionary of the present invention;

[0028] Figure 3 Provides a model training strategy framework for the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0030] The object of the present invention is to provide a simple and efficient Mandarin-Cantonese unified recognition method, which can use a single model to simultaneously recognize high-resource Mandarin and low-resource Cantonese.

[0031] This invention proposes a simple and efficient Mandarin-Cantonese unified recognition method, which can be applied to scenarios where Mandarin and Cantonese are recognized simultaneously. This invention adopts an end-to-end CTC-Attention joint training method, builds a Mandarin-Cantonese unified recognition dictionary, and combines the language identification LID (Language Identification) module to alleviate the problem of context recognition confusion in multi-language recognition.

[0032] End-to-end speech recognition systems can directly map speech signals to graphemes or word sequences, which simplifies the speech recognition training process and has significant advantages compared to non-end-to-end architectures. Currently, the most commonly used end-to-end speech recognition architecture is an encoder-decoder network that combines the CTC objective function as an auxiliary task.

[0033] The core of CTC technology is that it can handle the prediction problem of sequences of unequal lengths, without the need to explicitly align the speech signal and the label data in advance, thereby simplifying the model structure and improving recognition efficiency and accuracy. The CTC loss function learns to align the input sequence with the target sequence, facilitating model training based on sequence tasks. It introduces a special blank symbol <blk>Each predicted category corresponds to a peak in the entire speech, and other positions that are not peaks are considered blank symbols. <blk>, which is done by copying the blank symbol <blk>To construct a CTC path with the same length as the input sequence.

[0034] The core of the self-attention mechanism technology applied to end-to-end speech recognition is that it allows the model to dynamically adjust the degree of attention to different time steps or different parts when processing audio input, thereby improving the robustness and recognition accuracy of the model. The Conformer-based encoder architecture combines the advantages of convolutional neural network (CNN) and self-attention mechanism, using deep separable convolution to effectively capture local features and adopting self-attention mechanism to capture global dependencies within the input sequence; the decoder based on the attention mechanism uses a higher-level feature representation and combines the previous prediction label to predict the symbol of the next output.

[0035] The encoder-decoder end-to-end speech recognition framework based on joint CTC-Attention combines the advantages of CTC and self-attention mechanisms, and can effectively model long-range dependencies in speech signals while maintaining computational efficiency. Therefore, the neural network architecture used in the present invention adopts a structure based on joint CTC-Attention.

[0036] There are words in Mandarin and Cantonese that have similar or identical pronunciations but different meanings, that is, a Mandarin word may have the same pronunciation as a Cantonese word with a completely different meaning from the Mandarin word, so when the speech recognition model generates corresponding labels, it may cause confusion between Mandarin and Cantonese, that is, a Mandarin sentence is mixed with Cantonese or a Cantonese sentence is mixed with Mandarin, resulting in a reduction in recognition rate. The present invention introduces the LID module as an auxiliary task to map the output tags to the corresponding language tags to alleviate the problem of confused recognition.

[0037] The present invention provides an end-to-end unified recognition method for Mandarin and low-resource Cantonese, comprising the following steps:

[0038] S1. Prepare training datasets D for Mandarin and Cantonese 普 and D 粤 , clean the data and organize the text to generate the training data set file.

[0039] First, filter out data that is too short or too long in the training audio, and limit the audio duration to 0.1s to 10s. Data that is too short contains less useful information, and data that is too long will cause insufficient GPU memory during model training. Both of these data are not conducive to model learning.

[0040] Secondly, the annotated text file of the audio is cleaned to remove punctuation, spaces, special symbols, etc., the text is converted from traditional Chinese to simplified Chinese, the Arabic numerals in the text are converted into Chinese characters, and the upper and lower case expressions of English are unified into upper or lower case.

[0041] Finally, the audio data and its corresponding annotated text are combined to generate a training dataset file, where the format of the audio-text pair is as follows:

[0042] The annotation text corresponding to the audio ID

[0043] S2. Since Cantonese is a low-resource language, there is less training data available, and it is quite different from Mandarin data. When using the same model to train Mandarin and Cantonese recognition, in order to avoid the recognition results being biased towards Mandarin due to excessive imbalance of data, the sorted Cantonese data is copied twice and put into the Mandarin and Cantonese data sets to generate the total training data set D. 总 , to increase the proportion of the Cantonese dataset in the total dataset and balance the dataset.

[0044] S3. The structure of the end-to-end Mandarin and low-resource Cantonese unified recognition model in the present invention includes: a shared encoder, an attention decoder, a CTC decoder, and a LID module. The LID module is added to this structure to alleviate the confusion problem in multi-language recognition.

[0045] The shared encoder can be viewed as an acoustic model that maps the input speech signal x to a hidden feature representation h. enc , the decoder converts the hidden feature representation into a recognition unit, where the CTC decoder helps the convergence of the attention decoder. The whole process can be expressed by the following formula:

[0046] h enc =SharedEncoder(x) (1)

[0047] P Att (y m |x)=AttentionDecoder(h enc ,y 1:m-1 ) (2)

[0048] P CTC (y,x)=CTCDecoder(h enc ) (3)

[0049] Where P represents the probability, y represents the result of speech recognition, that is, the annotated text, and m represents the mth recognition unit decoded when the decoding time step is m (the recognition unit is expressed in English as recognition units, which is the smallest representation unit of the speech recognition output result. It can be a phoneme, word, subword or character, etc., which is equivalent to the modeling unit in the dictionary construction part below, but it appears in the decoding part at this time, so it is called a recognition unit). The attention decoder generates decoding outputs one by one according to the time step. Through the self-attention mechanism, each newly generated output result will become the input information of the subsequent output, so the decoder can use the previous prediction results to decode the new recognition unit. As shown in formula (2), when the mth recognition unit is decoded, the recognition result y generated by the previous 1:m-1 decoding time steps will be used. 1:m-1 .

[0050] The CTC decoder is used to promote the convergence of the attention decoder;

[0051] The LID module assists in selecting the correct language corresponding to the acoustic features during the decoding process, which can be expressed as follows:

[0052] P LID (y LID ,x)=LIDModule(h enc ) (4)

[0053] where y LID Indicates the language category to which the speech data belongs, that is, Mandarin or Cantonese.

[0054] The structure of the entire model is as follows Figure 1 shown.

[0055] like Figure 1 As shown in the figure and combined with formulas (1) to (4), the speech signal x(x1, x2, …, x T ) is input to the shared encoder to generate the hidden feature representation h enc , the hidden feature is represented by h enc They are input into the attention decoder, CTC decoder and LID module respectively. These three modules represent h with hidden features. enc As input. The attention decoder (Attention Decoder) and CTC decoder (CTCDecoder) pass their respective loss functions loss ATT and loss CTC Calculate the speech recognition result y(y1,y2,…,y M ), the LID module (LID Module) uses the loss function loss LID Calculate the loss between the language category to which the speech recognition result y belongs, and finally generate the fusion loss through the weighted coefficient λ, and use this loss to train the model. Figure 1 middle, <sos eos>Indicates the start / end of the recognition result y, <blk>represents the blank symbol in the CTC decoder, M represents the total number of CTC decoding units in the speech signal, <man>Indicates that the category is Mandarin, <can>Indicates that the category is Cantonese. The total fusion loss function loss MTL As shown below:

[0056] loss MTL =λ Att loss Att +λ CTC loss CTC +λ LID loss LID (5)

[0057] Among them, in the present invention, ATT =0.7,λ CTC =0.2,λ LID = 0.1 makes the model achieve the best effect.

[0058] Furthermore, the present invention also includes constructing a dictionary of unified recognition models;

[0059] The selection of acoustic modeling units, i.e., dictionaries, is a key component of an end-to-end speech recognition model. The dictionary of an end-to-end Mandarin and low-resource Cantonese unified recognition model of the present invention needs to include both Mandarin and Cantonese. For Chinese automatic speech recognition ASR, the modeling unit has undergone a transformation from phonemes, context-dependent phonemes, syllables to characters. In recent years, characters have become the most commonly used modeling units. Therefore, the model dictionary of the present invention is constructed with characters as modeling units.

[0060] As two important languages ​​of the Chinese language system, Mandarin and Cantonese share similarities as well as some differences. Both languages ​​use Chinese characters as the basic unit, but they differ in pronunciation, vocabulary, and grammar. Mandarin has a relatively simple syllable structure with four tones, while Cantonese has a more complex syllable structure with six to nine tones. In addition, there are differences in the vocabulary of the two languages, and the same character may exist in the vocabulary of both languages, but the pronunciation and meaning are completely different. At the same time, there are differences in grammar and usage between Mandarin and Cantonese, and the word order of Cantonese tends to be more flexible compared to Mandarin.

[0061] The construction of the dictionary of the unified recognition model of Mandarin and Cantonese in the present invention is divided into the following three steps:

[0062] S201, Statistical Mandarin Dataset D 普 How many words are contained in the annotated text? These words are combined into a Mandarin dictionary Dic 普 ;

[0063] S202, Statistical Cantonese Dataset D 粤 How many words are contained in the annotated text? These words are combined into a Cantonese dictionary Dic 粤 ;

[0064] S203, Put Mandarin and Cantonese dictionary Dic 普 and Dick 粤 The same characters in the dictionary are merged together, and the different characters are retained separately to build a unified modeling dictionary Dic. The schematic diagram of the construction of the unified dictionary is as follows: Figure 2 As shown:

[0065] The format of the generated dictionary is as follows:

[0066] Label ID

[0067] The construction method of the above modeling unit can utilize the similarity between Mandarin and Cantonese, so that the recognition of Mandarin and Cantonese can promote each other and achieve better results than single language recognition. 普 and D 粤 The annotated text in is converted into Label ID word by word for model training. When the model outputs the recognition result, the dictionary is used to convert the Label ID of the recognition result into text to generate the final recognized text.

[0068] Furthermore, before model training, the dataset D 总 The audio data in the data enhancement operation is performed, and the data enhancement operation includes the following steps:

[0069] a) Speed ​​perturbation: The speed perturbation operation in speech recognition training is an effective data enhancement technology. It generates new training samples by changing the playback speed of the speech signal, thereby increasing the diversity of training data and the generalization ability of the model. Speed ​​perturbation factors of 1.1 and 0.9 are selected for speech signal enhancement operations. The speed perturbation factor of 1.1 means increasing the speed of the speech signal by 10%, and the speed perturbation factor of 0.9 means reducing the speed of the speech signal by 10%.

[0070] b) SpecAugment data enhancement: The core idea of ​​SpecAugment is to perform data enhancement at the spectrogram level. By simulating actual conditions such as noise, distortion, and silent segments in audio, the model is forced to learn to extract useful information in complex environments, thereby improving its anti-interference and generalization capabilities.

[0071] SpecAugment mainly includes the following three main data enhancement strategies:

[0072] Frequency Masking: Randomly select a continuous area on the frequency axis of the spectrogram and set it to zero. This operation simulates short-term noise or distortion in the audio, which helps the model learn to correctly recognize speech when some frequency information is missing.

[0073] Time Masking: Perform a similar operation on the time axis, i.e. randomly select consecutive time segments and set them to zero. This can train the model to adapt to silence segments of different lengths and improve the robustness of the model in the time dimension.

[0074] Frequency Block Masking and Time Block Masking: Instead of simply masking continuous frequency or time domain segments, these two variants mask rectangular blocks of a certain size.

[0075] Since frequency block masking and time block masking can more naturally simulate interference in the actual environment and further increase the generalization ability of the model, the SpecAugment data enhancement method in the present invention adopts frequency block masking and time block masking methods.

[0076] Furthermore, the training data of the present invention is input into the model structure in step S3 for training after velocity perturbation and SpecAugment data enhancement until the model converges. The shared encoder in step S3 is composed of multiple conformer layers, and the size of the cnn_module_kernel of the conformer layer is 15; the attention decoder adopts a bidirectional Transformer structure for decoding tasks. The number of shared encoder layers and attention decoder layers are 12 and 3 respectively, and the number of attention heads is 8. The CTC decoder adopts the CTC objective function, and the LID module also adopts the CTC objective function. The difference between it and the CTC decoder is that the final output layer label of the LID module is the corresponding language category, rather than the LabelID of the recognized text. During the training phase, the dropout rate was set to 0.1, and the Adam optimization method was used for optimization.

[0077] Embodiment 1:

[0078] The present invention is an end-to-end unified recognition method for Mandarin and low-resource Cantonese.

[0079] Figure 3 The training strategy framework diagram of this method is shown in Figure 1. First, the original training data sets, Mandarin and Cantonese, are cleaned. Second, the Cantonese data set is doubled to address the problem of too few Cantonese data sets. The annotated text in the training data set is then used to construct a dictionary. Finally, the training data is enhanced and sent to Figure 1 In the training network shown, the loss function is as shown in formula (5). It is judged whether the total number of training rounds epochs is reached. If it is reached, the trained model file is exported. Otherwise, the weights are updated by back propagation and the loss is calculated again.

[0080] An end-to-end unified recognition method for Mandarin and low-resource Cantonese, the steps are as follows:

[0081] 1) Prepare training datasets D for Mandarin and Cantonese 普 and D 粤 , to clean the data and organize the text. First, filter out data that is too short or too long in the training audio, and the audio duration is limited to 0.1s to 10s. Data that is too short contains less useful information, and data that is too long will cause insufficient GPU memory during model training. Data in both cases is not conducive to model learning. Secondly, clean the annotated text file of the audio, remove punctuation, spaces, and special symbols in the text, convert the text from traditional Chinese to simplified Chinese, convert the Arabic numerals in the text into Chinese characters, and unify the case of English into uppercase or lowercase. Finally, combine the audio data and its corresponding annotated text to generate a training data set file, where the format of the audio-text pair is as follows:

[0082] The annotation text corresponding to the audio ID

[0083] 2) Since Cantonese is a low-resource language, there is less training data available, and it is quite different from Mandarin data. When using the same model to train Mandarin and Cantonese recognition, in order to avoid the recognition results being biased towards Mandarin due to excessive data imbalance, the sorted Cantonese data is doubled and put into the Mandarin and Cantonese data sets to generate the total training data set D 总 .

[0084] 3) The structure of the end-to-end Mandarin and low-resource Cantonese unified recognition model in the present invention includes a shared encoder (SharedEncoder), an attention decoder (AttentionDecoder) and a CTC decoder (CTC Decoder). In addition, a LID module (LID Module) is added to this structure to alleviate the confusion problem in multi-language recognition.

[0085] The shared encoder can be viewed as an acoustic model that maps the input speech signal x to a hidden feature representation h enc , the decoder converts the hidden feature representation into a recognition unit, where the CTC decoder helps the convergence of the attention model. The whole process can be expressed as follows:

[0086] h enc = SharedEncoder(x) (1)

[0087] P Att (y m |x)=AttentionDecoder(h enc ,y 1:m-1 ) (2)

[0088] P CTC (y,x)=CTCDecoder(h enc ) (3)

[0089] Where m represents the decoded mth recognition unit. The LID module assists in selecting the correct language corresponding to the acoustic features during the decoding process, which can be expressed by the following formula:

[0090] P LID (y LID ,x)=LIDModule(h enc ) (4)

[0091] The structure of the entire model is as follows Figure 1 shown.

[0092] The loss function of the end-to-end Mandarin and low-resource Cantonese unified recognition model in the present invention is as follows:

[0093] loss MTL =λ Att loss Att +λ CTC loss CTC +λ LID loss LID (5)

[0094] 4) The selection of acoustic modeling units, i.e., dictionaries, is a key component of the end-to-end speech recognition model. The dictionary of the end-to-end Mandarin and low-resource Cantonese unified recognition model of the present invention needs to include both Mandarin and Cantonese. For Chinese ASR, the modeling unit has undergone a transformation from phonemes, context-dependent phonemes, syllables to characters. In recent years, characters have become the most commonly used modeling units. Therefore, the model dictionary of the present invention is constructed with characters as modeling units.

[0095] As two important languages ​​of the Chinese language system, Mandarin and Cantonese share similarities as well as some differences. Both languages ​​use Chinese characters as the basic unit, but they differ in pronunciation, vocabulary, and grammar. Mandarin has a relatively simple syllable structure with four tones, while Cantonese has a more complex syllable structure with six to nine tones. In addition, there are differences in the vocabulary of the two languages, and the same character may exist in the vocabulary of both languages, but the pronunciation and meaning are completely different. At the same time, there are differences in grammar and usage between Mandarin and Cantonese, and the word order of Cantonese tends to be more flexible compared to Mandarin.

[0096] The construction of the dictionary of the unified recognition model of Mandarin and Cantonese in the present invention is divided into the following three steps:

[0097] a) Statistical Mandarin Dataset D 普 How many words are contained in the annotated text? These words are combined into a Mandarin dictionary Dic 普 ;

[0098] b) Statistical Cantonese Dataset D 粤 How many words are contained in the annotated text? These words are combined into a Cantonese dictionary Dic 粤 ;

[0099] c) Dic 普 and Dick 粤 The same characters in the dictionary are merged together, and the different characters are retained separately to build a unified modeling dictionary Dic. The schematic diagram of the construction of the unified dictionary is as follows: Figure 2 shown.

[0100] The construction method of the above modeling units can utilize the similarities between Mandarin and Cantonese, so that the recognition of Mandarin and Cantonese can promote each other to achieve better results than single language recognition.

[0101] 5) Before model training, the dataset D 总 The audio data in step 5) is subjected to data enhancement operation, wherein the data enhancement operation in step 5) comprises the following steps:

[0102] a) Speed ​​perturbation: The speed perturbation operation in speech recognition training is an effective data enhancement technology. It generates new training samples by changing the playback speed of the speech signal, thereby increasing the diversity of training data and the generalization ability of the model. Speed ​​perturbation factors of 1.1 and 0.9 are selected for speech signal enhancement operations. The speed perturbation factor of 1.1 means increasing the speed of the speech signal by 10%, and the speed perturbation factor of 0.9 means reducing the speed of the speech signal by 10%.

[0103] b) SpecAugment data enhancement: The core idea of ​​SpecAugment is to perform data enhancement at the spectrogram level. By simulating actual conditions such as noise, distortion, and silent segments in audio, the model is forced to learn to extract useful information in complex environments, thereby improving its anti-interference and generalization capabilities.

[0104] SpecAugment mainly includes the following three main data enhancement strategies:

[0105] Frequency Masking: Randomly select a continuous area on the frequency axis of the spectrogram and set it to zero. This operation simulates short-term noise or distortion in the audio, which helps the model learn to correctly recognize speech when some frequency information is missing.

[0106] Time Masking: Perform a similar operation on the time axis, i.e. randomly select consecutive time segments and set them to zero. This can train the model to adapt to silence segments of different lengths and improve the robustness of the model in the time dimension.

[0107] Frequency Block Masking and Time Block Masking: Instead of simply masking continuous frequency or time domain segments, these two variants mask rectangular blocks of a certain size.

[0108] Since frequency block masking and time block masking can more naturally simulate interference in the actual environment and further increase the generalization ability of the model, the SpecAugment data enhancement method in the present invention adopts this method.

[0109] 6) After the training data of the present invention is enhanced by velocity perturbation and SpecAugment data, it is input into the model structure in step 3) for training until the model converges. The shared encoder in step 3) is composed of multiple conformer layers, the size of the cnn_module_kernel of the conformer layer is 15, and the attention decoder adopts a bidirectional Transformer structure for decoding tasks. The number of shared encoder layers and attention decoder layers are 12 and 3 respectively, and the number of attention heads is 8. The CTC decoder uses a linear layer of the CTC objective function, and the LID module also uses a linear layer based on the CTC objective function. The difference from the CTC decoder is that the final output layer of the LID module is the corresponding language category. During the training phase, the dropout rate was set to 0.1, and the Adam optimization method was used for optimization.

[0110] The present invention realizes the unified recognition of Mandarin and low-resource Cantonese by constructing an end-to-end speech recognition neural network model architecture, and proposes a construction method of a modeling unit to improve the convergence performance of the model. The end-to-end speech recognition method can complete the needs of speech recognition in the scenario where Mandarin and Cantonese appear at the same time, and achieve good performance.

[0111] The advantages of the present invention compared with the prior art are:

[0112] The present invention provides a simple and efficient end-to-end unified recognition method for Mandarin and Cantonese, which is suitable for scenarios where high-resource Mandarin and low-resource Cantonese coexist. The method can use a unified model to simultaneously recognize Mandarin and Cantonese, and can also utilize the similarities between Mandarin and Cantonese, so that the convergence between Mandarin and Cantonese in the unified recognition model training can promote each other. In addition, the present invention proposes a method for constructing a Mandarin and Cantonese modeling dictionary, which solves the modeling unit problem of the unified recognition model, and adds a LID module to alleviate the context recognition confusion problem caused by multi-language training.

[0113] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / can> < / man> < / blk> < / sos> < / blk> < / blk> < / blk>

Claims

1. An end-to-end unified recognition method for Mandarin and low-resource Cantonese, characterized in that: The method comprises the following steps: S1. Prepare training datasets D for Mandarin and Cantonese 普 and D 粤 , clean the data and organize the text to generate the training data set file; S2. Copy the sorted Cantonese data twice and put it into the Mandarin and Cantonese data sets to generate the total training data set D 总 ; S3. Design an end-to-end unified Mandarin and low-resource Cantonese recognition model, including a shared encoder, an attention decoder, a CTC decoder, and a LID module. The whole process is expressed by the following formula: h enc =SharedEncoder(x) (1) P Att (y m |x)=AttentionDecoder(h enc ,y 1:m-1 ) (2) P CTC (y,x)=CTCDecoder(h enc ) (3) The shared encoder is used to map the input speech signal x into a hidden feature representation h enc ; P represents the probability, y represents the result of speech recognition, that is, the annotated text, and m represents the mth recognition unit decoded when the decoding time step is m; The attention decoder generates decoding outputs one by one according to the time step. Through the self-attention mechanism, each newly generated output result will become the input information of the subsequent output. The decoder can use the previous prediction results to decode the new recognition unit. As shown in formula (2), when decoding the mth recognition unit, the recognition result y generated by the previous 1:m-1 decoding time steps will be used. 1:m-1 ; The CTC decoder is used to promote the convergence of the attention decoder; The LID module assists in selecting the correct language corresponding to the acoustic features during the decoding process, which is expressed as follows: P LID (y LID ,x)=LIDModule(h enc ) (4) where y LID Indicates the language category to which the speech data belongs, that is, Mandarin or Cantonese.

2. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 1, characterized in that: The S1 specifically includes: First, filter out data that is too short or too long in the training audio, and limit the audio length to 0.1s to 10s; Secondly, the audio annotation text files are cleaned to remove punctuation, spaces and special symbols in the text, convert the text from traditional Chinese to simplified Chinese, convert the Arabic numerals in the text into Chinese characters, and unify the uppercase and lowercase expressions of English into uppercase or lowercase. Finally, combine the audio data and its corresponding annotated text to generate a training dataset file, where the format of the audio-text pair is as follows: The annotation text corresponding to the audio ID.

3. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 1, characterized in that: Speech signal x(x1,x2,…,x T ) is input to the shared encoder to generate the hidden feature representation h enc , the hidden feature is represented by h enc They are input into the attention decoder, CTC decoder and LID module respectively. These three modules represent h with hidden features. enc As input; among them, the attention decoder and CTC decoder pass their respective loss functions loss ATT and loss CTC Calculate the speech recognition result y(y1,y2,…,y M ), the LID module uses the loss function loss LID Calculate the loss between the language category to which the speech recognition result y belongs, and finally generate the fusion loss through the weighted coefficient λ, and use this loss to train the model; the total fusion loss function loss MTL As shown below: loss MTL =λ Att loss Att +λ CTC loss CTC +λ LID loss LID (5) Among them, λ ATT ,λ CTC ,λ LID are the weights of the attention decoder, CTC decoder and LID module respectively.

4. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 3, characterized in that: l ATT =0.7,λ CTC =0.2,λ LID = 0.

1.

5. The end-to-end Mandarin and low-resource Cantonese unified recognition method according to any one of claims 1 to 4, characterized in that: The dictionary of the end-to-end Mandarin and low-resource Cantonese unified recognition model contains both Mandarin and Cantonese. The construction of the dictionary uses characters as modeling units and includes the following steps: S201, Statistical Mandarin Dataset D 普 How many words are contained in the annotated text? These words are combined into a Mandarin dictionary Dic 普 ; S202, Statistical Cantonese Dataset D 粤 How many words are contained in the annotated text? These words are combined into a Cantonese dictionary Dic 粤 ; S203, Put Mandarin and Cantonese dictionary Dic 普 and Dick 粤 The same characters in are merged together, and the different characters are retained separately to build a unified modeling dictionary Dic. The format of the generated dictionary is: Word Label ID.

6. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 5, characterized in that: Use this dictionary to train the dataset D 普 and D 粤 The annotated text in is converted into Label ID word by word for model training. When the model outputs the recognition result, the dictionary is used to convert the Label ID of the recognition result into text to generate the final recognized text.

7. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 5, characterized in that: Before model training, the dataset D 总 The audio data in is enhanced by data augmentation, which includes the following steps: speed perturbation and SpecAugment data augmentation.

8. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 7, characterized in that: The speed perturbation includes: generating new training samples by changing the playback speed of the speech signal, thereby increasing the diversity of training data and the generalization ability of the model; selecting speed perturbation factors 1.1 and 0.9 to perform speech signal enhancement operations, where the speed perturbation factor 1.1 means increasing the speech signal speed by 10%, and the speed perturbation factor 0.9 means reducing the speech signal speed by 10%.

9. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 7, characterized in that: The SpecAugment data enhancement includes: Frequency masking: randomly select a continuous area on the frequency axis of the spectrogram and then set this area to zero; Time masking: Randomly select consecutive time segments on the time axis and set them to zero; Frequency block masking and time block masking: Instead of simply masking continuous frequency or time domain segments, these two variants mask rectangular blocks of a certain size.

10. The end-to-end Mandarin and low-resource Cantonese unified recognition method as claimed in claim 7, characterized in that: After the training data is subjected to velocity perturbation and SpecAugment data enhancement, it is input into the model structure in step S3 for training until the model converges; the shared encoder in step S3 is composed of multiple conformer layers, and the size of the cnn_module_kernel of the conformer layer is 15; the attention decoder adopts a bidirectional Transformer structure for decoding tasks; the number of shared encoder layers and attention decoder layers are 12 and 3 respectively, and the number of attention heads is 8; the CTC decoder adopts the CTC objective function, and the LID module also adopts the CTC objective function. The difference between it and the CTC decoder is that the final output layer label of the LID module is the corresponding language category, rather than the Label ID of the recognized text; in the training stage, the dropout rate is set to 0.1, and the Adam optimization method is used for optimization.

Images