Dialect speech recognition method, device, equipment and storage medium

Abstract

The present application discloses a dialect speech recognition method, device, equipment and storage medium. The present application pre-configures a dialect speech recognition model, which is obtained by training speech samples of various types of languages ​​(including Mandarin and various types of dialects) and using the recognition text of the training samples after syntactic analysis and phone-level annotation, and the phone-level annotated text carrying syntactic information as labels. Compared with the existing word-level end-to-end model, the present application is based on the phoneme phone level for modeling, which improves the distinguishability of modeling units between various types of dialects and between dialects and Mandarin from the pronunciation level, and reduces the crosstalk between dialects, thereby improving the recognition effect of each dialect. Furthermore, the introduction of syntactic information in the annotation enables the model to learn the grammatical information of each dialect, further improving the recognition effect of each dialect. In addition, due to the addition of grammatical information, the readability of the model recognition text can also be improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and more specifically, to a dialect speech recognition method, device, equipment and storage medium. Background Art

[0002] China has a vast territory, and there are many dialects of Chinese and ethnic minority languages. Modern Chinese has various dialects, and they are distributed in a wide area. The differences between modern Chinese dialects are reflected in pronunciation, vocabulary, and grammar, especially in pronunciation. Given the diversity and differences of dialects, higher requirements are placed on speech recognition technology.

[0003] Existing dialect speech recognition technology mainly trains acoustic and language models based on target dialect training data. The modeling unit generally uses word-level end-to-end modeling to identify dialects by extracting word-level features. However, unlike Mandarin, for most dialects, the amount of labeled training data is small, and word-level end-to-end modeling relies on a large amount of data, which results in poor performance of the model on low-resource dialects and low accuracy of dialect recognition results. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a dialect speech recognition method, device, equipment and storage medium to solve the problem of low recognition accuracy of existing dialect speech recognition technology.

[0005] The specific plan is as follows:

[0006] A dialect speech recognition method, comprising:

[0007] Obtain the speech to be recognized in the target language type;

[0008] Inputting the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model;

[0009] The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects;

[0010] Decoding the phone-level annotated text carrying syntactic information using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information;

[0011] The character-level annotated text carrying syntactic information is regularized to remove the syntactic information therein and obtain a dialect speech recognition text.

[0012] Preferably, the process of determining the labels of the training samples in the dialect speech recognition model training process includes:

[0013] Obtaining the recognition text corresponding to the training sample;

[0014] Performing syntactic analysis on the recognized text, and annotating the analyzed syntactic information into the recognized text to obtain a character-level annotated text carrying the syntactic information;

[0015] Using a phone-level pronunciation dictionary that matches the language type of the training sample, determining a phone-level annotation corresponding to the recognized text;

[0016] The characters in the character-level annotated text carrying syntactic information are replaced with corresponding phone-level annotations to obtain phone-level annotated text carrying syntactic information.

[0017] Preferably, the process of determining the phone-level pronunciation dictionary matching each language type includes:

[0018] For Mandarin and Cantonese, the corresponding phone-level pronunciation dictionary is used directly;

[0019] For the remaining language types except Mandarin and Cantonese, the phone-level pronunciation dictionary of Mandarin is reused to form a phone-level pronunciation dictionary matching the respective language types.

[0020] Preferably, for each language type other than Mandarin and Cantonese, the phone-level pronunciation dictionary of Mandarin is reused to form a phone-level pronunciation dictionary matching each language type, including:

[0021] Numbering the remaining language types respectively to obtain a number corresponding to each language type;

[0022] For any of the remaining language types:

[0023] A number corresponding to the language type is added after each phone in the phone-level pronunciation dictionary of Mandarin to obtain a phone-level pronunciation dictionary matching the language type.

[0024] Preferably, the dialect speech recognition model includes an encoding module and a decoding module, the encoding module is used to encode the input speech to obtain speech coding features; the decoding module is used to predict the phone-level annotated text carrying syntactic information corresponding to the input speech based on the speech coding features.

[0025] Preferably, the encoding module is trained by pre-training; wherein the encoding module pre-training process includes:

[0026] Acquire a training data set, wherein the training data set includes training speech of various types of languages, wherein the various types of languages ​​include Mandarin and various types of dialects;

[0027] In the training data set, the training speech of the same type of language constitutes the positive sample pair, and the training speech of different types of languages ​​constitutes the negative sample pair. The encoding module is trained using a contrastive learning strategy. During the training process, the goal is to maximize the similarity between the speech coding features of the positive sample pair and minimize the similarity between the speech coding features of the negative sample pair until the set training end condition is met to obtain the trained encoding module.

[0028] Preferably, the acquiring of the training data set comprises:

[0029] Get the original training data set;

[0030] Adjusting the ratio of the amount of training speech data of each language in the original training data set so that in the adjusted training data set, the ratio of the amount of training speech data of Mandarin to the amount of training speech data of the other dialects does not exceed a set ratio, and the amount of training speech data of the dialects remains consistent;

[0031] The adjusted training data set is randomly processed so that the distribution of training speech of each type of language in the training data set is random.

[0032] Preferably, the process of establishing a decoding network corresponding to the target language type includes:

[0033] The decoding network corresponding to the target language type is trained using a phone-level pronunciation dictionary that matches the target language type and a text corpus of the target language type that carries syntactic information.

[0034] A dialect speech recognition device, comprising:

[0035] A speech acquisition unit to be recognized, used to acquire the speech to be recognized in the target language type;

[0036] A model processing unit, used for inputting the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model;

[0037] The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects;

[0038] A character decoding unit, configured to decode the phone-level annotated text carrying syntactic information by using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information;

[0039] The text regularization unit is used to perform text regularization on the character-level annotated text carrying syntactic information to remove the syntactic information therein and obtain the dialect speech recognition text.

[0040] A dialect speech recognition device, comprising: a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement the various steps of the dialect speech recognition method as described above.

[0043] A storage medium stores a computer program, which, when executed by a processor, implements the various steps of the dialect speech recognition method described above.

[0044] By means of the above technical scheme, the present application pre-configures a dialect speech recognition model, which is obtained by using speech samples of various types of languages ​​(including Mandarin and various types of dialects) as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation to obtain phone-level annotated text carrying syntactic information as labels for training. Compared with the existing character-level end-to-end model, the present application performs modeling based on the phoneme phone level, which improves the distinguishability of modeling units between various types of dialects and between dialects and Mandarin from the pronunciation level, and reduces the crosstalk between dialects, thereby improving the recognition effect of each dialect.

[0045] Furthermore, based on phone-level modeling, this application introduces syntactic information in the label annotation of training data, which enables the model to learn the grammatical information of each dialect, that is, it can make full use of the phone-level and grammatical features of the training data to further improve the recognition effect of each dialect. In addition, due to the addition of grammatical information, the readability of the model recognition text can also be improved.

[0046] Based on the trained dialect speech recognition model, the speech to be recognized in the target language type is recognized to obtain the phone-level annotated text with syntactic information output by the model. On this basis, the phone-level annotated text with syntactic information is decoded using the preset decoding network corresponding to the target language type to obtain the character-level annotated text with syntactic information. The text is regularized to remove the syntactic information and obtain the dialect speech recognition text. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 A schematic diagram of a flow chart of a dialect speech recognition method provided in an embodiment of the present application;

[0049] Figure 2 An overall flow chart of a dialect speech recognition method is illustrated;

[0050] Figure 3 A schematic diagram of the structure of a dialect speech recognition device provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of a dialect speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The present application provides a dialect speech recognition solution that can be applied to various scenarios where speech recognition is required, especially scenarios where various types of dialect speech need to be recognized.

[0054] In an optional embodiment, the present application solution can classify various types of dialects into seven categories according to the standard voice area division method, including:

[0055] Northern dialect, Wu dialect, Xiang dialect, Gan dialect, Hakka dialect, Cantonese dialect, and Min dialect. The detailed classification is as follows:

[0056] Category 1: "Northern dialect". It is the common basic dialect of the Han nationality, and the population using it accounts for 70% of the total Han population. "Northern dialect" is divided into four dialects: Northeast, Northwest, Southwest and Jianghuai.

[0057] Category 2: "Wu dialect". Distributed in Shanghai, Jiangsu, south of the Yangtze River, east of Zhenjiang, and most of Zhejiang, represented by Suzhou dialect.

[0058] Category 3: “Xiang dialect”: Distributed in most areas of Hunan Province, represented by Changsha dialect.

[0059] Category 4: Gan dialect. Distributed in most areas of Jiangxi Province, represented by Nanchang dialect.

[0060] Category 5: Hakka dialect. Mainly distributed in eastern Guangdong, western Fujian, southeastern Jiangxi and southern Guangxi, with Meixian dialect in Guangdong as the representative.

[0061] Category 6: “Min dialect”: It spans four provinces, including most of Fujian Province, parts of northeastern Guangdong Province and southern Zhejiang Province, as well as most of the Han Chinese-inhabited areas in Taiwan Province.

[0062] Category 7: “Cantonese dialect”: Mainly distributed in the central and southern parts of Guangdong Province, the eastern and southern parts of Guangxi, as well as Hong Kong and Macao, with Cantonese as the representative.

[0063] Among them, the Cantonese dialect has its own standard pronunciation system, and the corresponding phone pronunciation dictionary contains 88 phones.

[0064] Of course, the above is only an optional dialect classification method, and the present application may also adopt other dialect classification methods to classify dialects.

[0065] The present application solution can be implemented based on a terminal with data processing capabilities, which can be a mobile phone, computer, server, cloud, etc.

[0066] Next, combine Figure 1 The dialect speech recognition method of the present application may include the following steps:

[0067] Step S100: Acquire speech to be recognized in the target language type.

[0068] The target language type may be Mandarin or various types of dialects. The present application may determine the target language type of the acquired speech to be recognized.

[0069] Step S110: input the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model.

[0070] Among them, the dialect speech recognition model is obtained by using speech samples of various types of languages ​​(including Mandarin and various types of dialects) as training samples, and the recognition text of the training samples is subjected to syntactic analysis and phone-level annotation, and the phone-level annotated text carrying syntactic information is used as label training.

[0071] In this embodiment, the phone-level modeling unit is used in model building. Compared with the character-level modeling unit, the distinguishability of the modeling units between different types of dialects and between dialects and Mandarin is improved from the pronunciation level, and the crosstalk between dialects is reduced, thereby improving the recognition effect of each dialect. At the same time, compared with the character-level modeling unit, the phone-level modeling unit has lower requirements for the amount of training data.

[0072] On this basis, the present application further introduces syntactic information into the label annotation, that is, after the recognition text of the training sample is subjected to text syntactic analysis, the syntactic information obtained is added to the label annotation, so that the model can learn the grammatical information of each dialect and further improve the recognition effect of each dialect. In addition, due to the addition of grammatical information, the readability of the model recognition text can also be improved.

[0073] The process of performing syntactic analysis on the recognition text of the training sample may be to mark the part of speech and syntactic information of the recognition text. For example, a syntactic analysis tool such as Stanford Parser may be used to perform syntactic analysis on the recognition text.

[0074] Furthermore, the process of performing phone-level annotation on the recognized text may be to perform phone-level annotation according to the language type of the training sample using a corresponding phone-level pronunciation dictionary.

[0075] Step S120: Decode the phone-level annotated text carrying syntactic information using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information.

[0076] The decoding network corresponding to the target language type can be trained using a phone-level pronunciation dictionary matching the target language type and a text corpus of the target language type carrying syntactic information. For Cantonese, the corresponding decoding network can be trained using a phone-level pronunciation dictionary of Cantonese and a Cantonese text corpus carrying syntactic information; for other dialects and Mandarin except Cantonese, the decoding network can be trained using a phone-level pronunciation dictionary of other dialects and Mandarin and a dialect or Mandarin text corpus carrying syntactic information.

[0077] For the results of the model output in the previous step, the decoding network is used to decode the phone-level annotations into character-level annotations, and then the character-level annotation text carrying syntactic information is obtained.

[0078] Step S130: regularize the character-level annotated text carrying syntactic information to remove the syntactic information and obtain a dialect speech recognition text.

[0079] Specifically, the user may not need to know the syntactic information, so the character-level annotated text carrying the syntactic information obtained in the previous step can be subjected to text regularization to remove the syntactic information contained therein and obtain the remaining dialect speech recognition text.

[0080] The dialect speech recognition method provided in the embodiment of the present application is pre-configured with a dialect speech recognition model. The model is obtained by using speech samples of various types of languages ​​(including Mandarin and various types of dialects) as training samples, and using the phone-level annotated text carrying syntactic information after syntactic analysis and phone-level annotation of the recognition text of the training samples as labels for training. Compared with the existing word-level end-to-end model, the present application performs modeling based on the phoneme phone level, which improves the distinguishability of modeling units between various types of dialects and between dialects and Mandarin from the pronunciation level, and reduces the crosstalk between dialects, thereby improving the recognition effect of each dialect.

[0081] Furthermore, based on phone-level modeling, this application introduces syntactic information in the label annotation of training data, which enables the model to learn the grammatical information of each dialect, that is, it can make full use of the phone-level and grammatical features of the training data to further improve the recognition effect of each dialect. In addition, due to the addition of grammatical information, the readability of the model recognition text can also be improved.

[0082] In some embodiments of the present application, the training process of the above-mentioned dialect speech recognition model is introduced.

[0083] The training data of the model includes speech samples of various types of languages ​​as training samples, and the recognition text of the training samples is subjected to syntactic analysis and phone-level annotation, and the phone-level annotated text carrying syntactic information is used as sample labels.

[0084] First, the process of determining sample labels is introduced:

[0085] S1. Obtain the recognition text corresponding to the training sample.

[0086] S2. Perform syntactic analysis on the recognized text, and annotate the analyzed syntactic information into the recognized text to obtain a character-level annotated text carrying syntactic information.

[0087] For example, if we perform syntactic analysis on the following recognition text {hello everyone}, the resulting syntactic tree structure is as follows:

[0088]

[0089] The annotation result after the above recognition text annotation syntactic information can be expressed as:

[0090] { <ip> <np> <pn>Everyone <pn> <np> <vp> <va>good <va> <vp> <ip>}

[0091] For ease of understanding, this application introduces several ways of representing syntactic information as follows:

[0092] IP represents a simple clause, PP represents a prepositional phrase, P represents a preposition, LCP represents a locative phrase, NP represents a noun phrase, NN represents an oral noun, LC represents a locative, PN represents a pronoun, AD represents an adverb, VV represents a modal verb, VP represents a verb phrase, and ADVP represents an adverbial phrase.

[0093] Taking the way of representing syntactic information in the above example as an example, there are a total of 12 modeling units corresponding to the syntactic information.

[0094] S3. Use a phone-level pronunciation dictionary that matches the language type of the training samples to determine the phone-level annotation corresponding to the recognition text.

[0095] Among them, for the two language types of Mandarin and Cantonese, directly use their respective phone-level pronunciation dictionaries.

[0096] For several other types of dialects, the phone-level pronunciation dictionary of Mandarin can be reused to form a phone-level pronunciation dictionary that matches their respective language types.

[0097] Among them, Mandarin has two pronunciation systems, namely the main vowel system and the initial-final system. Among them, the main vowel system contains 81 phones, and the initial-final system contains 177 phones. The initial-final system is the pinyin annotation that we are familiar with. Taking "Hello everyone" as an example, in the pronunciation dictionary of the initial-final system, the pronunciation annotation is {d a4 j ia1 h ao3}, and in the pronunciation dictionary of the main vowel system, the pronunciation annotation is {d a4 j y a1 h aw3}. In this application, for the pronunciation dictionary of Mandarin, either of the above two systems can be adopted. The following takes the initial-final system as an example for illustration.

[0098] For the remaining several dialects other than Mandarin and Cantonese, the process of reusing the phone-level pronunciation dictionary of Mandarin to form a phone-level pronunciation dictionary that matches their respective language types may include:

[0099] S31. Number each of the remaining language types to obtain the number corresponding to each language type.

[0100] S32. For any one of the remaining language types:

[0101] Add the number corresponding to the language type after each phone in the phone-level pronunciation dictionary of Mandarin to obtain a phone-level pronunciation dictionary that matches the language type.

[0102] Taking "Hello everyone" as an example, if the annotation result in the Mandarin phone-level pronunciation dictionary is {d a4 j ia1 hao3}, then for the Nth type of dialect, the annotation result in the matching phone-level pronunciation dictionary is {d_N a4_N j_N ia1_Nh_N ao3_N}.

[0103] S4. Replace the characters in the character-level annotation text carrying syntactic information with the corresponding phone-level annotations to obtain a phone-level annotation text carrying syntactic information.

[0104] Specifically, using the character-level annotation text carrying syntactic information obtained in step S2 above { <ip> <np> <pn>Everyone <pn> <np> <vp> <va>good <va> <vp> <ip>} as an example, the results of phone-level annotation replacement are as follows when the training samples are Mandarin or the other seven dialects:

[0105] Class 0 Mandarin: { <sos> <ip> <np> <pn>d a4 j ia1 <pn> <np> <vp> <va>h ao3 <va> <vp> <ip> <eos>}

[0106] Category 1 Northern dialect: { <sos> <ip> <np> <pn>d_1a4_1j_1ia1_1 <pn> <np> <vp> <va>h_1ao3_1 <va> <vp> <ip> <eos>}

[0107] Class 2 Wu dialect is marked as { <sos> <ip> <np> <pn><h2 style=";text-align:left;direction:ltr">d_2a4_2j_2ia1_2<h2 style=";text-align:left;direction:ltr"> <pn> <np> <vp> <va><h2 style=";text-align:left;direction:ltr">h_2ao3_2<h2 style=";text-align:left;direction:ltr"> <va> <vp> <ip> <eos>}

[0108] …

[0109] Category 7 Cantonese dialect is marked as { <sos> <ip> <np> <pn><h2 style=";text-align:left;direction:ltr">d_7aa6_7iT_7g_7 aa1_7<h2 style=";text-align:left;direction:ltr"> <pn> <np> <vp> <va><h2 style=";text-align:left;direction:ltr">h_7o2_7uT_7<h2 style=";text-align:left;direction:ltr"> <va> <vp> <ip> <eos>}

[0110] Among them, the Cantonese dialect uses its own corresponding phone-level pronunciation dictionary for phone-level annotation, and there is no need to reuse the phone-level pronunciation dictionary of Mandarin.

[0111] in, <sos> 、 <eos>The start and end characters for the set annotation.

[0112] Combined with the previous introduction, the phone-level pronunciation dictionary of the Cantonese dialect contains 88 phones, and the phone-level pronunciation dictionary under the vowel-vowel system of Mandarin contains 177 phones. Except for Cantonese, the other 6 dialects reuse the phone-level pronunciation dictionary of Mandarin to contain 177*6 phones. There are 12 modeling units when annotating syntactic information, in addition to a silent note sil, two start and end symbols <sos> 、 <eos>, the total number of modeling units that can be used in the dialect speech recognition model modeling process is: 88+177+177*6+12+1+2=1342.

[0113] In some embodiments of the present application, the dialect speech recognition model may include two parts: an encoding module and a decoding module.

[0114] Among them, the encoding module is used to encode the input speech to obtain speech coding features; the decoding module is used to predict the phone-level annotated text carrying syntactic information corresponding to the input speech based on the speech coding features.

[0115] It is understandable that it is relatively simple to obtain Mandarin training data, and a large amount of supervised training data can be accumulated. However, for dialects, the amount of supervised dialect training data is relatively small. To this end, a pre-training scheme is provided in this embodiment to pre-train the encoding module, so that the encoding module is pre-trained using unsupervised training data, and then the encoding module can be fine-tuned using supervised training data.

[0116] Next, the pre-training process of the encoding module is explained:

[0117] S1. Obtain training data set.

[0118] The training data set includes training speech of various types of languages, and the various types of languages ​​include Mandarin and various types of dialects.

[0119] In an optional manner, the process of obtaining a training data set may include:

[0120] S11. Obtain the original training data set.

[0121] S12. Adjust the ratio of the amount of training speech data of each language in the original training data set so that in the adjusted training data set, the ratio of the amount of training speech data of Mandarin to the amount of training speech data of the other dialects does not exceed a set ratio, and the amount of training speech data of the dialects remains consistent.

[0122] The ratio of Mandarin to dialect training speech data may not exceed 2:1, so as to ensure that the speech data of various types of languages ​​in the training data set are balanced while making full use of Mandarin training speech data.

[0123] Specific adjustment methods may include enhancing the training speech of each type of dialect by adding noise, fast speaking speed, tfmask, etc., so as to expand the amount of dialect training speech data.

[0124] S13, performing randomization on the adjusted training data set so that the distribution of training speech of each type of language in the training data set is random.

[0125] Specifically, in order to ensure that when pre-training the encoding model based on the training data set, the training speech data of each language type can be extracted as training data, the adjusted training data set can be randomized in this step to disrupt the sorting order of the training speech of each language type in the set, so that the distribution of training speech of each type of language is random.

[0126] S2. In the training data set, the training speech of the same type of language constitutes the positive sample pair, and the training speech of different types of language constitutes the negative sample pair. The encoding module is trained using a contrastive learning strategy.

[0127] Specifically, during the training process, the goal is to maximize the similarity between the speech coding features of the positive sample pairs and minimize the similarity between the speech coding features of the negative sample pairs until the set training end condition is reached to obtain the trained coding module.

[0128] In this embodiment, a contrastive learning strategy is adopted to train the encoding module, which can make full use of Mandarin resources to optimize the recognition of low-resource dialects, improve the recognition effect of low-resource dialects, and reduce the cost of training data.

[0129] At the same time, it also enables the encoding module to learn the differences between different language types.

[0130] Among them, the training loss function can be as follows:

[0131]

[0132] Where N represents the number of training speech in the training data set. Of course, if the training is performed in batches, N can also be represented by the number of training speech in a batch. i , z j and z k They represent the encoding features of the i, j, and k training speech after being encoded by the encoding module. sim() represents the similarity calculation. When lid j ≠lid i When , it means that the language type of the i-th training speech and the j-th training speech is the same. The value is 1, when lid k≠ ≠lid i When , it means that the language types of the i-th training speech and the k-th training speech are different. The value is 0, and τ is the set network parameter.

[0133] It can be seen from the above formula that the numerator only calculates the similarity between positive sample pairs, and the denominator only calculates the similarity between negative sample pairs. The greater the similarity between positive sample pairs and the smaller the similarity between negative sample pairs, the smaller the overall loss.

[0134] This embodiment provides a pre-training method based on comparative learning between Mandarin and dialects, which can improve the feature representation ability of the encoding module for different types of dialects and Mandarin at the acoustic level, thereby achieving the purpose of improving the recognition effect of various dialects by using Mandarin data in the context of low dialect resources.

[0135] Next, combine Figure 2 , introducing an optional overall flow chart of the dialect speech recognition method of the present application.

[0136] For the dialect speech recognition model, it includes an encoding module encoder and a decoding module decoder.

[0137] The present application can adopt a comparative learning strategy based on dialect speech training data and Mandarin speech training data to pre-train the encoding module, and the pre-trained encoding module and decoding module form a dialect speech recognition model.

[0138] During the training process of the dialect speech recognition model, speech samples of various languages ​​are used as training samples. The recognition text of the training samples is subjected to syntactic analysis and phone-level annotation, and the phone-level annotation text carrying syntactic information is used as the sample label. The process of determining the sample label is as follows: Figure 2 As shown, the recognized text is first subjected to syntactic analysis to obtain a word-level annotated text with syntactic information.

[0139] For all types of dialects except Cantonese, the phone-level pronunciation dictionary of Mandarin can be reused, and Cantonese and Mandarin can directly use their respective corresponding phone-level pronunciation dictionaries.

[0140] According to the language type to which the speech sample corresponding to the recognized text belongs, the recognized text is annotated at the phone level using the phone-level pronunciation dictionary of the language type, and the characters in the word-level annotated text with syntactic information are replaced with the phone-level annotations to obtain the phone-level annotated text with syntactic information.

[0141] The speech samples corresponding to the recognized text and the phone-level annotated text with syntactic information are used as training data to train the dialect speech recognition model composed of the encoding module and the decoding module. The training process can use a cross entropy loss function, etc.

[0142] After training the dialect speech recognition model, the speech to be recognized in the target language type is input into the model to obtain the phone-level annotated text carrying syntactic information after decoding. The phone-level annotated text carrying syntactic information is input into the decoding network corresponding to the target language type for decoding to obtain the decoded character-level annotated text carrying syntactic information. After text regularization and removal of syntactic information, the character-level recognized text is obtained.

[0143] The following is a description of the dialect speech recognition device provided in an embodiment of the present application. The dialect speech recognition device described below and the dialect speech recognition method described above can be referenced to each other.

[0144] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a dialect speech recognition device disclosed in an embodiment of the present application.

[0145] like Figure 3 As shown, the device may include:

[0146] A speech acquisition unit 11 for acquiring speech to be recognized in a target language type;

[0147] The model processing unit 12 is used to input the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model;

[0148] The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects;

[0149] A character decoding unit 13 is used to decode the phone-level annotated text carrying syntactic information by using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information;

[0150] The text regularization unit 14 is used to perform text regularization on the character-level annotated text carrying syntactic information to remove the syntactic information therein and obtain the dialect speech recognition text.

[0151] Optionally, the apparatus of the present application further includes a model training unit, which is used to train and obtain a dialect speech recognition model, wherein the process of determining the labels of training samples in the dialect speech recognition model training process may include:

[0152] Obtaining the recognition text corresponding to the training sample;

[0153] Performing syntactic analysis on the recognized text, and annotating the analyzed syntactic information into the recognized text to obtain a character-level annotated text carrying the syntactic information;

[0154] Using a phone-level pronunciation dictionary that matches the language type of the training sample, determining a phone-level annotation corresponding to the recognized text;

[0155] The characters in the character-level annotated text carrying syntactic information are replaced with corresponding phone-level annotations to obtain phone-level annotated text carrying syntactic information.

[0156] Optionally, the process of determining the phone-level pronunciation dictionary matching each language type by the model training unit may include:

[0157] For Mandarin and Cantonese, the corresponding phone-level pronunciation dictionary is used directly;

[0158] For the remaining language types except Mandarin and Cantonese, the phone-level pronunciation dictionary of Mandarin is reused to form a phone-level pronunciation dictionary matching the respective language types.

[0159] Optionally, the model training unit reuses the phone-level pronunciation dictionary of Mandarin for each language type other than Mandarin and Cantonese to form a phone-level pronunciation dictionary matching each language type, which may include:

[0160] Numbering the remaining language types respectively to obtain a number corresponding to each language type;

[0161] For any of the remaining language types:

[0162] A number corresponding to the language type is added after each phone in the phone-level pronunciation dictionary of Mandarin to obtain a phone-level pronunciation dictionary matching the language type.

[0163] Optionally, the dialect speech recognition model may include an encoding module and a decoding module, the encoding module being used to encode the input speech to obtain speech encoding features; the decoding module being used to predict the phone-level annotated text carrying syntactic information corresponding to the input speech based on the speech encoding features.

[0164] Optionally, the apparatus of the present application may further include a pre-training unit for pre-training the encoding module. The encoding module pre-training process may include:

[0165] Acquire a training data set, wherein the training data set includes training speech of various types of languages, wherein the various types of languages ​​include Mandarin and various types of dialects;

[0166] In the training data set, the training speech of the same type of language constitutes the positive sample pair, and the training speech of different types of languages ​​constitutes the negative sample pair. The encoding module is trained using a contrastive learning strategy. During the training process, the goal is to maximize the similarity between the speech coding features of the positive sample pair and minimize the similarity between the speech coding features of the negative sample pair until the set training end condition is met to obtain the trained encoding module.

[0167] Optionally, the process of the pre-training unit acquiring the training data set may include:

[0168] Get the original training data set;

[0169] Adjusting the ratio of the amount of training speech data of each language in the original training data set so that in the adjusted training data set, the ratio of the amount of training speech data of Mandarin to the amount of training speech data of the other dialects does not exceed a set ratio, and the amount of training speech data of the dialects remains consistent;

[0170] The adjusted training data set is randomly processed so that the distribution of training speech of each type of language in the training data set is random.

[0171] Optionally, the apparatus of the present application may further include a decoding network establishing unit, which is used to establish a decoding network corresponding to the target language type. The process may include:

[0172] The decoding network corresponding to the target language type is trained using a phone-level pronunciation dictionary that matches the target language type and a text corpus of the target language type that carries syntactic information.

[0173] The dialect speech recognition device provided in the embodiment of the present application can be applied to dialect speech recognition devices, such as terminals: mobile phones, computers, etc. Optionally, Figure 4 The hardware structure diagram of the dialect speech recognition device is shown in FIG. Figure 4 ,The hardware structure of the dialect speech recognition device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0174] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0175] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0176] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0177] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0178] Obtain the speech to be recognized in the target language type;

[0179] Inputting the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model;

[0180] The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects;

[0181] Decoding the phone-level annotated text carrying syntactic information using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information;

[0182] The character-level annotated text carrying syntactic information is regularized to remove the syntactic information therein and obtain a dialect speech recognition text.

[0183] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0184] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0185] Obtain the speech to be recognized in the target language type;

[0186] Inputting the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model;

[0187] The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects;

[0188] Decoding the phone-level annotated text carrying syntactic information using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information;

[0189] The character-level annotated text carrying syntactic information is regularized to remove the syntactic information therein and obtain a dialect speech recognition text.

[0190] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0191] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0192] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0193] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / eos> < / sos> < / eos> < / sos> < / eos> < / ip> < / vp> < / va> < / va> < / vp> < / np> < / pn> < / pn> < / np> < / ip> < / sos> < / eos> < / ip> < / vp> < / va> < / va> < / vp> < / np> < / pn> < / pn> < / np> < / ip> < / sos> < / eos> < / ip> < / vp> < / va> < / va> < / vp> < / np> < / pn> < / pn> < / np> < / ip> < / sos> < / eos> < / ip> < / vp> < / va> < / va> < / vp> < / np> < / pn> < / pn> < / np> < / ip> < / sos> < / ip> < / vp> < / va> < / va> < / vp> < / np> < / pn> < / pn> < / np> < / ip> < / ip> < / vp> < / va> < / va> < / vp> < / np> < / pn> < / pn> < / np> < / ip>

Claims

1. A dialect speech recognition method, characterized in that: include: Obtain the speech to be recognized in the target language type; Inputting the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model; The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects; Decoding the phone-level annotated text carrying syntactic information using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information; Performing text regularization on the character-level annotated text carrying syntactic information to remove the syntactic information therein and obtain a dialect speech recognition text; The process of establishing a decoding network corresponding to the target language type includes: The decoding network corresponding to the target language type is trained using a phone-level pronunciation dictionary that matches the target language type and a text corpus of the target language type that carries syntactic information.

2. The method according to claim 1, characterized in that The process of determining the labels of the training samples in the dialect speech recognition model training process includes: Obtaining the recognition text corresponding to the training sample; Performing syntactic analysis on the recognized text, and annotating the analyzed syntactic information into the recognized text to obtain a character-level annotated text carrying the syntactic information; Using a phone-level pronunciation dictionary that matches the language type of the training sample, determining a phone-level annotation corresponding to the recognized text; The characters in the character-level annotated text carrying syntactic information are replaced with corresponding phone-level annotations to obtain phone-level annotated text carrying syntactic information.

3. The method according to claim 2, characterized in that The process of determining the phone-level pronunciation dictionary that matches each language type includes: For Mandarin and Cantonese, the corresponding phone-level pronunciation dictionary is used directly; For the remaining language types except Mandarin and Cantonese, the phone-level pronunciation dictionary of Mandarin is reused to form a phone-level pronunciation dictionary matching the respective language types.

4. The method according to claim 3, characterized in that For each language type other than Mandarin and Cantonese, the phone-level pronunciation dictionary of Mandarin is reused to form a phone-level pronunciation dictionary matching each language type, including: Numbering the remaining language types respectively to obtain a number corresponding to each language type; For any of the remaining language types: A number corresponding to the language type is added after each phone in the phone-level pronunciation dictionary of Mandarin to obtain a phone-level pronunciation dictionary matching the language type.

5. The method according to claim 1, characterized in that The dialect speech recognition model includes an encoding module and a decoding module. The encoding module is used to encode the input speech to obtain speech encoding features; the decoding module is used to predict the phone-level annotated text carrying syntactic information corresponding to the input speech based on the speech encoding features.

6. The method according to claim 5, characterized in that The encoding module is trained by pre-training; wherein the encoding module pre-training process includes: Acquire a training data set, wherein the training data set includes training speech of various types of languages, wherein the various types of languages ​​include Mandarin and various types of dialects; In the training data set, the training speech of the same type of language constitutes the positive sample pair, and the training speech of different types of languages ​​constitutes the negative sample pair. The encoding module is trained using a contrastive learning strategy. During the training process, the goal is to maximize the similarity between the speech coding features of the positive sample pair and minimize the similarity between the speech coding features of the negative sample pair until the set training end condition is met to obtain the trained encoding module.

7. The method according to claim 6, characterized in that The obtaining of the training data set comprises: Get the original training data set; Adjusting the ratio of the amount of training speech data of each language in the original training data set so that in the adjusted training data set, the ratio of the amount of training speech data of Mandarin to the amount of training speech data of the other dialects does not exceed a set ratio, and the amount of training speech data of the dialects remains consistent; The adjusted training data set is randomly processed so that the distribution of training speech of each type of language in the training data set is random.

8. A dialect speech recognition device, characterized in that: include: A speech acquisition unit to be recognized, used to acquire the speech to be recognized in the target language type; A model processing unit, used for inputting the speech to be recognized into a preset dialect speech recognition model to obtain a phoneme-level annotated text carrying syntactic information output by the model; The dialect speech recognition model is obtained by using speech samples of various types of languages ​​as training samples, and using the recognition text of the training samples after syntactic analysis and phone-level annotation with syntactic information as labels for training; the various types of languages ​​include Mandarin and various types of dialects; A decoding network building unit is used to build a decoding network corresponding to the target language type, including: using a phone-level pronunciation dictionary matching the target language type and a text corpus of the target language type carrying syntactic information to train a decoding network corresponding to the target language type; A character decoding unit, configured to decode the phone-level annotated text carrying syntactic information by using a preset decoding network corresponding to the target language type to obtain a character-level annotated text carrying syntactic information; The text regularization unit is used to perform text regularization on the character-level annotated text carrying syntactic information to remove the syntactic information therein and obtain the dialect speech recognition text.

9. A dialect speech recognition device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the dialect speech recognition method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the dialect speech recognition method as described in any one of claims 1 to 7 is implemented.

Images