Language similarity evaluation method based on forced decoding, speech recognition method and related equipment

Abstract

The invention provides a language similarity evaluation method based on forced decoding, a speech recognition method and related equipment. The language similarity evaluation method based on forced decoding comprises the following steps: S101, selecting a multi-language voice model, and appointing a target language of the multi-language voice model; the multi-language speech model is used for decoding an input source language speech into a target language text; s102, inputting a source language voice into the multi-language voice model so as to forcibly decode the source language voice into a target language text, and obtaining a normalized posterior probability of the target language text; and S103, calculating a language similarity score between the source language and the target language based on the normalized posterior probability. Based on the method, the invention also provides a speech recognition method based on language similarity and related equipment. The source language speech is compulsively decoded into the target language text, the similarity of cross-language intrinsic representation is implicitly calculated, and speech recognition is performed based on the similarity score, so that the recognition performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a language similarity evaluation method based on forced decoding, a speech recognition method and related equipment. Background Art

[0002] In the field of speech recognition, although different languages ​​differ in pronunciation structure and expression, their high-dimensional features have similar distribution patterns, that is, similar semantics in different languages ​​have similar high-dimensional features. Multilingual joint training can make full use of the inherent relationship between different languages, so that multilingual deep information can promote and enhance each other, and assist in improving the speech recognition performance of the target language. Studies have shown that improving the model's ability to distinguish between languages ​​is more conducive to improving multilingual speech recognition performance, and how to introduce effective language-related knowledge during the multilingual joint training of the model is the key to improving its language discrimination ability.

[0003] At present, the methods of introducing language information can be divided into methods based on prior knowledge, methods based on data-driven, and methods based on pre-trained models. In general, methods based on prior knowledge require the experience and expertise of experts in the field. Although they are effective, they are expensive and cannot adapt to today's intelligent trends. Data-driven methods rely on training data. Only when there is a sufficient amount of data can the mapping network show good performance and obtain reliable similarity results. This data dependence makes it less applicable. The method based on pre-trained models has been the mainstream method in recent years, but current research only judges language similarity based on the explicit information of the language type of speech specified by humans, and fails to explore the role of implicit information between languages ​​in similarity assessment. Therefore, when using existing language similarity methods for subsequent speech recognition, the accuracy of speech recognition results needs to be improved. Summary of the invention

[0004] In order to solve the problem that the existing language similarity evaluation technology does not explore the implicit information between languages ​​for similarity evaluation, and the problem that speech recognition based on the existing language similarity method has a low recognition result accuracy, the present invention provides a language similarity evaluation method based on forced decoding and a speech recognition method based on language similarity.

[0005] In a first aspect, the present invention provides a language similarity assessment method based on forced decoding, comprising:

[0006] S101: Select a multilingual speech model and specify a target language of the multilingual speech model; the multilingual speech model is used to decode the input source language speech into the target language text;

[0007] S102: Inputting the source language speech into the multilingual speech model to forcibly decode it into a target language text, and obtaining a normalized posterior probability of the target language text;

[0008] S103: Calculating a language similarity score between the source language and the target language based on the normalized posterior probability.

[0009] Furthermore, the calculation formula of the language similarity score is as follows:

[0010]

[0011] Among them, α represents the language similarity score, J st represents the number of words in the decoded target language text, p(·) represents the normalized posterior probability, Indicates that the nth speech in source language s is decoded into the jth word in the text of target language t.

[0012] In a second aspect, the present invention provides a speech recognition method based on language similarity, comprising:

[0013] S201: Build a multilingual speech recognition dataset;

[0014] S202: In the multilingual speech recognition dataset, specify a target language, and use the remaining languages ​​as candidate source languages, and use the language similarity evaluation method as described in claim 1 or 2 to obtain a language similarity score between each candidate source language and the target language;

[0015] S203: Selecting source language data with a higher language similarity score and mixing them with target language data as a training data set, and performing multilingual joint fine-tuning on a given multilingual speech model based on the training data set to obtain a speech recognition model;

[0016] S204: Inputting the target language speech to be recognized into the speech recognition model to obtain a speech recognition result.

[0017] Furthermore, the multilingual speech recognition data set includes speech data and language text data; wherein the language text data is mainly composed of the language text corresponding to each piece of speech data.

[0018] Furthermore, the multilingual speech model adopts an MMS model.

[0019] In a third aspect, the present invention provides a language recognition system based on language similarity, comprising:

[0020] A data set construction module, used to obtain multilingual speech data and construct a multilingual speech data set; wherein the multilingual speech data set includes speech data and text data, and each language data has corresponding text data;

[0021] A language similarity evaluation module, used to designate a target language and take the remaining languages ​​as candidate source languages ​​in the multilingual speech recognition dataset, and obtain a language similarity score between each candidate source language and the target language using a language similarity evaluation method;

[0022] The model training module is used to sort the language similarity scores and select the source language data with high scores to mix with the target language data, perform multilingual joint fine-tuning on the given multilingual speech model, and obtain the speech recognition model;

[0023] The speech recognition module is used to input the target language speech to be recognized into the speech recognition model to obtain the speech recognition text.

[0024] In a fourth aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0025] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the method described above when executed by a processor.

[0026] Beneficial effects of the present invention:

[0027] The present invention provides a speech recognition method based on language similarity, which can effectively evaluate the similarity between languages ​​and apply it to multilingual speech recognition, thereby improving the speech recognition performance of the target language. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of a language similarity assessment method based on forced decoding provided by an embodiment of the present invention;

[0029] Figure 2 A flow chart of a speech recognition method based on language similarity provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the structure of a speech recognition system based on language similarity provided by an embodiment of the present invention;

[0031] Figure 4A structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, an embodiment of the present invention provides a language similarity evaluation method based on forced decoding, comprising:

[0034] S101: Select a multilingual speech model and specify a target language of the multilingual speech model;

[0035] Specifically, it can be understood that the multilingual speech model is an intelligent model that decodes the input source language speech into the target language text. Generally speaking, the original target language of the pre-trained multilingual speech model is uncertain and can be multiple languages ​​at the same time. The embodiment of the present invention is intended to evaluate the similarity between the source language and a target language of a specific language, therefore, it is necessary to specify the language of the target language in this step.

[0036] S102: Inputting the source language speech into the multilingual speech model to forcibly decode it into the target language text, and obtaining the normalized posterior probability of the target language text;

[0037] Specifically, it has been mentioned above that when the target language of the multilingual speech model is not specified, the language of the target language is uncertain. There is a situation where the target language output by the model by default is not the target language you want to study. Therefore, this step is based on the previous step and decodes it into the text of the specified target language through forced decoding.

[0038] S103: Calculate a language similarity score between the source language and the target language based on the normalized posterior probability.

[0039] The language similarity method provided in the embodiment of the present invention can effectively evaluate the similarity between languages ​​by forcibly decoding the source language speech into the target language text and using its normalized posterior as the basis for language similarity determination, implicitly calculating the similarity of cross-language intrinsic representations. It should be noted that in addition to being applied to speech recognition, this similarity method can also be used for training translation models to effectively improve machine translation performance.

[0040] In one embodiment, the language similarity score calculation formula is as follows:

[0041]

[0042] Among them, α represents the similarity score, J st represents the number of words in the target language text to be decoded, p(·) represents the posterior probability during the decoding process, Indicates the jth word in the text corresponding to the nth speech after the source language s is decoded into the target language t.

[0043] Specifically, the larger the α is, the stronger the similarity between the phonetic information of the source language and the target language is.

[0044] like Figure 2 As shown, an embodiment of the present invention provides a speech recognition method based on language similarity, comprising:

[0045] S201: Build a multilingual speech recognition dataset;

[0046] Specifically, the multilingual speech recognition dataset includes speech data and language text data; the language text data is mainly composed of the language text corresponding to each speech data. The total vocabulary corresponding to the multilingual text in the dataset is V; the multilingual speech dataset is represented by the following formula:

[0047]

[0048] in,

[0049]

[0050] in, is the data of the lth language, and the corresponding vocabulary of the lth language is V l , L is the total number of languages ​​in the multilingual speech dataset, x nl is the voice data, y nl The language text, each voice data has a corresponding language text;

[0051] Voice datax nl It is expressed as follows:

[0052]

[0053] Among them, x nl is the multidimensional vector of the nth speech data of the lth language in the multilingual speech recognition dataset, is the component of the i-th dimension of the vector;

[0054] Understandable, x nl It can be the original speech waveform or the extracted speech features, depending on the input data mode of the multilingual speech model.

[0055] The corresponding language text data is:

[0056]

[0057] Among them, y nl For x nl The corresponding language text data, that is, the text corresponding to the nth speech data in the lth language in the multilingual speech recognition dataset, represents the jth word of the language text data corresponding to the nth voice data in the lth language, and

[0058] S202: In a multilingual speech recognition dataset, a target language is specified, and the remaining languages ​​are taken as candidate source languages, and the above-mentioned language similarity evaluation method is used to obtain a language similarity score between each candidate source language and the target language;

[0059] Specifically, in the multilingual speech recognition dataset constructed in step S201, if the target language is specified, the remaining language types in the dataset are all source languages, and the language similarity evaluation method based on forced decoding is used to obtain the language similarity score between each source language and the target language.

[0060] It can be understood that the data corresponding to the target language in the multilingual speech recognition dataset is deleted to obtain the source language data:

[0061]

[0062] in, Represents source language data, represents a multilingual speech dataset, Represents the data of the target language, t∈1,2,...,L;

[0063] Specifically, the source language data Language data in Forced to decode the text into the target language data, the decoding result is The above process can be expressed as follows:

[0064]

[0065] in,

[0066]

[0067] in, Indicates the jth word in the text corresponding to the nth speech after the source language data is decoded into the target language t.

[0068] S203: Selecting source language data with a higher language similarity score and mixing them with target language data as a training data set, and performing multilingual joint fine-tuning on a given multilingual speech model based on the training data set to obtain a speech recognition model;

[0069] Specifically, the language similarity scores α between the source languages ​​and the target languages ​​calculated in step S202 are sorted. The larger the α, the stronger the similarity between the speech information of the source language and the target language. The source language data with high language similarity scores α are selected and mixed with the target language data as the training data set, and the multilingual joint fine-tuning is performed on the given multilingual speech model.

[0070] In this embodiment, 1 hour of data with a high language similarity score α is selected, the MMS model is used as the baseline model, and multi-language joint fine-tuning is performed to obtain a speech recognition model.

[0071] It can be understood that the language similarity score α can not only represent the similarity between languages ​​in terms of language type, but also measure the similarity through the high-dimensional implicit representation of speech data calculated layer by layer by the model. Compared with only using the output posterior of the language identification (LID) model as an explicit indicator as the evaluation criterion, α also gives full play to the model's implicit integration and extraction capabilities of multilingual knowledge on this basis. When the model has strong multilingual understanding and abstract representation capabilities, the superiority of α will be fully demonstrated.

[0072] S204: Input the target language speech to be recognized into the speech recognition model to obtain the speech recognition result.

[0073] like Figure 3 As shown, the embodiment of the present invention also provides a speech recognition system based on language similarity, including:

[0074] A data set construction module is used to obtain multilingual speech data and construct a multilingual speech data set; wherein the multilingual speech data set includes speech data and text data, and each language data has corresponding text data;

[0075] A language similarity assessment module is used to specify a target language in a multilingual speech recognition dataset and use the remaining languages ​​as candidate source languages, and use a language similarity assessment method to obtain a language similarity score between each candidate source language and the target language;

[0076] The model training module is used to sort the language similarity scores and select the source language data with high scores to mix with the target language data, perform multilingual joint fine-tuning on the given multilingual speech model, and obtain the speech recognition model;

[0077] The speech recognition module is used to input the target language speech to be recognized into the speech recognition model to obtain the speech recognition text.

[0078] like Figure 4 As shown, an embodiment of the present invention further provides an electronic device, which may include: a processor (processor) 401, a communication interface (Communications Interface) 402, a memory (memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. The processor 401 may call the logic instructions in the memory 403 to execute the method provided in the above embodiment, for example, including:

[0079] S101: Select a multilingual speech model and specify the target language of the multilingual speech model; the multilingual speech model is used to decode the input source language speech into the target language text; S102: Input the source language speech into the multilingual speech model to force it to be decoded into the target language text, and obtain the normalized posterior probability of the target language text; S103: Calculate the language similarity score between the source language and the target language based on the normalized posterior probability. For example, it also includes:

[0080] S201: Construct a multilingual speech recognition dataset; S202: In the multilingual speech recognition dataset, specify the target language, and use the remaining languages ​​as candidate source languages, and use the language similarity evaluation method to obtain the language similarity score between each candidate source language and the target language; S203: Select the source language data with a higher language similarity score and mix it with the target language data as a training dataset, and perform multilingual joint fine-tuning on the given multilingual speech model based on the training dataset to obtain a speech recognition model; S204: Input the target language speech to be recognized into the speech recognition model to obtain the speech recognition result.

[0081] In addition, when the logic instructions in the above-mentioned memory 403 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0082] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiment is implemented, for example, including:

[0083] S101: Select a multilingual speech model and specify the target language of the multilingual speech model; the multilingual speech model is used to decode the input source language speech into the target language text; S102: Input the source language speech into the multilingual speech model to force it to be decoded into the target language text, and obtain the normalized posterior probability of the target language text; S103: Calculate the language similarity score between the source language and the target language based on the normalized posterior probability. For example, it also includes:

[0084] S201: Construct a multilingual speech recognition dataset; S202: In the multilingual speech recognition dataset, specify the target language, and use the remaining languages ​​as candidate source languages, and use the language similarity evaluation method to obtain the language similarity score between each candidate source language and the target language; S203: Select the source language data with a higher language similarity score and mix it with the target language data as a training dataset, and perform multilingual joint fine-tuning on the given multilingual speech model based on the training dataset to obtain a speech recognition model; S204: Input the target language speech to be recognized into the speech recognition model to obtain the speech recognition result.

[0085] In order to verify the effectiveness of the speech recognition method based on language similarity proposed in this invention, the large-scale multilingual speech model MMS-300M released by Meta AI is used as the baseline model. Since the model needs to be fine-tuned to have speech recognition capabilities, all comparison methods are based on fine-tuning technology. The difference between the methods lies in the different training data selected for fine-tuning. The comparison methods include:

[0086] FT: Single language fine-tuning. Fine-tune the model using 1 hour of randomly selected target language training data.

[0087] ML-FT: Multilingual Joint Fine-tuning. Mix randomly selected 1 hour of training data from all 6 languages, and then fine-tune the model in multiple languages. Finally, fine-tune the model using randomly selected 1 hour of target language training data.

[0088] LID-FT: Data selection is performed using a method based on the posterior output of the language identification model. The language identification model we used is MMS-1B. For the target language, 1 hour of training data is randomly selected. For the source language, their 1 hour of training data is not randomly selected, but the posterior output of the language identification model is used as the selection basis. The 1 hour of data with the largest normalized posterior of the target language obtained by the language identification model output is selected. After mixing the 6 language training data, multi-language joint fine-tuning is performed. Finally, the model is fine-tuned using the randomly selected 1 hour of target language training data.

[0089] FD-FT: Use the method proposed in this invention to select data. For the target language, randomly select 1 hour of training data. For the source language data, select 1 hour of data with the largest α. Mix the 6 language training data and perform multi-language joint fine-tuning. Finally, use the randomly selected 1 hour of target language training data to fine-tune the model.

[0090] A single 32GB NVIDIA V100 GPU was used for training, with a batch size of 8 and gradient accumulation every 2 steps. The AdamW optimization algorithm was used for model training for 500 rounds. In the first 50 rounds, the learning rate increased linearly from 4e-6 to 3e-5, then maintained at 3e-5 for 300 rounds, and then decreased linearly to 5e-7 for the last 150 rounds. A model checkpoint was saved every 50 rounds during training. The word error rate (WER) was used as an indicator to measure the accuracy of speech recognition. During the training process, the word error rate on the validation set was always paid attention to, and finally the checkpoint with the smallest word error rate on the validation set was selected as the trained model.

[0091] Table 1 Test results of all compared methods.

[0092]

[0093] Table 1 shows the test results of all the comparison methods, and we can get:

[0094] (1) The test results of multi-language joint fine-tuning are consistently better than those of single-language fine-tuning, and the average WER has decreased by 15.23%. This shows that the similarity between languages ​​can effectively help the model improve its recognition ability of the target language. It can be considered that the source language data is pseudo-enhanced data of the target language data, which indirectly increases the amount of target language data. In addition, through multi-language joint fine-tuning, the model obtains an initial state that is more suitable for target language speech recognition, so after fine-tuning the target language, there can be a relatively large performance improvement.

[0095] (2) Compared with single language fine-tuning and multi-language joint fine-tuning with randomly selected training data, the method based on the output posteriors of the language recognition model achieved consistent WER reduction, and the average WER was relatively reduced by 18.95% and 4.39%, respectively. This shows that screening data in the source language data that is more similar to the target language is conducive to improving the ASR performance of the model. This further proves the correctness of some previous work conclusions.

[0096] (3) Compared with single language fine-tuning, multi-language joint fine-tuning and the method based on language recognition model output posteriors, the proposed method based on forced decoding achieved consistent WER reduction, and the average WER was relatively reduced by 21.74%, 7.68% and 3.45%, respectively. This result shows that the speech recognition method based on language similarity proposed in the present invention can better reflect language similarity than the method based on language recognition model output posteriors, and its advantage lies in the utilization of implicit information of the model. Different from the method based on language recognition model output posteriors that explicitly displays language similarity, the method proposed in the present invention uses the normalized posteriors of forced decoding as an indicator for judging similarity, which is not only explicit similarity at the level of language types, but also similarity at the level of implicit information of the model's internal representation across languages. Comprehensive comparison of objective data and subjective analysis confirms the effectiveness of the speech recognition method based on language similarity proposed in the present invention.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A language similarity assessment method based on forced decoding, characterized in that: include: S101: Select a multilingual speech model and specify a target language of the multilingual speech model; the multilingual speech model is used to decode the input source language speech into the target language text; S102: Inputting the source language speech into the multilingual speech model to forcibly decode it into a target language text, and obtaining a normalized posterior probability of the target language text; S103: Calculating a language similarity score between the source language and the target language based on the normalized posterior probability.

2. The language similarity assessment method based on forced decoding according to claim 1, characterized in that: The calculation formula of the language similarity score is as follows: Among them, α represents the language similarity score, J st represents the number of words in the decoded target language text, p(·) represents the normalized posterior probability, Indicates that the nth speech in source language s is decoded into the jth word in the text of target language t.

3. A speech recognition method based on language similarity, characterized in that: include: S201: Build a multilingual speech recognition dataset; S202: In the multilingual speech recognition dataset, specify a target language, and use the remaining languages ​​as candidate source languages, and use the language similarity evaluation method as described in claim 1 or 2 to obtain a language similarity score between each candidate source language and the target language; S203: Selecting source language data with a higher language similarity score and mixing them with target language data as a training data set, and performing multilingual joint fine-tuning on a given multilingual speech model based on the training data set to obtain a speech recognition model; S204: Inputting the target language speech to be recognized into the speech recognition model to obtain a speech recognition result.

4. The speech recognition method based on language similarity according to claim 3, characterized in that: The multilingual speech recognition data set includes speech data and language text data; wherein the language text data is mainly composed of the language text corresponding to each piece of speech data.

5. The speech recognition method based on language similarity according to claim 3, characterized in that: The multilingual speech model adopts the MMS model.

6. A speech recognition system based on language similarity, characterized in that: include: A data set construction module, used to obtain multilingual speech data and construct a multilingual speech data set; The multilingual speech data set includes speech data and text data, and each piece of language data has corresponding text data; A language similarity evaluation module, used to designate a target language and take the remaining languages ​​as candidate source languages ​​in the multilingual speech recognition dataset, and obtain a language similarity score between each candidate source language and the target language using a language similarity evaluation method; The model training module is used to sort the language similarity scores and select the source language data with high scores to mix with the target language data, perform multilingual joint fine-tuning on the given multilingual speech model, and obtain the speech recognition model; The speech recognition module is used to input the target language speech to be recognized into the speech recognition model to obtain the speech recognition text.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is performed.

Images