Low-resource language speech recognition and model training method, device and program product

Abstract

The application discloses a text processing method and device, related equipment and computer program product. A data set composed of three different data of a target language is used to train a model in a LoRA fine-tuning manner to obtain three low-rank adaptive models corresponding to the three data sets respectively. The first data set uses real voice-text pair data, the second data set has the same text as the first data set and the voice is a synthesized voice, and the third data set includes collected high-resource text corpus and the voice is a synthesized voice. A task arithmetic merging strategy is used to calculate the sum of the first and third low-rank adaptive models and the difference of the second low-rank adaptive model to obtain a merged low-rank adaptive model. The voice recognition model of the target language is composed of the merged low-rank adaptive model and a pre-trained voice recognition model. The model is optimized by fully utilizing the synthesized data without affecting the real data effect, and the voice recognition effect of the low-resource language is improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a text processing method, apparatus, related equipment, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, automatic speech recognition technology has been commercially applied on a large scale in mainstream languages ​​with abundant resources, such as Chinese, English, and Japanese. Speech recognition technology heavily relies on large-scale, high-quality annotated speech corpora as the foundation for model training. However, among the thousands of languages ​​worldwide, the vast majority are low-resource languages—languages ​​with very limited amounts of available real-world speech-text pairing data, such as some minority languages ​​and less commonly spoken languages. The scarcity of training data for low-resource languages ​​makes it difficult to meet the traditional speech recognition model's requirements for "large quantity, high quality, and comprehensive scenarios" of training data.

[0003] To alleviate this problem, some existing technologies employ transfer learning, transferring parameters from pre-trained speech recognition models in mainstream languages ​​to low-resource language recognition tasks, attempting to reduce reliance on low-resource language data through knowledge transfer from mainstream languages. However, due to significant differences in pronunciation rules (such as phoneme composition and tone changes) and grammatical structures among different languages, direct transfer can easily lead to "negative transfer" of the model. That is, knowledge from mainstream languages ​​not only fails to help with low-resource language recognition but also interferes with the model's learning of low-resource language features, ultimately resulting in recognition accuracy that fails to meet practical application requirements. Other existing technologies employ data augmentation techniques, generating "pseudo-data" by perturbing the existing limited amount of low-resource language speech data (such as adding random noise, adjusting speech rate, or changing pitch) to expand the training data scale. However, such methods can only expand data within the existing data distribution range and cannot supplement new speech scenarios or pronunciation features. Furthermore, excessive augmentation can easily lead to data distortion, causing the features learned by the model to deviate from real speech features, further reducing the model's robustness. Some existing technologies employ speech synthesis, synthesizing large amounts of speech data from readily available text data in the corresponding language. The synthesized speech-text pairs are then merged with a small amount of existing real speech-text pairs, and the merged training data is used to train a speech recognition model. However, because synthesized speech data differs significantly from real speech data, directly merging the training data to train the model can easily cause the learned features to deviate from real speech features, affecting the speech recognition performance. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a text processing method, apparatus, related equipment, and computer program product to improve the speech recognition performance of low-resource languages ​​and reduce reliance on large amounts of labeled data. The specific solution is as follows:

[0005] Firstly, a method for training a low-resource language speech recognition model is provided, including:

[0006] First, second, and third datasets for the target language are obtained. The first dataset includes real speech-text pairs, and the second and third datasets each include synthesized speech-text pairs. The text in the second dataset is the same as the text in the first dataset, and the text in the third dataset is the collected text corpus of the target language, and the number of texts is greater than the number of texts in the first dataset.

[0007] A training model consisting of an initial low-rank adaptation model and a pre-trained speech recognition model is obtained. The training model is then fine-tuned using LoRA with the first, second, and third datasets to obtain a first low-rank adaptation model corresponding to the first dataset, a second low-rank adaptation model corresponding to the second dataset, and a third low-rank adaptation model corresponding to the third dataset.

[0008] The first low-rank adaptation model, the second low-rank adaptation model, and the third low-rank adaptation model are merged to obtain a merged low-rank adaptation model, which is the difference between the sum of the first low-rank adaptation model and the third low-rank adaptation model and the second low-rank adaptation model.

[0009] The merged low-rank adaptation model is combined with the pre-trained speech recognition model to obtain the speech recognition model for the target language.

[0010] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of obtaining the first, second, and third datasets for the target language includes:

[0011] Obtain a first dataset, which includes real speech-text pairs in the target language;

[0012] Speech synthesis is performed on the text in the first dataset to obtain the synthesized speech corresponding to the text. The synthesized speech and the corresponding text constitute the second dataset.

[0013] A third dataset is formed by acquiring text corpus in the target language, performing speech synthesis on the text corpus to obtain synthesized speech, and the text corpus and the corresponding synthesized speech. The number of text corpus ...

[0014] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of fine-tuning the model to be trained using a third dataset to obtain the third low-rank adaptive model includes:

[0015] The second dataset is merged into the third dataset, and the merged third dataset is used to fine-tune the model to be trained using LoRA to obtain the third low-rank adaptation model.

[0016] In one possible design, in another implementation of the first aspect of the embodiments of this application, during the process of fine-tuning the model to be trained using LoRA with first, second and third datasets respectively, the loss function includes a gradient penalty term, which is used to constrain the norm of the gradient vector of the LoRA parameters to approach a set small amount.

[0017] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of merging the first low-rank adaptation model, the second low-rank adaptation model, and the third low-rank adaptation model to obtain a merged low-rank adaptation model includes:

[0018] Calculate the difference between the third low-rank adaptive model and the second low-rank adaptive model;

[0019] The first low-rank adaptive model and the difference are weighted and summed to obtain the merged low-rank adaptive model, wherein the weight of the first low-rank adaptive model is greater than the weight of the difference.

[0020] In one possible design, in another implementation of the first aspect of the embodiments of this application, the initial low-rank adaptation model is obtained in the following way:

[0021] The randomly initialized low-rank adaptive model is used as the initial low-rank adaptive model;

[0022] or,

[0023] For a randomly initialized low-rank adaptive model, the first, second, or third dataset is used to fine-tune the randomly initialized low-rank adaptive model by setting a small number of steps, which serves as the initial low-rank adaptive model.

[0024] In one possible design, in another implementation of the first aspect of the embodiments of this application, the pre-trained speech recognition model adopts a Large Language Model (LLM) structure, including:

[0025] Audio encoder, mapping module, and large language model LLM;

[0026] The audio encoder is used to extract audio features from the input audio, and the mapping module is used to map the audio features to the input space of the large language model, and use the mapped features as the input of the large language model to obtain the recognized text through decoding by the large language model.

[0027] Secondly, a low-resource language speech recognition method is provided, including:

[0028] Obtain the speech to be recognized in the target language;

[0029] The speech to be recognized is processed by the configured speech recognition model of the target language to obtain the speech recognition result; wherein the speech recognition model is a model trained by the low-resource language speech recognition model training method described in any of the first aspects of this application.

[0030] Thirdly, an electronic device is provided, comprising: a memory and a processor;

[0031] The memory is used to store programs;

[0032] The processor is configured to execute the program to implement the various steps of the low-resource language speech recognition model training method described in any of the first aspects of this application, or to implement the various steps of the low-resource language speech recognition method described in the second aspect of this application.

[0033] Fourthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the various steps of the low-resource language speech recognition model training method described in any of the first aspects of this application, or implements the various steps of the low-resource language speech recognition method described in the second aspect of this application.

[0034] Using the above technical solutions, this application constructs three datasets. The first dataset includes real speech-text pairs in the target language (low-resource language). The text in the second dataset is the same as the text in the first dataset, and the speech is synthesized from the text. The third dataset includes a large amount of collected text data in the target language and the corresponding synthesized speech. It is understandable that, compared to real speech, a large amount of text data in the target language can be easily collected, and a large-scale third dataset can be constructed through speech synthesis. Based on this, this application adopts a LoRA fine-tuning scheme. The training model consists of an initial low-rank adaptation model and a pre-trained speech recognition model. LoRA fine-tuning is performed on the training model using the first, second, and third datasets respectively. During the process, the parameters of the pre-trained speech recognition model are kept frozen, while the parameters of the low-rank adaptation model are updated, thus obtaining three fine-tuned low-rank adaptation models corresponding to the three datasets, namely, the first, second, and third low-rank adaptation models. This application employs a task-arithmetic merging strategy, combining three low-rank adaptation models. Specifically, it calculates the difference between the sum of the first and third low-rank adaptation models and the second low-rank adaptation model to obtain a merged low-rank adaptation model. This merged low-rank adaptation model, along with a pre-trained speech recognition model, forms the target language's speech recognition model. The difference between the first and second low-rank adaptation models can be understood as the difference between real and synthesized speech, excluding the influence of resource availability. Adding the third low-rank adaptation model allows the merged low-rank adaptation model to simulate the same training effect as high-resource real speech-text pairs, even with only high-resource synthesized speech-text pairs. This achieves optimal model optimization using synthesized data without affecting real-data performance, improving speech recognition performance in low-resource languages ​​and reducing reliance on large amounts of labeled data. Attached Figure Description

[0035] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0036] Figure 1 A schematic diagram of an implementation system architecture for the low-resource language speech recognition method and model training method provided in this application embodiment;

[0037] Figure 2 A schematic diagram of a low-resource language speech recognition model training method provided in this application embodiment;

[0038] Figure 3 A schematic diagram of a training framework for a low-resource language speech recognition model provided in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] Current optimization schemes for speech recognition in low-resource languages ​​(minor languages) generally involve expanding low-resource data through data augmentation and speech synthesis techniques. For example, in speech recognition of the Tibetan Kham dialect, some schemes use noise addition to simulate complex sound interference in the real environment; others train a speech synthesis model using existing data, then use the speech synthesis model to synthesize speech from the low-resource language text, adding the synthesized speech-text pairs to a small existing dataset of real speech-text pairs, and finally training the speech recognition model for the low-resource language using the merged dataset.

[0042] However, speech data obtained through data synthesis or data augmentation differs from real speech data. Directly using data augmentation and synthesized data can cause the features learned by the model to deviate from real speech features, affecting performance in real-world scenarios.

[0043] This application proposes a low-resource speech recognition optimization method based on low-rank adaptive merging, which fully utilizes synthetic data to optimize the speech recognition model without affecting the performance of real data.

[0044] This application provides a method for training a low-resource language speech recognition model, and a corresponding low-resource language speech recognition method, which can be applied to, for example... Figure 1 The system architecture shown may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0045] Either terminal 100 or server 200 can be used independently to execute the low-resource language speech recognition model training method or the low-resource language speech recognition method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the low-resource language speech recognition model training method or the low-resource language speech recognition method provided in the embodiments of this application.

[0046] The following description Figure 1The product form of the mid-terminal 100;

[0047] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, translator, learning machine, teaching large screen, wearable device, vehicle-mounted device, conference terminal, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0048] First, the training method for low-resource language speech recognition models provided in the embodiments of this application will be introduced. Taking the application of this method to a computer device as an example, the computer device can specifically be... Figure 1 The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 2 The training method for this low-resource language speech recognition model specifically includes the following steps:

[0049] Step S100: Obtain the first, second, and third datasets for the target language.

[0050] The target language can be a low-resource language, that is, a language with very little real voice-text pairing data available, such as some minority languages ​​or less commonly spoken languages.

[0051] The first dataset includes real speech-text pairs. For target languages ​​with limited resources, real speech-text pairs are generally scarce; therefore, the first dataset contains a small amount of real speech-text pairs, which can be manually annotated or obtained through open-source methods. The speech included is authentic speech in the target language, such as recordings by real people or speech from real people obtained from the internet. The corresponding text is the recognized text.

[0052] The second and third datasets each contain synthesized speech-text pairs. The text in the second dataset is the same as the text in the first dataset. The text in the third dataset consists of collected text corpora in the target language, and the number of texts in the third dataset is greater than that in the first dataset.

[0053] If we define the real speech-text pairs in the first dataset as (X1, Y1), then the synthesized speech-text pairs in the second dataset are represented as (X2, Y1), and the synthesized speech-text pairs in the third dataset are represented as (X3, Y3), and the number of Y3 pairs is generally much greater than the number of Y1 pairs.

[0054] Combination Figure 3 As shown, in some possible implementations, speech synthesis technology can be used to synthesize the text in the first dataset to obtain the synthesized speech corresponding to the text, and the synthesized speech and the corresponding text constitute the second dataset.

[0055] For the third dataset, one could be text corpus in the target language. Compared to real speech-text pair datasets, which are difficult to obtain and costly, acquiring abundant text corpus in the target language is much easier. Methods include collecting books, newspapers, and other media in the target language, as well as using web crawlers to scrape open-source online articles and social media data. Compared to real speech-text pair datasets in the target language, these methods can easily yield more than 10 times the amount of text corpus compared to real datasets.

[0056] Speech synthesis technology is used to synthesize speech from the acquired text corpus, and the third dataset consists of the text corpus and the corresponding synthesized speech.

[0057] Based on the composition of the first, second, and third datasets mentioned above, each dataset can be summarized as shown in Table 1 below:

[0058] Table 1

[0059]

[0060] Step S110: Obtain the training model consisting of the initial low-rank adaptation model and the pre-trained speech recognition model. Use the first, second and third datasets to fine-tune the training model using LoRA to obtain the first low-rank adaptation model, the second low-rank adaptation model and the third low-rank adaptation model.

[0061] The initial low-rank adaptive model can be represented as ΔW0. Pre-trained speech recognition models can employ various structures, such as the Transformer structure and the Large Language Model (LLM) structure. Taking the LLM structure as an example, it can include an audio encoder, a mapping module, and the LLM itself. The audio encoder extracts audio features from the input audio, the mapping module maps these features to the input space of the LLM, and uses the mapped features as input to the LLM. The LLM then decodes these features to obtain the recognized text.

[0062] In this embodiment, a LoRA fine-tuning strategy is used for model training. Therefore, the model to be trained consists of an initial low-rank adaptive model ΔW0 and a pre-trained speech recognition model. Figure 3As shown, LoRA fine-tuning was performed on the training model using the first, second, and third datasets, respectively. According to the LoRA training principle, the backbone network parameters are frozen during the fine-tuning process, and only the parameters of the low-rank adaptation model (LoRA) are updated. That is, the parameters of the pre-trained speech recognition model are kept frozen, while the parameters of the low-rank adaptation model (LoRA) are updated.

[0063] After fine-tuning and training with the first dataset, the first low-rank adaptive model ΔW1 is obtained; after fine-tuning and training with the second dataset, the second low-rank adaptive model ΔW2 is obtained; and after fine-tuning and training with the third dataset, the third low-rank adaptive model ΔW3 is obtained.

[0064] Step S120: Merge the first low-rank adaptation model, the second low-rank adaptation model, and the third low-rank adaptation model to obtain the merged low-rank adaptation model.

[0065] Specifically, in this embodiment, a task arithmetic merging strategy is adopted to merge the three low-rank adaptation models obtained after fine-tuning training on the three datasets. The merged low-rank adaptation model is the difference between the sum of the first low-rank adaptation model ΔW1 and the third low-rank adaptation model ΔW3 and the second low-rank adaptation model ΔW2, i.e., the merged low-rank adaptation model ΔWnew:

[0066] ΔWnew=ΔW3+ΔW1 -ΔW2

[0067] Based on the principles of task arithmetic, we know that:

[0068] Suppose we have domain dataset A and domain dataset B, with corresponding task vectors respectively. and The two vectors are weighted and summed to obtain a new task vector, which is then added to the pre-trained model. This new model may simultaneously provide gains on both domain task A and domain task B. This can also be viewed as... Add on the basis This allows a model that is originally good at domain A to add functionality for domain B.

[0069] However, when A and B are tasks in different domains, Add on the basis It will not necessarily provide further gains on top of the performance of model A; in fact, it may even suppress the performance of model A when there are task conflicts or significant differences between the data of A and B. Furthermore, the task vector size under low resource conditions will be smaller than the task vector size under high resource conditions, for example... The response is much greater than At the same time, it will also inhibit The response.

[0070] Taking speech recognition as an example, the task vector corresponding to a model trained with a small amount of real speech data. Task vectors corresponding to models trained using synthesized speech data Because the distributions of real and synthesized speech differ significantly, adding the two datasets together does not improve the processing of real speech data. Traditional methods directly mix synthesized and real speech to train the speech recognition model, similar to the vector addition mentioned above. However, synthesized speech may not necessarily improve the speech recognition model and could even introduce negative effects.

[0071] This application considers the case of low-resource real speech, where a large amount of synthetic data would weaken the impact of real speech data on the final model.

[0072] Based on the principle of task arithmetic, it can be further understood that if task B is similar to task A and task D is similar to task C, then if there are task vectors... , , In this case, a new task vector can be obtained by performing operations on the three task vectors using task arithmetic methods:

[0073]

[0074] This new task vector can then be applied to task D. This embodiment utilizes the principle of task arithmetic. For low-resource scenarios, low-resource real speech data and low-resource synthesized speech data can be considered as tasks B and A, respectively; high-resource real speech data and high-resource synthesized speech data can be considered as tasks D and C. Therefore, given the existing first, second, and third datasets (corresponding to low-resource real speech data, low-resource synthesized speech data, and high-resource synthesized speech data, respectively), by fine-tuning the first, second, and third low-rank adaptation models trained on each of the three datasets, a merged low-rank adaptation model can be obtained according to the task arithmetic merging method. This merged low-rank adaptation model can achieve an effect close to that of the low-rank adaptation model fine-tuned and trained on high-resource real speech data. In other words, even in situations where high-resource real speech data is scarce, a low-rank adaptation model with similar performance can be obtained.

[0075] The above formula ΔWnew=ΔW3+ΔW1-ΔW2 can be interpreted from the following two dimensions:

[0076] The first type:

[0077] ΔWnew = ΔW3 + (ΔW1 - ΔW2)

[0078] The difference between the first low-rank adaptation model ΔW1 and the second low-rank adaptation model ΔW2 can be understood as the difference between real speech and synthesized speech excluding the influence of resource quantity. In addition, the third low-rank adaptation model ΔW3 can simulate the same training effect as the high-resource real speech-text pair data when only high-resource synthesized speech-text pair data is available. This achieves the goal of fully utilizing synthesized data to optimize the model without affecting the real data effect, improving the speech recognition effect of low-resource languages ​​and reducing the dependence on a large amount of labeled data.

[0079] The second type:

[0080] ΔWnew = ΔW1 + (ΔW3 - ΔW2)

[0081] The difference between the third low-rank adaptation model ΔW3 and the second low-rank adaptation model ΔW2 can be understood as the difference in information contained between different resource quantities when excluding the influence of the speech source (i.e., whether it is synthetic or real). Combined with the first low-rank adaptation model ΔW1, the merged low-rank adaptation model can simulate the same training effect as the high-resource real speech-text pair data when only low-resource real speech data is available. This achieves the goal of fully utilizing synthetic data to optimize the model without affecting the real data effect, thereby improving the speech recognition effect of low-resource languages ​​and reducing the dependence on a large amount of labeled data.

[0082] Step S130: Merge the merged low-rank adaptation model with the pre-trained speech recognition model to obtain the speech recognition model for the target language.

[0083] Combination Figure 3 As shown, the merged low-rank adaptation model obtained through the above steps is equivalent to the low-rank adaptation model trained with high-resource real speech-text pairs. Therefore, merging it with the pre-trained speech recognition model can yield a speech recognition model for the target language, which can greatly improve the speech recognition performance for low-resource target languages.

[0084] The method provided in this application embodiment enables the optimization of the model by fully utilizing synthetic data without affecting the performance of real data, thereby improving the speech recognition performance of low-resource languages ​​and reducing the reliance on a large amount of labeled data.

[0085] In some embodiments of this application, considering that the model merging method is based on the premise that each model to be merged needs to be fine-tuned from the same pre-trained initial model, the aforementioned embodiments of this application use the same initial low-rank adaptation model and pre-trained speech recognition model to form the model to be trained, serving as the starting point for training the three different datasets respectively. The initial low-rank adaptation model can be obtained in the following way:

[0086] 1. Use the randomly initialized low-rank adaptive model as the initial low-rank adaptive model.

[0087] 2. For the randomly initialized low-rank adaptive model, use the first, second, or third dataset to fine-tune the randomly initialized low-rank adaptive model by setting a small number of steps, and use it as the initial low-rank adaptive model.

[0088] By using a dataset to fine-tune a randomly initialized low-rank adaptive model in a few steps, the initial low-rank adaptive model can acquire basic capabilities, thus accelerating the speed of subsequent fine-tuning training.

[0089] In some embodiments of this application, a dataset constraint strategy is provided:

[0090] To improve the stability of merging low-rank adaptive models, the results of the second low-rank adaptive model ΔW2 trained on the second dataset and the third low-rank adaptive model ΔW3 trained on the third dataset should not deviate too much from each other on the same data. Therefore, during the fine-tuning training process of the third low-rank adaptive model ΔW3, the second dataset can be merged into the third dataset, and the merged third dataset can be used to fine-tune the model to be trained using LoRA to obtain the third low-rank adaptive model ΔW3.

[0091] By merging the second dataset into the third dataset, the fine-tuned third low-rank adaptation model ΔW3 can be made to have similar responses to the same speech data as the second low-rank adaptation model ΔW2, thereby improving the stability of subsequent low-rank adaptation model merging.

[0092] In some embodiments of this application, a gradient penalty constraint strategy is provided:

[0093] In this embodiment, to reduce the risk of merging low-rank adaptation models and ensure their stability after merging, one possible approach is to fine-tune the proportional relationship between the input and output distances of each low-rank adaptation model after fine-tuning for different data from the same distribution, i.e., satisfying the Lipschitz condition:

[0094]

[0095] Where f() represents the function representation of the low-rank adaptive model, and K is a small value, such as 0.01 or other values.

[0096] The above formula means that similar samples should produce similar outputs. Since gradient is the derivative, the above equation can be transformed into:

[0097]

[0098] Therefore, in the process of fine-tuning the LoRA model using the first, second, and third datasets respectively, the loss function can be increased with a gradient penalty term, which is used to constrain the norm of the gradient vector of the LoRA parameters to approach a small amount K.

[0099] The LoRA parameters that need to be updated during model fine-tuning are the low-rank adaptive weights ΔW = B × A, i.e., the low-rank matrices B and A. Therefore, the gradients of B and A are constrained to be less than or equal to K when training and updating the weights.

[0100] In this embodiment, by adding gradient penalty constraints to the fine-tuning training process of the model to be trained, the distances between the fine-tuned models and the input and output distances of different data from the same distribution can satisfy the Lipschitz condition, thereby improving the stability of the merged low-rank adaptive models after fine-tuning.

[0101] In some embodiments of this application, a weighting strategy for the low-rank adaptive model merging process is further provided, mainly for the interpretation of the second dimension.

[0102] Specifically, based on the properties of the matrix spectral norm, a larger spectral norm has a greater impact on the final result. In the training of the low-rank adaptation model, the larger the training steps, the larger the spectral norm of ΔW. Since the amount of synthesized speech data in the third dataset is much larger than the amount of real speech data in the first dataset, to suppress the influence of the third low-rank adaptation model trained on the third dataset, it can be scaled. Therefore, the calculation formula for the merged low-rank adaptation model can be adjusted as follows:

[0103] ΔWnew=ΔW1 +u×(ΔW3-ΔW2)

[0104] Where u is a value greater than 0 and less than 1. By suppressing the influence of the third low-rank adaptation model in the merged low-rank adaptation model, the influence of the low-rank adaptation models trained on high-resource synthetic audio data and real speech data in the merged low-rank adaptation model can be balanced.

[0105] Corresponding to the calculation formula of the merged low-rank adaptive model mentioned above, the process of merging low-rank adaptive models to obtain the merged low-rank adaptive model includes:

[0106] Calculate the difference between the third low-rank adaptive model ΔW3 and the second low-rank adaptive model ΔW2;

[0107] The first low-rank adaptive model ΔW1 and the difference are weighted and summed to obtain the merged low-rank adaptive model ΔWnew, wherein the weight of the first low-rank adaptive model ΔW3 is greater than the weight of the difference.

[0108] In some embodiments of this application, based on the target language speech recognition model provided by the low-resource language speech recognition model training method provided in any of the foregoing embodiments, this embodiment further provides a low-resource language speech recognition method, including:

[0109] Obtain the speech to be recognized in the target language.

[0110] The target language is a low-resource language.

[0111] The speech recognition model for the target language, trained using any of the foregoing embodiments, is used to process the speech to be recognized, resulting in a speech recognition result.

[0112] As can be seen from the foregoing embodiments, the method of this application can fully utilize synthetic data to optimize the speech recognition model of the target language without affecting the effect of real data, thereby improving the recognition effect of the speech recognition model of low-resource languages.

[0113] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, tablets, learning machines, translators, large-screen teaching displays, wearable devices, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0114] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2 or a program loaded from a storage device 8 into a random access memory (RAM) 3, to implement the low-resource language speech recognition model training method or the low-resource language speech recognition method of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 3 also stores various programs and data required for the operation of the electronic device. The processing unit 1, ROM 2, and RAM 3 are interconnected via a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.

[0115] Typically, the following devices can be connected to I / O interface 5: input devices 6 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 7 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 8 including, for example, memory cards, hard drives, etc.; and communication devices 9. Communication device 9 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0116] This application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device enables the electronic device to implement any of the low-resource language speech recognition model training methods or low-resource language speech recognition methods provided in this application.

[0117] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the low-resource language speech recognition model training methods or low-resource language speech recognition methods provided in this application.

[0118] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0120] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0121] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0122] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A method for training a low-resource language speech recognition model, characterized in that, include: First, second, and third datasets for the target language are obtained. The first dataset includes real speech-text pairs, and the second and third datasets each include synthesized speech-text pairs. The text in the second dataset is the same as the text in the first dataset, and the text in the third dataset is the collected text corpus of the target language, and the number of texts is greater than the number of texts in the first dataset. A training model consisting of an initial low-rank adaptation model and a pre-trained speech recognition model is obtained. The training model is then fine-tuned using LoRA with the first, second, and third datasets to obtain a first low-rank adaptation model corresponding to the first dataset, a second low-rank adaptation model corresponding to the second dataset, and a third low-rank adaptation model corresponding to the third dataset. The first low-rank adaptation model, the second low-rank adaptation model, and the third low-rank adaptation model are merged to obtain a merged low-rank adaptation model, which is the difference between the sum of the first low-rank adaptation model and the third low-rank adaptation model and the second low-rank adaptation model. The merged low-rank adaptation model is combined with the pre-trained speech recognition model to obtain the speech recognition model for the target language.

2. The method according to claim 1, characterized in that, The process of obtaining the first, second, and third datasets for the target language includes: Obtain a first dataset, which includes real speech-text pairs in the target language; Speech synthesis is performed on the text in the first dataset to obtain the synthesized speech corresponding to the text. The synthesized speech and the corresponding text constitute the second dataset. A third dataset is formed by acquiring text corpus in the target language, performing speech synthesis on the text corpus to obtain synthesized speech, and the text corpus and the corresponding synthesized speech. The number of text corpus ...

3. The method according to claim 1, characterized in that, The process of fine-tuning the model to be trained using LoRA with a third dataset to obtain the third low-rank adaptive model includes: The second dataset is merged into the third dataset, and the merged third dataset is used to fine-tune the model to be trained using LoRA to obtain the third low-rank adaptation model.

4. The method according to claim 1, characterized in that, In the process of fine-tuning the model to be trained using the first, second and third datasets respectively, the loss function includes a gradient penalty term, which is used to constrain the norm of the gradient vector of the LoRA parameters to approach a set small amount.

5. The method according to claim 1, characterized in that, The process of merging the first low-rank adaptive model, the second low-rank adaptive model, and the third low-rank adaptive model to obtain the merged low-rank adaptive model includes: Calculate the difference between the third low-rank adaptive model and the second low-rank adaptive model; The first low-rank adaptive model and the difference are weighted and summed to obtain the merged low-rank adaptive model, wherein the weight of the first low-rank adaptive model is greater than the weight of the difference.

6. The method according to any one of claims 1-5, characterized in that, The initial low-rank adaptation model is obtained in the following way: The randomly initialized low-rank adaptive model is used as the initial low-rank adaptive model; or, For a randomly initialized low-rank adaptive model, the first, second, or third dataset is used to fine-tune the randomly initialized low-rank adaptive model by setting a small number of steps, which serves as the initial low-rank adaptive model.

7. The method according to any one of claims 1-5, characterized in that, The pre-trained speech recognition model adopts a Large Language Model (LLM) architecture, including: Audio encoder, mapping module, and large language model LLM; The audio encoder is used to extract audio features from the input audio, and the mapping module is used to map the audio features to the input space of the large language model, and use the mapped features as the input of the large language model to obtain the recognized text through decoding by the large language model.

8. A low-resource language speech recognition method, characterized in that, include: Obtain the speech to be recognized in the target language; The speech to be recognized is processed by the configured speech recognition model of the target language to obtain the speech recognition result; wherein the speech recognition model is a model trained by the low-resource language speech recognition model training method of any one of claims 1 to 7.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the low-resource language speech recognition model training method as described in any one of claims 1 to 7, or to implement each step of the low-resource language speech recognition method as described in claim 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the low-resource language speech recognition model training method as described in any one of claims 1 to 7, or implements the steps of the low-resource language speech recognition method as described in claim 8.

Images