Low-resource Language Speech Recognition System with Multi-source Corpus Fusion and Its Training Method

Abstract

The present invention relates to the technical field of low-resource language speech recognition systems, and specifically to a low-resource language speech recognition system that integrates multi-source corpora and its training method, including a multi-source language framework that uses labeled speech data of high-resource and low-resource languages to train a model for recognizing the speech of the low-resource language. The multi-source language framework includes three parts: an acoustic layer, a language layer, and a pronunciation layer. Through the innovative multi-source language framework and advanced feature fusion technology, the above-mentioned technical difficulties are successfully overcome. It can not only effectively utilize the knowledge of high-resource languages to enhance the recognition effect of low-resource languages, but also accurately capture the commonalities and differences between different languages. In addition, the present invention performs well in processing multilingual mixed environments, greatly improving the adaptability and generalization ability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of low-resource language speech recognition systems, and in particular to a low-resource language speech recognition system integrating multi-source corpora and a training method thereof. Background Art

[0002] In today's globalized language landscape, speech recognition technology for low-resource languages is gaining increasing attention. However, the development of this technology continues to face numerous challenges. Traditional speech recognition systems typically rely on large amounts of labeled data for training, which is a significant obstacle for languages with smaller populations or limited digital resources. Despite researchers' tireless efforts in this field in recent years, existing technologies still have numerous shortcomings.

[0003] Currently, the most recent state-of-the-art approaches to speech recognition in low-resource languages primarily employ two approaches. The first is the single-language model approach, which uses only limited target language data for training. This approach is straightforward, but due to insufficient training data, it often struggles to build a high-performing recognition model. This is especially true for endangered languages with extremely small populations, where collecting sufficient training data is nearly impossible.

[0004] The second approach is a simple multilingual model, which mixes data from multiple languages for training. This approach alleviates the data scarcity problem to some extent, but it ignores the differences between languages and fails to effectively leverage the relationships between them to improve recognition of low-resource languages. Moreover, this simple mixing often leads to reduced model performance when dealing with languages with very different phonetic characteristics.

[0005] The main technical issues with these existing methods can be summarized as follows: First, they are unable to effectively leverage knowledge from high-resource languages to assist in the recognition of low-resource languages. Second, they lack the ability to fine-tune the acoustic and linguistic features between different languages. Third, they perform poorly in mixed multilingual environments, struggling to accurately distinguish and recognize different languages. Finally, these methods also suffer from significant deficiencies in computational efficiency and model generalization.

[0006] Faced with these challenges, it is particularly necessary to develop speech recognition systems that can effectively utilize multi-source corpora and accurately recognize low-resource languages. Summary of the Invention

[0007] The present invention addresses these technical challenges. It proposes a multi-source corpus-integrated, low-resource speech recognition system and its training method to address this urgent need. It aims to address key issues in existing technologies, such as low data utilization efficiency, insufficient feature extraction, and weak cross-language knowledge transfer capabilities.

[0008] The present invention provides a low-resource language speech recognition system using multi-source corpus fusion, including a multi-source language framework, using labeled high-resource language and low-resource language speech data to train a model for recognizing the low-resource language speech, the multi-source language framework includes three parts: an acoustic layer, a language layer, and a pronunciation layer, wherein:

[0009] The acoustic layer maps each acoustic feature to a common vocal tract coding subspace representing each language-independent feature and a vocal tract residual space representing language-specific information through ASVD. The former is used to predict the phoneme probability distribution of each frame, and the latter is used to predict the phoneme probability distribution of each frame for each language. The decoding module propagates the probability distribution in each dimension through the speech network, language network, and pronunciation network, and outputs the final prediction result.

[0010] The acoustic model uses ASVD to divide the training data into two parts: a common channel encoder and a channel residual encoder. The former uses one ASVD component for acoustic training, while the latter uses the remaining ASVD components for training, thereby gaining the ability to extract multiple acoustic information.

[0011] Language model, used to model language uncertainty and improve the performance of language perception tasks;

[0012] The pronunciation model aligns each phoneme with a different language marker, increasing the model's language awareness while avoiding confusion between phonemes and languages.

[0013] Feature fusion solutions, including multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and feature fusion technologies enhanced by graph neural networks for cross-language knowledge propagation.

[0014] Specifically, in the multi-scale contrastive learning, for the vocal tract coding features of speech, multi-scale windows are used in the ASVD acoustic features to constrain the common vocal tract encoder and the vocal tract residual encoder to obtain vocal tract coding characteristics of multiple scales in multiple languages; in the hierarchical attention alignment mechanism, the global and local alignment of multiple languages is automatically learned based on the pronunciation module and decoding model, and a hierarchical attention mechanism is used to align the vocal tract coding features between different languages.

[0015] Specifically, in the feature fusion scheme, the technology of adversarial feature decoupling and recombining is to use adversarial learning and nonlinear decoder to separate and recombine the speech layer and language layer features of the common acoustic channel coding through decoupling learning in the common channel coding space; and the feature fusion technology of graph neural network enhanced cross-language knowledge propagation is to use graph neural network based on regularization and orthogonal constraints to propagate and fuse language-specific channel residual features in the decoding of each layer.

[0016] Specifically, the self-supervised training strategy of the multi-source speech recognition encoder is as follows: for a low-resource language ι1 and a high-resource language ι2, the training data ν of each speech is divided into a common channel encoder V by ASVD l:c and channel residual encoder V l:r Two parts; then simultaneously train the common channel encoder parameters V l:c ~p(V l:c ) and the channel residual encoder parameters V l:r ~p(V l:r ) and a common model d(V l:c θ l:c ) and a language model dv(V l :r;θ v ); Different languages ι correspond to different channel residual encoder parameters V during training l:r and language model parameters θ v ; Among them, the two languages ι1 and ι2 can simultaneously train the common channel encoder and the channel residual encoder, and realize effective fusion of information through the feature fusion technology.

[0017] The training method of the low-resource language speech recognition system using multi-source corpus fusion comprises the following steps:

[0018] Step 1) Based on the multi-source language framework, we design a feature fusion scheme that combines multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and graph neural network-enhanced cross-lingual knowledge propagation. We use this scheme to train an encoder for a low-resource language l, where X is a high-resource language dataset and Yl is a low-resource language dataset.

[0019] Step 1-1) Design a multi-scale contrastive learning framework to extract acoustic information from the channel coding layer, wherein a fixed-size sliding window is used to sample from a fixed-size input channel coding; then, the corresponding channel coding is divided into multiple segments, and the common channel encoder and the language channel encoder respectively use these coding segments to predict the implicit semantic information of each channel coding of each frame;

[0020] Steps 1-2) Use a hierarchical attention alignment mechanism to learn multilingual alignment from both global and local perspectives. Design a self-supervised MAB algorithm to find the global and local alignment matrices. Use the MAB algorithm to learn the alignment matrix.

[0021] Steps 1-3) Adaptively split and reassemble vocal tract coding features using adversarial learning and decoders in multilingual acoustic features;

[0022] Steps 1-4), cross-language knowledge dissemination feature fusion technology using graph neural network enhanced cross-language knowledge dissemination;

[0023] Step 2) For the trained encoder, obtain the output of the decoder according to its characteristics.

[0024] Specifically, the design of multi-scale contrastive learning in step 1-1) includes first using ASVD to divide each language in the training dataset V into a common channel encoder V lcc and the vocal channel residual encoder V for each language l：r :

[0025] V l：c =V l -V l：r , l∈{l1,l2}

[0026] In addition, the MAB approach is used to learn the decoding matrix A for each frame of each language l and the common channel encoding and decoding matrix A cl Used to decode common channel coded semantic vectors:

[0027] z l =A l ·V l：c

[0028] z cl =A cl ·V l：c

[0029] Decompose each channel encoding vector into a common channel encoding phoneme embedding and a channel residual phoneme embedding:

[0030] e l =z l +z cl

[0031] The matching loss is used to calculate the loss between the residual representation of the common channel encoding and the residual representation of the residual channel encoding; the contrast loss is used to calculate the matching loss between the phoneme embedding of the common channel encoding and the phoneme embedding of the channel residual, where V is the training dataset, ι is the language identifier, ι1 and ι2 represent low-resource language and high-resource language respectively, and V l：c For common channel encoding, V l : r is the channel residual coding, A l is the decoding matrix for each language and each frame, A cl is the common channel encoding and decoding matrix, z l Encode semantic vectors for language-specific vocal tracts, z d Encode semantic vector for common channel, e l is the final channel encoding vector.

[0032] Specifically, the steps 1-2) are to make each language and decoder learn language context features, construct two hierarchical global and local attention alignment matrices in the common channel coding space, and make the decoder learn the attention of multiple languages; find the two language and decoding hierarchical matrices A={A l , 0, ..., A l , N} and {A l，c , 0, ..., A l，c , N}, through the MAB algorithm, learn the alignment matrix; where A is the alignment matrix set, A l , i (i = 0, ..., N) is the i-th layer local attention alignment matrix of the l-th language, A l , c,i (i=0,...,N) is the i-th layer global attention alignment matrix of the l-th language, N is the total number of layers, and the MAB algorithm is used to learn these alignment matrices.

[0033] Specifically, steps 1-3) include adversarial learning, allowing the learned decoder to generate channel residual embedding, and calculating the final channel residual embedding through the decoder model; the generative adversarial network consists of three parts: a reconstructor, a generator, and a discriminator. The generator reconstructs the channel residual embedding from the common encoder embedding, and the latter receives the channel residual embedding generated by the reconstructor and the common channel encoding, and determines whether the common encoding comes from the reconstructor or the discriminator; in the adversarial loss, the reconstructor is trained by minimizing the adversarial loss between the reconstructor and the generator, and the generator is trained by minimizing the adversarial loss between the generator and the discriminator; the channel residual embedding uses a function to decouple learning; and a separation function is used to decouple learning of the channel residual embedding.

[0034] Specifically, for the channel residual embedding, decoupling learning is performed through the function f, which uses the MAB method to learn the alignment matrix to separate language features; the contrast loss is used to constrain the embedding loss between the language decoder and the channel residual embedding frame; decoupling learning is implemented using the contrast loss function Lcontrast:

[0035]

[0036] Among them, f is the decoupling function, e1 and e I′ is the vocal tract residual embedding of different enhanced versions of the same speech, e k is the vocal tract residual embedding of other samples in the batch, sim() is the cosine similarity function, and τ is the temperature parameter.

[0037] Specifically, the enhanced cross-language knowledge propagation feature fusion technology in steps 1-4) includes a graph neural network enhanced cross-language knowledge propagation feature fusion technology. After extracting the common channel code of each language, the graph neural network is designed using the common channel code set of each decoupling layer for all languages, and the common channel code matrix C is trained using SVD and ASVD. cl And each channel coding matrix C lr , and finally use the encoding matrix C after matrix decomposition lc and C cr Finally, further learning for each language is performed using another matching loss and contrastive cosine loss at each layer:

[0038]

[0039] Among them, C cl is the common channel coding matrix, C lr is the language-specific vocal tract encoding matrix, C lc and C cr is the encoding matrix after matrix decomposition, ||·||F represents the Frobenius norm, cos() is the cosine similarity function, and λ is the weight parameter;

[0040] The step 2 also includes using a reconstruction model based on a decoding network to reconstruct the speech segment, and solving the open set problem by minimizing the ASVD reconstruction loss, specifically using an ASVD encoder encoding step; for the speech input D* in the test set, the trained model is recognized to generate a recognition result of the target low-resource language based on multi-source learning.

[0041] Advantages and beneficial effects of the present invention:

[0042] The multi-source corpus-integrated low-resource language speech recognition system proposed in this paper successfully overcomes the aforementioned technical challenges through an innovative multi-source language framework and advanced feature fusion technology. It not only effectively leverages knowledge from high-resource languages to enhance recognition of low-resource languages, but also accurately captures the commonalities and differences between different languages. Furthermore, the present invention performs exceptionally well in mixed multilingual environments, significantly improving the system's adaptability and generalization capabilities.

[0043] The implementation of the present invention brings about many beneficial effects. First, it significantly improves the speech recognition accuracy of low-resource languages, making it possible to effectively process some languages that were previously difficult to reliably recognize. Second, the cross-language knowledge transfer capability of the present invention provides a new tool for linguistic research, helping to better understand the relationship between different languages. Furthermore, it provides strong technical support for the protection and research of endangered languages, which is conducive to maintaining linguistic diversity. Finally, the efficient computing method of the present invention enables it to operate in resource-constrained environments, paving the way for the application of language technology in remote areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall logic block diagram of the system of the present invention.

[0045] Figure 2 It is a logic flow chart of the training method of the present invention.

[0046] Figure 3 It is a logic flow chart of the multi-source language framework of the present invention.

[0047] Figure 4 This is a logic flow chart of the feature fusion solution of the present invention. DETAILED DESCRIPTION

[0048] To further illustrate the technical means and effects employed by the present invention to achieve its intended objectives, the following detailed description, along with the accompanying drawings and preferred embodiments, includes a detailed description of the specific implementations, structures, features, and effects thereof. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0050] See Figure 1-4 This paper proposes a low-resource language speech recognition system that utilizes multi-source corpus fusion. The core of this system is a multi-source language framework 1, whose primary purpose is to use annotated speech data from both high-resource and low-resource languages to train a model capable of recognizing speech in the low-resource language. This framework specifically comprises three key components: an acoustic layer 11, a language layer 12, and a pronunciation layer 13.

[0051] First, the acoustic layer 11 uses ASVD to map each acoustic feature into two key spaces: a common vocal tract encoding subspace representing each language-independent feature, and a vocal tract residual space representing language-specific information. The former is used to predict the phoneme probability distribution for each frame, while the latter is used to predict the phoneme probability distribution for each language in each frame. This decomposition approach enables the system to better capture the commonalities and differences between different languages.

[0052] Next, the decoding module 14 propagates the probability distribution across each dimension through the speech network, language network, and pronunciation network, ultimately outputting a prediction result. This multi-network collaborative approach comprehensively considers all aspects of speech, language, and pronunciation, thereby improving recognition accuracy.

[0053] Third, the acoustic model uses ASVD to split the training data into two parts: a common channel encoder and a channel residual encoder. The former uses one ASVD component for acoustic training, while the latter uses the remaining ASVD components for training, thereby gaining the ability to extract diverse acoustic information. This separation allows the system to more effectively utilize the large amount of data from high-resource languages to assist model training for low-resource languages.

[0054] The language model is then used to model the uncertainty of the language and improve the performance of language perception tasks. This is particularly important for processing multilingual speech environments, as it can help the system more accurately determine the language of the speech.

[0055] Next, the pronunciation model aligns each phoneme with a different language marker, increasing the model's language awareness while avoiding confusion between phonemes and languages. This step is a critical bridge between the acoustic and linguistic layers, effectively processing phonemes that are similar but not identical across languages.

[0056] Finally, the feature fusion solution includes multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and cross-lingual knowledge propagation enhanced by graph neural networks. These advanced technologies work together to significantly improve the system's performance when processing multilingual data.

[0057] First, it's important to clarify that the acoustic layer 11, language layer 12, and pronunciation layer 13 are components of the multi-source language framework 1, and the acoustic model, language model, and pronunciation model are the specific models used in these layers. The acoustic layer 11 is the part of the framework responsible for processing acoustic features. It uses ASVD (Adaptive Singular Value Decomposition) to map acoustic features into the common vocal tract coding subspace and the vocal tract residual space. The acoustic model, the specific model used in this layer, is responsible for converting acoustic features into the probability distribution of phonemes or other acoustic units. The features extracted by the acoustic layer 11 through ASVD are input into the acoustic model, which then outputs a phoneme-level probability distribution. This information is passed to the language layer 12 and pronunciation layer 13. The language layer 12 is responsible for modeling language uncertainty to improve the performance of language perception tasks. The language model, the specific model used in this layer, predicts the probability distribution of word sequences, helping the system understand the structure and grammar of the language. The language layer 12 receives the phoneme probability distribution from the acoustic layer 11 and uses the language model to evaluate the likelihood of different word sequences. This information is combined with the output of the pronunciation layer 13 to influence the final decoding results. The pronunciation layer 13 is responsible for aligning each phoneme with different linguistic markers. The pronunciation model is a specific model used in this layer. It describes the standard pronunciation of a word, that is, how a word is broken down into a phoneme sequence. The pronunciation layer 13 uses the pronunciation model to combine the phoneme probabilities output by the acoustic layer 11 with the word sequence likelihood provided by the language layer 12 to determine the most likely word-phoneme correspondence.

[0058] The connection and collaboration between these three layers is achieved through the decoding module 14. The decoding module 14 integrates information from the acoustic layer 11, the language layer 12, and the pronunciation layer 13, propagates the probability distribution in each dimension through the speech network, the language network, and the pronunciation network, and finally outputs the recognition result.

[0059] In a multi-language environment, the advantages of this layered structure are: the acoustic layer 11 can capture common acoustic features across different languages, facilitating cross-lingual knowledge transfer. The language layer 12 can handle mixed multilingual environments and effectively distinguish the grammatical structures of different languages. The pronunciation layer 13 can address pronunciation differences between languages, improving the system's adaptability to low-resource languages.

[0060] In this way, the multi-source language framework can effectively leverage knowledge from high-resource languages to enhance recognition of low-resource languages, while maintaining sensitivity to the idiosyncrasies of various languages. This design allows the system to fully leverage relevant information from high-resource languages when processing low-resource languages, significantly improving recognition performance.

[0061] Preferably, in one embodiment of the present invention, when the system processes speech data in Chinese and Uyghur, the common vocal tract coding subspace may capture the phoneme characteristics shared by both languages, while the vocal tract residual space retains the tonal information of Chinese and the unique vowel characteristics of Uyghur, respectively. This approach enables the system to maximize knowledge sharing between different languages while maintaining language specificity.

[0062] Preferably, in one embodiment of the present invention, the present invention further refines the specific implementation of multi-scale contrastive learning 21 and the hierarchical attention alignment mechanism. First, multi-scale contrastive learning targets the vocal tract coding characteristics of speech. In the ASVD acoustic feature, multi-scale windows are used to constrain the common vocal tract encoder and the vocal tract residual encoder to obtain vocal tract coding characteristics at multiple scales for multiple languages. This approach allows the system to capture acoustic features at different time scales, thereby better adapting to the acoustic characteristics of different languages.

[0063] Secondly, a hierarchical attention alignment mechanism automatically learns global and local alignment across multiple languages based on the pronunciation module and decoding model, using a hierarchical attention mechanism to align vocal tract encoding features across different languages. This mechanism is particularly helpful for language pairs with significant differences in grammatical structure, aligning features at different levels and thus improving cross-lingual learning.

[0064] Preferably, in one embodiment of the present invention, when processing language pairs with significantly different grammatical structures, such as Chinese and Tibetan, multi-scale contrastive learning may use a smaller time window (e.g., 10 milliseconds) to capture rapidly changing phonemes, while using a larger time window (e.g., 50 milliseconds) to capture more stable phonemes. This flexible approach enables the system to better adapt to the different acoustic characteristics of the two languages.

[0065] Preferably, in one embodiment of the present invention, the present invention further describes in detail two key technologies in the feature fusion solution: adversarial feature decoupling and reconstruction technology and graph neural network enhanced cross-language knowledge propagation technology.

[0066] First, adversarial feature decoupling and recombining technology uses adversarial learning and a nonlinear decoder to separate and recombine the speech-layer and language-layer features encoded in the common acoustic channel encoding space through decoupled learning. This method can effectively separate language-independent acoustic features from language-specific features, thereby improving the system's performance in cross-lingual tasks.

[0067] Secondly, the cross-lingual knowledge propagation technique enhanced by graph neural networks uses graph neural networks based on regularization and orthogonality constraints to propagate and fuse language-specific vocal tract residual features in each layer's decoding. This approach effectively transfers knowledge between different languages, especially when processing languages from closely related language families.

[0068] Preferably, in one embodiment of the present invention, when processing the three East Asian languages of Chinese, Japanese and Korean, the cross-language knowledge propagation technology enhanced by graph neural networks can better capture the common features and subtle differences between them, thereby improving the recognition performance of each language.

[0069] Preferably, in one embodiment of the present invention, a self-supervised training strategy for a multi-source speech recognition encoder is described in detail. Specifically, for a low-resource language ι1 (such as Qiang) and a high-resource language ι2 (such as Chinese), the system divides the training data V of each speech into a common channel encoder V and a common channel encoder V through ASVD. l：c and channel residual encoder V l：r Two parts.

[0070] Then, the system simultaneously trains the common channel encoder parameters V l：c ~p(V l：c ) and the channel residual encoder parameters V l：r ~p(V l：r ) and a common model d(V l：c θ l：c ) and a language model dv(V l :r;θ v ). During training, different languages ι correspond to different channel residual encoder parameters V l：r and language model parameters θ v Of particular note, the common vocal tract encoder and the vocal tract residual encoder can be trained simultaneously for the two languages ι1 and ι2, and information can be effectively integrated through feature fusion techniques. This parallel training strategy not only improves training efficiency but also maximizes knowledge sharing between different languages while maintaining language specificity.

[0071] Preferably, in one embodiment of the present invention, when the system processes Qiang (a low-resource language) and Chinese (a high-resource language) simultaneously, the system can leverage the extensive Chinese training data to improve the Qiang acoustic model while preserving Qiang's unique phonetic features. This approach significantly improves Qiang recognition accuracy without compromising the system's performance for Chinese.

[0072] Preferably, in one embodiment of the present invention, a method for training a low-resource language speech recognition system using source corpus fusion is proposed. The method comprises the following steps:

[0073] First, based on a multi-source language framework, we design a feature fusion scheme that combines multi-scale contrastive learning, a hierarchical attention alignment mechanism, adversarial feature decoupling and reorganization, and graph neural network-enhanced cross-lingual knowledge propagation. This scheme is then used to train an encoder for a low-resource language l, where X is a high-resource language dataset and Yl is a low-resource language dataset.

[0074] Next, a multi-scale contrastive learning framework is designed to extract acoustic information from the channel code layer. Specifically, a fixed-size sliding window is used to sample fixed-size input channel codes. The corresponding channel codes are then divided into multiple segments, and the common channel encoder and language channel encoder use these segments to predict the implicit semantic information of each channel code in each frame.

[0075] Next, a hierarchical attention alignment mechanism is used to learn multilingual alignment from both a global and local perspective. In this step, a self-supervised MAB algorithm is designed to find global and local alignment matrices and then used to learn the alignment matrix. This approach captures the correspondence between languages at different levels, thereby improving the effectiveness of cross-lingual learning.

[0076] Next, the system uses adversarial learning and a decoder to adaptively split and reassemble vocal tract encoding features within multilingual acoustic features. This step helps the system better separate language-independent acoustic features from language-specific features, thereby improving performance in cross-lingual tasks.

[0077] Finally, the system uses graph neural network-enhanced cross-lingual knowledge propagation feature fusion technology, which effectively transfers knowledge between different languages, especially when dealing with languages from similar language families.

[0078] After completing the above steps, for the trained encoder, the system obtains the output of the decoder based on its characteristics.

[0079] Preferably, in one embodiment of the present invention, when processing Yi (a low-resource language) and Chinese (a high-resource language), the training method can fully utilize the large amount of Chinese data to enhance the Yi speech recognition model. For example, in the multi-scale contrastive learning step, the system may use time windows of different sizes to capture the tonal characteristics of Yi and the syllable structure of Chinese, respectively, thereby improving the recognition accuracy of both languages.

[0080] Preferably, in one embodiment of the present invention, the design process of multi-scale contrastive learning is further described in detail. First, the system uses ASVD to divide each language in the training data set V into a common channel encoder V l:c and the vocal channel residual encoder V for each language l:r. This can be expressed by the following formula:

[0081] V l：c =V l -V l：r , l∈{l1,l2}

[0082] Among them, ι1 represents low-resource languages and ι2 represents high-resource languages.

[0083] Then, the system uses MAB (Multi-Head Attention Block) to learn the decoding matrix A for each frame of each language l and the common channel encoding and decoding matrix A cl , used to decode the common channel encoded semantic vector. This process can be expressed by the following formula:

[0084] Decode common channel coded semantic vectors:

[0085] z l =A l ·V l：c

[0086] z cl =A cl ·V l：c

[0087] Next, the system decomposes each channel encoding vector into a common channel encoding phoneme embedding and a channel residual phoneme embedding:

[0088] e l =z l +z cl

[0089] Where V is the training dataset, ι is the language identifier, ι1 and ι2 represent low-resource language and high-resource language respectively, V l：c For common channel encoding, V l : r is the channel residual coding, A l is the decoding matrix for each language and each frame, A cl is the common channel encoding and decoding matrix, z l Encode semantic vectors for language-specific vocal tracts, z d Encode semantic vector for common channel, e l Finally, the system uses matching loss to calculate the loss between the residual representation of the common channel encoding and the residual representation of the residual channel encoding, and uses contrastive loss to calculate the matching loss between the phoneme embedding of the common channel encoding and the channel residual phoneme embedding.

[0090] Preferably, in one embodiment of the present invention, when processing Zhuang (a low-resource language) and Chinese (a high-resource language), this multi-scale contrastive learning method can effectively capture both languages' common features (such as certain consonants) and their unique features (such as Zhuang's tone patterns). This method not only improves Zhuang recognition accuracy but also enhances the system's robustness when processing mixed speech.

[0091] Preferably, one embodiment of the present invention describes a method for implementing a hierarchical attention alignment mechanism. First, the system enables each language and decoder to learn language context features. Then, in the common channel coding space, hierarchical global and local attention alignment matrices are constructed, enabling the decoder to learn attention across multiple languages.

[0092] Then, the system finds two languages and the decoding level matrix A={A l , 0, ..., A l , N} and {A l，c , 0, ..., A l，c , N}. Finally, the alignment matrix is learned through the MAB algorithm.

[0093] In this process A l , i (i = 0, ..., N) represents the i-th layer local attention alignment matrix of the l-th language, A l , c,i (i=0,...,N) represents the i-th layer global attention alignment matrix of the l-th language, and N is the total number of layers.

[0094] Preferably, in one embodiment of the present invention, when processing parallel corpora of Tibetan and Chinese, the underlying A l , may focus on phoneme-level alignment, while high-level A l , N may focus on semantic alignment at the sentence level. This hierarchical structure enables the system to capture the correspondence between languages at different levels, thereby improving the effect of cross-language learning, especially when dealing with language pairs with large differences in grammatical structure.

[0095] In one embodiment of the present invention, adversarial learning is implemented as follows: First, the system uses the learned decoder to generate channel residual embeddings, and then uses the decoder model to calculate the final channel residual embeddings. In this process, the generative adversarial network consists of three key components: a reconstructor, a generator, and a discriminator.

[0096] Specifically, the generator reconstructs the channel residual embedding from the common encoder embedding. The discriminator receives the channel residual embedding and the common channel encoding generated by the reconstructor and determines whether the common encoding comes from the reconstructor or the discriminator. In the calculation of the adversarial loss, the system trains the reconstructor by minimizing the adversarial loss between the reconstructor and the generator, and trains the generator by minimizing the adversarial loss between the generator and the discriminator.

[0097] In addition, the system uses a separation function to decouple the learning of the vocal tract residual embedding using a specific function. This method can effectively separate language-independent acoustic features from language-specific features.

[0098] Preferably, in one embodiment of the present invention, when processing two closely related languages, Dai and Thai, this adversarial learning approach can help the system better distinguish subtle differences between them. For example, the system may learn subtle differences in tonal patterns between Dai and Thai, thereby improving recognition accuracy for both languages.

[0099] Preferably, one embodiment of the present invention further refines the decoupled learning process for channel residual embedding. First, the system performs decoupled learning using the function f, which uses a MAB approach to learn an alignment matrix to separate language features. Then, the system uses a contrastive loss to constrain the embedding loss between the language decoder and the channel residual embedding frame.

[0100] Finally, decoupling learning is implemented using the contrast loss function Lcontrast, which can be expressed as:

[0101]

[0102] Among them, f is the decoupling function, e1 and e l′ is the vocal tract residual embedding of different enhanced versions of the same speech, e k is the vocal tract residual embedding of other samples in the batch, sim() is the cosine similarity function, and τ is the temperature parameter.

[0103] Preferably, in one embodiment of the present invention, when processing the three minority languages of Miao, Dong, and Bouyei, this decoupled learning approach can help the system capture both common features and subtle differences between them. For example, the system might learn the similarities in the tonal systems of the three languages while also identifying the differences in their consonant systems, thereby improving recognition performance for each language.

[0104] The cross-language knowledge propagation feature fusion technology enhanced by graph neural networks first extracts the common vocal tract encoding for each language. The system then designs a graph neural network using the common vocal tract encoding set of each decoupled layer for all languages. The system then uses SVD and ASVD to train the common vocal tract encoding matrix Ccl and the vocal tract encoding matrix Clr for each language.

[0105] Next, the system uses the encoding matrix C after matrix decomposition lc and C cr Finally, another matching loss and contrastive cosine loss are used at each layer to further learn each language, which can be expressed by the following formula:

[0106]

[0107] Among them, C cl is the common channel coding matrix, C lr is the language-specific vocal tract encoding matrix, C cl and C cr is the encoding matrix after matrix decomposition, ||·||F represents the Frobenius norm, cos() is the cosine similarity function, and λ is the weight parameter;

[0108] Preferably, in one embodiment of the present invention, when processing three Turkic languages—Uyghur, Kazakh, and Kyrgyz—this graph neural network-enhanced cross-language knowledge transfer technology can effectively capture the linguistic connections between them. For example, the system may learn similarities in word order, grammatical structure, and phonological systems among these languages, thereby improving the recognition accuracy of each language.

[0109] In the specific implementation of step 2, the system first uses a reconstruction model based on a decoding network to reconstruct the speech segment. This method solves the open set problem by minimizing the ASVD reconstruction loss, specifically using the ASVD encoder encoding step.

[0110] The advantage of this approach is that it can effectively handle unseen speech samples, improving the system's generalization capabilities. By minimizing the ASVD reconstruction loss, the system can learn more robust feature representations, thereby maintaining high recognition accuracy in the face of various accents, dialects, and noisy environments.

[0111] Preferably, in one embodiment of the present invention, when the system processes the Hani language, spoken in Yunnan Province, this ASVD-based reconstruction method can help the system better adapt to the different dialects and accents of the Hani language. For example, the system may learn the pronunciation variations of Hani language in different regions, thereby maintaining a high recognition accuracy rate when dealing with speakers from different villages. This is of great significance for the protection and research of endangered languages like Hani.

[0112] Specific operations for the test set in step 2: First, the system uses the trained model to recognize the speech input D* in the test set. Then, the system generates recognition results for the target low-resource language l based on multi-source learning.

[0113] This process fully utilizes the cross-lingual knowledge learned in the previous training phase, enabling the system to better handle recognition tasks in low-resource languages. Through multi-source learning, the system can transfer knowledge learned in high-resource languages to the recognition of low-resource languages, thereby improving recognition performance.

[0114] Preferably, in one embodiment of the present invention, when the system processes the Tibetan dialect of Qinghai Province, it can leverage the extensive data of standard Tibetan and Chinese to improve recognition accuracy. For example, the system might initialize a model for the Qinghai Tibetan dialect using the acoustic and language models of standard Tibetan, and then fine-tune it using limited Qinghai Tibetan dialect data. This approach can significantly improve recognition performance for the Qinghai Tibetan dialect, achieving good results even with very limited training data.

[0115] Furthermore, this approach can help the system better handle code-mixing situations, where speakers mix Tibetan and Chinese in the same sentence. The system can use the knowledge learned from Chinese to assist in identifying Chinese loanwords or phrases in Tibetan, thereby improving overall recognition accuracy.

[0116] The above method is applied to low-resource language speech recognition systems and non-pathological research fields. Specifically, the system integrates self-supervised learning data of high-resource and low-resource languages into a unified acoustic space, thereby training a multilingual model and performing encoder learning and speech recognition.

[0117] The advantage of this approach is that it can fully utilize the large amount of data from high-resource languages to assist in model training for low-resource languages, thereby achieving good recognition results even with limited low-resource language data. Furthermore, by integrating data from different languages into a unified acoustic space, the system can learn more versatile and robust feature representations, which is particularly beneficial for speech recognition tasks in multilingual environments.

[0118] Preferably, in one embodiment of the present invention, this method can be applied to the research and protection of minority languages in southwest China. For example, when studying Yi, Naxi, and Bai, the system can use Chinese as a high-resource language and map the acoustic features of these languages into the same acoustic space. This not only improves the recognition accuracy of these low-resource languages, but also helps linguists better understand the relationships and differences between these languages.

[0119] Furthermore, this method can also be applied to language acquisition research. For example, when studying the pronunciation characteristics of native Chinese speakers learning English, the system can use a large amount of data from both Chinese and English to train a bilingual model, and then use this model to analyze the learner's pronunciation. This can help researchers more accurately identify the learner's pronunciation errors and whether these errors are influenced by their native language (Chinese).

[0120] This approach also has potential applications in non-pathological research. For example, when studying certain speech disorders, such as stuttering, the system can utilize the abundant data of normal speech as high-resource data, while using the limited data of pathological speech as low-resource data. In this way, the system can better capture the subtle differences between normal and pathological speech, providing a more accurate tool for early diagnosis and evaluation of treatment effectiveness.

[0121] In summary, the multi-source corpus-based speech recognition system and training method proposed in this paper successfully address the data scarcity issue in low-resource language speech recognition through an innovative multi-source language framework and advanced feature fusion technology. This system not only improves the recognition accuracy of low-resource languages but also provides new insights into the development of cross-language speech recognition technology. This approach has broad application prospects in linguistic research, minority language protection, cross-cultural communication, and the popularization of speech technology in multilingual regions.

[0122] For example, in education, this system could be used to develop more intelligent and personalized language learning software. For students learning minority languages, this system can provide accurate pronunciation guidance and assessment, even if there aren't many learning resources available. Furthermore, for minority students learning Mandarin or other mainstream languages, the system can provide more targeted learning suggestions based on the characteristics of their native language.

[0123] In terms of cultural preservation, this system can help record and preserve endangered languages. Many minority languages face challenges such as declining speakers and younger generations failing to use them. This efficient speech recognition system can make it easier for researchers to collect and transcribe large amounts of audio materials, such as oral histories and folk tales, providing valuable digital resources for the preservation of these languages and cultures.

[0124] In the healthcare sector, this system can help develop more accurate remote diagnostic tools. In some minority areas, there may be a shortage of medical specialists who speak specific languages. This multilingual speech recognition system allows patients' descriptions to be accurately converted into text, which can then be translated into a language familiar to doctors, overcoming language barriers and providing better medical services.

[0125] In the commercial sector, this system can help companies better tap into multilingual markets. For example, when developing voice assistants or customer service systems for minority regions, even if training data for a particular language is scarce, data from other related languages can be leveraged to improve service quality. This not only helps companies expand their market but also provides a better service experience for minority users.

[0126] In short, this invention holds enormous promise. It not only advances the technology of speech recognition for low-resource languages but also has the potential to positively impact social issues such as language equality, the preservation of cultural diversity, and educational equity. With the further development and application of this technology, we can expect to see the formation of a more diverse and inclusive speech technology ecosystem.

[0127] In order to verify the superiority of the present invention, a comprehensive comparison was conducted between an embodiment and two comparative examples. These tests were carried out around the core innovation of the present invention, fully demonstrating the advantages of the multi-source corpus fusion low-resource language speech recognition system.

[0128] Example 1 employed the complete solution of the present invention, including a multi-source language framework, multi-scale contrastive learning, a hierarchical attention alignment mechanism, adversarial feature decoupling and reorganization, and cross-lingual knowledge propagation enhanced by graph neural networks. Tibetan was selected as the target low-resource language, with Chinese and Nepali as auxiliary high-resource languages.

[0129] Comparative Example 1 uses a traditional monolingual speech recognition model trained with limited Tibetan data, which represents a common approach for processing low-resource languages.

[0130] Comparative Example 2 uses a simple multilingual model, mixing Tibetan, Chinese, and Nepali data for training, but does not use the feature fusion technology proposed in this invention. The following key indicators were selected to evaluate the performance of the system:

[0131] 1. Word Error Rate (WER): This is a standard metric for measuring the accuracy of a speech recognition system.

[0132] 2. Phoneme Error Rate (PER): This can more carefully reflect the system's recognition ability at the phoneme level.

[0133] 3. Cross-lingual Transfer Efficiency (CTE): This is a new metric proposed to measure the system's ability to leverage high-resource language knowledge to improve low-resource language recognition.

[0134] 4. Computational Efficiency (CE): measured by training time and inference speed.

[0135] The test method is as follows: a speech dataset containing 1,000 hours of Tibetan, 5,000 hours of Chinese, and 3,000 hours of Nepali was used. The Tibetan data was randomly divided into a training set (800 hours), a validation set (100 hours), and a test set (100 hours). Three models were trained under the same hardware conditions and evaluated on the Tibetan test set. The following are detailed test results:

[0136] index Example 1 Comparative Example 1 Comparative Example 2 WER 12.3％ 25.7％ 18.9％ PER 8.7％ 19.2％ 14.1％ CTE 0.85 N / A 0.42 CE 1.2x 1.0x 1.1x

[0137] From these results, we can clearly see the superiority of the present invention. Example 1 is significantly better than the two comparative examples in all key indicators. Let's analyze these results one by one:

[0138] First, in terms of word error rate (WER), our approach reduced errors by 52.1% compared to a traditional single-language model (Comparison Example 1) and by 34.9% compared to a simple multilingual model (Comparison Example 2). This means the system is able to more accurately recognize Tibetan speech, which is of great significance for practical applications such as speech translation or voice assistants.

[0139] Secondly, the significant reduction in the phoneme error rate (PER) (54.7% reduction compared to Comparative Example 1 and 38.3% reduction compared to Comparative Example 2) demonstrates that the system has more accurate modeling capabilities at the phoneme level. This not only improves overall recognition accuracy but also provides a more reliable tool for linguistic research. For example, when studying differences in Tibetan dialects, the system is able to capture more subtle pronunciation changes.

[0140] Cross-language transfer efficiency (CTE) is a key innovation of this invention. The CTE for Example 1 reached 0.85, significantly higher than the 0.42 for Comparative Example 2. This demonstrates that the system can more effectively leverage knowledge from Chinese and Nepali to improve Tibetan recognition. This efficient knowledge transfer capability makes the system particularly suitable for processing resource-scarce languages, providing strong technical support for the preservation of linguistic diversity.

[0141] In terms of computational efficiency (CE), although the method slightly increases computational complexity (20% compared to Comparative Example 1), the significant improvement in recognition accuracy makes this extra cost completely worthwhile. Moreover, compared to the simple multilingual model (Comparative Example 2), the method only increases computational complexity by 9%, yet brings significant performance improvements.

[0142] These results fully demonstrate the superiority of this invention for speech recognition tasks in low-resource languages. The method not only significantly improves recognition accuracy but also demonstrates excellent cross-lingual knowledge transfer capabilities. This means that even for languages with extremely limited training data, the system can provide reliable recognition by leveraging knowledge from related languages.

[0143] The best embodiment is undoubtedly Example 1, which fully demonstrates all the advantages of the present invention. In practical applications, this comprehensive solution is recommended, especially when dealing with minority languages with extremely scarce language resources. For example, for some endangered languages with fewer than 10,000 speakers, a system may be the only viable solution for achieving reliable speech recognition.

[0144] Overall, these test results strongly support our innovative philosophy: by integrating multi-source corpora and advanced feature fusion techniques, speech recognition performance for low-resource languages can be significantly improved. This is not only a technological advancement, but also provides a powerful tool for protecting linguistic equality and cultural diversity.

Claims

1. A low-resource language speech recognition system utilizing multi-source corpus fusion, characterized by: It includes a multi-source language framework, which uses labeled speech data of high-resource languages and low-resource languages to train a model for recognizing speech in the low-resource language. The multi-source language framework includes three parts: acoustic layer, language layer and pronunciation layer, among which, The acoustic layer maps each acoustic feature to a common vocal tract coding subspace representing each language-independent feature and a vocal tract residual space representing language-specific information through ASVD. The former is used to predict the phoneme probability distribution of each frame, and the latter is used to predict the phoneme probability distribution of each frame for each language. The decoding module propagates the probability distribution in each dimension through the speech network, language network, and pronunciation network, and outputs the final prediction result. The acoustic model uses ASVD to divide the training data into two parts: a common channel encoder and a channel residual encoder. The former uses one ASVD component for acoustic training, while the latter uses the remaining ASVD components for training, thereby gaining the ability to extract multiple acoustic information. Language model, used to model language uncertainty and improve the performance of language perception tasks; The pronunciation model aligns each phoneme with a different language marker, increasing the model's language awareness while avoiding confusion between phonemes and languages. Feature fusion solutions, including multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and feature fusion of cross-lingual knowledge propagation enhanced by graph neural networks.

2. The low-resource language speech recognition system according to claim 1, characterized in that: In the multi-scale contrastive learning, multi-scale windows are used to constrain the common vocal tract encoder and the vocal tract residual encoder in the ASVD acoustic features for the vocal tract coding features of speech, so as to obtain vocal tract coding characteristics of multiple scales in multiple languages. In the hierarchical attention alignment mechanism, global and local alignment of multiple languages is automatically learned based on the pronunciation module and decoding model, and a hierarchical attention mechanism is used to align the vocal tract coding features between different languages.

3. The low-resource language speech recognition system according to claim 2, wherein: In the feature fusion scheme, the technology of adversarial feature decoupling and recombining is specifically to use adversarial learning and nonlinear decoder to separate and recombine the speech layer and language layer features of the common acoustic channel coding through decoupling learning in the common channel coding space; and the feature fusion technology of graph neural network enhanced cross-language knowledge propagation is specifically to use graph neural network based on regularization and orthogonal constraints to propagate and fuse language-specific channel residual features in the decoding of each layer.

4. The low-resource language speech recognition system according to claim 1, wherein: The self-supervised training strategy of the multi-source speech recognition encoder is specifically for a low-resource language and a high-resource language , the training data of each speech is transformed into Common Channel Encoder Sum channel residual encoder Two parts; then train the common channel encoder parameters simultaneously Sum channel residual encoder parameters And a public model and a language model ; Different languages during training Corresponding to different channel residual encoder parameters and language model parameters Two of the languages 、 The common channel encoder and the channel residual encoder can be trained simultaneously, and effective fusion of information can be achieved through the feature fusion technology.

5. A method for training a low-resource language speech recognition system using multi-source corpus fusion according to any one of claims 1 to 4, characterized in that: The method comprises the following steps: Step 1) Based on the multi-source language framework, we design a feature fusion scheme that combines multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and graph neural network-enhanced cross-lingual knowledge propagation. We use this scheme to train an encoder for a low-resource language l, where X is a high-resource language dataset and Yl is a low-resource language dataset. Step 1-1) Design a multi-scale contrastive learning framework to extract acoustic information from the channel coding layer, wherein a fixed-size sliding window is used to sample from a fixed-size input channel coding; then, the corresponding channel coding is divided into multiple segments, and the common channel encoder and the language channel encoder respectively use these coding segments to predict the implicit semantic information of each channel coding of each frame; Steps 1-2) Use a hierarchical attention alignment mechanism to learn multilingual alignment from both global and local perspectives. Design a self-supervised MAB algorithm to find the global and local alignment matrices. Use the MAB algorithm to learn the alignment matrix. Steps 1-3) Adaptively split and reassemble vocal tract coding features using adversarial learning and decoders in multilingual acoustic features; Steps 1-4), cross-language knowledge dissemination feature fusion technology using graph neural network enhanced cross-language knowledge dissemination; Step 2) For the trained encoder, obtain the output of the decoder based on its characteristics.

6. The training method according to claim 5, characterized in that: In step 1-1), the design of multi-scale contrastive learning includes first using ASVD to transform the training dataset into Each language in the common channel encoder and the vocal tract residual encoder for each language : In addition, the MAB approach is used to learn the decoding matrix for each frame of each language and common channel encoding and decoding matrix , used to decode the common channel coded semantic vector: Decompose each channel encoding vector into a common channel encoding phoneme embedding and a channel residual phoneme embedding: The matching loss is used to calculate the loss between the residual representation of the common channel encoding and the residual representation of the residual channel encoding; the contrast loss is used to calculate the matching loss between the phoneme embedding of the common channel encoding and the channel residual phoneme embedding, where, is the training data set, is the language identifier, and Represent low-resource languages and high-resource languages, respectively. For common channel encoding, is the channel residual coding, is the decoding matrix for each language and each frame, is the common channel encoding and decoding matrix, Encode semantic vectors for language-specific vocal tracts, Encode semantic vectors for common channels, is the final channel encoding vector.

7. The training method according to claim 6, characterized in that: Steps 1-2) are to make each language and decoder learn language context features, construct two hierarchical global and local attention alignment matrices in the common channel coding space, and make the decoder learn the attention of multiple languages; find the two language and decoding hierarchical matrices by MAB method and ,Through the MAB algorithm, we learn the alignment matrix; is the set of alignment matrices, For the language Layer local attention alignment matrix For the language layer global attention alignment matrix, is the total number of layers, and the MAB algorithm is used to learn these alignment matrices.

8. The training method according to claim 7, characterized in that: Steps 1-3) include adversarial learning, allowing the learned decoder to generate channel residual embeddings, and calculating the final channel residual embeddings through the decoder model; The generative adversarial network consists of three parts: a reconstructor, a generator, and a discriminator. The generator reconstructs the channel residual embedding from the common encoder embedding, while the discriminator receives the channel residual embedding and common channel encoding generated by the reconstructor and determines whether the common encoding comes from the reconstructor or the discriminator. In the adversarial loss, the reconstructor is trained by minimizing the adversarial loss between the reconstructor and the generator, and the generator is trained by minimizing the adversarial loss between the generator and the discriminator. The channel residual embedding uses a function to decouple learning. A separation function is used to decouple the learned vocal channel residual embedding.

9. The training method according to claim 8, characterized in that: For the channel residual embedding, decoupling learning is performed through the function f, which uses the MAB method to learn the alignment matrix to separate language features; the contrast loss is used to constrain the embedding loss between the language decoder and the channel residual embedding frame; decoupling learning is implemented using the contrast loss function Lcontrast: Where f is the decoupling function, and The vocal tract residual embeddings for different enhanced versions of the same speech, is the channel residual embedding of other samples in the batch, () is the cosine similarity function, is the temperature parameter.

10. The training method according to claim 7, 8 or 9, characterized in that: The enhanced cross-language knowledge propagation feature fusion technology in steps 1-4) includes a graph neural network enhanced cross-language knowledge propagation feature fusion technology. After extracting the common channel code of each language, a graph neural network is designed using the common channel code set of each decoupling layer for all languages, and the common channel code matrix is trained using SVD and ASVD. And each channel coding matrix , and finally use the encoding matrix after matrix decomposition and Finally, further learning for each language is performed using another matching loss and contrastive cosine loss at each layer: in, is the common channel coding matrix, is the language-specific vocal tract encoding matrix, and is the encoding matrix after matrix decomposition, ||·||F represents the Frobenius norm, cos() is the cosine similarity function, and λ is the weight parameter; The step 2 also includes using a reconstruction model based on a decoding network to reconstruct the speech segment, using a method to minimize the ASVD reconstruction loss to solve the open set problem, specifically using an ASVD encoder encoding step; for the speech input D* in the test set, the trained model is used to recognize and generate a target low-resource language based on multi-source learning l The recognition result of .

Images