Multi-source corpus fusion low-resource language speech recognition system and training method thereof

Abstract

The invention relates to the technical field of low-resource language speech recognition systems, in particular to a multi-source corpus fused low-resource language speech recognition system and a training method thereof, comprising a multi-source language framework, training a model for recognizing the speech of a low-resource language by using speech data of labeled high-resource languages and low-resource languages, and training the model for recognizing the speech of the low-resource languages by using the speech data of the labeled high-resource languages and low-resource languages. The multi-source language framework comprises an acoustic layer, a language layer and a pronunciation layer, and the technical problem is successfully solved through the innovative multi-source language framework and the advanced feature fusion technology. The method not only can effectively utilize knowledge of high-resource languages to enhance the recognition effect of low-resource languages, but also can accurately capture generality and difference among different languages. In addition, the method is excellent in performance when a multi-language mixed environment is processed, and the adaptability and generalization ability of the system are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of low-resource language speech recognition systems, and in particular to a low-resource language speech recognition system integrating multi-source corpora and a training method thereof. Background Art

[0002] In today's global language environment, speech recognition technology for low-resource languages ​​has received increasing attention. However, the development of this technology has always faced many challenges. Traditional speech recognition systems usually rely on a large amount of labeled data for training, which is a huge obstacle for languages ​​with a small population or lack of digital resources. In recent years, although researchers have made unremitting efforts in this field, existing technologies still have many shortcomings.

[0003] Currently, the closest existing technologies mainly use two methods to deal with the speech recognition problem of low-resource languages. The first is the single language model method, which only uses limited target language data for training. This method is simple and direct, but it is often difficult to build a recognition model with good performance due to insufficient training data. In particular, for some endangered languages ​​with very small populations, collecting enough training data is almost an impossible task.

[0004] The second approach is a simple multilingual model that mixes data from multiple languages ​​together for training. This approach alleviates the problem of data scarcity to a certain extent, but it ignores the differences between different languages ​​and cannot effectively use the relationship between languages ​​to improve the recognition effect of low-resource languages. Moreover, when dealing with languages ​​with very different speech characteristics, this simple mixture often leads to a decrease in model performance.

[0005] The main technical problems of these existing methods can be summarized as follows: First, they cannot effectively use the knowledge in high-resource languages ​​to assist the recognition of low-resource languages. Second, they lack the ability to fine-tune the acoustic and linguistic features between different languages. Third, they perform poorly when dealing with multilingual mixed environments and have difficulty accurately distinguishing and recognizing different languages. Finally, these methods also have obvious deficiencies in computational efficiency and model generalization capabilities.

[0006] Faced with these challenges, it becomes particularly necessary to develop speech recognition systems that can effectively utilize multi-source corpora and accurately recognize low-resource languages. Summary of the invention

[0007] The present invention is proposed to solve these technical problems. The present invention is proposed to solve the key problems existing in the prior art, such as low data utilization efficiency, insufficient feature extraction, and weak cross-language knowledge transfer ability, by integrating multi-source corpus and low-resource language speech recognition system and its training method.

[0008] The present invention provides a low-resource language speech recognition system using multi-source corpus fusion, including a multi-source language framework, using labeled high-resource language and low-resource language speech data to train a model for recognizing the low-resource language speech, the multi-source language framework includes three parts: an acoustic layer, a language layer and a pronunciation layer, wherein:

[0009] The acoustic layer maps each acoustic feature to a common vocal tract coding subspace representing each language-independent feature and a vocal tract residual space representing language-specific information by means of ASVD. The former is used to predict the phoneme probability distribution of each frame, and the latter is used to predict the phoneme probability distribution of each frame of each language. The decoding module propagates the probability distribution in each dimension through the speech network, the language network and the pronunciation network, and outputs the final prediction result.

[0010] The acoustic model uses ASVD to divide the training data into two parts, which are composed of a common channel encoder and a channel residual encoder. The former uses one ASVD component for acoustic training, and the latter uses the remaining ASVD components for training, thereby obtaining the ability to extract a variety of acoustic information.

[0011] Language model, which is used to model the uncertainty of language and improve the effect of language perception tasks;

[0012] The pronunciation model aligns each phoneme with a different language marker, increasing the model's language awareness while avoiding confusion between phonemes and languages;

[0013] Feature fusion solutions include multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and feature fusion techniques for cross-language knowledge propagation enhanced by graph neural networks.

[0014] Specifically, in the multi-scale contrastive learning, for the vocal tract coding features of speech, a multi-scale window is used in the ASVD acoustic features to constrain the common channel encoder and the channel residual encoder to obtain the vocal tract coding characteristics of multiple scales in multiple languages; in the hierarchical attention alignment mechanism, the global and local alignment of multiple languages ​​is automatically learned according to the pronunciation module and the decoding model, and the hierarchical attention mechanism is used to align the vocal tract coding features between different languages.

[0015] Specifically, in the feature fusion scheme, the technology of adversarial feature decoupling and recombination, specifically, in the common vocal channel coding space, uses adversarial learning and nonlinear decoder to separate and reorganize the speech layer and language layer features of the common acoustic vocal channel coding through decoupling learning; and the feature fusion technology of graph neural network enhanced cross-language knowledge propagation, specifically, in the decoding of each layer, uses a graph neural network based on regularization and orthogonal constraints to propagate and fuse language-specific vocal channel residual features.

[0016] Specifically, the self-supervised training strategy of the multi-source speech recognition encoder is specifically, for a language ι of a low-resource language 1 and a high-resource language ι 2 , the training data ν of each speech is divided into the common channel encoder V by ASVD l:c and channel residual encoder V l:r Two parts; then train the common channel encoder parameters V at the same time l:c ~p(V l:c ) and the channel residual encoder parameters V l:r ~p(V l:r ) and a common model d(V l:c θ l:c ) and a language model dv(V l :r;θ v ); Different languages ​​ι correspond to different channel residual encoder parameters V during training l:r and language model parameters θ v ; Two of the languages 1 , 2 The common channel encoder and the channel residual encoder can be trained simultaneously, and effective fusion of information can be achieved through the feature fusion technology.

[0017] The training method of the low-resource language speech recognition system using multi-source corpus fusion comprises the following steps:

[0018] Step 1), based on the multi-source language framework, design a feature fusion scheme of multi-scale contrastive learning, hierarchical attention alignment mechanism, adversarial feature decoupling and reorganization, and cross-language knowledge propagation enhanced by graph neural network, and use the scheme to train the encoder of low-resource language l, where X is a high-resource language dataset and Yl is a low-resource language dataset;

[0019] Step 1-1), design a multi-scale contrastive learning framework for extracting acoustic information of the channel coding layer, wherein a fixed-size sliding window is used to sample from a fixed-size input channel coding; then, the corresponding channel coding is divided into multiple segments, and the common channel encoder and the language channel encoder respectively use these coding segments to predict the implicit semantic information of each channel coding of each frame;

[0020] Step 1-2), learn multilingual alignment from both global and local perspectives through a hierarchical attention alignment mechanism, design a self-supervised MAB algorithm to find global and local alignment matrices, and use the MAB algorithm to learn the alignment matrix;

[0021] Steps 1-3), in multi-language acoustic features, using adversarial learning and decoders to adaptively split and reassemble vocal tract coding features;

[0022] Steps 1-4), cross-language knowledge propagation feature fusion technology using graph neural network enhanced cross-language knowledge propagation;

[0023] Step 2) For the trained encoder, the output of the decoder is obtained according to its characteristics.

[0024] Specifically, the design of multi-scale contrastive learning in step 1-1) includes first using ASVD to divide each language in the training dataset V into a common channel encoder V lcc and the channel residual encoder V for each language l：r :

[0025] V l：c =V l -V l：r , l∈{l 1 , l 2}

[0026] In addition, the MAB approach is used to learn the decoding matrix A for each frame of each language l and the common channel encoding and decoding matrix A cl For decoding common channel coded semantic vectors:

[0027] z l =A l ·V l：c

[0028] z cl =A cl ·V l：c

[0029] Decompose each channel encoding vector into a common channel encoding phoneme embedding and a channel residual phoneme embedding:

[0030] e l =z l +z cl

[0031] The matching loss is used to calculate the loss between the residual representation of the common channel encoding and the residual representation of the residual channel encoding; the contrast loss is used to calculate the matching loss between the phoneme embedding of the common channel encoding and the phoneme embedding of the channel residual, where V is the training data set, ι is the language identifier, ι 1 and 2 Represent low-resource languages ​​and high-resource languages, V l：c For common channel encoding, V l :r is the channel residual coding, A lis the decoding matrix for each language and each frame, A cl is the common channel encoding and decoding matrix, z l Encode semantic vectors for language-specific vocal channels, z d Encode semantic vector for common channel, e l is the final channel encoding vector.

[0032] Specifically, the steps 1-2) are to make each language and decoder learn language context features, construct two hierarchical global and local attention alignment matrices in the common channel coding space, and make the decoder learn the attention of multiple languages; find the two language and decoding hierarchical matrices A={A l , 0, ..., A l , N} and {A l，c , 0, ..., A l，c , N}, through the MAB algorithm, learn the alignment matrix; where A is the alignment matrix set, A l , i (i = 0, ..., N) is the i-th layer local attention alignment matrix of the l-th language, A l , c,i (i=0,...,N) is the i-th layer global attention alignment matrix of the l-th language, N is the total number of layers, and the MAB algorithm is used to learn these alignment matrices.

[0033] Specifically, the steps 1-3) include adversarial learning, allowing the learned decoder to generate channel residual embedding, and calculating the final channel residual embedding through the decoder model; the generative adversarial network consists of three parts: a reconstructor, a generator and a discriminator. The generator reconstructs the channel residual embedding from the common encoder embedding, and the latter receives the channel residual embedding generated by the reconstructor and the common channel encoding, and determines whether the common encoding comes from the reconstructor or the discriminator; in the adversarial loss, the reconstructor is trained by minimizing the adversarial loss between the reconstructor and the generator, and the generator is trained by minimizing the adversarial loss between the generator and the discriminator; the channel residual embedding uses a function to decouple the learning; a separation function is used to decouple the learning of the channel residual embedding.

[0034] Specifically, for the channel residual embedding, decoupling learning is performed through function f, and function f uses the MAB method to learn the alignment matrix to separate language features; the contrast loss is used to constrain the embedding loss between the language decoder and the channel residual embedding frame; decoupling learning is implemented using the contrast loss function Lcontrast:

[0035]

[0036] Among them, f is the decoupling function, e 1 and e I′ is the vocal tract residual embedding of different enhanced versions of the same speech, ek is the channel residual embedding of other samples in the batch, sim() is the cosine similarity function, and τ is the temperature parameter.

[0037] Specifically, the enhanced cross-language knowledge propagation feature fusion technology in step 1-4) includes a cross-language knowledge propagation feature fusion technology enhanced by a graph neural network. After extracting the common channel code of each language, a graph neural network is designed using the common channel code set of each decoupling layer for all languages, and the common channel code matrix C is trained using SVD and ASVD. cl And each channel coding matrix C lr , and finally use the encoding matrix C after matrix decomposition lc and C cr ; Finally, another matching loss and contrastive cosine loss are used at each layer to further learn each language:

[0038]

[0039] Among them, C cl is the common channel coding matrix, C lr is the language-specific channel encoding matrix, C lc and C cr is the encoding matrix after matrix decomposition, ||·||F represents the Frobenius norm, cos() is the cosine similarity function, and λ is the weight parameter;

[0040] The step 2 also includes using a reconstruction model based on a decoding network to reconstruct the speech segment, and adopting a method of minimizing the ASVD reconstruction loss to solve the open set problem, specifically using an ASVD encoder encoding step; for the speech input D* in the test set, the trained model is used for recognition, and a recognition result of a target low-resource language based on multi-source learning is generated.

[0041] Advantages and beneficial effects of the present invention:

[0042] The low-resource language speech recognition system with multi-source corpus fusion proposed by the present invention successfully overcomes the above technical difficulties through an innovative multi-source language framework and advanced feature fusion technology. It can not only effectively utilize the knowledge of high-resource languages ​​to enhance the recognition effect of low-resource languages, but also accurately capture the commonalities and differences between different languages. In addition, the present invention performs well in processing multi-language mixed environments, greatly improving the adaptability and generalization ability of the system.

[0043] The implementation of the present invention brings about many beneficial effects. First, it significantly improves the accuracy of speech recognition of low-resource languages, so that some languages ​​that were previously difficult to achieve reliable recognition can now be effectively processed. Secondly, the cross-language knowledge transfer capability of the present invention provides a new tool for linguistic research, which helps to better understand the relationship between different languages. Furthermore, it provides strong technical support for the protection and research of endangered languages, which is conducive to maintaining language diversity. Finally, the efficient computing method of the present invention enables it to operate in a resource-constrained environment, paving the way for the application of language technology in remote areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall logic block diagram of the system of the present invention.

[0045] Figure 2 It is a logic flow chart of the training method of the present invention.

[0046] Figure 3 It is a logic flow chart of the multi-source language framework of the present invention.

[0047] Figure 4 It is a logic flow chart of the feature fusion solution of the present invention. DETAILED DESCRIPTION

[0048] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0050] See also Figure 1-4 The present invention proposes a low-resource language speech recognition system using multi-source corpus fusion. The core of the system is a multi-source language framework 1, whose main purpose is to use the labeled speech data of high-resource languages ​​and low-resource languages ​​to train a model that can recognize the speech of the low-resource language. The framework specifically includes three key parts: an acoustic layer 11, a language layer 12, and a pronunciation layer 13.

[0051] First, the acoustic layer 11 maps each acoustic feature to two important spaces by ASVD: one is the common channel coding subspace representing each language-independent feature, and the other is the channel residual space representing language-specific information. The former is used to predict the phoneme probability distribution of each frame, while the latter is used to predict the phoneme probability distribution of each frame for each language. This decomposition method enables the system to better capture the commonalities and differences between different languages.

[0052] Secondly, the decoding module 14 propagates the probability distribution in each dimension through the speech network, language network and pronunciation network, and finally outputs the prediction result. This multi-network collaborative working method can comprehensively consider all aspects of speech, language and pronunciation, thereby improving the accuracy of recognition.

[0053] Thirdly, the acoustic model uses ASVD to divide the training data into two parts, which are composed of a common channel encoder and a channel residual encoder. The former uses one ASVD component for acoustic training, and the latter uses the remaining ASVD components for training, thereby obtaining the ability to extract a variety of acoustic information. This separation allows the system to more effectively utilize a large amount of data from high-resource languages ​​to assist model training for low-resource languages.

[0054] The language model is then used to model the uncertainty of the language and improve the performance of language perception tasks. This is particularly important for dealing with multilingual speech environments, as it helps the system more accurately determine which language the speech belongs to.

[0055] Next, the pronunciation model aligns each phoneme with different language markers, increasing the model's language perception while avoiding confusion between phonemes and languages. This step is a key bridge between the acoustic layer and the language layer, and can effectively process phonemes that are similar but not exactly the same between different languages.

[0056] Finally, the feature fusion solution includes multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and feature fusion technology enhanced by graph neural network cross-language knowledge propagation. These advanced technologies work together to greatly improve the system's performance in processing multilingual data.

[0057] First of all, it should be clear that the acoustic layer 11, language layer 12 and pronunciation layer 13 are components of the multi-source language framework 1, and the acoustic model, language model and pronunciation model are specific models used in these layers. The acoustic layer 11 is the part of the framework responsible for processing acoustic features. It uses ASVD (adaptive singular value decomposition) technology to map acoustic features to the common vocal tract coding subspace and the vocal tract residual space. The acoustic model is a specific model used in this layer, which is responsible for converting acoustic features into the probability distribution of phonemes or other acoustic units; the features extracted by the acoustic layer 11 through ASVD are input into the acoustic model, and the acoustic model then outputs the probability distribution of the phoneme level, which is passed to the language layer 12 and the pronunciation layer 13. The language layer 12 is responsible for modeling the uncertainty of the language and improving the effect of the language perception task. The language model is a specific model used in this layer, which predicts the probability distribution of word sequences and helps the system understand the structure and grammar of the language. The language layer 12 receives the phoneme probability distribution from the acoustic layer 11 and uses the language model to evaluate the possibility of different word sequences. This information is combined with the output of the pronunciation layer 13 to jointly influence the final decoding result. The pronunciation layer 13 is responsible for aligning each phoneme with different language markers. The pronunciation model is a specific model used in this layer, which describes the standard pronunciation of a word, that is, how a word is decomposed into a phoneme sequence. The pronunciation layer 13 uses the pronunciation model to combine the phoneme probability output by the acoustic layer 11 with the word sequence possibility provided by the language layer 12 to determine the most likely word-phoneme correspondence.

[0058] The connection and collaboration between these three layers is achieved through the decoding module 14. The decoding module 14 integrates the information from the acoustic layer 11, the language layer 12 and the pronunciation layer 13, propagates the probability distribution in each dimension through the speech network, the language network and the pronunciation network, and finally outputs the recognition result.

[0059] In a multi-source language environment, the advantages of this hierarchical structure are: the acoustic layer 11 can capture the common acoustic features between different languages, which is conducive to cross-language knowledge transfer. The language layer 12 can handle multi-language mixed environments and effectively distinguish the grammatical structures of different languages. The pronunciation layer 13 can handle the pronunciation differences between different languages ​​and improve the system's adaptability to low-resource languages.

[0060] In this way, the multi-source language framework 1 can effectively use the knowledge of high-resource languages ​​to enhance the recognition effect of low-resource languages, while maintaining sensitivity to various language characteristics. This design enables the system to fully utilize relevant information from high-resource languages ​​when processing low-resource languages, thereby significantly improving recognition performance.

[0061] Preferably, in one embodiment of the present invention, when the system processes speech data of Chinese and Uyghur, the common vocal tract coding subspace may capture the phoneme features common to the two languages, while the vocal tract residual space retains the tone information of Chinese and the special vowel features of Uyghur respectively. This method enables the system to maximize knowledge sharing between different languages ​​while maintaining language specificity.

[0062] Preferably, in one embodiment of the present invention, the present invention further refines the specific implementation of multi-scale contrastive learning 21 and hierarchical attention alignment mechanism. First, multi-scale contrastive learning targets the vocal tract coding features of speech, and uses multi-scale windows in the ASVD acoustic features to constrain the common vocal tract encoder and the vocal tract residual encoder to obtain vocal tract coding characteristics of multiple scales in multiple languages. This method allows the system to capture acoustic features at different time scales, thereby better adapting to the acoustic characteristics of different languages.

[0063] Secondly, the hierarchical attention alignment mechanism automatically learns global and local alignment of multiple languages ​​based on the pronunciation module and decoding model, and uses a hierarchical attention mechanism to align the vocal tract encoding features between different languages. This mechanism is particularly helpful for processing language pairs with large differences in grammatical structure, and can align features at different levels, thereby improving the effect of cross-language learning.

[0064] Preferably, in one embodiment of the present invention, when processing a language pair with large grammatical structure differences such as Chinese and Tibetan, multi-scale contrastive learning may use a smaller time window (such as 10 milliseconds) to capture rapidly changing phonemes, while using a larger time window (such as 50 milliseconds) to capture more stable phonemes. This flexible approach enables the system to better adapt to the different acoustic characteristics of the two languages.

[0065] Preferably, in one embodiment of the present invention, the present invention further describes in detail two key technologies in the feature fusion scheme: adversarial feature decoupling and recombination technology and graph neural network enhanced cross-language knowledge propagation technology.

[0066] First, the adversarial feature decoupling and reorganization technology uses adversarial learning and nonlinear decoders to separate and reorganize the speech layer and language layer features of the common acoustic channel encoding through decoupling learning in the common channel encoding space. This method can effectively separate language-independent acoustic features and language-specific features, thereby improving the system's performance in cross-language tasks.

[0067] Secondly, the cross-language knowledge propagation technology enhanced by graph neural network uses graph neural network propagation based on regularization and orthogonal constraints and integrates language-specific vocal tract residual features in the decoding of each layer. This method can effectively transfer knowledge between different languages, especially when dealing with languages ​​with similar language families.

[0068] Preferably, in one embodiment of the present invention, when processing the three East Asian languages ​​of Chinese, Japanese and Korean, the graph neural network enhanced cross-language knowledge propagation technology can better capture the common features and subtle differences between them, thereby improving the recognition performance of each language.

[0069] Preferably, in one embodiment of the present invention, a self-supervised training strategy for a multi-source speech recognition encoder is described in detail. Specifically, for a low-resource language, 1 (such as Qiang) and a high-resource language ι 2 (such as Chinese), the system uses ASVD to divide the training data V of each voice into a common channel encoder V l：c and channel residual encoder V l：r Two parts.

[0070] Then, the system simultaneously trains the common channel encoder parameters V l：c ~p(V l：c ) and the channel residual encoder parameters V l：r ~p(V l：r ) and a common model d(V l：c θ l：c ) and a language model dv(V l :r;θ v ). Different languages ​​ι correspond to different channel residual encoder parameters V during training. l：r and language model parameters θ v Among them, it is worth noting that the two languages ​​ι 1 , 2 The common channel encoder and the channel residual encoder can be trained simultaneously, and information can be effectively fused through feature fusion technology. This parallel training strategy not only improves training efficiency, but also maximizes knowledge sharing between different languages ​​while maintaining language specificity.

[0071] Preferably, in one embodiment of the present invention, when the system processes Qiang (a low-resource language) and Chinese (a high-resource language) at the same time, a large amount of training data of Chinese can be used to improve the acoustic model of Qiang while retaining the unique speech features of Qiang. This method can significantly improve the recognition accuracy of Qiang without affecting the recognition performance of Chinese.

[0072] Preferably, in one embodiment of the present invention, a method for training a low-resource language speech recognition system with source corpus fusion is proposed. The method comprises the following steps:

[0073] First, based on the multi-source language framework, we design a feature fusion scheme that combines multi-scale contrastive learning, hierarchical attention alignment mechanism, adversarial feature decoupling and reorganization, and cross-language knowledge propagation enhanced by graph neural networks. Then, we use the scheme to train the encoder of the low-resource language l, where X is a high-resource language dataset and Yl is a low-resource language dataset.

[0074] Next, a multi-scale contrastive learning framework is designed to extract the acoustic information of the channel coding layer. Specifically, a fixed-size sliding window is used to sample from a fixed-size input channel coding. Then, the corresponding channel coding is divided into multiple segments, and the common channel encoder and the language channel encoder use these coding segments to predict the implicit semantic information of each channel coding of each frame.

[0075] Then, the hierarchical attention alignment mechanism is used to learn multilingual alignment from both global and local perspectives. In this step, the system designs a self-supervised MAB algorithm to find global and local alignment matrices, and uses the MAB algorithm to learn the alignment matrix. This method can capture the correspondence between languages ​​at different levels, thereby improving the effect of cross-language learning.

[0076] Next, in the multilingual acoustic features, the system uses adversarial learning and decoders to adaptively split and reassemble the vocal tract encoding features. This step helps the system better separate language-independent acoustic features and language-specific features, thereby improving performance in cross-language tasks.

[0077] Finally, the system uses graph neural network-enhanced cross-language knowledge propagation feature fusion technology, which can effectively transfer knowledge between different languages, especially when dealing with languages ​​with similar language families.

[0078] After completing the above steps, for the trained encoder, the system obtains the output of the decoder based on its characteristics.

[0079] Preferably, in one embodiment of the present invention, when processing Yi (a low-resource language) and Chinese (a high-resource language), the training method can make full use of a large amount of Chinese data to enhance the speech recognition model of Yi. For example, in the multi-scale contrastive learning step, the system may use time windows of different sizes to capture the tone features of Yi and the syllable structure of Chinese, respectively, thereby improving the recognition accuracy of the two languages.

[0080] Preferably, in one embodiment of the present invention, the design process of multi-scale contrastive learning is further described in detail. First, the system uses ASVD to divide each language in the training data set V into a common channel encoder V l:c and the channel residual encoder V for each language l:r . This can be expressed by the following formula:

[0081] V l：c =V l -V l：r , l∈{l 1 , l 2}

[0082] Among them, ι 1 Represents a low-resource language, ι 2 Represents a high-resource language.

[0083] Then, the system uses MAB (Multi-Head Attention Block) to learn the decoding matrix A for each frame in each language l and the common channel encoding and decoding matrix A cl , used to decode the common channel encoded semantic vector. This process can be expressed by the following formula:

[0084] Decode common channel encoded semantic vector:

[0085] z l =A l ·V l：c

[0086] z cl =A cl ·V l：c

[0087] Next, the system decomposes each channel encoding vector into a common channel encoding phoneme embedding and a channel residual phoneme embedding:

[0088] e l =z l +z cl

[0089] Where V is the training data set, ι is the language identifier, 1 and 2 Represent low-resource languages ​​and high-resource languages, V l：c For common channel encoding, V l :r is the channel residual coding, A l is the decoding matrix for each language and each frame, A cl is the common channel encoding and decoding matrix, z l Encode semantic vectors for language-specific vocal channels, z d Encode semantic vector for common channel, el Finally, the system uses a matching loss to calculate the loss between the residual representation of the common channel encoding and the residual representation of the residual channel encoding, and uses a contrastive loss to calculate the matching loss between the phoneme embedding of the common channel encoding and the channel residual phoneme embedding.

[0090] Preferably, in one embodiment of the present invention, when processing Zhuang (a low-resource language) and Chinese (a high-resource language), this multi-scale contrastive learning method can effectively capture the common features of the two languages ​​(such as certain consonants) and their respective unique features (such as the tone pattern of Zhuang). This method not only improves the recognition accuracy of Zhuang, but also enhances the robustness of the system when processing mixed speech.

[0091] Preferably, in one embodiment of the present invention, a method for implementing a hierarchical attention alignment mechanism is described. First, the system enables each language and decoder to learn language context features. Then, in the common channel coding space, two hierarchical global and local attention alignment matrices are constructed to enable the decoder to learn the attention of multiple languages.

[0092] Then, the system finds two languages ​​and the decoding level matrix A = {A l , 0, ..., A l , N} and {A l，c , 0, ..., A l，c , N}. Finally, the alignment matrix is ​​learned through the MAB algorithm.

[0093] In this process A l , i (i = 0, ..., N) represents the i-th layer local attention alignment matrix of the l-th language, A l , c,i (i=0,...,N) represents the i-th layer global attention alignment matrix of the l-th language, and N is the total number of layers.

[0094] Preferably, in one embodiment of the present invention, when processing parallel corpora of Tibetan and Chinese, the underlying A l , may focus on phoneme-level alignment, while high-level A l , N may focus on semantic alignment at the sentence level. This hierarchical structure enables the system to capture the correspondence between languages ​​at different levels, thereby improving the effect of cross-language learning, especially when dealing with language pairs with large differences in grammatical structures.

[0095] Preferably, in one embodiment of the present invention, the specific implementation steps of adversarial learning are as follows: first, the system allows the learned decoder to generate channel residual embedding, and calculates the final channel residual embedding through the decoder model. In this process, the generative adversarial network consists of three key parts: a reconstructor, a generator, and a discriminator.

[0096] Specifically, the generator reconstructs the channel residual embedding from the common encoder embedding. The discriminator receives the channel residual embedding and the common channel encoding generated by the reconstructor, and determines whether the common encoding comes from the reconstructor or the discriminator. In the calculation of the adversarial loss, the system trains the reconstructor by minimizing the adversarial loss between the reconstructor and the generator, and trains the generator by minimizing the adversarial loss between the generator and the discriminator.

[0097] In addition, the vocal tract residual embedding uses a specific function to decouple the learning, and the system uses a separation function to decouple the learning of the vocal tract residual embedding. This method can effectively separate language-independent acoustic features from language-specific features.

[0098] Preferably, in one embodiment of the present invention, when processing two similar languages, Dai and Thai, this adversarial learning method can help the system better distinguish their subtle differences. For example, the system may learn the slight differences in the tone patterns of Dai and Thai, thereby improving the recognition accuracy of the two languages.

[0099] Preferably, in one embodiment of the present invention, the decoupled learning process of the channel residual embedding is further refined. First, the system performs decoupled learning through function f, and function f uses the MAB method to learn the alignment matrix to separate the language features. Then, the system constrains the embedding loss between the language decoder and the channel residual embedding frame through the contrast loss.

[0100] Finally, decoupled learning is implemented using the contrast loss function Lcontrast, which can be expressed as:

[0101]

[0102] Among them, f is the decoupling function, e 1 and e l′ is the vocal tract residual embedding of different enhanced versions of the same speech, e k is the channel residual embedding of other samples in the batch, sim() is the cosine similarity function, and τ is the temperature parameter.

[0103] Preferably, in one embodiment of the present invention, when processing the three minority languages ​​of Miao, Dong and Bouyei, this decoupled learning method can help the system capture the common features and subtle differences between them. For example, the system may learn the similarities of the three languages ​​in the tone systems, and also recognize the differences in their consonant systems, thereby improving the recognition performance of each language.

[0104] The cross-language knowledge propagation feature fusion technology enhanced by graph neural network, first, after extracting the common channel coding of each language, the system uses the common channel coding set of each decoupled layer to design the graph neural network for all languages. Then, the system uses SVD and ASVD to train the common channel coding matrix Ccl and the channel coding matrix Clr of each language.

[0105] Next, the system uses the encoding matrix C after matrix decomposition lc and C cr Finally, another matching loss and contrastive cosine loss are used at each layer to further learn each language, which can be expressed by the following formula:

[0106]

[0107] Among them, C cl is the common channel coding matrix, C lr is the language-specific channel encoding matrix, C cl and C cr is the encoding matrix after matrix decomposition, ||·||F represents the Frobenius norm, cos() is the cosine similarity function, and λ is the weight parameter;

[0108] Preferably, in one embodiment of the present invention, when processing the three languages ​​of Uyghur, Kazakh and Kirgiz, this graph neural network enhanced cross-language knowledge propagation technology can effectively capture the linguistic connections between them. For example, the system may learn the similarities between these languages ​​in word order, grammatical structure and phonological system, thereby improving the recognition accuracy of each language.

[0109] The specific implementation method of step 2 is as follows: first, the system uses a reconstruction model based on a decoding network to reconstruct the speech segment. This method solves the open set problem by minimizing the ASVD reconstruction loss, and specifically uses the ASVD encoder encoding step.

[0110] The advantage of this method is that it can effectively process unseen speech samples and improve the generalization ability of the system. By minimizing the ASVD reconstruction loss, the system can learn a more robust feature representation, thereby maintaining a high recognition accuracy in the face of various accents, dialects or noisy environments.

[0111] Preferably, in one embodiment of the present invention, when the system processes the Hani language in Yunnan Province, this ASVD-based reconstruction method can help the system better adapt to different dialects and accents of the Hani language. For example, the system may learn the pronunciation variants of the Hani language in different regions, so that it can still maintain a high recognition accuracy when facing speakers from different villages. This is of great significance for the protection and research of endangered languages ​​such as the Hani language.

[0112] Specific operations for the test set in step 2. First, for the speech input D* in the test set, the system uses the trained model for recognition. Then, the system generates the recognition results of the target low-resource language l based on multi-source learning.

[0113] This process makes full use of the cross-language knowledge learned in the previous training phase, enabling the system to better handle the recognition task of low-resource languages. Through multi-source learning, the system can transfer the knowledge learned in high-resource languages ​​to the recognition of low-resource languages, thereby improving the recognition performance.

[0114] Preferably, in one embodiment of the present invention, when the system processes the Tibetan dialect of Qinghai Province, a large amount of data of standard Tibetan and Chinese can be used to improve the recognition accuracy. For example, the system may use the acoustic and language models of standard Tibetan to initialize the model of the Qinghai Tibetan dialect, and then use limited Qinghai Tibetan dialect data for fine-tuning. This method can significantly improve the recognition performance of the Qinghai Tibetan dialect, and can achieve good results even when the training data is very limited.

[0115] In addition, this approach can help the system better handle code-mixing situations, where speakers mix Tibetan and Chinese in the same sentence. The system can use the knowledge learned from Chinese to assist in identifying Chinese loanwords or phrases in Tibetan, thereby improving the overall recognition accuracy.

[0116] The above method is applied in low-resource language speech recognition systems and non-pathological research fields. Specifically, the system integrates the self-supervised learning high-resource language and low-resource language data into a unified acoustic space, and then trains a multilingual model and performs encoder learning and speech recognition.

[0117] The advantage of this method is that it can make full use of a large amount of data from high-resource languages ​​to assist in model training for low-resource languages, thereby achieving better recognition results even under limited low-resource language data conditions. At the same time, by integrating data from different languages ​​into a unified acoustic space, the system can learn more general and robust feature representations, which is particularly beneficial for speech recognition tasks in multilingual environments.

[0118] Preferably, in one embodiment of the present invention, the method can be applied to the research and protection of minority languages ​​in southwest China. For example, when studying Yi, Naxi and Bai, the system can use Chinese as a high-resource language and map the acoustic features of these languages ​​to the same acoustic space. This can not only improve the recognition accuracy of these low-resource languages, but also help linguists better understand the relationship and differences between these languages.

[0119] In addition, this method can also be applied to the field of language acquisition research. For example, when studying the pronunciation characteristics of Chinese native speakers when learning English, the system can use a large amount of data from Chinese and English to train a bilingual model, and then use this model to analyze the learner's pronunciation. This can help researchers more accurately identify learners' pronunciation errors and whether these errors are affected by their native language (Chinese).

[0120] This method also has potential application value in non-pathological research fields. For example, when studying certain language disorders (such as stuttering), the system can use a large amount of normal speech data as high-resource data and limited pathological speech data as low-resource data. In this way, the system can better capture the subtle differences between normal and pathological speech, thereby providing a more accurate tool for early diagnosis and evaluation of treatment effects.

[0121] In general, the low-resource language speech recognition system and its training method proposed by the present invention successfully solve the problem of data scarcity in low-resource language speech recognition through an innovative multi-source language framework and advanced feature fusion technology. It not only improves the recognition accuracy of low-resource languages, but also provides new ideas for the development of cross-language speech recognition technology. This method has broad application prospects in the fields of linguistic research, protection of minority languages, cross-cultural communication, and the popularization of speech technology in multilingual areas.

[0122] For example, in the field of education, this system can be used to develop more intelligent and personalized language learning software. For students who want to learn minority languages, this system can provide accurate pronunciation guidance and evaluation even if there is a lack of a large number of learning resources on the market. At the same time, for minority students learning Mandarin or other mainstream languages, the system can provide more targeted learning suggestions based on the characteristics of the students' native language.

[0123] In terms of cultural protection, this system can help record and preserve endangered languages. Many minority languages ​​face problems such as a decrease in the number of users and the younger generation no longer using them. Through this efficient speech recognition system, researchers can more easily collect and transcribe a large amount of oral history, folk tales and other audio materials, providing valuable digital resources for the protection of these languages ​​and cultures.

[0124] In the field of healthcare, this system can help develop more accurate remote diagnostic tools. In some minority areas, there may be a lack of medical experts who speak specific languages. Through this multilingual speech recognition system, the patient's description can be accurately converted into text and then translated into a language familiar to the doctor, thus overcoming language barriers and providing better medical services.

[0125] In the commercial field, this system can help companies better develop multilingual markets. For example, when developing voice assistants or customer service systems for minority areas, even if there is little training data in a certain language, the quality of service can be improved by using data in other related languages. This can not only help companies expand their markets, but also provide better service experience for minority users.

[0126] In short, the application prospects of this invention are very broad. It not only promotes the development of speech recognition for low-resource languages ​​in terms of technology, but also has a positive impact on social issues such as language equality, protection of cultural diversity, and educational equity. With the further development and application of this technology, we can expect to see the formation of a more diverse and inclusive speech technology ecosystem.

[0127] In verifying the superiority of the present invention, an embodiment and two comparative examples were selected for comprehensive comparison. These tests were carried out around the core innovation of the present invention, fully demonstrating the advantages of the multi-source corpus fusion low-resource language speech recognition system.

[0128] Example 1 adopts the complete solution of the present invention, including a multi-source language framework, multi-scale contrastive learning, a hierarchical attention alignment mechanism, adversarial feature decoupling and reorganization, and graph neural network enhanced cross-language knowledge propagation technology. Tibetan was selected as the target low-resource language, and Chinese and Nepali were selected as auxiliary high-resource languages.

[0129] Comparative Example 1 uses a traditional monolingual speech recognition model, trained with only limited Tibetan data, which represents a common approach to dealing with low-resource languages.

[0130] Comparative Example 2 uses a simple multilingual model to train Tibetan, Chinese and Nepali data together, but does not use the feature fusion technology proposed in the present invention. The following key indicators are selected to evaluate the performance of the system:

[0131] 1. Word Error Rate (WER): This is a standard metric for measuring the accuracy of a speech recognition system.

[0132] 2. Phoneme Error Rate (PER): This can more carefully reflect the system's recognition ability at the phoneme level.

[0133] 3. Cross-lingual Transfer Efficiency (CTE): This is a new metric proposed to measure the system’s ability to use high-resource language knowledge to improve low-resource language recognition.

[0134] 4. Computational Efficiency (CE): measured by training time and inference speed.

[0135] The detection method is as follows: a speech dataset containing 1000 hours of Tibetan, 5000 hours of Chinese, and 3000 hours of Nepali was used. The Tibetan data was randomly divided into a training set (800 hours), a validation set (100 hours), and a test set (100 hours). Three models were trained under the same hardware conditions and evaluated on the Tibetan test set. The following are the detailed test results:

[0136] index Example 1 Comparative Example 1 Comparative Example 2 WER 12.3％ 25.7％ 18.9％ PER 8.7％ 19.2％ 14.1％ CTE 0.85 N / A 0.42 CE 1.2x 1.0x 1.1x

[0137] From these results, it is clear that the present invention is superior. Example 1 is significantly better than the two comparative examples in all key indicators. Let's analyze these results one by one:

[0138] First, in terms of word error rate (WER), our method reduces errors by 52.1% compared to the traditional single-language model (Comparison Example 1) and by 34.9% compared to the simple multi-language model (Comparison Example 2). This means that the system can recognize Tibetan speech more accurately, which is of great significance for practical applications such as speech translation or voice assistants.

[0139] Secondly, the significant reduction in the phoneme error rate (PER) (54.7% reduction compared to comparison example 1 and 38.3% reduction compared to comparison example 2) shows that the system has more accurate modeling capabilities at the phoneme level. This not only improves the overall recognition accuracy, but also provides a more reliable tool for linguistic research. For example, when studying the differences in Tibetan dialects, the system is able to capture more subtle pronunciation changes.

[0140] Cross-language transfer efficiency (CTE) is an important innovation of the present invention. The CTE of Example 1 reaches 0.85, which is much higher than 0.42 of Comparative Example 2. This shows that the system can more effectively use the knowledge in Chinese and Nepali to improve the recognition effect of Tibetan. This efficient knowledge transfer capability makes the system particularly suitable for processing languages ​​with scarce resources, providing strong technical support for the protection of language diversity.

[0141] In terms of computational efficiency (CE), although the method slightly increases the computational complexity (20% more than in Comparative Example 1), this extra cost is completely worthwhile considering the significant improvement in recognition accuracy. Moreover, compared with the simple multilingual model (Comparative Example 2), the method only increases the computational complexity by 9%, but brings significant performance improvement.

[0142] These results fully demonstrate the superiority of the present invention in dealing with low-resource language speech recognition tasks. The method not only significantly improves the recognition accuracy, but also demonstrates excellent cross-language knowledge transfer capabilities. This means that even for languages ​​with extremely limited training data, the system can provide reliable recognition services by leveraging knowledge of related languages.

[0143] The best embodiment is undoubtedly embodiment 1, which fully demonstrates all the advantages of the present invention. In practical applications, it is recommended to adopt this complete solution, especially when dealing with minority languages ​​with extremely scarce language resources. For example, for some endangered languages ​​with less than 10,000 speakers, a system may be the only feasible solution to achieve reliable speech recognition.

[0144] In general, these test results strongly support the innovative concept of the company: through the fusion of multi-source corpora and advanced feature fusion technology, the speech recognition performance of low-resource languages ​​can be significantly improved. This is not only a technological advancement, but also provides a powerful tool for the protection of language equality and cultural diversity.

Claims

1. A low-resource language speech recognition system using multi-source corpus fusion, characterized by: It includes a multi-source language framework, which uses the annotated speech data of high-resource languages ​​and low-resource languages ​​to train a model for recognizing the speech of the low-resource language. The multi-source language framework includes three parts: an acoustic layer, a language layer, and a pronunciation layer, wherein: The acoustic layer maps each acoustic feature to a common vocal tract coding subspace representing each language-independent feature and a vocal tract residual space representing language-specific information by means of ASVD. The former is used to predict the phoneme probability distribution of each frame, and the latter is used to predict the phoneme probability distribution of each frame of each language. The decoding module propagates the probability distribution in each dimension through the speech network, the language network and the pronunciation network, and outputs the final prediction result. The acoustic model uses ASVD to divide the training data into two parts, which are composed of a common channel encoder and a channel residual encoder. The former uses one ASVD component for acoustic training, and the latter uses the remaining ASVD components for training, thereby obtaining the ability to extract a variety of acoustic information. Language model, which is used to model the uncertainty of language and improve the effect of language perception tasks; The pronunciation model aligns each phoneme with a different language marker, increasing the model's language awareness while avoiding confusion between phonemes and languages; Feature fusion schemes, including feature fusion of multi-scale contrastive learning, hierarchical attention alignment, adversarial feature decoupling and reorganization, and graph neural network enhanced cross-language knowledge propagation.

2. The low-resource language speech recognition system according to claim 1, characterized in that: In the multi-scale contrastive learning, for the vocal tract coding features of speech, a multi-scale window is used in the ASVD acoustic features to constrain the common vocal tract encoder and the vocal tract residual encoder to obtain vocal tract coding characteristics of multiple scales in multiple languages; in the hierarchical attention alignment mechanism, the global and local alignment of multiple languages ​​is automatically learned based on the pronunciation module and the decoding model, and the hierarchical attention mechanism is used to align the vocal tract coding features between different languages.

3. The low-resource language speech recognition system according to claim 2, characterized in that: In the feature fusion scheme, the technology of adversarial feature decoupling and recombination is specifically, in a common vocal channel coding space, adversarial learning and a nonlinear decoder are used to separate and reorganize the speech layer and language layer features of the common acoustic vocal channel coding through decoupling learning; and the feature fusion technology of graph neural network enhanced cross-language knowledge propagation is specifically, in the decoding of each layer, a graph neural network based on regularization and orthogonal constraints is used to propagate and fuse language-specific vocal channel residual features.

4. The low-resource language speech recognition system according to claim 1, characterized in that: Specifically, for a low-resource language ι1 and a high-resource language ι2, the training data ν of each speech is divided into a common channel encoder V by ASVD. l:c and channel residual encoder V l:r Two parts; then train the common channel encoder parameters V at the same time l:c ~p(V l:c ) and the channel residual encoder parameters V l:r ~p(V l:r ) and a common model d(V l:c θ l:c and a language model dv(V l :r;θ v ); Different languages ​​ι correspond to different channel residual encoder parameters V during training l:r and language model parameters θ v ; The two languages ​​ι1 and ι2 can simultaneously train the common channel encoder and the channel residual encoder, and effectively fuse the information through the feature fusion technology.

5. A training method for a low-resource language speech recognition system comprising the multi-source corpus fusion method according to any one of claims 1 to 4, characterized in that: The method comprises the following steps: Step 1), based on the multi-source language framework, design a feature fusion scheme of multi-scale contrastive learning, hierarchical attention alignment mechanism, adversarial feature decoupling and reorganization, and cross-language knowledge propagation enhanced by graph neural network, and use the scheme to train the encoder of low-resource language l, where X is a high-resource language dataset and Yl is a low-resource language dataset; Step 1-1), design a multi-scale contrastive learning framework for extracting acoustic information of the channel coding layer, wherein a fixed-size sliding window is used to sample from a fixed-size input channel coding; then, the corresponding channel coding is divided into multiple segments, and the common channel encoder and the language channel encoder respectively use these coding segments to predict the implicit semantic information of each channel coding of each frame; Step 1-2), learn multilingual alignment from both global and local perspectives through a hierarchical attention alignment mechanism, design a self-supervised MAB algorithm to find global and local alignment matrices, and use the MAB algorithm to learn the alignment matrix; Steps 1-3), in multi-language acoustic features, using adversarial learning and decoders to adaptively split and reassemble vocal tract coding features; Steps 1-4), cross-language knowledge propagation feature fusion technology using graph neural network enhanced cross-language knowledge propagation; Step 2) For the trained encoder, the output of the decoder is obtained according to its characteristics.

6. The training method according to claim 5, characterized in that: The design of multi-scale contrastive learning in step 1-1) includes first using ASVD to divide each language in the training dataset V into a common channel encoder V lcc and the channel residual encoder V for each language l:r : V lc =V l -V l:r ,l∈{l1,l2} In addition, the MAB approach is used to learn the decoding matrix A for each frame of each language l and the common channel encoding and decoding matrix A cl For decoding common channel coded semantic vectors: from l =A l ·In l:c from cl =A cl ·In l:c Decompose each channel encoding vector into a common channel encoding phoneme embedding and a channel residual phoneme embedding: yes l =with l +with cl The matching loss is used to calculate the loss between the residual representation of the common channel encoding and the residual representation of the residual channel encoding; the contrast loss is used to calculate the matching loss between the phoneme embedding of the common channel encoding and the phoneme embedding of the channel residual, where V is the training dataset, ι is the language identifier, ι1 and ι2 represent low-resource language and high-resource language respectively, and V l:c For common channel encoding, V l :r is the channel residual coding, A l is the decoding matrix for each language and each frame, A cl is the common channel encoding and decoding matrix, z l Encode semantic vectors for language-specific vocal channels, z d Encode semantic vector for common channel, e l is the final channel encoding vector.

7. The training method according to claim 6, characterized in that: The step 1-2) is to make each language and decoder learn language context features, construct two hierarchical global and local attention alignment matrices in the common channel coding space, and make the decoder learn the attention of multiple languages; find the two language and decoding hierarchical matrices A={A l ,0,…,A l ,N} and {A l,c ,0,…,A l,c ,N}, through the MAB algorithm, learn the alignment matrix; where A is the alignment matrix set, A l ,i(i=0,…,N) is the i-th layer local attention alignment matrix of the l-th language, A l ,c,i(i=0,…,N) is the i-th layer global attention alignment matrix of the l-th language, N is the total number of layers, and the MAB algorithm is used to learn these alignment matrices.

8. The training method according to claim 7, characterized in that: The steps 1-3) include adversarial learning, allowing the learned decoder to generate channel residual embedding, and calculating the final channel residual embedding through the decoder model; The generative adversarial network consists of three parts: a reconstructor, a generator, and a discriminator. The generator reconstructs the channel residual embedding from the common encoder embedding, while the latter receives the channel residual embedding and the common channel encoding generated by the reconstructor to determine whether the common encoding comes from the reconstructor or the discriminator. In the adversarial loss, the reconstructor is trained by minimizing the adversarial loss between the reconstructor and the generator, and the generator is trained by minimizing the adversarial loss between the generator and the discriminator. The channel residual embedding uses a function to decouple learning. A separation function is used to decouple the learned vocal tract residual embedding.

9. The training method according to claim 8, characterized in that: For the channel residual embedding, decoupling learning is performed through function f, which uses the MAB method to learn the alignment matrix to separate language features; the contrast loss is used to constrain the embedding loss between the language decoder and the channel residual embedding frame; decoupling learning is implemented using the contrast loss function Lcontrast: Among them, f is the decoupling function, e1 and e l′ is the vocal tract residual embedding of different enhanced versions of the same speech, e k is the channel residual embedding of other samples in the batch, sim() is the cosine similarity function, and τ is the temperature parameter.

10. The training method according to claim 7, 8 or 9, characterized in that: The enhanced cross-language knowledge propagation feature fusion technology in step 1-4) includes a graph neural network enhanced cross-language knowledge propagation feature fusion technology. After extracting the common channel code of each language, a graph neural network is designed using the common channel code set of each decoupling layer for all languages, and the common channel code matrix C is trained using SVD and ASVD. cl And each channel coding matrix C lr , and finally use the encoding matrix C after matrix decomposition lc and C cr ; Finally, another matching loss and contrastive cosine loss are used at each layer to further learn each language: Among them, C cl is the common channel coding matrix, C lr is the language-specific channel encoding matrix, C lc and C cr is the encoding matrix after matrix decomposition, ||·||F represents the Frobenius norm, cos() is the cosine similarity function, and λ is the weight parameter; The step 2 also includes using a reconstruction model based on a decoding network to reconstruct the speech segment, and adopting a method of minimizing the ASVD reconstruction loss to solve the open set problem, specifically using an ASVD encoder encoding step; for the speech input D* in the test set, the trained model performs recognition to generate a recognition result of the target low-resource language l based on multi-source learning.

Images