Pre-training method, system and electronic device of multilingual self-supervised model

Abstract

Embodiments of the present application provide a multilingual self-supervised model pre-training method, system and electronic equipment. The method comprises: inputting unpaired unsupervised speech data selected from a multilingual data set to a language identification network to build a language classifier; extracting target language speech embedding and speech sentence embedding from a target language speech and speech sentence set; determining a training difficulty standard of extended dynamic curriculum learning of the multilingual self-supervised model based on an initial running loss of the multilingual self-supervised model, the target language speech embedding and the speech sentence embedding; and the language classifier performing pre-training of the multilingual self-supervised model in extended dynamic curriculum learning based on the training difficulty standard and a training set of different data amounts dynamically determined from the multilingual data set. The method of the embodiments of the present application makes the multilingual data set more efficient, eliminates potential harmful data on low-resource target language speech, and improves the performance of multilingual self-supervised learning downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech, and more particularly to a pre-training method, system, and electronic device for a multilingual self-supervised model. Background Technology

[0002] Thanks to abundant training pairing data, E2E (end-to-end) ASR (automatic speech recognition) models achieve excellent results on languages ​​with rich resources. However, when E2E ASR models are applied to low-resource languages, their performance drops significantly because prepared corpora are almost unavailable for low-resource languages.

[0003] To address the aforementioned issues, existing technologies typically employ the following methods: self-supervised speech processing models; masked speech modeling; and self-supervised pre-training data selection methods based on discrete representations.

[0004] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0005] Self-supervised speech processing models treat all unsupervised data as equally important, without considering the importance and relevance of the data. The methods described above train a very large, general-purpose model that lacks specificity for certain languages.

[0006] Mask-based speech modeling focuses more on frame-level data selection and less on sentence-level selection, thus failing to effectively reduce data volume and training time.

[0007] The technique of filtering data using language models based on discrete speech representations relies more on the accuracy of discrete speech representations, and the selection of language models based on machine labeling is not the best way to select data. Summary of the Invention

[0008] In order to at least address the problems in existing technologies where pre-training does not distinguish between the similarity between languages ​​or sentences, does not select data based on the sentences themselves, and does not effectively reduce the amount of training data and improve training efficiency.

[0009] In a first aspect, embodiments of the present invention provide a pre-training method for a multilingual self-supervised model, comprising:

[0010] Unpaired unsupervised speech data will be selected from the multilingual dataset and input into the language recognition network. A language classifier will be constructed based on the bottleneck vector containing speech sentence information extracted from the unsupervised data.

[0011] Determine the target language speech in the multilingual dataset, extract a speech sentence set from the target language speech, and extract the corresponding target language speech embedding and speech sentence embedding from the target language speech and the speech sentence set;

[0012] The training difficulty standard for the extended dynamic curriculum-based learning of the multilingual self-supervised model is determined based on the preset initial running loss of the multilingual self-supervised model, the target language speech embedding, and the speech sentence embedding.

[0013] The multilingual self-supervised model is pre-trained using extended dynamic curriculum-based learning by utilizing training sets of different amounts of data dynamically determined from the multilingual dataset based on the training difficulty criteria by the language classifier.

[0014] Secondly, embodiments of the present invention provide a pre-training system for a multilingual self-supervised model, comprising:

[0015] The language classifier building module is used to select unpaired unsupervised speech data from a multilingual dataset and input it into a language recognition network, and build a language classifier based on the bottleneck vector containing speech sentence information extracted from the unsupervised data;

[0016] An embedding determination module is used to determine the target language speech in the multilingual dataset, extract a speech sentence set from the target language speech, and extract the corresponding target language speech embedding and speech sentence embedding from the target language speech and the speech sentence set.

[0017] An extended dynamic curriculum-based learning program module is used to determine the training difficulty standard for the extended dynamic curriculum-based learning of the multilingual self-supervised model based on the preset initial running loss of the multilingual self-supervised model, the target language speech embedding, and the speech sentence embedding.

[0018] The pre-training module is used to perform extended dynamic curriculum-based pre-training on the multilingual self-supervised model by using training sets of different amounts of data dynamically determined from the multilingual dataset based on the training difficulty criteria by the language classifier.

[0019] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a pre-training method for a multilingual self-supervised model according to any embodiment of the present invention.

[0020] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the pre-training method for a multilingual self-supervised model according to any embodiment of the present invention.

[0021] The beneficial effects of this invention are as follows: This method trains a language classifier using a small amount of unsupervised data and calculates the similarity between sentences using bottleneck vectors containing more speech sentence information. The language classifier allows for more efficient use of multilingual datasets and eliminates potentially harmful data in low-resource target language speech, while also significantly improving the performance of downstream tasks in multilingual self-supervised learning. Furthermore, based on data similarity selection, a curriculum-based learning method is employed to allow the model to learn from simple data to more difficult data. The difficulty of the data is assessed based on the model's forward propagation loss and sentence-level similarity weighting; this curriculum-based learning method makes the model converge more easily. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a pre-training method for a multilingual self-supervised model provided in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of multilingual speech sentence embedding, which is a pre-training method for a multilingual self-supervised model provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the architecture of a pre-training method for a multilingual self-supervised model provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the total duration and distribution of multilingual pre-training data for a pre-training method of a multilingual self-supervised model provided in an embodiment of the present invention (the histogram shows the language distribution of selected pre-training data for each target language speech under different data selection thresholds);

[0027] Figure 5 This is a schematic diagram comparing the word error rates of different pre-training data in the selection of a pre-training method for a multilingual self-supervised model according to an embodiment of the present invention.

[0028] Figure 6This is a schematic diagram illustrating the changes in relative word error rate and convergence steps during the training process of a pre-training method for a multilingual self-supervised model provided in an embodiment of the present invention.

[0029] Figure 7 This is a schematic diagram of the structure of a pre-training system for a multilingual self-supervised model provided in an embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram of an embodiment of an electronic device for pre-training a multilingual self-supervised model, as provided in one embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] like Figure 1 The diagram shown is a flowchart of a pre-training method for a multilingual self-supervised model according to an embodiment of the present invention, including the following steps:

[0033] S11: Select unpaired unsupervised speech data from the multilingual dataset and input it into the language recognition network. Construct a language classifier based on the bottleneck vector containing speech sentence information extracted from the unsupervised data.

[0034] S12: Determine the target language speech in the multilingual dataset, extract a speech sentence set from the target language speech, and extract the corresponding target language speech embedding and speech sentence embedding from the target language speech and the speech sentence set;

[0035] S13: Based on the preset initial running loss of the multilingual self-supervised model, the target language speech embedding, and the speech sentence embedding, determine the training difficulty standard for the extended dynamic curriculum learning of the multilingual self-supervised model.

[0036] S14: Using the language classifier to perform extended dynamic curriculum-based pre-training on the multilingual self-supervised model, training sets of different data volumes dynamically determined from the multilingual dataset based on the training difficulty standard.

[0037] For step S11, this method found that an LID (Language Identification) network trained with a small number of unpaired voices in a multilingual dataset can effectively distinguish data from different languages.

[0038] In one implementation, the language recognition network is constructed from a time-delay neural network.

[0039] In this embodiment, a Time Delay Neural Network (TDNN) is used to construct the LID network, and a language classifier is built by extracting bottleneck features from the sentences in the dialogue. The constructed language classifier can then be used to evaluate the similarity between the dialogue and the target language speech based on the speech sentence embedding and the target language speech embedding.

[0040] A constructed language classifier is used to select data from a multilingual dataset for training a multilingual self-supervised model. Regarding data selection, existing techniques primarily focus on language-level pre-training data selection, which involves excluding all data from languages ​​not selected during the pre-training process. However, this method argues that this approach may not be ideal, as it may lead to the discarding of potentially useful data. The multilingual dataset used in this method consists of 400 audio recordings from each of five languages: Catalan (ca), Basque (eu), French (fr), Italian (it), and Portuguese (pt).

[0041] For step S12, the target language speech is represented as l, and l is embedded as E. l Let L be the set of all dialogues in l, and let E be the embedding of dialogue u in the set of all dialogues. u .

[0042] Based on the determined target language speech embedding E l and speech sentence embedding E u Determine E u and E l The similarity between them was evaluated as their cosine distance:

[0043]

[0044] Language embedding E l Defined as the average embedding of all dialogues u∈L:

[0045]

[0046] After extracting the target language speech embedding E l and speech sentence embedding Eu Next, T-SNE (t-distributed stochastic neighbor embedding) analysis was performed to visualize the linguistic distance between dialogues in different languages. The results of the analysis are as follows: Figure 2 As shown. Although Italian and French embeddings are generally more similar, this method found that the number of Portuguese embeddings is within the range of Italian. Furthermore, a small subset of embeddings from each language is distributed throughout the embedding space, suggesting that dialogue-level selection produces more plausible data than language-level selection in terms of similarity to the target language speech, regardless of the actual language. This method further provides a quantitative analysis of the actual language distribution of selected data similar to specific target language speech in the experiments.

[0047] For step S13, prior art pre-training typically uses DCL (Dynamic Curriculum Learning). In DCL, the difficulty metric is evaluated by dynamically monitoring the running loss throughout the training process. However, this method finds that DCL randomly selects a subset of training data at the start of training, which may result in subsets of unknown difficulty appearing in the critical early stages of training.

[0048] To address the aforementioned issues, this method improves upon DCL, extending it to EDCL (Extended Dynamic Curriculum Learning), which incorporates an additional language similarity term in its difficulty assessment. This modification helps create a more reasonable training subset in the early stages of training, when operational loss is still unstable. Specifically, this method applies the training difficulty criterion H of dialogue u... u Defined as the target language speech l and the corresponding length-normalized running loss L u Weighted sum of embedding distances between:

[0049]

[0050] The training difficulty standard H is obtained through the above method. u .

[0051] For step S14, in the initial stage of training, this method will set the initial running loss L. u Set to 0, and determine the initial subset for training based on dialogue-level language similarity. Periodically use H... uThe difficulty of the data is reassessed, and the training set is adjusted accordingly. This process allows EDCL to gradually include more challenging data as training progresses. This method limits the size of the dynamically determined training set to exclude data with low similarity to the target language speech and high runtime loss, which could be detrimental to the training process. The overall structure of this method is as follows: Figure 3 As shown.

[0052] The above steps are described in detail in the experimental setup. Regarding the data, the experiments were conducted using the COMMONVOICE dataset, which is a large multilingual speech corpus primarily composed of articles from Wikipedia. This method selected five languages ​​for the experiments: Catalan (ca), Basque (eu), French (fr), Italian (it), and Portuguese (pt), totaling 1145 hours. Figure 4 The total duration section summarizes the data duration for each language. Regarding data selection, this method investigates different percentages of the selected data. The selected dataset for the target language speech l is... Where R∈{12.5%, 25%, 50%, 100%} is the percentage of the selected data. It should be noted that for a specific target language speech l, Always effective.

[0053] The LID network consists of a TDNN and a classifier. A three-layer fully connected classifier with a ReLU (Rectified Linear Unit) activation function is used, employing an Adam optimizer with a learning rate of 0.0005. To train the language classifier, only three hours of data per language are selected as training data, with the number of time periods set to 100. Training the language classifier can be efficiently completed in two hours using only a single GPU.

[0054] This method uses FairSeq as a toolkit for pre-training and fine-tuning experiments. During the pre-training phase, a base model with a 12-layer structure, wav2vec2.0, is used, following the hyperparameters of the base model, with a maximum update size of 400k and a learning rate of 0.0002.

[0055] As one implementation method, after pre-training the multilingual self-supervised model with extended dynamic curriculum learning, the method further includes:

[0056] The multilingual self-supervised model is subjected to connectionist temporal classification.

[0057] In this implementation, for fine-tuning, the method selects the highest-performing pre-trained model checkpoints to fine-tune the target language speech. CTC (Connectionist Temporal Classification) was used and fine-tuned, updating it 20k times with a learning rate of 0.00005. To create a low-resource setting, only 10 hours of target language speech pairing data were used for fine-tuning.

[0058] As one implementation method, the training strategy for pre-training of the extended dynamic curriculum-based learning includes:

[0059] A first training strategy for pre-training the multilingual self-supervised model using a training set comprising 25% of the data determined from the multilingual dataset based on the training difficulty criterion by the language classifier, and...

[0060] A second training strategy is to pre-train the multilingual self-supervised model using a training set dynamically determined from the multilingual dataset based on the training difficulty standard, with the training set size increased from 25% to 50%.

[0061] In this implementation, the experiment explored two different EDCL strategies, denoted as EDCL. 25％ and EDCL 25％→50％ The former strategy involves adjusting the training set during each EDCL evaluation while maintaining a fixed percentage of data selection at 25%. The latter strategy starts with the initial dataset D. 25％ Start by gradually increasing the selected data to D. 50％ EDCL (25％→50％) The percentage changes of the selected data R are as follows:

[0062]

[0063] Where t represents the number of training updates, a0 represents the initial data volume, and β represents the update interval between two EDCL evaluations. For the data selection criteria in each evaluation, α is set to 0.1, and σ is set to 0.9.

[0064] This method reconstructs the language model using KenLM's 4-gram language model, training on the text portion of each language. A beam size of 50 is used for the Wav2letter++ beam search decoder. Experiments validate that the language model weights and word insertion penalty are set to 3.2 and -0.8, respectively. Experiments show that with 25% of the data, this method can quickly and efficiently pre-train multilingual self-supervised models with low resources and achieve relatively good performance. As the data volume increases from 25% to 50%, this method can efficiently pre-train multilingual self-supervised models with relatively low resources and achieve even better performance.

[0065] As can be seen from this implementation, this method trains a language classifier using a small amount of unsupervised data and calculates sentence similarity using bottleneck vectors containing more speech and sentence information. The language classifier allows for more efficient use of multilingual datasets and eliminates potentially harmful data in low-resource target languages, while also significantly improving the performance of downstream tasks in multilingual self-supervised learning. Furthermore, based on data similarity selection, a curriculum-based learning method is employed to allow the model to learn from simple to difficult data. The difficulty of the data is assessed by weighting the loss from forward propagation and sentence-level similarity, making the model converge more easily.

[0066] Regarding the experimental results of this method Figure 4 The table shows the language distribution of the selected pre-training data for each target language speech at different data selection thresholds (12.5%, 25%, 50%) (where the histograms are from top to bottom: ca, eu, fr, it, pt). Although the imbalance in the total duration of different languages ​​makes analyzing language similarity from the histograms challenging, this method still yields some conclusions consistent with T-SNE results. For example, although the total amount of Catalan (ca) and French (fr) data is similar, at 12.5%, Catalan (ca) is selected for Basque (eu). This indicates that Catalan (ca) is more similar to Basque (eu) than French (fr), consistent with the T-SNE embedding distance. Similarly, although the total amount of Italian (it) and Basque (eu) data is comparable, Italian (it) is selected for French (fr).

[0067] This method presents results by selecting different amounts of data based on the language similarity criterion of the aforementioned cosine distance formula, such as... Figure 5 As shown in the first four lines. For ease of comparison, Figure 6The chart shows the relative reduction in WER and the average number of convergence steps for the five languages ​​at different data percentages. It can be observed that reducing the data size from 100% to 50% results in a decrease in WER, but no decrease in convergence steps, indicating an improvement in data quality, but some of the selected data may be hindering convergence. Further reducing the data to 25% leads to a decrease in both convergence steps and a further reduction in WER, demonstrating the effectiveness of the data selection. However, when the data is reduced to 12.5%, a significant performance drop can be observed due to the exclusion of a large amount of beneficial training data.

[0068] Furthermore, this method also found that the quality of the Italian data was relatively poor, with a large proportion of non-Italian symbols. Experimental results also showed that retraining the Italian language with the language model did not significantly improve scores. It was also noted that the degree of improvement achieved through data selection varied by language and was related to the duration D of that language in the original dataset. 100％ Related. For example, D 100％ The data for Catalonia, which has the most data, comes from D. 100％ Reduce to D 25％ This resulted in a relative improvement of 13% / 11% for the development / test set. However, the improvement rate for Portuguese increased to 22% / 20%, which is D 100％ The one with the fewest data points. Assume D. 100％ Includes more noisy data specific to the language, while D 100％ There is relatively little data available for this type of language. Therefore, data selection is even more critical for this type of language, as it can filter out more harmful data. This observation applies to all languages ​​except Italian, which has noisy data in its own dataset.

[0069] like Figure 5 The last two lines demonstrate the effectiveness of incorporating the EDCL strategy to improve ASR (Automatic Speech Recognition) performance and reduce the number of steps required for convergence. When trained with a fixed 25% of the data, the third line shows that using EDCL25% results in 55k fewer convergence steps compared to training without a curriculum-based learning strategy, while maintaining ASR performance. This indicates that the criterion proposed in the weighted sum formula for embedding distance is a useful estimate of dialogue difficulty, enabling the model to converge more efficiently.

[0070] In such Figure 5 In the last line, this method uses EDCL. 25％→50％ The training set was selected incrementally from 25% to 50%. This led to a further improvement in ASR performance. Compared to the second row, which used 50% of the data without a dedicated course, EDCL... 25％→50％This achieves better convergence steps and WER performance. This demonstrates that the EDCL strategy provides a more efficient method for ASR pre-training with limited training data, thereby improving performance in fewer steps.

[0071] The training efficiency and ASR performance of this method were analyzed. Figure 6 This shows the WER improvement relative to the baseline, which is D 100％ The pre-trained data, as well as the actual update steps before convergence. Each column group in the graph corresponds to, for example, the pre-trained data and the actual update steps before convergence. Figure 5 The average result for each row in the table.

[0072] The first three columns represent the trade-off between convergence steps and ASR performance. As the amount of selected data decreases from 50% to 12.5%, the number of steps required for model convergence decreases, but this comes at the cost of decreased ASR performance. While this trade-off may be unavoidable, a better trade-off is achieved by incorporating a curriculum-based learning strategy, as shown in the last two groups. This strategy effectively reduces the difficulty of the early stages of pre-training, making it easier for the model to converge in fewer steps. The comparison between the first and last columns best illustrates this point.

[0073] The proposed EDCL strategy enables more efficient and effective pre-training of ASR for low-resource target language speech, reducing training steps and improving speech recognition performance.

[0074] In summary, this paper proposes a pre-training data selection method for SSL (self-supervised learning) based on dialogue-level language similarity. This method effectively reduces the amount of data required and the number of training steps, while eliminating potentially harmful ASR data on low-resource target language speech. Furthermore, this method introduces a curriculum-based learning strategy that utilizes specialized estimates of language similarity and churn as criteria for dialogue difficulty. Experiments were conducted on five languages ​​in a 10-hour low-resource setting on the COMMONVOICE dataset. By combining these techniques, superior performance is achieved on low-resource target language speech, requiring only 25% of the data for pre-training compared to a baseline where all data was used for pre-training, saving 60% of the training steps. Experimental results highlight the effectiveness and efficiency of this method in ASR SSL pre-training and its potential applications in various other SSL tasks.

[0075] like Figure 7 The diagram shown is a structural schematic of a pre-training system for a multilingual self-supervised model provided in an embodiment of the present invention. The system can execute the pre-training method for the multilingual self-supervised model described in any of the above embodiments and is configured in a terminal.

[0076] This embodiment provides a pre-training system 10 for a multilingual self-supervised model, which includes: a language classifier construction module 11, an embedding determination module 12, an extended dynamic curriculum learning module 13, and a pre-training module 14.

[0077] The language classifier construction module 11 is used to select unpaired unsupervised speech data from the multilingual dataset and input it into the language recognition network, and construct a language classifier based on the bottleneck vector containing speech sentence information extracted from the unsupervised data; the embedding determination module 12 is used to determine the target language speech in the multilingual dataset, extract a set of speech sentences from the target language speech, and determine the target language speech embedding and the speech sentence embedding of the speech sentence set; the extended dynamic curriculum learning module 13 is used to determine the training difficulty standard of the extended dynamic curriculum learning of the multilingual self-supervised model based on the preset initial running loss of the multilingual self-supervised model, the target language speech embedding, and the speech sentence embedding; the pre-training module 14 is used to pre-train the multilingual self-supervised model with extended dynamic curriculum learning using training sets of different amounts of data dynamically determined from the multilingual dataset based on the training difficulty standard.

[0078] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the pre-training method of the multilingual self-supervised model in any of the above method embodiments.

[0079] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0080] Unpaired unsupervised speech data will be selected from the multilingual dataset and input into the language recognition network. A language classifier will be constructed based on the bottleneck vector containing speech sentence information extracted from the unsupervised data.

[0081] Determine the target language speech in the multilingual dataset, extract a speech sentence set from the target language speech, and extract the corresponding target language speech embedding and speech sentence embedding from the target language speech and the speech sentence set;

[0082] The training difficulty standard for the extended dynamic curriculum-based learning of the multilingual self-supervised model is determined based on the preset initial running loss of the multilingual self-supervised model, the target language speech embedding, and the speech sentence embedding.

[0083] The multilingual self-supervised model is pre-trained using extended dynamic curriculum-based learning by utilizing training sets of different amounts of data dynamically determined from the multilingual dataset based on the training difficulty criteria by the language classifier.

[0084] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the pre-training method of the multilingual self-supervised model in any of the above method embodiments.

[0085] Figure 8 This is a schematic diagram of the hardware structure of an electronic device for a pre-training method of a multilingual self-supervised model provided in another embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0086] One or more processors 810 and memory 820, Figure 8 Taking a processor 810 as an example, the device for pre-training a multilingual self-supervised model may also include an input device 830 and an output device 840.

[0087] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0088] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the pre-training method of the multilingual self-supervised model in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the pre-training method of the multilingual self-supervised model in the above-described method embodiments.

[0089] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0090] Input device 830 can receive input numerical or character information. Output device 840 may include display devices such as a display screen.

[0091] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, they execute the pre-training method of the multilingual self-supervised model in any of the above method embodiments.

[0092] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0093] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0094] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the pre-training method for a multilingual self-supervised model according to any embodiment of this invention.

[0095] The electronic devices described in this application exist in various forms, including but not limited to:

[0096] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0097] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0098] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0099] (4) Other electronic devices with data processing functions.

[0100] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pre-training method for a multilingual self-supervised model, comprising: Unpaired unsupervised speech data will be selected from the multilingual dataset and input into the language recognition network. A language classifier will be constructed based on the bottleneck vector containing speech sentence information extracted from the unsupervised speech data. Determine the target language speech in the multilingual dataset, extract a speech sentence set from the target language speech, and extract the corresponding target language speech embedding and speech sentence embedding from the target language speech and the speech sentence set; Based on the similarity between the target language speech embedding and the speech sentence embedding, and the running loss of the multilingual self-supervised model, the training difficulty standard for the extended dynamic curriculum learning of the multilingual self-supervised model is determined by weighted summation. In the initial stage of pre-training, the initial running loss is set to zero, and the initial training set is determined based on the similarity. The multilingual self-supervised model is pre-trained using extended dynamic curriculum-based learning by utilizing training sets of different amounts of data dynamically determined from the multilingual dataset based on the training difficulty criteria by the language classifier.

2. The method according to claim 1, wherein, The training strategies for pre-training in the extended dynamic curriculum-based learning include: A first training strategy for pre-training the multilingual self-supervised model using a training set comprising 25% of the data determined from the multilingual dataset based on the training difficulty criterion by the language classifier, and... A second training strategy is to pre-train the multilingual self-supervised model using a training set dynamically determined from the multilingual dataset based on the training difficulty standard, with the training set size increased from 25% to 50%.

3. The method according to claim 1, wherein, After pre-training the multilingual self-supervised model with extended dynamic curriculum learning, the method further includes: The multilingual self-supervised model is subjected to connectionist temporal classification.

4. The method according to claim 1, wherein, The language recognition network is constructed from a time-delay neural network.

5. A pre-training system for a multilingual self-supervised model, comprising: The language classifier building module is used to select unpaired unsupervised speech data from a multilingual dataset and input it into a language recognition network, and build a language classifier based on the bottleneck vector containing speech sentence information extracted from the unsupervised speech data. An embedding determination module is used to determine the target language speech in the multilingual dataset, extract a speech sentence set from the target language speech, and extract the corresponding target language speech embedding and speech sentence embedding from the target language speech and the speech sentence set. An extended dynamic curriculum-based learning module is used to determine the training difficulty standard of the extended dynamic curriculum-based learning of the multilingual self-supervised model by weighted sum based on the similarity between the target language speech embedding and the speech sentence embedding, and the running loss of the multilingual self-supervised model. In the initial stage of pre-training, the initial running loss is set to zero, and the initial training set is determined based on the similarity. The pre-training module is used to perform extended dynamic curriculum-based pre-training on the multilingual self-supervised model by using training sets of different amounts of data dynamically determined from the multilingual dataset based on the training difficulty criteria by the language classifier.

6. The system according to claim 5, wherein, The training strategies for pre-training in the extended dynamic curriculum-based learning include: A first training strategy for pre-training the multilingual self-supervised model using a training set comprising 25% of the data determined from the multilingual dataset based on the training difficulty criterion by the language classifier, and... A second training strategy is to pre-train the multilingual self-supervised model using a training set dynamically determined from the multilingual dataset based on the training difficulty standard, with the training set size increased from 25% to 50%.

7. The system according to claim 5, wherein, The system also includes a fine-tuning module for performing connectionist temporal classification on the multilingual self-supervised model.

8. The system according to claim 5, wherein, The language recognition network is constructed from a time-delay neural network.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-4.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-4.

Images