Multilingual speech recognition methods, devices, equipment and media

Abstract

This invention relates to the fields of artificial intelligence and fintech, and discloses a multilingual speech recognition method. The method involves: acquiring a pre-trained basic speech recognition model; configuring adapters for each language; configuring multiple language feature expert modules; acquiring training speech data; selecting the appropriate language feature expert module and inputting speech samples; determining the target low-resource language; freezing the parameters of other adapters; training only the adapter for the target language; selecting the corresponding adapter based on language identifiers and training it; and continuing training on all batches of data to generate the target speech recognition model. This invention, by configuring language-dependent adapters and language feature expert modules, allows for targeted processing of the characteristics of different languages, significantly improving the model's adaptability and recognition accuracy. By freezing adapters for non-target languages, unnecessary resource consumption is reduced, focusing on optimizing the target low-resource language and ensuring the efficiency of the training process.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a multilingual speech recognition method, apparatus, device and storage medium. Background Technology

[0002] In the financial sector, Automatic Speech Recognition (ASR) systems are widely used in customer service, voice banking, and real-time transaction monitoring. Multilingual Automatic Speech Recognition (MASR) systems, in particular, can handle voice input in different languages, helping financial institutions meet the needs of multilingual customers. However, existing multilingual ASR systems have technical shortcomings in several key areas, limiting their widespread application in the financial industry.

[0003] To improve recognition accuracy, existing multilingual ASR systems typically require increasing model size. However, as the number of model parameters increases—for example, the USM model reaches 2 billion parameters and the MMS model exceeds 1 billion parameters—the models become enormous and difficult to run efficiently. These massive models face high computational resource consumption and significant latency issues when processing real-time voice data in the financial sector, failing to meet the real-time response requirements of streaming processing. This latency particularly negatively impacts applications requiring timely decision-making, such as high-frequency financial trading and real-time risk monitoring.

[0004] In multilingual speech recognition tasks, existing technologies often face performance disparities between different languages. Due to differences in data distribution, vocabulary size, and language structure for each language, a single model cannot balance performance across all languages ​​when processing multiple languages ​​simultaneously, resulting in some languages ​​performing significantly worse than others. This asynchronous performance can impact customer experience in financial services, particularly for multilingual customer service, where service quality may lag noticeably in certain languages. Furthermore, many existing systems rely on the mean word error rate (WER) as a performance metric, but this average often masks performance deficiencies in specific languages, potentially leading to biases in practical applications.

[0005] With increasingly stringent privacy regulations, the collection and use of data in many languages ​​are restricted, resulting in a lack of sufficient data for effective training of existing models. This is particularly challenging in the financial sector, where processing low-resource languages ​​becomes especially difficult. Financial institutions often need to handle customer needs from diverse language backgrounds, but data resources for some languages ​​are scarce, making it difficult for existing technologies to effectively address this challenge.

[0006] When new languages ​​need to be supported, existing technologies typically require large-scale retraining and tuning of the entire model, significantly increasing deployment and maintenance complexity. In financial institutions, language scalability becomes particularly important as business scope expands and the demand for cross-border services grows. However, adding new languages ​​to existing models often requires substantial computational resources, increasing the complexity and cost of expanding the system and hindering rapid iteration and optimization. Summary of the Invention

[0007] The main objective of this invention is to provide a multilingual speech recognition method, apparatus, device, and storage medium, aiming to solve the technical problems of uneven performance of existing multilingual automatic speech recognition systems when processing different languages, and the difficulty in efficiently fine-tuning for low-resource languages, resulting in low training efficiency and poor model scalability.

[0008] To achieve the above objectives, the present invention provides a multilingual speech recognition method, comprising:

[0009] Obtain a pre-trained basic speech recognition model that can recognize multiple languages, and freeze the parameters of the basic speech recognition model;

[0010] Configure a language dependency adapter in the basic speech recognition model so that each language has a corresponding language dependency adapter;

[0011] Multiple language feature expert modules are configured in the basic speech recognition model, and each language feature expert module is used to process the corresponding language features.

[0012] Acquire training speech data, and divide the training speech data into multiple batches of data, each batch of data containing multiple speech samples;

[0013] For each speech sample in the training speech data, select a corresponding language feature expert module, and input the speech sample into the corresponding language feature expert module to obtain the output result;

[0014] Each speech sample in each batch of data is assigned a corresponding language identifier to indicate the language type to which the speech sample belongs;

[0015] Identify the target low-resource language, freeze the parameters of language dependency adapters that do not belong to the target low-resource language, so as to train and optimize only the parameters of the language dependency adapters corresponding to the target low-resource language;

[0016] Based on the language identifiers of the speech samples in each batch of data, a corresponding low-resource language dependency adapter is selected, and the corresponding low-resource language dependency adapter is trained based on the output results corresponding to the speech samples in order to optimize and adjust the parameters of the low-resource language dependency adapter.

[0017] The training continues until all batches of data have been completed, generating a trained target language dependency adapter to obtain the target speech recognition model.

[0018] Speech recognition is performed based on the target speech recognition model.

[0019] Furthermore, to achieve the above objectives, the present invention provides a multilingual speech recognition device, comprising:

[0020] The basic model acquisition module is used to acquire a pre-trained basic speech recognition model, which is capable of recognizing multiple languages, and to freeze the parameters of the basic speech recognition model.

[0021] The language adapter configuration module is used to configure a language dependency adapter in the basic speech recognition model so that each language has a corresponding language dependency adapter.

[0022] An expert module configuration module is used to configure multiple language feature expert modules in the basic speech recognition model, and each language feature expert module is used to process the corresponding language feature.

[0023] The data batch division module is used to acquire training speech data and divide the training speech data into multiple batches of data, each batch of data containing multiple speech samples.

[0024] The training data processing module is used to select a corresponding language feature expert module for each speech sample in the training speech data, and input the speech sample into the corresponding language feature expert module to obtain the output result.

[0025] The language identifier allocation module is used to assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0026] The low-resource language adapter control module is used to determine the target low-resource language and freeze the parameters of language dependency adapters that do not belong to the target low-resource language, so as to train and optimize only the parameters of the language dependency adapters corresponding to the target low-resource language.

[0027] The low-resource language adapter training module is used to select the corresponding low-resource language dependency adapter based on the language identifier of the speech samples in each batch of data, and to train the corresponding low-resource language dependency adapter based on the output results of the speech samples, so as to optimize and adjust the parameters of the low-resource language dependency adapter.

[0028] The adapter training completion module is used to generate a trained target language dependency adapter until all batches of data have been trained, so as to obtain the target speech recognition model.

[0029] A speech recognition module is used to perform speech recognition based on the target speech recognition model.

[0030] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multilingual speech recognition program stored in the memory and executable on the processor, wherein the multilingual speech recognition program, when executed by the processor, implements the steps of the multilingual speech recognition method as described above.

[0031] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multilingual speech recognition program, which, when executed by a processor, implements the steps of the multilingual speech recognition method as described above.

[0032] Beneficial Effects: This invention relates to the fields of artificial intelligence and fintech, and discloses a multilingual speech recognition method. By acquiring a pre-trained basic speech recognition model, the model can recognize multiple languages ​​and its parameters are frozen. Language-dependent adapters are configured, with each language corresponding to an adapter. Multiple language feature expert modules are configured to handle different language features. Training speech data is acquired, and the corresponding language feature expert module is selected and speech samples are input. The training speech data is divided into multiple batches, and a language identifier is assigned to each sample. The target low-resource language is determined, the parameters of other adapters are frozen, and only the adapter for the target language is trained. Based on the language identifier, the corresponding adapter is selected and trained until the training of all batches of data is completed, generating the target speech recognition model. This invention, by configuring language-dependent adapters and language feature expert modules, enables targeted processing of the characteristics of different languages, significantly improving the model's adaptability and recognition accuracy. By freezing adapters for non-target languages, unnecessary resource consumption is reduced, focusing on the optimization of the target low-resource language and ensuring the efficiency of the training process. Furthermore, the scalability of the model is enhanced, enabling rapid adaptation to the addition of new languages ​​and reducing the complexity of deployment and maintenance. Attached Figure Description

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0034] Figure 1 This is a schematic diagram of an application environment for a multilingual speech recognition method according to an embodiment of the present invention;

[0035] Figure 2 This is a flowchart illustrating an embodiment of the multilingual speech recognition method of the present invention;

[0036] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multilingual speech recognition device of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0038] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0039] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0040] The multilingual speech recognition method provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can obtain a pre-trained basic speech recognition model from the user terminal. The model can recognize multiple languages ​​and its parameters are frozen. Language dependency adapters are configured, with each language corresponding to an adapter. Multiple language feature expert modules are configured to handle different language features. Training speech data is acquired, the appropriate language feature expert module is selected, and speech samples are input. The training speech data is divided into multiple batches, and a language identifier is assigned to each sample. The target low-resource language is determined, the parameters of other adapters are frozen, and only the adapter for the target language is trained. Based on the language identifier, the corresponding adapter is selected and trained until all batches of data have been trained, generating the target speech recognition model. This invention, by configuring language dependency adapters and language feature expert modules, enables targeted processing of the features of different languages, significantly improving the model's adaptability and recognition accuracy. By freezing adapters for non-target languages, unnecessary resource consumption is reduced, focusing on the optimization of the target low-resource language and ensuring the efficiency of the training process. Furthermore, the model's scalability is enhanced, enabling rapid adaptation to the addition of new languages ​​and reducing the complexity of deployment and maintenance. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0041] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multilingual speech recognition method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0042] like Figure 2 As shown, the multilingual speech recognition method proposed in this invention includes the following steps:

[0043] S10, Obtain a pre-trained basic speech recognition model, which is capable of recognizing multiple languages, and freeze the parameters of the basic speech recognition model;

[0044] In this embodiment, a pre-trained basic speech recognition model is loaded from an existing model library. This model can handle speech input in multiple languages ​​and can be further fine-tuned or applied. Having undergone extensive pre-training, this model possesses a certain level of speech recognition capability, particularly in multilingual environments.

[0045] A pre-trained multilingual speech recognition model can be loaded using deep learning frameworks such as TensorFlow or PyTorch. The model's pre-training typically relies on large-scale multilingual speech datasets and is performed through supervised learning.

[0046] This model's ability to recognize multiple languages ​​means that its parameters, after specific training, are capable of processing and understanding the speech features of different languages. In multilingual recognition tasks, the model needs to identify the language type of the speech input and process it based on specific language characteristics.

[0047] The model is trained on multilingual datasets (such as CommonVoice and LibriSpeech) to learn the acoustic and linguistic features of multiple languages. The model is typically built based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformer architectures.

[0048] Freezing model parameters means that the parameters of the base model will remain unchanged during subsequent fine-tuning and will not be used in training. This effectively avoids readjusting the entire model and reduces the computational burden.

[0049] In training frameworks, you can freeze the parameters of certain layers by setting "requires_grad=False". For example, in PyTorch, you can iterate through the layers of the model and select the parts that need to be frozen to lock their parameters.

[0050] By loading a pre-trained multilingual base speech recognition model and freezing its parameters, the accuracy of existing models can be maintained while avoiding the need to repeatedly train large-scale base models, thus reducing computational resource consumption. This allows for focused fine-tuning of specific language parts, significantly improving training efficiency and deployment speed, especially demonstrating remarkable performance optimization in multilingual tasks.

[0051] S20, Configure a language dependency adapter in the basic speech recognition model so that each language is configured with a corresponding language dependency adapter;

[0052] In this embodiment, a language-dependent adapter is introduced into the basic speech recognition model. These adapters are designed to optimize for the characteristics of each language, independently processing speech input in different languages. By processing the acoustic and grammatical features of a specific language, the language-dependent adapter enhances the model's recognition capability in that language.

[0053] Multiple language adapter modules are incorporated into the model. Each adapter processes the input signal through projection onto a low-dimensional feature space and an activation function, then returns an optimized output. The adapter parameters are trained based on language-specific data to ensure they effectively capture the characteristics of that language. For example, adapters can be used to extract and process features of phonemes and intonation in certain languages.

[0054] Configure an adapter independently for each language, ensuring that voice input in each language is processed by a specific adapter, avoiding the situation where multiple languages ​​share a single adapter. This avoids interference between the characteristics of different languages, thereby improving their respective recognition accuracy.

[0055] In the model architecture, a mechanism is set up to automatically select the corresponding language adapter for processing after recognizing the input speech language. The adapter selection is based on the language identifier in the speech signal, so the system can dynamically switch to the appropriate adapter module.

[0056] Example Explanation: In global banking, customers in different countries may interact with the bank using multiple languages, such as through telephone banking or voice recognition systems. By configuring language-dependent adapters, the bank's voice system can automatically select the appropriate adapter for processing when it receives customer voice input in different languages. For example, when receiving a request from a Spanish user, the system can automatically switch to the Spanish adapter for recognition and processing, ensuring service accuracy and a consistent user experience.

[0057] By configuring independent language dependency adapters for each language, the system can be specifically optimized for the characteristics of each language, avoiding interference between different language features. This not only improves the system's efficiency in processing multiple languages ​​but also significantly enhances the recognition accuracy of each language. Furthermore, since each adapter is optimized independently, only minor adjustments are needed for adapters for newly added languages, reducing system complexity and computational resource requirements.

[0058] S30, multiple language feature expert modules are configured in the basic speech recognition model, and each language feature expert module is used to process the corresponding language features;

[0059] In this embodiment, multiple language feature expert modules are introduced into the basic speech recognition model. Each expert module focuses on the features of a specific language, ensuring that the model can recognize and process the unique acoustic features, grammatical structures, and vocabulary systems of different languages. These expert modules are independent and designed to provide optimized processing for different languages.

[0060] Each expert module can be implemented using specially designed neural networks trained on datasets of different languages. Expert modules can be dynamically selected based on the input language identifiers and language feature processing can be performed in the intermediate or output layers of the model. Expert modules can use neural network architectures such as Convolutional Neural Networks (CNNs) or Long Short-Term Memory Networks (LSTMs) to capture language features.

[0061] Each expert module is designed to handle the features of a specific language. These features include tone, grammar, phonemes, and semantic structure. By configuring a dedicated expert module for each language, the model ensures that the characteristics of each language are handled specifically, avoiding interference between languages.

[0062] During training, the expert module is optimized based on a language-specific dataset to capture the language's unique attributes. During inference, the input speech signal is first identified by language type, and then the corresponding expert module is activated to process the language-specific features, ensuring recognition accuracy.

[0063] Example Explanation: In global financial operations, customers may interact with banks using different languages. In customer service or telephone banking, by configuring multiple language-specific expert modules, the system can dynamically select the appropriate module to handle voice input in different languages. For example, when a bank receives a call from a German-speaking customer, the system will invoke the German expert module to process the specific features of the German voice without affecting the system's processing of other languages. This enables financial institutions to efficiently handle customer needs from diverse language backgrounds, improving customer experience and service quality.

[0064] By configuring multiple language feature expert modules, the system can address the unique characteristics of different languages, avoiding the performance degradation issues that occur when a single model handles multiple languages. The introduction of expert modules allows the model to be specifically optimized for the acoustic and semantic features of different languages, improving recognition accuracy. Simultaneously, the independence of the expert modules ensures that the introduction of new languages ​​does not affect the recognition performance of other languages, enhancing the model's scalability.

[0065] S40, acquire training speech data, divide the training speech data into multiple batches of data, each batch of data containing multiple speech samples;

[0066] In this embodiment, a speech dataset for training is obtained from a data source. This dataset contains speech samples in multiple languages ​​for subsequent model training and fine-tuning. This speech data is typically labeled, indicating the corresponding language type and other relevant information.

[0067] The large amount of training speech data is divided into several batches. Since it is impossible to process all the data at once when training on large-scale datasets, batching effectively reduces memory usage, increases computational efficiency, and ensures that the model can gradually update its weights. The amount of data in each batch is determined by factors such as the system's computing power and memory size.

[0068] In deep learning frameworks (such as TensorFlow and PyTorch), batch processing is typically implemented using DataLoader or Dataset. These tools can automatically divide large datasets into predetermined batch sizes and progressively pass them to the model for training. Each batch is fed into the neural network independently during training, completing one forward and one backward propagation cycle.

[0069] Each batch of data consists of several speech samples, typically from datasets of different languages ​​or the same language. By batching speech samples, the model can process multiple speech inputs in parallel, thereby improving training efficiency and the model's generalization ability.

[0070] The number of speech samples in a batch is typically controlled by the batch size parameter. The batch size can be adjusted based on hardware performance and training task requirements. Common batch sizes are 16, 32, and 64. Each speech sample is preprocessed into a feature vector or Mel-frequency cepstral coefficients (MFCCs) before being fed into the neural network layers of the model.

[0071] Example Explanation: In a global financial services voice customer service system, a large amount of voice input data from different countries and regions needs to be processed. By dividing the training voice data into multiple batches, the bank's voice recognition system can efficiently process data from different languages ​​at once. For example, the system retrieves voice data from customers in multiple countries from a database, divides it into several batches, and each batch contains voice input from multiple countries. This not only improves training efficiency but also ensures that the system can recognize voice input from different languages ​​in a balanced way when dealing with multilingual customers, optimizing the global customer service experience.

[0072] By dividing the training speech data into multiple batches, the system can effectively avoid memory overflow issues while increasing training stability and parallel efficiency. Batch processing allows each training iteration to use only a portion of the data, better randomizing model updates, preventing the model from getting trapped in local optima, and improving the model's generalization ability. Especially in multilingual data training, batch processing ensures that each batch contains a balanced representation of speech samples from different languages, thereby enhancing the model's multilingual adaptability.

[0073] S50, select a corresponding language feature expert module for each speech sample in the training speech data, and input the speech sample into the corresponding language feature expert module to obtain the output result;

[0074] In this embodiment, speech samples are extracted from existing speech datasets (such as LibriSpeech, CommonVoice, etc.). Each speech sample is accompanied by a language label in the dataset to facilitate language classification during subsequent training. These speech data can be acquired and processed from different data sources through an automated data processing pipeline.

[0075] After acquiring speech samples, the system selects an appropriate language feature expert module based on the language identifier of each sample. This step ensures that each speech sample is processed by the expert module best suited to its language features, avoiding feature confusion between different languages.

[0076] By using language identification or language detection algorithms (such as text-based or phoneme-based detection), the system can determine the language type of each speech sample. Based on the determined language type, the system dynamically selects the corresponding language feature expert module to ensure that the speech sample is processed by the expert module for that specific language.

[0077] Each selected speech sample is input into a pre-selected language feature expert module for processing. Each expert module is trained on a specific language, enabling it to capture the unique features of that language and perform corresponding analysis and processing on the input speech.

[0078] After a speech sample is input into the expert module, the expert module's neural network extracts features from the speech, such as Mel-frequency cepstral coefficients (MFCC), pitch, and phonemes. Then, the expert module processes the speech according to its specific linguistic characteristics to generate an optimized output.

[0079] After processing the speech samples, the language feature expert module outputs the corresponding results. This output can be a feature vector or a direct speech recognition result, depending on the module design and task requirements.

[0080] The expert module processes speech samples through a series of convolutional or recurrent neural network layers to generate high-dimensional feature vectors or language model probability distributions. The output can be used for further classification, recognition, or downstream tasks, such as speech-to-text conversion.

[0081] Example Explanation: In cross-border financial services, customers from different countries use different languages ​​for voice input. A bank's voice recognition system first determines the customer's language by acquiring their voice data. The system then automatically selects a language feature expert module that matches the customer's language to process the voice input. The expert module optimizes the input based on the customer's language characteristics, generating accurate recognition results. This allows the system to more accurately recognize customer requests, regardless of whether the customer is speaking English, French, or Chinese, thereby improving the efficiency and quality of global customer service.

[0082] By selecting the most suitable language feature expert module for each speech sample, the system can perform specialized optimization for speech data in different languages, ensuring the accuracy and efficiency of speech recognition. This dynamic selection mechanism based on language features improves the model's adaptability and avoids confusion and performance degradation in multilingual scenarios. Furthermore, the expert module processes language features, making the recognition results for different languages ​​more accurate and reducing the error rate in cross-language recognition.

[0083] S60, assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0084] In this embodiment, the system needs to assign an identifier corresponding to the language type of each speech sample in the batch. This means that the system will identify the language of each speech sample and mark which language the sample belongs to. This language identifier helps the system dynamically select the appropriate language processing module or adapter during training or inference.

[0085] The system assigns a language identifier to each speech sample using language detection algorithms or metadata-based annotation. For example, identifiers can be directly assigned based on predefined metadata (such as a "language type" field) in the speech dataset. Alternatively, the language type can be inferred during training by analyzing the phoneme features or acoustic models of the speech samples and then labeled as a specific language. Commonly used identification methods include ISO language codes (e.g., EN for English, ZH for Chinese, etc.).

[0086] The assigned language identifier indicates the language type of each speech sample, enabling the model to process it according to the language type. This ensures that the system uses the correct language feature modules or adapters during processing, avoiding interference between languages ​​and thus improving recognition accuracy.

[0087] A "language identifier" field is appended to the data structure of each speech sample and passed to the model. During processing, the system determines which language feature expert module or language dependency adapter to use based on the language identifier. This transmission of the language identifier can be automated through the input pipeline of the training data and passed to the model along with other information from the batch.

[0088] Example Explanation: In global banking operations, customers interact with the bank's system via voice input. The system receives voice requests in different language contexts, such as English, Chinese, and French. By assigning a corresponding language identifier to each voice sample, the bank's voice recognition system can quickly identify the customer's language type and call the appropriate language feature module for processing. For example, when a customer makes a voice request in Spanish, the system automatically identifies and selects the Spanish adapter for processing, ensuring the accuracy of voice recognition. This mechanism can greatly improve the efficiency of multilingual customer service and customer satisfaction.

[0089] By assigning a corresponding linguistic identifier to each speech sample, the system can classify different language samples in a multilingual environment. Assigning linguistic identifiers not only improves the system's efficiency during training and inference but also ensures that the system selects the most appropriate module or model path when processing different languages, avoiding confusion between language features. Furthermore, the explicitness of the linguistic identifiers enhances the system's flexibility, enabling it to handle multilingual input and improve recognition accuracy.

[0090] S70, determine the target low-resource language, freeze the parameters of the language dependency adapter that does not belong to the target low-resource language, so as to train and optimize only the parameters of the language dependency adapter corresponding to the target low-resource language;

[0091] In this embodiment, low-resource languages ​​that require optimization and training are identified and determined. Low-resource languages ​​typically refer to languages ​​where data is scarce and model training is insufficient in the field of speech recognition. The goal of identifying low-resource languages ​​is to focus on optimizing the recognition capabilities of these languages ​​and improving their performance in the model.

[0092] By analyzing speech datasets or system usage, we can identify which languages ​​are considered low-resource languages. For example, we can dynamically or manually select target low-resource languages ​​for optimization based on metrics such as the amount of language data, the quality of speech training samples, and model performance (e.g., a high word error rate, WER).

[0093] "Freezing" refers to locking the parameters of other language adapters that are not part of the target low-resource language, keeping them unchanged during training to avoid affecting the performance of other language adapters. After freezing the parameters, training will only optimize for the target low-resource language adapter.

[0094] In model training frameworks, parameter freezing is typically achieved by setting `requires_grad=False` for specific language adapter modules. Once frozen, these modules will not update their parameters during training, thus avoiding impacting the performance of other languages. This ensures that fine-tuning and optimization are only performed on the adapter for the target low-resource language.

[0095] This approach focuses on training and optimizing the adapter parameters for the target low-resource language, avoiding the computational overhead of training adapters for all languages ​​simultaneously. It concentrates on optimizing the target language to improve recognition accuracy. During training, only the adapter module corresponding to the target language is activated, and its parameters are optimized through backpropagation. The model is trained using data from the target low-resource language, and the adapter parameters for that language are gradually optimized to an ideal state through multiple iterations.

[0096] Example Explanation: In global banking operations, some bank customers may use less common, low-resource languages, such as certain minority languages ​​or regional languages. The bank's speech recognition system may perform poorly in these languages. By identifying target low-resource languages, the system can specifically optimize its recognition capabilities for these languages, ensuring that the system can more accurately recognize customer requests when they are using these languages ​​for voice operations. For example, the system can automatically freeze the adapter parameters for mainstream languages ​​and train only the Nepali language adapter, improving its performance in speech recognition and ensuring efficient financial services and customer satisfaction.

[0097] By training and optimizing only the adapter for the target low-resource language, the system can concentrate resources and computing power to improve the recognition performance of low-resource languages ​​without affecting the recognition performance of other languages. This mechanism of freezing adapters for other languages ​​not only significantly reduces computational costs but also improves the model accuracy for low-resource languages, solving the problem of low efficiency in processing low-resource languages ​​in existing technologies. Especially in multilingual scenarios, this method enables efficient model optimization and maintenance.

[0098] S80, select the corresponding low-resource language dependency adapter based on the language identifier of the speech samples in each batch of data, and train the corresponding low-resource language dependency adapter based on the output result corresponding to the speech samples to optimize and adjust the parameters of the low-resource language dependency adapter;

[0099] In this embodiment, each speech sample in the batch data has its corresponding language identifier, indicating the language type to which the sample belongs. This step refers to the system automatically selecting a suitable dependency adapter for that language based on the language identifier of the speech sample, especially for samples of low-resource languages, ensuring that the system can call the corresponding low-resource language adapter for processing and training.

[0100] In the data processing pipeline, the linguistic identifiers of speech samples are extracted and passed to the model's language adapter selection module. This module matches and activates the corresponding low-resource language adapters based on the linguistic identifiers, ensuring that the speech samples are processed by the appropriate adapter module. This can be achieved through a lookup table or a dynamic adapter selection mechanism.

[0101] After the adapter for the low-resource language is determined, speech samples are input into the corresponding language adapter for processing, and the output is obtained. This step refers to using the output of the speech samples to perform backpropagation training of the adapter. Through this process, the system can gradually optimize the parameters of the low-resource language adapter and improve its performance.

[0102] The model performs forward propagation based on the input speech sample and its corresponding language adapter, generating an output. Then, the system calculates the loss function based on this output and updates and optimizes the parameters of the low-resource language adapter using a backpropagation algorithm (such as gradient descent). This training process is repeated until the parameters of the low-resource language adapter are sufficiently optimized.

[0103] Through backpropagation and parameter update training, the system can gradually optimize and adjust the parameters of the low-resource language adapter to better adapt it to the characteristics of low-resource languages. In this way, the performance of low-resource languages ​​in speech recognition tasks can be improved, and the error rate reduced.

[0104] In each round of training, the parameters of the low-resource language adapter are updated through feedback from the loss function. The optimization algorithm used may be SGD (Stochastic Gradient Descent), Adam optimizer, etc., and the parameter updates are fine-tuned based on the error of each speech sample, ultimately enabling the adapter to achieve the expected performance.

[0105] Example Description: In global banking operations, systems need to handle customer requests from different languages. For customers using low-resource languages ​​(such as certain minority languages ​​or local dialects), speech recognition systems may face significant challenges. By configuring separate adapters for these low-resource languages ​​and dynamically selecting based on the language identifiers of the customer's voice input, the system can perform accurate training and optimization for these languages. For example, when a customer uses Urdu for voice input, the system will select the corresponding Urdu adapter and optimize its training based on the customer's voice input, gradually improving recognition accuracy and ensuring the precision of banking services and customer satisfaction.

[0106] By selecting a low-resource language adapter based on speech sample language identifiers and performing targeted training using the output results, the system can focus on optimizing the recognition capability of low-resource languages. This approach effectively avoids interference from high-resource languages ​​to low-resource languages ​​in multilingual systems, while also reducing the waste of computational resources. Optimizing the parameters of the low-resource language adapter not only improves the recognition accuracy of low-resource languages ​​but also enhances the system's adaptability in multilingual scenarios, thereby improving overall performance.

[0107] S90, until all batches of data have been trained, a trained target language dependency adapter is generated to obtain the target speech recognition model;

[0108] In this embodiment, the entire training process continues until all batches of data have been used for training. During training, the system processes the training speech data in batches, with each batch used to optimize and update the parameters of the low-resource language-dependent adapter. The completion of this step signifies that speech samples from all batches of data have been used in training.

[0109] In deep learning frameworks, this step is accomplished by iterating through all batches of data. The system updates the model parameters after each batch has completed training, until all batches have been processed one or more times. Common implementations use for loops or iterators built into the training framework to process the dataset.

[0110] After training on all batches of data, the dependency adapter for the low-resource language is fully optimized, ultimately generating a pre-trained adapter module. This adapter is specifically optimized for the characteristics of the target language, adapting to its acoustic and semantic features, and can work efficiently in speech recognition tasks.

[0111] After each training round, the adapter's parameters are updated and stored. Once the entire training process is complete, the system saves the final adapter state. At this point, the generated adapter contains an optimized set of parameters specifically designed for processing the target low-resource language.

[0112] After the adapter is trained and optimized, the adapter module and the base model together constitute the final target speech recognition model in the entire speech recognition system. This model can accurately recognize and process speech input in the target low-resource language while maintaining the ability to recognize other languages.

[0113] The final speech recognition model integrates a base speech recognition model with a pre-trained language dependency adapter. This model can be used in the inference phase to recognize low-resource speech input by invoking the pre-trained adapter. The system can save this model as the final inference model for use in subsequent speech recognition tasks.

[0114] Example Explanation: In global financial services, banks need to handle voice requests from different countries and regions, some of which are spoken in less common, low-resource languages. By training the system with voice data in these low-resource languages ​​in batches, the system can generate specially optimized language dependency adapters. After all batches of training are completed, the resulting target speech recognition model will be able to accurately recognize these low-resource languages. For example, after multiple rounds of training and optimization, the system generates language dependency adapters for Filipino or Urdu and integrates them with the base speech recognition model, ensuring that the system can handle voice requests from customers speaking these languages. This improves the bank's global customer service experience, ensuring that users of different languages ​​can enjoy efficient speech recognition services.

[0115] By training all data in batches, the system ensures that the adapter for the target low-resource language is fully optimized, ultimately generating an efficient dependency adapter suitable for the specific language. This training mechanism not only improves the recognition accuracy of the target language but also reduces the computational burden on high-resource languages. The final integrated target speech recognition model combines the base model and the optimized adapter, enabling it to effectively handle speech input in low-resource languages ​​in multilingual scenarios and improving the overall performance of the system.

[0116] S100, Speech recognition is performed based on the target speech recognition model.

[0117] In this embodiment, a target speech recognition model that has been trained and integrated with a language dependency adapter is used to process speech input. The target speech recognition model includes a base speech recognition model and an optimized language dependency adapter, specifically designed for speech recognition tasks in multilingual scenarios, with particular optimization for low-resource languages.

[0118] During the inference phase, the target speech recognition model receives the input speech signal and selects a suitable language-dependent adapter module based on the signal's linguistic identifier for processing and recognition. The model uses pre-trained parameters and structure, leveraging language characteristics and acoustic models to analyze and decode the speech signal, outputting the final recognition result.

[0119] The model processes the input speech signal and outputs text or other forms of recognition results. The speech recognition process typically includes steps such as speech signal preprocessing, feature extraction, language model inference, and final text decoding. The target speech recognition model uses the parameters and characteristics learned during training to perform feature analysis on the speech signal and generate corresponding text or semantic output.

[0120] In speech recognition, the system first receives and preprocesses the input speech signal, then extracts and decodes features using a target speech recognition model. The model uses an optimized language-dependent adapter to further process the input speech features and generate the recognition result. The final output can be text, command execution, or input from other applications.

[0121] Example Description: In a bank's global customer service, customers can communicate with the bank via telephone or voice system. The system uses a target speech recognition model to process the customer's voice requests and selects the appropriate language dependency adapter based on the customer's language. Regardless of whether the customer is speaking English, Chinese, or a low-resource language (such as Swahili), the system can accurately recognize the customer's speech content and generate a corresponding text or instruction output. For example, a customer can check their account balance or make a transfer via voice; the system will generate accurate recognition results based on the target speech recognition model, ensuring the smooth execution of the operation.

[0122] By using a target speech recognition model, the system can efficiently and accurately process user voice input in multilingual scenarios, achieving particularly better results in low-resource languages. The target speech recognition model combines a base model with an optimized language dependency adapter, enabling the system to be multilingually adaptable and dynamically select and process voice input in different languages, thus improving the accuracy and efficiency of speech recognition. This efficient speech recognition capability allows the system to be applied to more multilingual scenarios, enhancing user experience and interactivity.

[0123] This invention relates to the fields of artificial intelligence and fintech, and discloses a multilingual speech recognition method. It involves acquiring a pre-trained basic speech recognition model capable of recognizing multiple languages ​​and freezing its parameters. Language-dependent adapters are configured, with each language corresponding to an adapter. Multiple language feature expert modules are configured to handle different language features. Training speech data is acquired, and the appropriate language feature expert module is selected and speech samples are input. The training speech data is divided into multiple batches, and a language identifier is assigned to each sample. The target low-resource language is determined, the parameters of other adapters are frozen, and only the adapter for the target language is trained. The corresponding adapter is selected based on the language identifier and trained until all batches of data have been trained, generating the target speech recognition model. This invention, by configuring language-dependent adapters and language feature expert modules, allows for targeted processing of the characteristics of different languages, significantly improving the model's adaptability and recognition accuracy. By freezing adapters for non-target languages, unnecessary resource consumption is reduced, focusing on optimizing the target low-resource language and ensuring the efficiency of the training process. Furthermore, the model's scalability is enhanced, enabling rapid adaptation to the addition of new languages ​​and reducing the complexity of deployment and maintenance.

[0124] In one embodiment, after obtaining the training speech data, step S40 above further includes:

[0125] S401, The training speech data is compressed using a lower projection matrix to reduce the feature dimension of the training speech data;

[0126] S402, the compressed training speech data is processed using an activation function;

[0127] S403, the training speech data after activation function processing is processed by the up projection matrix to restore the original feature dimension of the training speech data.

[0128] In this embodiment, a lower projection matrix is ​​used to process the training speech data to reduce the feature dimensionality of the data. Reducing dimensionality can reduce computational overhead while preserving key information. The projection matrix maps high-dimensional features to a low-dimensional space through linear transformation, a process that helps improve training efficiency, especially when dealing with complex datasets.

[0129] Using a linear transformation formula, the input high-dimensional data X is transformed through a projection matrix D to obtain low-dimensional data X' = D*X. This projection matrix D is set before training and is usually implemented through matrix operations or dimensionality reduction algorithms (such as PCA). Its function is to extract the most representative features from the high-dimensional data while reducing the data dimensionality.

[0130] Nonlinear activation is applied to the compressed data to further process its nonlinear features. Activation functions are typically used to enhance the model's ability to express complex relationships. Commonly used activation functions include ReLU, Sigmoid, and Tanh, which aim to introduce nonlinearity so that the model can learn and fit complex speech features.

[0131] After compression using the downprojection matrix, the low-dimensional feature data is processed by applying the activation function ReLU(x) = max(0,x) or other nonlinear functions. The activation function preserves the effective positive signal in the data and enables the model to handle complex variations in speech data through nonlinear transformation.

[0132] After activation function processing, the data is restored to its original feature dimensions using an upprojection matrix. This step ensures that the data can continue to be processed in subsequent layers of the model without information loss due to dimensionality reduction. The upprojection matrix uses an inverse linear transformation to restore the data from the low-dimensional space to its original dimensions, maintaining consistency between input and output.

[0133] The inverse transformation is performed using the upprojection matrix U, with the formula X” = U * X', where X' is the low-dimensional data processed by the activation function, and X” is the recovered high-dimensional data. This is done to further process the speech data within the model, ensuring that subsequent steps can handle data of the same dimension as the input features.

[0134] This embodiment uses down-projection and up-projection matrices to process data, enabling the system to maintain the integrity of speech data while reducing computational complexity. The use of activation functions further enhances the model's ability to process complex speech features. This structure not only improves the efficiency of training and inference but also avoids information loss due to dimensionality compression. Especially when processing large-scale speech data or real-time speech tasks, it can effectively improve the system's response speed and performance.

[0135] In one embodiment, the above S50 includes:

[0136] S504, the matching score between each speech sample in the training speech data and each language feature expert module is determined by the router module;

[0137] S505, Select the preset number of language feature expert modules ranked first by matching score;

[0138] S506: Input the speech samples into each selected language feature expert module to obtain multiple expert output results;

[0139] S507, Based on the matching score, the output results of the multiple experts are weighted to obtain the weighted final output result.

[0140] In this embodiment, the router module is a key component in the multilingual speech recognition system, responsible for dynamically selecting and allocating among multiple language feature expert modules or computational paths. Its core function is to select the most suitable language feature expert module based on the characteristics of the input speech sample (such as language identifiers, speech features, etc.) and route the input signal to these modules for processing.

[0141] The main function of the router module is to dynamically select the most suitable expert module for processing based on the input speech sample and the adaptability of each language feature expert module. For example, in a multilingual environment, different expert modules specialize in handling the features of different languages, and the router module will select the appropriate module based on the language identifier of the input speech. The selection can be based on specific language features, acoustic features of the speech, or preset model parameters.

[0142] The router module determines which expert module is best suited to process the current input data by calculating a matching score between each input sample and different expert modules. Typically, the matching score can be calculated using methods such as vector similarity or Softmax. A higher matching score indicates that the expert module is likely to perform better on the current speech sample.

[0143] Based on the matching score, the router module selects the top-ranked expert modules and sends the input data to these modules for processing. The router module acts as a "middleman," ensuring that the input speech sample is processed by the most suitable module. The router module determines the data flow and is crucial for the collaborative work of modules in the system.

[0144] After selecting multiple language feature expert modules, the router module needs to perform weighted processing on the outputs of these modules to obtain the final result. This weighting is typically based on the matching score of the expert modules; that is, expert modules with higher scores have a greater impact on the final output.

[0145] For example, the router module is responsible for calculating the matching score between the input speech sample and different language feature expert modules based on the sample's features. This score measures the adaptability of each expert module to the speech sample. The matching score selects the most suitable language feature expert module based on the language features in the speech sample. The router module can use a Softmax equal probability distribution algorithm to calculate the matching score matrix based on the input feature vector (such as speech features, language identifiers, etc.) and the preset weights of each language feature expert module. The score calculation formula is usually based on vector dot product or similarity measurement, and this score represents the degree of adaptability between the speech sample and each expert module.

[0146] After calculating the matching score, the system selects the top-ranked expert modules. The preset number refers to selecting a certain number of expert modules from the matching score rankings for speech processing. Typically, the top two or a few expert modules are selected to effectively distribute the computational burden and improve the accuracy of the results. After determining the matching score of each speech sample with all expert modules, the system sorts these scores and selects the top preset number of expert modules (e.g., the top 2 or top 3). The selection process is usually accomplished using a sorting function to ensure that the system selects the most suitable expert module for the current speech sample.

[0147] Each selected expert module receives the input speech sample and processes it independently. The processing result of each expert module is recorded, generating multiple outputs. Each output reflects the specific processing and analysis of the speech sample by that module. Speech samples are input into each selected language feature expert module, and each module processes the sample according to its own specific language feature model. Expert modules typically use optimized deep learning models, such as Convolutional Neural Networks (CNNs) or Long Short-Term Memory Networks (LSTMs), to extract features and classify the speech, generating their respective outputs.

[0148] The outputs of each expert module need to be comprehensively processed. The system weights the outputs of each expert module based on its matching score. The higher the matching score of an expert, the greater the weight of its output in the final result. Through weighted processing, the system generates a comprehensive and optimized final output. The system combines the outputs of each expert module based on its matching score, using a weighted average or other weighting method. The higher the weight of an expert's output, the greater its proportion in the final result. The formula is: Final_output = Σ(Wi*Outputi), where Wi is the expert's weight and Outputi is the corresponding expert's output. The final weighted result is the final processing result of the speech sample.

[0149] This embodiment uses a router module to dynamically select the language feature expert module that best matches the speech sample. This allows the system to process the speech sample more effectively, improving the accuracy of speech recognition. Weighted processing ensures that the output of each expert module is reasonably distributed in the final result, avoiding the limitations of single-module processing. Parallel operation of multiple expert modules further improves processing speed and reduces interference between language features.

[0150] In one embodiment, S10 includes:

[0151] S101 collects speech datasets containing multiple languages;

[0152] S102, Construct a deep learning model for multilingual speech recognition;

[0153] S103, The deep learning model is trained using the speech dataset, and the model parameters are adjusted through multiple rounds of iteration to obtain the basic speech recognition model;

[0154] S104, Freeze the parameters of the basic speech recognition model, including the parameters of the input layer, hidden layer and output layer.

[0155] In this embodiment, the first step is to collect a speech dataset containing multiple languages ​​to train the basic speech recognition model. This dataset needs to cover all languages ​​supported by the system's target and include speech samples for each language. These speech samples are used for model training, helping the model learn how to recognize and distinguish the speech features of different languages. The dataset can be constructed using existing speech datasets (such as LibriSpeech, Common Voice, etc.) or a custom dataset, ensuring coverage of different languages ​​and accents. The collection of speech samples can include various pronunciation scenarios, such as daily conversations, reading text, and question-and-answer sessions.

[0156] This paper describes how to build a basic speech recognition model capable of recognizing multiple languages ​​using deep learning techniques. The model should be able to handle inputs from different languages ​​and extract and classify the speech features of each language. Common model architectures include Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), and Transformers, which typically perform well in speech recognition. A multilingual speech recognition model can be built using neural network frameworks (such as TensorFlow and PyTorch). The model's input is speech features (such as MFCCs and Mel-spectrum data), and the output is a text sequence or phoneme labels for the speech. The model structure can include multi-layer convolutional networks and recurrent networks to capture the temporal and spatial features of the speech.

[0157] A deep learning model is trained using collected multilingual speech datasets, and its parameters are tuned to enable the model to learn and master the speech features of multiple languages. During training, the model parameters are iteratively optimized through multiple rounds until the model's recognition performance on all target languages ​​reaches the expected level. The training process typically involves multiple epochs. In each epoch, the dataset is divided into multiple batches and fed into the model, which updates its parameters through backpropagation. Optimization algorithms (such as Adam or SGD) are used to adjust network weights and reduce the model's loss function (such as cross-entropy loss) to improve speech recognition accuracy.

[0158] After model training is complete, freezing the parameters of the base speech recognition model means that all parameters will no longer be updated. Freezing the parameters of the input, hidden, and output layers ensures that the model remains stable during subsequent fine-tuning and does not affect the performance of other training modules. The freezing operation primarily reduces computational burden, allowing focus on training other modules that depend on this base model (such as language dependency adapters). By setting `requires_grad = False`, the parameters of each layer are frozen, ensuring that the parameters of all layers of the base speech recognition model (including the input, hidden, and output layers) remain unchanged during subsequent training. After freezing, the model's weights will not be updated during backpropagation, thus concentrating computational resources on fine-tuning other modules.

[0159] This embodiment collects datasets in multiple languages ​​and constructs a multilingual speech recognition model, enabling the system to accurately recognize user speech input in multilingual environments. Freezing the parameters of the basic speech recognition model ensures that other modules can be further optimized on a stable basis during subsequent fine-tuning without affecting the overall performance of the model. This training and freezing strategy significantly reduces training time and improves the scalability of the speech recognition system, especially in multilingual scenarios, where it can significantly improve recognition accuracy and processing efficiency.

[0160] In one embodiment, S20 includes:

[0161] S201 defines the structure of a language dependency adapter for each language;

[0162] S202, For each language, use the corresponding speech dataset to train the corresponding language dependency adapter, so that the language dependency adapter learns the acoustic features and language features specific to each language;

[0163] S203, Integrate the language dependency adapter into the basic speech recognition model;

[0164] S204 configures a unique identifier for the language dependency adapter for each language, which is used to correctly select and apply the appropriate language dependency adapter during model training.

[0165] In this embodiment, a separate Language Dependency Adapter (LDA) is set up for each target language in the multilingual speech recognition system. These adapters are specifically designed to handle the characteristics and speech features of each language. The adapter structure is primarily designed to capture the acoustic and semantic features of different languages ​​in order to make accurate recognition when processing inputs from different languages. The system can define a specific language adapter network structure for each language. This structure can be a simplified deep neural network layer specifically designed to handle the features of that language. The structure of each language adapter will differ in its input, hidden layers, and output layers to adapt to the characteristics of each language.

[0166] Each language-dependent adapter needs to be trained on a specialized speech dataset to learn the acoustic and linguistic features of that language. These features include the frequency distribution of speech sounds, phoneme combinations, and grammatical patterns of the speech stream. Through training, the language adapter will be able to accurately process and recognize speech input in that language. By inputting the dedicated speech dataset for each language into the corresponding language adapter for training, the adapter will progressively optimize its parameters through multiple iterations. The training process employs backpropagation and optimization algorithms such as Adam or SGD. The training objective of each language adapter is to match its output with the specific features of that language, thereby optimizing its language recognition capabilities.

[0167] Language dependency adapters do not operate independently but need to be integrated with the underlying speech recognition model. The integration process refers to combining each language dependency adapter with certain modules within the model (such as intermediate layers and output layers), enabling the model to process the input language characteristics during inference. Within the structure of the underlying speech recognition model, each language dependency adapter is embedded into the model through interfaces or modularization. For example, the base model processes basic speech features, while when a specific language is recognized, the system invokes the corresponding language dependency adapter to further process the speech data.

[0168] To ensure the model correctly selects the adapter for each language during training and inference, the system needs to configure a unique identifier (ID) for each language-dependent adapter. This identifier helps the model automatically select and activate the appropriate adapter when faced with multilingual speech input, ensuring that each speech sample is processed by the appropriate module. The system can assign a unique identifier to each language-dependent adapter through a configuration file or a database. Whenever the model receives a speech sample, the system selects the appropriate adapter based on the sample's language type. The identifier can be associated with a language code (e.g., "en" for English, "fr" for French) to ensure correct mapping between samples and adapters.

[0169] This embodiment enables the system to optimize processing for the characteristics of different languages ​​by configuring language-dependent adapters for each language. Each adapter learns the unique acoustic and linguistic features of that language, ensuring recognition accuracy in multilingual environments. Configuring unique identifiers ensures that the model can quickly and accurately select and call the corresponding adapter, enhancing the system's multilingual processing capabilities. After integrating the adapters, the speech recognition model can maintain the efficiency of the basic model while making additional optimizations according to the needs of specific languages, improving the overall performance of the system.

[0170] In one embodiment, S30 includes:

[0171] S301, determine a list of languages ​​and define corresponding language characteristics for each language in the list;

[0172] S302, configure a corresponding language feature expert module for each language feature.

[0173] S303, integrate the language feature expert module into the intermediate or output layer of the basic speech recognition model.

[0174] In this embodiment, the system needs to explicitly list the supported languages, each with its specific phonetic characteristics (such as tone, syllable structure, grammatical patterns, etc.). A dedicated language characteristic expert module is defined based on the characteristics of each language to better handle the speech input of that language. By analyzing each language, its phonetic features and semantic rules are identified. Feature vectors can be extracted from the speech dataset to define language characteristics such as frequency distribution, prosodic patterns, and phoneme combinations. For example, the tonal characteristics of Chinese and the consonant cluster combination characteristics of English. The language list can include commonly used languages ​​(such as English, French, Spanish, etc.) as well as low-resource languages.

[0175] A dedicated language feature expert module is assigned to each language feature, ensuring that this module can handle the specific features of that language. A language feature expert module is a neural network or model module specifically designed to process and optimize the specific speech features of that language, ensuring that input speech samples can be processed according to those specific language features. By creating independent expert modules for each language feature, the model can perform speech processing according to the needs of different languages. The parameters and structure of each expert module are specifically designed and trained for the speech features of a particular language, ensuring that the module can capture and process the unique features of that language. Expert modules can be part of models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Transformers.

[0176] The language feature expert module does not operate independently but needs to be integrated with the base speech recognition model. Integration can occur at the intermediate or output layer, combining the expert module's processing results with the base model so that the base model can utilize the expert module's output for final speech recognition. Structurally, this can be achieved by inserting the language feature expert module into an intermediate or output layer of the base model. Specifically, this can be implemented using a multi-task learning framework, concatenating or fusing the output of each expert module with other layers of the base model. The integrated model can further process speech data based on the outputs of expert modules with different language features, improving the recognition performance for specific languages.

[0177] This embodiment, by configuring independent language feature expert modules for each language, enables the system to more accurately capture and process the specific speech features of each language, thus improving the accuracy of multilingual speech recognition. By integrating these expert modules with the base model, the system can optimize the processing capabilities for specific languages ​​without affecting overall performance. The use of multilingual feature expert modules gives the system greater adaptability and flexibility when facing different languages, significantly improving recognition accuracy, especially in multilingual or low-resource language environments.

[0178] In one embodiment, S100 includes:

[0179] S1001, receives the voice signal to be recognized;

[0180] S1002, perform preprocessing operations on the speech signal to be recognized, the preprocessing operations include noise reduction and normalization;

[0181] S1003, Extract speech features from the speech signal to be recognized after preprocessing, the speech features including Mel frequency cepstral coefficients and pitch features;

[0182] S1004, Based on the language type of the speech signal to be recognized, determine the corresponding language dependency adapter in the target speech recognition model;

[0183] S1005, The speech features are processed by the language dependency adapter, and the processing includes feature space mapping, nonlinear activation and normalization.

[0184] S1006, The processed speech features are identified using the target speech recognition model to generate a recognition result.

[0185] In this embodiment, the system receives a voice input signal as the start of the voice recognition task. The voice signal can come from a recording device, a telephone, or other audio input device, and the system needs to process the signal to complete the voice recognition. Real-time voice data can be captured through a microphone or audio interface, or the voice signal can be read from a pre-recorded audio file. The voice signal is input to the voice recognition system in digital form.

[0186] To improve the accuracy of speech recognition, the system first preprocesses the input speech signal. Preprocessing steps include removing background noise (denoising) and adjusting the signal amplitude to a standard range (normalization). These operations help improve the performance of subsequent feature extraction and recognition. Denoising can use filters (such as low-pass filters or spectral subtraction) to remove background noise from the speech signal. Normalization scales the amplitude of the speech signal to a fixed range, avoiding recognition errors caused by differences in signal strength.

[0187] Feature extraction is a crucial step in speech recognition systems. MFCC (Medium Frequency Cepstral Coefficients) is one of the commonly used features in speech recognition, capturing the frequency characteristics of speech signals. Pitch features are used to identify pitch information in speech, which is particularly important for processing emotional and intonation variations. The MFCC algorithm is used to perform frequency domain analysis on the speech signal to extract its cepstral coefficients. First, the speech signal is subjected to a Fourier transform to generate a spectrogram. Then, the spectrogram is transformed using the Mel filter bank to obtain the Mel frequency cepstral coefficients (MFCC). Pitch features can be used to extract pitch information in speech through frequency tracking algorithms.

[0188] The system selects an appropriate language dependency adapter based on the language type of the speech signal. Adapters are modules specifically designed for different languages, optimizing speech recognition performance for those languages. The language dependency adapter processes the speech signal according to language characteristics to improve recognition accuracy. By analyzing the linguistic identifiers or features of the speech signal, the system selects the language dependency adapter that matches that language. The language type can be detected during the preprocessing stage or inferred from user input or context. Each language has a corresponding language dependency adapter, which performs speech processing based on specific language features.

[0189] In speech recognition, a language-dependent adapter further processes the input speech features. Feature space mapping projects the original speech features onto a new feature space to better suit subsequent speech recognition tasks. Nonlinear activation functions enhance the model's learning ability, while normalization ensures that the output features are within a reasonable range. Feature space mapping transforms speech features into a feature representation suitable for model processing. Nonlinear activation functions (such as ReLU) enhance the nonlinearity of the processing. Normalization adjusts feature values ​​to a uniform range, preventing excessively large or small feature values ​​from affecting recognition performance.

[0190] The system uses a target speech recognition model to recognize the processed speech features and generate the final recognition result. The recognition result is usually in text form, but can also be the output of other speech understanding tasks (such as command or intent recognition). The processed speech features are then classified or decoded using a trained speech recognition model (such as a model based on convolutional neural networks or recurrent neural networks). The system outputs the corresponding text or semantic result based on the feature extraction results. This stage utilizes the core inference part of the model, mapping speech features to specific speech labels or text representations.

[0191] This embodiment, through preprocessing of speech signals, feature extraction, and language-dependent adapter-based processing, enables the system to more accurately capture and process speech input in different languages. Feature space mapping, nonlinear activation, and normalization further enhance the recognition performance, resulting in higher recognition accuracy in multilingual scenarios.

[0192] In one embodiment, a multilingual speech recognition device is provided, which corresponds one-to-one with the multilingual speech recognition methods described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multilingual speech recognition device of the present invention. The modules include: a basic model acquisition module 10, a language adapter configuration module 20, an expert module configuration module 30, a data batch partitioning module 40, a training data processing module 50, a language identifier allocation module 60, a low-resource language adapter control module 70, a low-resource language adapter training module 80, an adapter training completion module 90, and a speech recognition module 100. Detailed descriptions of each functional module are as follows:

[0193] The basic model acquisition module 10 is used to acquire a pre-trained basic speech recognition model, which is capable of recognizing multiple languages, and to freeze the parameters of the basic speech recognition model.

[0194] Language adapter configuration module 20 is used to configure language dependency adapters in the basic speech recognition model so that each language is configured with a corresponding language dependency adapter;

[0195] Expert module configuration module 30 is used to configure multiple language feature expert modules in the basic speech recognition model, and each language feature expert module is used to process the corresponding language feature.

[0196] The data batch division module 40 is used to acquire training speech data and divide the training speech data into multiple batches of data, wherein each batch of data contains multiple speech samples.

[0197] The training data processing module 50 is used to select a corresponding language feature expert module for each speech sample in the training speech data, and input the speech sample into the corresponding language feature expert module to obtain the output result.

[0198] The language identifier allocation module 60 is used to assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0199] The low-resource language adapter control module 70 is used to determine the target low-resource language and freeze the parameters of language dependency adapters that do not belong to the target low-resource language, so as to train and optimize only the parameters of the language dependency adapters corresponding to the target low-resource language.

[0200] The low-resource language adapter training module 80 is used to select the corresponding low-resource language dependency adapter based on the language identifier of the speech samples in each batch of data, and to train the corresponding low-resource language dependency adapter based on the output results corresponding to the speech samples, so as to optimize and adjust the parameters of the low-resource language dependency adapter.

[0201] The adapter training completion module 90 is used to generate a trained target language dependency adapter until all batches of data have been trained to obtain the target speech recognition model.

[0202] The speech recognition module 100 is used to perform speech recognition based on the target speech recognition model.

[0203] In one embodiment, the training data processing module 40 is specifically used for:

[0204] The training speech data is compressed using a downward projection matrix to reduce the feature dimension of the training speech data.

[0205] The compressed training speech data is processed using an activation function;

[0206] The training speech data, after being processed by the activation function, is then processed by an upprojection matrix to restore the original feature dimensions of the training speech data.

[0207] In one embodiment, the training data processing module 50 is specifically used for:

[0208] The matching score between each speech sample in the training speech data and each language feature expert module is determined by the router module.

[0209] Select the preset number of language feature expert modules ranked by matching score;

[0210] Input the speech samples into each selected language feature expert module to obtain multiple expert output results;

[0211] Based on the matching score, the output results of the multiple experts are weighted to obtain the weighted final output result.

[0212] In one embodiment, the basic model acquisition module 10 is specifically used for:

[0213] Collect speech datasets containing multiple languages;

[0214] Build a deep learning model for multilingual speech recognition;

[0215] The deep learning model is trained using the speech dataset, and the model parameters are adjusted through multiple iterations to obtain the basic speech recognition model.

[0216] Freeze the parameters of the basic speech recognition model, including the parameters of the input layer, hidden layer, and output layer.

[0217] In one embodiment, the language adapter configuration module 20 is specifically used for:

[0218] Define a structure for a language dependency adapter for each language;

[0219] For each language, a corresponding language dependency adapter is trained using the corresponding speech dataset, enabling the language dependency adapter to learn the acoustic and linguistic features specific to each language;

[0220] The language dependency adapter is integrated into the basic speech recognition model;

[0221] Configure a unique identifier for the language dependency adapter for each language to correctly select and apply the appropriate language dependency adapter during model training.

[0222] In one embodiment, the expert module configuration module 30 is specifically used for:

[0223] Determine a list of languages ​​and define corresponding language characteristics for each language in the list;

[0224] Configure a corresponding language feature expert module for each language feature.

[0225] The language feature expert module is integrated into the intermediate or output layer of the basic speech recognition model.

[0226] In one embodiment, the speech recognition module 100 is specifically used for:

[0227] Receive the voice signal to be recognized;

[0228] The speech signal to be recognized is subjected to preprocessing operations, including noise reduction and normalization.

[0229] Speech features are extracted from the preprocessed speech signal to be recognized, including Mel-frequency cepstral coefficients and pitch features;

[0230] Based on the language type of the speech signal to be recognized, determine the corresponding language dependency adapter in the target speech recognition model;

[0231] The speech features are processed by the language dependency adapter, and the processing includes feature space mapping, nonlinear activation, and normalization.

[0232] The processed speech features are identified using the target speech recognition model to generate recognition results.

[0233] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multilingual speech recognition method on the server side.

[0234] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multilingual speech recognition method on the user side.

[0235] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0236] Obtain a pre-trained basic speech recognition model that can recognize multiple languages, and freeze the parameters of the basic speech recognition model;

[0237] Configure a language dependency adapter in the basic speech recognition model so that each language has a corresponding language dependency adapter;

[0238] Multiple language feature expert modules are configured in the basic speech recognition model, and each language feature expert module is used to process the corresponding language features.

[0239] Acquire training speech data, and divide the training speech data into multiple batches of data, each batch of data containing multiple speech samples;

[0240] For each speech sample in the training speech data, select a corresponding language feature expert module, and input the speech sample into the corresponding language feature expert module to obtain the output result;

[0241] Each speech sample in each batch of data is assigned a corresponding language identifier to indicate the language type to which the speech sample belongs;

[0242] Identify the target low-resource language, freeze the parameters of language dependency adapters that do not belong to the target low-resource language, so as to train and optimize only the parameters of the language dependency adapters corresponding to the target low-resource language;

[0243] Based on the language identifiers of the speech samples in each batch of data, a corresponding low-resource language dependency adapter is selected, and the corresponding low-resource language dependency adapter is trained based on the output results corresponding to the speech samples in order to optimize and adjust the parameters of the low-resource language dependency adapter.

[0244] The training continues until all batches of data have been completed, generating a trained target language dependency adapter to obtain the target speech recognition model.

[0245] Speech recognition is performed based on the target speech recognition model.

[0246] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0247] Obtain a pre-trained basic speech recognition model that can recognize multiple languages, and freeze the parameters of the basic speech recognition model;

[0248] Configure a language dependency adapter in the basic speech recognition model so that each language has a corresponding language dependency adapter;

[0249] Multiple language feature expert modules are configured in the basic speech recognition model, and each language feature expert module is used to process the corresponding language features.

[0250] Acquire training speech data, and divide the training speech data into multiple batches of data, each batch of data containing multiple speech samples;

[0251] For each speech sample in the training speech data, select a corresponding language feature expert module, and input the speech sample into the corresponding language feature expert module to obtain the output result;

[0252] Each speech sample in each batch of data is assigned a corresponding language identifier to indicate the language type to which the speech sample belongs;

[0253] Identify the target low-resource language, freeze the parameters of language dependency adapters that do not belong to the target low-resource language, so as to train and optimize only the parameters of the language dependency adapters corresponding to the target low-resource language;

[0254] Based on the language identifiers of the speech samples in each batch of data, a corresponding low-resource language dependency adapter is selected, and the corresponding low-resource language dependency adapter is trained based on the output results corresponding to the speech samples in order to optimize and adjust the parameters of the low-resource language dependency adapter.

[0255] The training continues until all batches of data have been completed, generating a trained target language dependency adapter to obtain the target speech recognition model.

[0256] Speech recognition is performed based on the target speech recognition model.

[0257] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0258] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0259] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0260] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multilingual speech recognition method, characterized in that, Includes the following steps: Obtain a pre-trained basic speech recognition model that can recognize multiple languages, and freeze the parameters of the basic speech recognition model; Define a structure for a language dependency adapter for each language; For each language, a corresponding language dependency adapter is trained using the corresponding speech dataset, enabling the language dependency adapter to learn the acoustic and linguistic features specific to each language; The language dependency adapter is integrated into the basic speech recognition model; Configure a unique identifier for the language dependency adapter for each language to correctly select and apply the appropriate language dependency adapter during model training; Multiple language feature expert modules are configured in the basic speech recognition model, and each language feature expert module is used to process the corresponding language features. Acquire training speech data and divide the training speech data into multiple batches, each batch containing multiple speech samples. For each speech sample in the training speech data, select a corresponding language feature expert module, and input the speech sample into the corresponding language feature expert module to obtain the output result; Each speech sample in each batch of data is assigned a corresponding language identifier to indicate the language type to which the speech sample belongs; Identify the target low-resource language, freeze the parameters of language dependency adapters that do not belong to the target low-resource language, and train and optimize the parameters of the language dependency adapters corresponding to the target low-resource language. Based on the language identifiers of the speech samples in each batch of data, a corresponding low-resource language dependency adapter is selected, and the corresponding low-resource language dependency adapter is trained based on the output results corresponding to the speech samples in order to optimize and adjust the parameters of the low-resource language dependency adapter. The training continues until all batches of data have been completed, generating a trained target language dependency adapter to obtain the target speech recognition model. Speech recognition is performed based on the target speech recognition model.

2. The multilingual speech recognition method as described in claim 1, characterized in that, After obtaining the training speech data, the following is also included: The training speech data is compressed using a downward projection matrix to reduce the feature dimension of the training speech data. The compressed training speech data is processed using an activation function; The training speech data, after being processed by the activation function, is processed by an upprojection matrix to restore the original feature dimensions of the training speech data.

3. The multilingual speech recognition method as described in claim 1, characterized in that, For each speech sample in the training speech data, a corresponding language feature expert module is selected, and the speech sample is input into the corresponding language feature expert module to obtain the output result, including: The matching score between each speech sample in the training speech data and each language feature expert module is determined by the router module. Select the preset number of language feature expert modules ranked by matching score; Input the speech samples into each selected language feature expert module to obtain multiple expert output results; Based on the matching score, the output results of the multiple experts are weighted to obtain the weighted final output result.

4. The multilingual speech recognition method as described in claim 1, characterized in that, Obtain a pre-trained basic speech recognition model capable of recognizing multiple languages, and freeze the parameters of the basic speech recognition model, including: Collect speech datasets containing multiple languages; Build a deep learning model for multilingual speech recognition; The deep learning model is trained using the speech dataset, and the model parameters are adjusted through multiple iterations to obtain the basic speech recognition model. Freeze the parameters of the basic speech recognition model, including the parameters of the input layer, hidden layer, and output layer.

5. The multilingual speech recognition method as described in claim 1, characterized in that, The basic speech recognition model is configured with multiple language feature expert modules, each of which is used to process corresponding language features, including: Determine a list of languages ​​and define corresponding language characteristics for each language in the list; Configure a corresponding language feature expert module for each language feature. The language feature expert module is integrated into the intermediate or output layer of the basic speech recognition model.

6. The multilingual speech recognition method as described in claim 1, characterized in that, Speech recognition based on the target speech recognition model includes: Receive the voice signal to be recognized; The speech signal to be recognized is subjected to preprocessing operations, including noise reduction and normalization. Speech features are extracted from the preprocessed speech signal to be recognized, including Mel-frequency cepstral coefficients and pitch features; Based on the language type of the speech signal to be recognized, determine the corresponding language dependency adapter in the target speech recognition model; The speech features are processed by the language dependency adapter, and the processing includes feature space mapping, nonlinear activation, and normalization. The processed speech features are identified using the target speech recognition model to generate recognition results.

7. A multilingual speech recognition device, characterized in that, The multilingual speech recognition device includes: The basic model acquisition module is used to acquire a pre-trained basic speech recognition model, which is capable of recognizing multiple languages, and to freeze the parameters of the basic speech recognition model. The language adapter configuration module is used to define the structure of a language dependency adapter for each language; for each language, the corresponding language dependency adapter is trained using the corresponding speech dataset, so that the language dependency adapter learns the acoustic and linguistic features unique to each language; the language dependency adapter is integrated into the basic speech recognition model; a unique identifier is configured for the language dependency adapter of each language, which is used to correctly select and apply the corresponding language dependency adapter during model training; An expert module configuration module is used to configure multiple language feature expert modules in the basic speech recognition model, and each language feature expert module is used to process the corresponding language feature. The data batch division module is used to acquire training speech data and divide the training speech data into multiple batches, each batch containing multiple speech samples. The training data processing module is used to select a corresponding language feature expert module for each speech sample in the training speech data, and input the speech sample into the corresponding language feature expert module to obtain the output result. The language identifier allocation module is used to assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs; The low-resource language adapter control module is used to determine the target low-resource language, freeze the parameters of language dependency adapters that do not belong to the target low-resource language, and train and optimize the parameters of the language dependency adapters corresponding to the target low-resource language. The low-resource language adapter training module is used to select the corresponding low-resource language dependency adapter based on the language identifier of the speech samples in each batch of data, and to train the corresponding low-resource language dependency adapter based on the output results of the speech samples, so as to optimize and adjust the parameters of the low-resource language dependency adapter. The adapter training completion module is used to generate a trained target language dependency adapter until all batches of data have been trained to obtain the target speech recognition model. A speech recognition module is used to perform speech recognition based on the target speech recognition model.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and a multilingual speech recognition program stored in the memory and executable on the processor, wherein the multilingual speech recognition program, when executed by the processor, implements the steps of the multilingual speech recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a multilingual speech recognition program, which, when executed by a processor, implements the steps of the multilingual speech recognition method as described in any one of claims 1-6.

Images