Multi-language speech recognition method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a multi-language speech recognition method, which comprises the following steps: acquiring a pre-trained basic speech recognition model; an adapter is correspondingly configured for each language; configuring a plurality of language characteristic expert modules; acquiring training voice data, selecting a corresponding language characteristic expert module and inputting a voice sample; and determining a target low-resource language, freezing parameters of other adapters, only training the adapter of the target language, selecting the corresponding adapter based on the language identifier and performing training until training of all batches of data is completed, and generating a target speech recognition model. According to the method, the language dependence adapter and the language characteristic expert module are configured, so that the characteristics of different languages are processed in a targeted manner, and the adaptability and the recognition accuracy of the model are remarkably improved. Through freezing the adapter of the non-target language, unnecessary resource consumption is reduced, optimization of the target low-resource language is focused, and high efficiency of the training process is ensured.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a multilingual speech recognition method, device, equipment and storage medium. Background Art

[0002] In the financial sector, automatic speech recognition (ASR) systems have been widely used in scenarios such as customer service, voice banking, and real-time transaction monitoring. In particular, multilingual automatic speech recognition (MASR) systems can process voice input in different languages, helping financial institutions to meet the needs of multilingual customers. However, existing multilingual ASR systems have technical deficiencies in several key areas, limiting their widespread application in the financial industry.

[0003] In order to improve recognition accuracy, existing multilingual ASR systems usually need to increase the size of the model. However, as the number of model parameters increases, for example, the USM model reaches 2 billion parameters and the MMS model exceeds 1 billion parameters, the model becomes large and difficult to run efficiently. When processing real-time voice data in the financial field, such a large model faces high computing resource consumption and significant latency issues, and cannot meet the needs of real-time response in streaming processing. This delay is particularly adversely affected in application scenarios that require timely decision-making, such as high-frequency financial transactions and real-time risk monitoring.

[0004] In multilingual speech recognition tasks, existing technologies often face performance differences between different languages. Due to the different data distribution, vocabulary, and language structure of each language, a single model cannot balance the performance of all languages ​​when processing multiple languages ​​at the same time, resulting in the recognition effect of some languages ​​being much lower than that of other languages. This asynchronous performance can affect customer experience in financial services, especially for multilingual customer service, where the service quality of some languages ​​may lag significantly behind. In addition, many existing systems rely on the average word error rate (WER) as a performance measure, but this average result often masks the poor performance of specific languages ​​and may cause deviations in actual operations.

[0005] As privacy protection regulations are strengthened, data collection and use in many languages ​​are restricted, resulting in a lack of sufficient data for effective training of existing models. In the financial sector in particular, processing low-resource languages ​​has become particularly difficult, as financial institutions often need to handle the needs of customers from different language backgrounds, while data resources for certain languages ​​are relatively scarce, making it difficult for existing technologies to effectively address this challenge.

[0006] When supporting new languages, existing technologies usually require large-scale retraining and adjustment of the entire model, which significantly increases the complexity of deployment and maintenance. In financial institutions, with the expansion of business scope and the growth of cross-border service needs, language scalability has become particularly important. However, when adding new languages ​​to existing models, it often takes a lot of computing resources, which increases the complexity and cost of expanding the system and is not conducive to the rapid iteration and optimization of the system. Summary of the invention

[0007] The main purpose of the present invention is to provide a multilingual speech recognition method, apparatus, device and storage medium, aiming to solve the technical problems that the existing multilingual automatic speech recognition system has uneven performance when processing different languages, and is difficult to efficiently fine-tune for low-resource languages, resulting in low training efficiency and poor model scalability.

[0008] To achieve the above object, the present invention provides a multilingual speech recognition method, comprising:

[0009] Acquire a pre-trained basic speech recognition model, where the basic speech recognition model can recognize multiple languages, and freeze parameters of the basic speech recognition model;

[0010] Configuring a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter;

[0011] A plurality of language feature expert modules are configured in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature;

[0012] Acquire training speech data, and divide the training speech data into a plurality of batches of data, each batch of data containing a plurality of speech samples;

[0013] Selecting a corresponding language feature expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language feature expert module to obtain an output result;

[0014] Assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0015] Determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language;

[0016] Selecting a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and training the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter;

[0017] Until the plurality of batches of data are trained, a trained target language dependent adapter is generated to obtain a target speech recognition model;

[0018] Speech recognition is performed based on the target speech recognition model.

[0019] Furthermore, to achieve the above object, the present invention provides a multilingual speech recognition device, comprising:

[0020] A basic model acquisition module, used to acquire a pre-trained basic speech recognition model, the basic speech recognition model being able to recognize multiple languages, and freeze the parameters of the basic speech recognition model;

[0021] A language adapter configuration module, used to configure a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter;

[0022] An expert module configuration module, used to configure a plurality of language feature expert modules in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature;

[0023] A data batch division module, used to obtain training speech data and divide the training speech data into a plurality of batches of data, each of which contains a plurality of speech samples;

[0024] A training data processing module, used for selecting a corresponding language characteristic expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language characteristic expert module to obtain an output result;

[0025] A language identifier assignment module is used to assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0026] A low-resource language adapter control module, configured to determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language;

[0027] A low-resource language adapter training module, used to select a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and train the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter;

[0028] An adapter training completion module, used to generate a trained target language dependent adapter until the plurality of batches of data are trained to obtain a target speech recognition model;

[0029] A speech recognition module is used to perform speech recognition based on the target speech recognition model.

[0030] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a multilingual speech recognition program stored in the memory and executable on the processor, wherein the multilingual speech recognition program implements the steps of the multilingual speech recognition method described above when executed by the processor.

[0031] Furthermore, to achieve the above object, the present invention also provides a computer-readable storage medium, on which a multilingual speech recognition program is stored, and when the multilingual speech recognition program is executed by a processor, the steps of the multilingual speech recognition method described above are implemented.

[0032] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a multilingual speech recognition method. By obtaining a pre-trained basic speech recognition model, the model can recognize multiple languages ​​and freeze its parameters. Configure a language-dependent adapter, and configure an adapter for each language. Configure multiple language feature expert modules to process different language features. Obtain training voice data, select a corresponding language feature expert module and input a voice sample, divide the training voice data into multiple batches of data, and assign a language identifier to each sample. Determine the target low-resource language, freeze the parameters of other adapters, only train the adapter of the target language, select the corresponding adapter based on the language identifier and train until the training of all batches of data is completed, and generate a target speech recognition model. The present invention configures a language-dependent adapter and a language feature expert module, so that the characteristics of different languages ​​are targeted, and the adaptability and recognition accuracy of the model are significantly improved. By freezing the adapter of the non-target language, unnecessary resource consumption is reduced, focusing on the optimization of the target low-resource language, and ensuring the efficiency of the training process. In addition, the scalability of the model is enhanced, and it can quickly adapt to the addition of new languages, reducing the complexity of deployment and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0034] Figure 1 A schematic diagram of an application environment of a multi-language speech recognition method according to an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a flow chart of an embodiment of a multilingual speech recognition method of the present invention;

[0036] Figure 3 A schematic diagram of functional modules of a preferred embodiment of a multi-language speech recognition device of the present invention;

[0037] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0038] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0040] The multilingual speech recognition method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can obtain a pre-trained basic speech recognition model through the user end, and the model can recognize multiple languages ​​and freeze its parameters. Configure a language-dependent adapter, and configure an adapter for each language. Configure multiple language feature expert modules to process different language features. Obtain training speech data, select a corresponding language feature expert module and input a speech sample, divide the training speech data into multiple batches of data, and assign a language identifier to each sample. Determine the target low-resource language, freeze the parameters of other adapters, only train the adapter of the target language, select the corresponding adapter based on the language identifier and train until the training of all batches of data is completed, and generate a target speech recognition model. The present invention configures a language-dependent adapter and a language feature expert module, so that the characteristics of different languages ​​are targeted, and the adaptability and recognition accuracy of the model are significantly improved. By freezing the adapter of the non-target language, unnecessary resource consumption is reduced, focusing on the optimization of the target low-resource language, and ensuring the efficiency of the training process. In addition, the scalability of the model is enhanced, and it can quickly adapt to the addition of new languages, reducing the complexity of deployment and maintenance. The user end may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server end may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0041] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a multilingual speech recognition method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0042] like Figure 2 As shown, the multilingual speech recognition method proposed by the present invention comprises the following steps:

[0043] S10, obtaining a pre-trained basic speech recognition model, where the basic speech recognition model can recognize multiple languages, and freezing parameters of the basic speech recognition model;

[0044] In this embodiment, a pre-trained basic speech recognition model is loaded from an existing model library. The model can handle speech input in multiple languages ​​and further fine-tune or apply it on this basis. The model has been pre-trained on a large scale and has a certain speech recognition capability, especially in a multi-language environment.

[0045] You can use a deep learning framework (such as TensorFlow or PyTorch) to load a pre-trained multilingual speech recognition model. The pre-training of the model usually relies on a large-scale multilingual speech dataset and is trained through supervised learning.

[0046] The model has the ability to recognize multiple languages, which means that the model's parameters have been specifically trained to process and understand the speech characteristics of different languages. In multilingual recognition tasks, the model needs to identify the language type of the speech input and process it based on specific language characteristics.

[0047] The model is trained using multilingual datasets (such as CommonVoice, LibriSpeech, etc.) to learn the acoustic and linguistic characteristics of multiple languages. The model is usually built based on a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer structure.

[0048] Freezing model parameters means that in the subsequent fine-tuning process, the parameters of the base model will remain unchanged and will not participate in training. This can effectively avoid re-adjusting the entire model and reduce the computational burden.

[0049] In the training framework, you can freeze the parameters of certain layers of the model by setting "requires_grad = False". For example, in PyTorch, you can traverse the layers of the model and select the parts that need to be frozen for parameter locking.

[0050] By loading the pre-trained multi-language basic speech recognition model and freezing its parameters, we can avoid repeated training of large-scale basic models and reduce computing resource consumption while ensuring the recognition accuracy of the existing model. We can focus on fine-tuning the specific language part, greatly improving training efficiency and deployment speed, especially in multi-language tasks, with significant performance optimization effects.

[0051] S20, configuring a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter;

[0052] In this embodiment, a language-dependent adapter is introduced into the basic speech recognition model. The role of these adapters is to optimize the characteristics of each language and independently process speech inputs in different languages. The language-dependent adapter enhances the recognition ability of the model in a specific language by processing the acoustic and grammatical features of the language.

[0053] Multiple language adapter modules are added to the model. Each adapter processes the input signal through projection and activation functions in a low-dimensional feature space and then returns an optimized output. The parameters of the adapter are trained based on specific language data to ensure that it can effectively capture the characteristics of the language. For example, the adapter is used to extract and process the features of phonemes and intonations of certain specific languages.

[0054] Configure an adapter for each language independently to ensure that the speech input of each language is processed by a specific adapter, avoiding the situation where multiple languages ​​share the same adapter. This can avoid interference between the characteristics of different languages, thereby improving the recognition accuracy of each.

[0055] In the model architecture, a mechanism is set up to automatically select the adapter of the corresponding language for processing after identifying the input speech language. The adapter is selected based on the language identifier in the speech signal, so that the system can dynamically switch to the appropriate adapter module.

[0056] Example: In global banking, customers in different countries may interact with banks in multiple languages, such as telephone banking or voice recognition systems. By configuring language-dependent adapters, the bank's voice system can automatically select the appropriate adapter for processing when receiving voice input from customers in different languages. For example, when receiving a request from a Spanish user, the system can automatically switch to the Spanish adapter for recognition and processing, ensuring the accuracy of the service and consistency of the user experience.

[0057] By configuring an independent language-dependent adapter for each language, the system can optimize the features of each language and avoid interference between different language features. This not only improves the system's processing efficiency for multiple languages, but also significantly improves the recognition accuracy of each language. In addition, since each adapter is optimized independently, only the adapter for the newly added language needs to be fine-tuned, reducing the complexity of the system and the demand for computing resources.

[0058] S30, configuring a plurality of language feature expert modules in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature;

[0059] In this embodiment, multiple language feature expert modules are introduced into the basic speech recognition model. Each expert module focuses on the characteristics of a specific language, ensuring that the model can recognize and process the unique acoustic features, grammatical structures and vocabulary systems of different languages. These expert modules are independent and are intended to provide optimized processing for different languages.

[0060] Each expert module can be implemented by specially designed neural networks that are trained based on data sets of different languages. Expert modules can be dynamically selected based on the input language identity and perform language feature processing in the intermediate layers or output layers of the model. Expert modules can use neural network structures such as convolutional neural networks (CNN) or long short-term memory networks (LSTM) to capture language features.

[0061] Each expert module is designed to process the characteristics of a specific language. These characteristics include the language's intonation, grammar, phonemes, and semantic structure. By configuring a dedicated expert module for each language, the model can ensure that the characteristics of each language are processed in a targeted manner to avoid mutual interference between languages.

[0062] During training, the expert module is optimized based on a language-specific dataset to capture the specific properties of that language. During inference, the input speech signal will first identify the language type, and then the corresponding expert module will be activated to process the characteristics of the specific language to ensure recognition accuracy.

[0063] Example description: In global financial business, customers may interact with banks in different languages. In customer service or telephone banking, by configuring multiple language feature expert modules, the system can dynamically select the appropriate module to process voice input in different languages. For example, when a bank receives a call from a German-speaking customer, the system will call the German expert module to process the specific features of German speech without affecting the system's processing of other languages. This enables financial institutions to efficiently handle the needs of customers from different language backgrounds and improve customer experience and service quality.

[0064] By configuring multiple language feature expert modules, the system can process the uniqueness of different languages, avoiding the performance degradation problem when a single model processes multiple languages. The introduction of expert modules enables the model to specifically optimize the acoustic and semantic features of different languages, improving recognition accuracy. At the same time, the independence of expert modules ensures that the introduction of new languages ​​will not affect the recognition effects of other languages, enhancing the scalability of the model.

[0065] S40, obtaining training speech data, and dividing the training speech data into a plurality of batches of data, each batch of data containing a plurality of speech samples;

[0066] In this embodiment, a speech data set for training is obtained from a data source. The data set contains speech samples in multiple languages ​​for subsequent model training and fine-tuning. These speech data are usually labeled to indicate the corresponding language type and other relevant information.

[0067] Divide a large amount of training speech data into several batches. Since it is impossible to process all the data at once when training large-scale data, batching can effectively reduce memory usage, increase computing efficiency, and ensure that the model can gradually update weights. The amount of data in each batch is determined by factors such as the system's computing power and memory size.

[0068] In deep learning frameworks (such as TensorFlow and PyTorch), batch processing is usually implemented through DataLoader or Dataset. These tools can automatically split large-scale datasets into predetermined batch sizes and gradually pass them to the model for training. Each batch is independently fed into the neural network during training to complete a forward and backward propagation.

[0069] The data in each batch consists of several speech samples, which usually come from datasets of different languages ​​or the same language. By processing speech samples in batches, the model can process multiple speech inputs in parallel, thereby improving training efficiency and the generalization ability of the model.

[0070] The number of speech samples in a batch is usually controlled by the parameter batch size. The batch size can be adjusted according to the hardware performance and the requirements of the training task. For example, commonly used batch sizes are 16, 32, 64, etc. Each speech sample is preprocessed into a feature vector or Mel-frequency cepstral coefficients (MFCC), and then input into the neural network layer of the model together.

[0071] Example description: In the voice customer service system of global financial services, a large amount of voice input data from different countries and regions needs to be processed. By dividing the training voice data into multiple batches, the bank's voice recognition system can efficiently process data in different languages ​​at one time. For example, the system obtains voice data from customers in multiple countries from the database and divides it into several batches, each of which contains voice input from multiple countries. This not only improves the training efficiency, but also ensures that the system can evenly recognize voice input in different languages ​​when facing multilingual customers, optimizing the global customer service experience.

[0072] By dividing the training speech data into multiple batches for processing, the system can effectively avoid memory overflow problems while increasing the stability and parallel efficiency of training. Batch processing allows only a portion of the data to be used for each training, which can better randomize model updates, prevent the model from falling into a local optimal solution, and improve the generalization ability of the model. Especially in the process of multilingual data training, batch processing can ensure that each batch contains speech samples of different languages ​​in a balanced manner, thereby enhancing the multilingual adaptability of the model.

[0073] S50, selecting a corresponding language feature expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language feature expert module to obtain an output result;

[0074] In this embodiment, voice samples are extracted from existing voice data sets (such as LibriSpeech, CommonVoice, etc.). Each voice sample is equipped with a language identifier in the data set to facilitate language classification during subsequent training. These voice data can be obtained and processed from different data sources through an automated data processing pipeline.

[0075] After acquiring the speech samples, the system selects the appropriate language feature expert module based on the language identifier of each sample. This step ensures that each speech sample is processed by the expert module that best suits its language features, avoiding feature confusion between different languages.

[0076] Through language identification or language detection algorithms (such as text-based or phoneme-based detection), the system can determine the language type of each speech sample. Based on the determined language type, the system dynamically selects the corresponding language feature expert module to ensure that the speech sample is processed by the expert module of the specific language.

[0077] Each selected speech sample is input into the selected language feature expert module for processing. Each expert module is trained in a specific language, can capture the unique characteristics of the language, and analyze and process the input speech accordingly.

[0078] After the speech sample is input into the expert module, the neural network of the expert module will extract the features of the speech, such as Mel-frequency cepstral coefficients (MFCC), pitch, phonemes, etc. Then, the expert module will process the speech according to the specific language characteristics and generate optimized output results.

[0079] After the language feature expert module processes the speech sample, the system will get the corresponding output result. This output result can be a feature vector or a direct speech recognition result, depending on the design of the module and the task requirements.

[0080] The expert module processes the speech sample through a series of convolutional or recurrent neural network layers to generate a high-dimensional feature vector or language model probability distribution. The output can be used for further classification, recognition or downstream tasks such as speech-to-text.

[0081] Example: In cross-border financial services, customers in different countries use different languages ​​for voice input. The bank's voice recognition system obtains the customer's voice data and first determines the language used by the customer. The system then automatically selects the language feature expert module that matches the customer's language to process the customer's voice input. The expert module generates accurate recognition results by performing specific optimization on the customer's language features. In this way, the system can more accurately recognize customer requests, whether the customer uses English, French or Chinese, thereby improving the efficiency and quality of global customer service.

[0082] By selecting the most suitable language feature expert module for each speech sample, the system can be optimized specifically for speech data in different languages ​​to ensure the accuracy and efficiency of speech recognition. This dynamic selection mechanism based on language features improves the adaptability of the model and can avoid confusion and performance degradation in multilingual scenarios. In addition, the expert module processes language features to make the recognition results of different languages ​​more accurate and reduce the error rate in cross-language recognition.

[0083] S60, assigning a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0084] In this embodiment, the system needs to assign an identifier corresponding to the language type of each speech sample in each batch. This means that the system will identify the language of each speech sample and mark which language the sample belongs to. This language identifier helps the system dynamically select the appropriate language processing module or adapter during training or inference.

[0085] The system tags each speech sample with a language identifier through a language detection algorithm or metadata-based annotation. For example, an identifier can be directly assigned based on predefined metadata of a speech dataset (such as a "language type" field). It is also possible to infer the language type during training by analyzing the phoneme features or acoustic model of a speech sample and label it as a specific language. Commonly used identification methods include ISO language codes (such as EN for English, ZH for Chinese, etc.).

[0086] The assigned language identifier is used to indicate the language type of each speech sample, so that the model can process it according to the language type. This ensures that the system uses the correct language feature module or adapter during processing, avoids interference between languages, and thus improves recognition accuracy.

[0087] A "language identifier" field is appended to the data structure of each speech sample and passed to the model. During processing, the system will decide which language feature expert module or language dependency adapter to use based on the language identifier. This language identifier can be automatically processed through the input pipeline of the training data and passed to the model together with other information in the batch.

[0088] Example description: In global banking, customers interact with banking systems through voice input. The system receives voice requests in different language contexts, such as English, Chinese, French, etc. By assigning a corresponding language identifier to each voice sample, the bank's speech recognition system can quickly identify the customer's language type and call the appropriate language feature module for processing. For example, when a customer makes a voice request in Spanish, the system automatically recognizes and selects the Spanish adapter for processing to ensure the accuracy of speech recognition. This mechanism can greatly improve the efficiency and customer satisfaction of multilingual customer service.

[0089] By assigning a corresponding language identifier to each speech sample, the system can classify and process samples of different languages ​​in a multilingual environment. Assigning language identifiers not only improves the efficiency of the system during training and reasoning, but also ensures that the system selects the most appropriate module or model path when processing different languages, avoiding confusion between language features. In addition, the clarity of language identifiers enhances the flexibility of the system, enabling it to cope with multilingual input and improve recognition accuracy.

[0090] S70, determining a target low-resource language, freezing parameters of language-dependent adapters that do not correspond to the target low-resource language, so as to train and optimize only parameters of the language-dependent adapter corresponding to the target low-resource language;

[0091] In this embodiment, low-resource languages ​​that need to be optimized and trained are identified and determined. Low-resource languages ​​generally refer to languages ​​for which data is relatively scarce and model training is insufficient in the field of speech recognition. The goal of identifying low-resource languages ​​is to focus on optimizing the recognition capabilities of these languages ​​and improve their performance in the model.

[0092] By analyzing the usage of speech datasets or systems, we can identify which languages ​​are low-resource languages. For example, we can dynamically or manually select target low-resource languages ​​for optimization based on indicators such as the amount of language data, the quality of speech training samples, and the performance of the model (such as a high word error rate WER).

[0093] "Freezing" means locking the parameters of other language adapters that are not the target low-resource language, so that they remain unchanged during training to avoid affecting the performance of other language adapters. After freezing the parameters, training will only be optimized for the target low-resource language adapter.

[0094] In the model training framework, parameter freezing is usually achieved by setting requires_grad = False for specific language adapter modules. After freezing, these modules will not update parameters during training to avoid affecting the performance of other languages. This ensures that only the adapters for the target low-resource language are fine-tuned and optimized.

[0095] Centrally train and optimize the adapter parameters of the target low-resource language. This approach avoids the computational overhead of training all language adapters at the same time and focuses on optimizing the target language to improve its recognition accuracy. During the training process, only the adapter module corresponding to the target language is activated, and its parameters are optimized through backpropagation. The model will be trained using the data of the target low-resource language, and the adapter parameters of the language will be gradually optimized to the ideal state through multiple rounds of iterations.

[0096] Example: In global banking, some bank customers may use some relatively scarce low-resource languages, such as some minority languages ​​or regional languages. The bank's speech recognition system may perform poorly on these low-resource languages. By identifying the target low-resource languages, the system can specifically optimize the recognition capabilities of these languages ​​to ensure that when customers use these languages ​​for voice operations, the system can more accurately recognize their requests. For example, the system can automatically freeze the adapter parameters for mainstream languages ​​and only train the Nepali language adapter to improve its performance in speech recognition, ensuring the efficiency of financial services and user satisfaction.

[0097] By training and optimizing only the adapter for the target low-resource language, the system can concentrate resources and computing power to improve the recognition of low-resource languages ​​without affecting the recognition performance of other languages. This mechanism of freezing adapters for other languages ​​not only greatly reduces computing costs, but also improves the model accuracy of low-resource languages, solving the problem of inefficient processing of low-resource languages ​​in existing technologies. Especially in multi-language scenarios, this method can achieve efficient model optimization and maintenance.

[0098] S80, selecting a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and training the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter;

[0099] In this embodiment, each speech sample in the batch data has its corresponding language identifier, which is used to indicate the language type to which the sample belongs. This step refers to the system automatically selecting a dependent adapter suitable for the language based on the language identifier of the speech sample, especially for samples of low-resource languages, to ensure that the system can call the corresponding low-resource language adapter for processing and training.

[0100] In the data processing pipeline, the language identifier of the speech sample is extracted and passed to the model's language adapter selection module. This module matches and activates the corresponding low-resource language adapter based on the language identifier to ensure that the speech sample can be processed by the appropriate adapter module. The specific implementation can be completed through a lookup table or a dynamic adapter selection mechanism.

[0101] After the low-resource language adapter is determined, the speech sample will be input into the corresponding language adapter for processing and output. This step refers to the back propagation training of the adapter through the output of the speech sample. Through this process, the system can gradually optimize the parameters of the low-resource language adapter and improve its performance.

[0102] The model will perform forward propagation based on the input speech sample and the language adapter corresponding to the sample to generate an output result. Then, the system will calculate the loss function based on this output result, and update and optimize the parameters of the low-resource language adapter through a back-propagation algorithm (such as gradient descent). The training process will be repeated until the parameters of the low-resource language adapter are fully optimized.

[0103] Through the training process of back propagation and parameter update, the system can gradually optimize and adjust the parameters of the low-resource language adapter to make it more adaptable to the characteristics of low-resource languages. In this way, the performance of low-resource languages ​​in speech recognition tasks can be improved and the error rate can be reduced.

[0104] In each round of training, the parameters of the low-resource language adapter are updated through feedback from the loss function. The optimization algorithm used may be SGD (stochastic gradient descent), Adam optimizer, etc. The parameter update is fine-tuned based on the error of each speech sample, so that the performance of the adapter can reach the expected effect.

[0105] Example description: In global banking, the system needs to handle customer requests from different languages. For customers who use low-resource languages ​​(such as certain minority languages ​​or local dialects), the speech recognition system may face greater challenges. By configuring independent adapters for these low-resource languages ​​and dynamically selecting the language identifier based on the customer's voice input, the system can accurately train and optimize for these languages. For example, when a customer uses Urdu for voice input, the system will select the corresponding Urdu adapter and optimize the training based on the voice input by the customer, gradually improving the recognition accuracy and ensuring the accuracy of banking services and customer satisfaction.

[0106] By selecting low-resource language adapters based on the language identification of speech samples and conducting targeted training based on the output results, the system can focus on optimizing the recognition capabilities of low-resource languages. This approach effectively avoids the interference of high-resource languages ​​on low-resource languages ​​in multilingual systems, while reducing the waste of computing resources. Optimizing and adjusting the parameters of the low-resource language adapter not only improves the recognition accuracy of low-resource languages, but also enhances the system's adaptability in multilingual scenarios and improves overall performance.

[0107] S90, until the plurality of batches of data are trained, generating a trained target language dependent adapter to obtain a target speech recognition model;

[0108] In this embodiment, the entire training process will continue until all batches of data are used for training. During the training process, the system will process the training speech data in batches, and each batch of data is used to optimize and update the parameters of the low-resource language dependent adapter. The completion of this step indicates that all speech samples of the batch data have participated in the training.

[0109] In the deep learning framework, this step is completed by looping through all batches of data. The system will update the model parameters after each batch is trained until all batches of data have been processed for one or more rounds. A common implementation is to process the dataset through a for loop or an iterator built into the training framework.

[0110] After the training process for all batches of data is completed, the dependent adapter for the low-resource language is fully optimized, and finally a trained adapter module is generated. This adapter is specifically optimized for the characteristics of the target language, adapted to the acoustic and semantic characteristics of the language, and can work efficiently in speech recognition tasks.

[0111] After each round of training, the adapter's parameters are updated and stored. After the entire training process is complete, the final adapter state is saved. At this point, the resulting adapter contains an optimized set of parameters specifically designed to handle the target low-resource language.

[0112] After the training and optimization of the adapter is completed, the adapter module in the entire speech recognition system and the base model together constitute the final target speech recognition model. This model can accurately recognize and process speech input in the target low-resource language while maintaining recognition capabilities for other languages.

[0113] The final speech recognition model contains the integration of the basic speech recognition model and the trained language-dependent adapter. This model can be used in the inference phase to recognize the input low-resource language speech by calling the trained adapter. The system can save this model as the final inference model for subsequent speech recognition tasks.

[0114] Example description: In global financial services, banks need to process voice requests from different countries and regions, some of which use relatively scarce, low-resource languages. By training the voice data of these low-resource languages ​​batch by batch, the system is able to generate specially optimized language-dependent adapters. After all batches of training are completed, the generated target speech recognition model will be able to accurately recognize these low-resource languages. For example, after multiple rounds of training optimization, the system generated a language-dependent adapter for Filipino or Urdu, and integrated it with the basic speech recognition model to ensure that the system can handle voice requests from customers in these languages. This can enhance the bank's global customer service experience and ensure that users of different languages ​​can enjoy efficient speech recognition services.

[0115] By training all data batch by batch, the system can ensure that the adapter of the target low-resource language is fully optimized, and finally generate an efficient dependent adapter suitable for a specific language. This training mechanism not only improves the recognition accuracy of the target language, but also reduces the computational burden on high-resource languages. The final integrated target speech recognition model combines the basic model and the optimized adapter, which can effectively process the speech input of low-resource languages ​​in multilingual scenarios, improving the overall performance of the system.

[0116] S100, performing speech recognition based on the target speech recognition model.

[0117] In this embodiment, a target speech recognition model that has been trained and integrated with a language-dependent adapter is used to process speech input. The target speech recognition model includes a basic speech recognition model and an optimized language-dependent adapter, and is specifically used to process speech recognition tasks in multilingual scenarios, especially for optimization of low-resource languages.

[0118] In the inference phase, the target speech recognition model receives the input speech signal and selects the appropriate language-dependent adapter module based on the language identifier of the signal for processing and recognition. The model uses the trained parameters and structure, language characteristics and acoustic models to analyze and decode the speech signal and output the final recognition result.

[0119] The model processes the input speech signal and outputs text or other forms of recognition results. The speech recognition process usually includes steps such as speech signal preprocessing, feature extraction, language model inference, and final text decoding. The target speech recognition model uses the parameters and characteristics learned during the training process to perform feature analysis on the speech signal and generate corresponding text or semantic output.

[0120] During speech recognition, the system first receives the input speech signal and performs preprocessing, and then extracts and decodes the features through the target speech recognition model. The model further processes the input speech features using an optimized language-dependent adapter to generate recognition results. The final output can be text, command execution, or input results of other applications.

[0121] Example description: In the global customer service of a bank, customers can communicate with the bank through telephone or voice system. The system uses the target speech recognition model to process the customer's voice request and selects the appropriate language-dependent adapter according to the language used by the customer. Whether the customer uses English, Chinese or a low-resource language (such as Swahili), the system can accurately recognize the customer's voice content and generate responsive text or instruction output. For example, a customer can inquire about the account balance or perform a transfer operation by voice, and the system will generate accurate recognition results based on the target speech recognition model to ensure the smooth operation.

[0122] By using the target speech recognition model, the system can efficiently and accurately process user voice input in multilingual scenarios, especially achieving better results in the recognition of low-resource languages. The target speech recognition model combines the basic model with an optimized language-dependent adapter, making the system multilingual adaptable and able to dynamically select and process voice input in different languages, improving the accuracy and efficiency of speech recognition. This efficient speech recognition capability enables the system to be applied to more multilingual scenarios, enhancing user experience and interaction capabilities.

[0123] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a multilingual speech recognition method. By obtaining a pre-trained basic speech recognition model, the model can recognize multiple languages ​​and freeze its parameters. Configure a language-dependent adapter, and configure an adapter for each language. Configure multiple language feature expert modules to process different language features. Obtain training voice data, select a corresponding language feature expert module and input a voice sample, divide the training voice data into multiple batches of data, and assign a language identifier to each sample. Determine the target low-resource language, freeze the parameters of other adapters, only train the adapter of the target language, select the corresponding adapter based on the language identifier and train until the training of all batches of data is completed, and generate a target speech recognition model. The present invention configures a language-dependent adapter and a language feature expert module, so that the characteristics of different languages ​​are processed in a targeted manner, and the adaptability and recognition accuracy of the model are significantly improved. By freezing the adapter of the non-target language, unnecessary resource consumption is reduced, focusing on the optimization of the target low-resource language, and ensuring the efficiency of the training process. In addition, the scalability of the model is enhanced, and it can quickly adapt to the addition of new languages, reducing the complexity of deployment and maintenance.

[0124] In one embodiment, in the above S40, after obtaining the training voice data, the following step is further included:

[0125] S401, compressing the training speech data through a down-projection matrix to reduce the feature dimension of the training speech data;

[0126] S402, using an activation function to process the compressed training speech data;

[0127] S403, processing the training speech data processed by the activation function through an up-projection matrix to restore the original feature dimension of the training speech data.

[0128] In this embodiment, the training speech data is processed using a down-projection matrix in order to reduce the feature dimension of the data. Reducing the dimension can reduce computational overhead while keeping key information intact. The projection matrix maps high-dimensional features to low-dimensional space through linear transformation, which helps improve training efficiency, especially when processing complex data sets.

[0129] Using the linear transformation formula, the input high-dimensional data X is transformed through the projection matrix D to obtain low-dimensional data X'=D*X. This lower projection matrix D is set before training, usually implemented through matrix operations or dimensionality reduction algorithms (such as PCA). Its function is to extract the most representative features from high-dimensional data while reducing the data dimension.

[0130] Nonlinear activation is performed on the compressed data to further process the nonlinear features in the data. Activation functions are usually used to enhance the model's ability to express complex relationships. Common activation functions include ReLU, Sigmoid, and Tanh, etc. The purpose is to introduce nonlinearity so that the model can learn and fit complex speech features.

[0131] On the low-dimensional feature data compressed by the down-projection matrix, the activation function ReLU(x)=max(0,x) or other nonlinear functions are applied to process the data. The activation function will retain the effective positive signal of the data and enable the model to handle the changes of complex speech data through nonlinear transformation.

[0132] After being processed by the activation function, the data will be restored to the original feature dimension through the up-projection matrix. This step is to ensure that the data can continue to be processed in the subsequent layers of the model without losing information due to dimensionality reduction. The up-projection matrix restores the data from the low-dimensional space to the original dimension through an inverse linear transformation to maintain the consistency of input and output.

[0133] The reverse transformation is performed through the projection matrix U, and the formula is X" = U*X', where X' is the low-dimensional data processed by the activation function, and X" is the restored high-dimensional data. The purpose of this is to further process the speech data in the model to ensure that the subsequent steps can process data of the same dimension as the input feature.

[0134] In this embodiment, by processing data using the lower projection and upper projection matrices, the system can maintain the integrity of the voice data while reducing the computational complexity. The use of the activation function further enhances the model's ability to process complex voice features. This structure not only improves the efficiency of training and reasoning, but also avoids information loss due to dimensional compression. In particular, when processing large-scale voice data or real-time voice tasks, the response speed and performance of the system can be effectively improved.

[0135] In one embodiment, the above S50 includes:

[0136] S504, determining, by a router module, a matching score between each speech sample in the training speech data and each language feature expert module;

[0137] S505, selecting a preset number of language feature expert modules ranked first in matching scores;

[0138] S506, inputting the speech samples into each selected language feature expert module respectively to obtain multiple expert output results;

[0139] S507: Based on the matching scores, weighted processing is performed on the output results of the multiple experts to obtain a weighted final output result.

[0140] In this embodiment, the router module (RouterModule) is a key component responsible for dynamically selecting and allocating between multiple language feature expert modules or calculation paths in the multilingual speech recognition system. Its core function is to select the most appropriate language feature expert module according to the input speech sample features (such as language identification, speech features, etc.), and route the input signal to these modules for processing.

[0141] The main function of the router module is to dynamically select the most appropriate expert module for processing based on the input speech sample and the adaptability of each language feature expert module. For example, in a multilingual environment, different expert modules specialize in processing the characteristics of different languages, and the router module will select the corresponding module based on the language identifier of the input speech. The basis for selection can be specific language features, acoustic features of speech, or preset model parameters.

[0142] The router module determines which expert module is most suitable for processing the current input data by calculating the matching score between each input sample and different expert modules. Usually, the matching score can be calculated by methods such as vector similarity or Softmax. The higher the matching score, the better the processing effect of the expert module on the current speech sample may be.

[0143] Based on the matching score, the router module selects the top-ranked expert modules and sends the input data to these modules for processing. The router module acts as a "middleman" to ensure that the input speech sample can be processed by the most appropriate module. The router module determines the flow of data and is the key to the collaborative work of modules in the system.

[0144] After selecting multiple language feature expert modules, the router module also needs to perform weighted processing on the outputs of these modules to obtain the final result. The weighted processing is usually based on the matching scores of the expert modules, that is, the expert modules with high scores have a greater impact on the final output results.

[0145] For example, the router module is responsible for calculating the matching score between the input speech sample and different language feature expert modules based on the features of the input speech sample. The score is used to measure the adaptability of each expert module to processing the speech sample. The matching score selects the most suitable language feature expert module based on the language features in the speech sample. The router module can generate a matching score matrix by calculating the input feature vector (such as speech features, language identifiers, etc.) and the preset weights of each language feature expert module through probability distribution algorithms such as Softmax. The score calculation formula is usually based on vector dot product or similarity measurement, and the score is used to represent the adaptability of the speech sample and each expert module.

[0146] After calculating the matching scores, the system selects the top-ranked expert modules. The preset number refers to selecting a certain number of expert modules for speech processing in the matching score ranking. Usually the first two or several expert modules are selected, which can effectively share the computational burden and improve the accuracy of the results. After determining the matching scores of each speech sample and all expert modules, the system sorts these scores and selects the first preset number of expert modules (such as the top 2 or top 3). The selection process is usually completed through a sorting function to ensure that the system selects the expert module that best suits the current speech sample.

[0147] The selected expert modules will each receive the input speech samples and process the samples independently. The processing results of each expert module will be recorded to generate multiple output results. Each output result reflects the specific processing and analysis of the speech sample by the module. The speech sample is input into each selected language feature expert module, and each module processes the sample according to its own specific language feature model. The expert modules usually use optimized deep learning models, such as convolutional neural networks (CNN) or long short-term memory networks (LSTM), to extract and classify speech features and generate their own output results.

[0148] The output results of each expert module need to be processed comprehensively, and the system weights the output of each expert module according to the matching score of each expert module. The higher the matching score of the expert, the greater the weight of its output in the final result. Through weighted processing, the system generates a comprehensive and optimized final output result. The system combines the output results of each expert module according to the matching score of each expert, using weighted average or other weighted methods. The higher the weight of the expert output, the greater the proportion in the final result. The formula is: Final_output = Σ(Wi*Outputi), where Wi is the expert weight and Outputi is the corresponding expert output. The final weighted result is the final processing result of the speech sample.

[0149] In this embodiment, by using the router module to dynamically select the language feature expert module that best matches the speech sample, the system can process the speech sample more specifically and improve the accuracy of speech recognition. Weighted processing ensures that the output of each expert module can be reasonably distributed in the final result, avoiding the limitations of single module processing. Multiple expert modules working in parallel can further improve the processing speed and reduce the interference between language features.

[0150] In one embodiment, the above S10 includes:

[0151] S101, collects speech datasets in multiple languages;

[0152] S102, building a deep learning model for multilingual speech recognition;

[0153] S103, training the deep learning model using the speech data set, and adjusting the model parameters through multiple rounds of iterations to obtain the basic speech recognition model;

[0154] S104, freezing the parameters of the basic speech recognition model, wherein the parameters include parameters of an input layer, a hidden layer, and an output layer.

[0155] In this embodiment, it is first necessary to collect a speech dataset containing multiple languages ​​for training the basic speech recognition model. This dataset needs to cover all languages ​​supported by the system target and contain speech samples in each language. These speech samples are used for model training to help the model learn how to recognize and distinguish speech features in different languages. The construction of the dataset can be achieved through existing speech datasets (such as LibriSpeech, Common Voice, etc.) or custom datasets to ensure coverage of different languages ​​and accents. The collection of speech samples can include various pronunciation scenarios, such as daily conversations, reading texts, questions and answers, etc.

[0156] Use deep learning technology to build a basic speech recognition model that can recognize multiple languages. The model should be able to process inputs in different languages ​​and be able to extract and classify speech features in each language. Common model structures include convolutional neural networks (CNN), long short-term memory networks (LSTM), Transformer, etc., which usually perform well in speech recognition. Use neural network frameworks (such as TensorFlow, PyTorch, etc.) to build a multilingual speech recognition model. The input of the model is speech features (such as MFCC, Mel spectrum, etc.), and the output is a text sequence or phoneme label of the speech. The model structure can include multi-layer convolutional networks and recurrent networks to capture the temporal and spatial characteristics of speech.

[0157] Using the collected multilingual speech datasets, the constructed deep learning model is trained and the model parameters are adjusted so that the model can learn and master the speech features of multiple languages. During the training process, the model parameters are gradually optimized through multiple rounds of iterations until the recognition performance of the model on all target languages ​​reaches the expected level. The training process is usually carried out through multiple rounds of iterations (epochs). In each round, the dataset is divided into multiple batches and input into the model, and the model updates the parameters through backpropagation. Optimization algorithms (such as Adam or SGD) are used to adjust the network weights and reduce the model's loss function (such as cross entropy loss) to improve the accuracy of speech recognition.

[0158] After the model training is completed, freezing the parameters of the basic speech recognition model means that all parameters of the model will no longer be updated. Freezing the parameters of the input layer, hidden layer, and output layer of the model can ensure that the model remains stable during the subsequent fine-tuning process and does not affect the performance of other training modules. The freezing operation is mainly to reduce the computational burden and focus on training other modules that depend on the basic model (such as language-dependent adapters). By setting requires_grad = False of the model layer to freeze the parameters of each layer, ensure that the parameters of all layers of the basic speech recognition model (including input layer, hidden layer, and output layer) remain unchanged during subsequent training. After freezing, the weights of the model will not be updated during the back-propagation process, thereby focusing computing resources on fine-tuning other modules.

[0159] This embodiment collects data sets in multiple languages ​​and builds a multilingual speech recognition model, so that the system can accurately recognize the user's voice input in a multilingual environment. Freezing the parameters of the basic speech recognition model ensures that other modules can be further optimized on a stable basis during subsequent fine-tuning without affecting the overall performance of the model. This training and freezing strategy greatly reduces training time and improves the scalability of the speech recognition system, especially in multilingual scenarios, which can significantly improve recognition accuracy and processing efficiency.

[0160] In one embodiment, the above S20 includes:

[0161] S201, define a language-dependent adapter structure for each language;

[0162] S202, for each language, using the corresponding speech data set to train a corresponding language-dependent adapter, so that the language-dependent adapter learns acoustic features and language features specific to each language;

[0163] S203, integrating the language-dependent adapter into the basic speech recognition model;

[0164] S204, configuring a unique identifier for the language-dependent adapter of each language, so as to correctly select and apply the corresponding language-dependent adapter during the model training process.

[0165] In this embodiment, in a multilingual speech recognition system, a separate language dependent adapter (LDA) is set for each target language, and these adapters will specifically process the characteristics and speech features of each language. The structure of the adapter is mainly designed to capture the acoustic and semantic features of different languages ​​​​in order to make accurate recognition when processing different language inputs. The system can define a specific language adapter network structure for each language, which can be a simplified deep neural network layer that specifically processes the characteristics of the language. The structure of each language adapter will be different in its input, hidden layer, and output layer to adapt to the characteristics of each language.

[0166] Each language-dependent adapter needs to be trained with a dedicated speech dataset to learn the acoustic and linguistic features of the language. These features include the speech frequency distribution, phoneme combinations, and grammatical patterns of speech streams in the language. Through training, the language adapter will be able to accurately process and recognize speech input of the language. By inputting the dedicated speech dataset of each language into the corresponding language adapter for training, the adapter will gradually optimize its parameters through multiple rounds of iterations. The training process uses a backpropagation mechanism and optimization algorithms such as Adam or SGD. The training goal of each language adapter is to match its output with the specific features of the language, thereby optimizing the recognition ability of the language.

[0167] The language-dependent adapter does not work independently, but needs to be combined with the basic speech recognition model. The integration process refers to combining the dependency adapter of each language with certain modules in the model (such as the middle layer and the output layer) so that it can be processed according to the input language characteristics during model reasoning. In the structure of the basic speech recognition model, each language-dependent adapter is embedded in the model through an interface or modularization. For example, the basic model processes basic speech features, and when a specific language is recognized, the system calls the corresponding language-dependent adapter to further process the speech data.

[0168] To ensure that the model can correctly select the adapter for each language during training and inference, the system needs to configure a unique identifier (ID) for each language-dependent adapter. The identifier helps the model automatically select and activate the corresponding adapter when faced with multilingual voice input, ensuring that each voice sample is processed by the appropriate module. The system can assign a unique identifier to each language-dependent adapter through a configuration file or database. Whenever the model receives a voice sample, the system selects the corresponding adapter based on the language type of the sample. The identifier can be associated with a language code (such as "en" for English and "fr" for French) to ensure the correct mapping of samples to adapters.

[0169] In this embodiment, by configuring a language-dependent adapter for each language, the system can optimize processing based on the characteristics of different languages. Each adapter learns the unique acoustic and linguistic features of the language to ensure recognition accuracy in a multilingual environment. Configuring a unique identifier ensures that the model can quickly and accurately select and call the corresponding adapter, enhancing the system's multilingual processing capabilities. After integrating the adapter, the speech recognition model can not only ensure the efficiency of the basic model, but also make additional optimizations based on the needs of specific languages, improving the overall performance of the system.

[0170] In one embodiment, the above S30 includes:

[0171] S301, determining a language list, and defining corresponding language characteristics for each language in the language list;

[0172] S302, configuring a corresponding language feature expert module for each language feature,

[0173] S303, integrating the language feature expert module into the middle layer or output layer of the basic speech recognition model.

[0174] In the present embodiment, the system needs to clearly support a list of languages, and each language has its specific voice characteristics (such as tone, syllable structure, grammatical pattern, etc.). Define its exclusive language characteristic expert module according to the characteristics of different languages, so as to better handle the voice input of the language. By analyzing each language, the voice characteristics and semantic rules of the language are identified. Feature vectors can be extracted based on the voice data set to define the characteristics of the language, such as frequency distribution, rhythmic pattern and phoneme combination, etc. For example, the tone characteristics of Chinese, the consonant group combination characteristics of English, etc. The language list can include commonly used languages ​​(such as English, French, Spanish, etc.) and low-resource languages.

[0175] Assign a dedicated language feature expert module to each language feature to ensure that the module can handle the specific features of the language. A language feature expert module is a neural network or model module that specializes in processing and optimizing the specific speech features of the language, ensuring that the input speech sample can be processed according to the specific language features. By creating independent expert modules for each language feature, the model can perform speech processing according to the needs of different languages. The parameters and structure of each expert module will be specifically designed and trained for the speech characteristics of a certain language to ensure that the module can capture and process the unique characteristics of the language. Expert modules can be part of models such as convolutional neural networks (CNN), recurrent neural networks (RNN), or Transformer.

[0176] The language feature expert module will not run independently, but needs to be integrated with the basic speech recognition model. The integration method can be to combine the processing results of the expert module with the basic model in the middle layer or output layer, so that the basic model can use the output of the expert module for the final speech recognition. In terms of model structure, the language feature expert module can be inserted into an middle layer or output layer of the basic model. The specific implementation can be to splice or fuse the output of each expert module with other layers of the basic model through a multi-task learning framework. The integrated model can further process speech data according to the output of the expert module with different language features and improve the recognition performance of a specific language.

[0177] By configuring an independent language feature expert module for each language, this embodiment enables the system to more accurately capture and process the specific speech features of each language, thereby improving the accuracy of multilingual speech recognition. By integrating these expert modules with the basic model, the system can optimize the processing capabilities of specific languages ​​without affecting overall performance. The use of multilingual feature expert modules makes the system more adaptable and flexible when facing different languages, especially in multilingual or low-resource language environments, significantly improving recognition accuracy.

[0178] In one embodiment, the above S100 includes:

[0179] S1001, receiving a voice signal to be recognized;

[0180] S1002, performing a preprocessing operation on the speech signal to be recognized, wherein the preprocessing operation includes denoising and normalization processing;

[0181] S1003, extracting speech features from the speech signal to be recognized after the preprocessing operation, wherein the speech features include Mel-frequency cepstral coefficients and pitch features;

[0182] S1004, determining a corresponding language-dependent adapter in the target speech recognition model based on the language type of the speech signal to be recognized;

[0183] S1005, processing the speech feature through the language-dependent adapter, wherein the processing includes feature space mapping, nonlinear activation and normalization processing;

[0184] S1006, recognizing the processed speech features through the target speech recognition model to generate a recognition result.

[0185] In this embodiment, the system receives a voice input signal as the beginning of the voice recognition task. The voice signal can come from a recording device, a phone or other audio input device, and the system needs to process the signal to complete the voice recognition. Real-time voice data can be captured through a microphone or an audio interface, or the voice signal can be read from a pre-recorded audio file. The voice signal is input to the voice recognition system in a digitized form.

[0186] To improve the accuracy of speech recognition, the system first preprocesses the input speech signal. The preprocessing steps include removing background noise (denoising) and adjusting the signal amplitude to a standard range (normalization). These operations help improve the effectiveness of subsequent feature extraction and recognition. Denoising can use filters (such as low-pass filters or spectral subtraction) to remove background noise from the speech signal. Normalization can scale the amplitude of the speech signal to a fixed range to avoid recognition errors caused by differences in signal strength.

[0187] Feature extraction is a key step in speech recognition systems. MFCC is one of the commonly used features in speech recognition, which can capture the frequency characteristics of speech signals. Tone features are used to identify pitch information in speech, which is particularly important for processing emotions and intonation changes in speech. The MFCC algorithm is used to perform frequency domain analysis on speech signals and extract their cepstral coefficients. First, the speech signal is Fourier transformed to generate a spectrogram, and then the spectrogram is transformed using a Mel filter bank to obtain the Mel frequency cepstral coefficients (MFCC). Tone features can extract pitch information in speech through a frequency tracking algorithm.

[0188] The system selects the appropriate language-dependent adapter based on the language type of the speech signal. Adapters are modules designed specifically for different languages ​​that optimize speech recognition performance for a specific language. Language-dependent adapters process speech signals based on language characteristics to improve recognition accuracy. By analyzing the language identity or language features of the speech signal, the system selects a language-dependent adapter that matches the language. The language type can be detected during the preprocessing phase or inferred from user input or context. Each language has a corresponding language-dependent adapter that performs speech processing based on specific language features.

[0189] During speech recognition, the language-dependent adapter further processes the input speech features. Feature space mapping refers to projecting the original speech features into a new feature space to better adapt to subsequent speech recognition tasks. Non-linear activation is used to enhance the learning ability of the model, while normalization ensures that the output features are within a reasonable range. The speech features are converted into feature representations suitable for model processing through feature space mapping. Non-linear activation functions (such as ReLU) are used to enhance the non-linear ability of processing. Normalization adjusts the feature values ​​to a uniform range to prevent features that are too large or too small from affecting the recognition effect.

[0190] The system uses the target speech recognition model to recognize the processed speech features and generate the final recognition results. The recognition results are usually in text form, or they can be the output of other speech understanding tasks (such as command or intent recognition). The processed speech features are classified or decoded through a trained speech recognition model (such as a model based on a convolutional neural network or a recurrent neural network). The system outputs the corresponding text or semantic results based on the results of feature extraction. This stage uses the core reasoning part of the model to map speech features to specific speech tags or text representations.

[0191] This embodiment can capture and process speech input in different languages ​​more accurately through preprocessing, feature extraction, and language-dependent adapter-based processing of speech signals. Feature space mapping, nonlinear activation, and normalization further enhance the recognition effect, making the system have higher recognition accuracy in multilingual scenarios.

[0192] In one embodiment, a multilingual speech recognition device is provided, and the multilingual speech recognition device corresponds one-to-one to the multilingual speech recognition method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the multi-language speech recognition device of the present invention. Basic model acquisition module 10, language adapter configuration module 20, expert module configuration module 30, data batch division module 40, training data processing module 50, language identification assignment module 60, low-resource language adapter control module 70, low-resource language adapter training module 80, adapter training completion module 90 and speech recognition module 100. The functional modules are described in detail as follows:

[0193] A basic model acquisition module 10 is used to acquire a pre-trained basic speech recognition model, the basic speech recognition model being able to recognize multiple languages, and freeze the parameters of the basic speech recognition model;

[0194] A language adapter configuration module 20, configured to configure a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter;

[0195] An expert module configuration module 30, configured to configure a plurality of language feature expert modules in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature;

[0196] The data batch division module 40 is used to obtain training speech data and divide the training speech data into a plurality of batches of data, each of which contains a plurality of speech samples;

[0197] The training data processing module 50 is used to select a corresponding language characteristic expert module for each speech sample in the training speech data, and input the speech sample into the corresponding language characteristic expert module to obtain an output result;

[0198] A language identifier assignment module 60 is used to assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0199] A low-resource language adapter control module 70 is used to determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language;

[0200] A low-resource language adapter training module 80 is used to select a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and train the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter;

[0201] The adapter training completion module 90 is used to generate a trained target language dependent adapter until the plurality of batches of data are trained to obtain a target speech recognition model;

[0202] The speech recognition module 100 is used to perform speech recognition based on the target speech recognition model.

[0203] In one embodiment, the training data processing module 40 is specifically used to:

[0204] The training speech data is compressed by a down-projection matrix to reduce the feature dimension of the training speech data;

[0205] The compressed training speech data is processed using an activation function;

[0206] The training speech data processed by the activation function is processed by an up-projection matrix to restore the original feature dimension of the training speech data.

[0207] In one embodiment, the training data processing module 50 is specifically used to:

[0208] Determining, by a router module, a matching score between each speech sample in the training speech data and each language feature expert module;

[0209] Select a preset number of language feature expert modules ranked top by matching scores;

[0210] Input the speech samples into each selected language feature expert module respectively to obtain multiple expert output results;

[0211] Based on the matching scores, the output results of the multiple experts are weighted to obtain a weighted final output result.

[0212] In one embodiment, the basic model acquisition module 10 is specifically used to:

[0213] Collect speech datasets in multiple languages;

[0214] Build deep learning models for multilingual speech recognition;

[0215] The deep learning model is trained using the speech data set, and the model parameters are adjusted through multiple rounds of iterations to obtain the basic speech recognition model;

[0216] Freeze the parameters of the basic speech recognition model, wherein the parameters include parameters of an input layer, a hidden layer, and an output layer.

[0217] In one embodiment, the language adapter configuration module 20 is specifically configured to:

[0218] Define a structure of language-dependent adapters for each language;

[0219] For each language, use the corresponding speech data set to train the corresponding language-dependent adapter, so that the language-dependent adapter learns the acoustic and linguistic features unique to each language;

[0220] Integrating the language-dependent adapter into the basic speech recognition model;

[0221] Configure a unique identifier for the language-dependent adapter of each language to correctly select and apply the corresponding language-dependent adapter during model training.

[0222] In one embodiment, the expert module configuration module 30 is specifically configured to:

[0223] Determine a language list, and define corresponding language features for each language in the language list;

[0224] Configure the corresponding language feature expert module for each language feature.

[0225] The language feature expert module is integrated into the middle layer or output layer of the basic speech recognition model.

[0226] In one embodiment, the speech recognition module 100 is specifically used for:

[0227] Receiving a voice signal to be recognized;

[0228] Performing a preprocessing operation on the speech signal to be recognized, wherein the preprocessing operation includes denoising and normalization processing;

[0229] Extracting speech features from the speech signal to be recognized after preprocessing, wherein the speech features include Mel frequency cepstral coefficients and pitch features;

[0230] Determining a corresponding language-dependent adapter in the target speech recognition model based on the language type of the speech signal to be recognized;

[0231] Processing the speech features through the language-dependent adapter, the processing including feature space mapping, non-linear activation and normalization processing;

[0232] The processed speech features are recognized by the target speech recognition model to generate a recognition result.

[0233] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a multilingual speech recognition method service side.

[0234] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multilingual speech recognition method on the user side.

[0235] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0236] Acquire a pre-trained basic speech recognition model, where the basic speech recognition model can recognize multiple languages, and freeze parameters of the basic speech recognition model;

[0237] Configuring a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter;

[0238] A plurality of language feature expert modules are configured in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature;

[0239] Acquire training speech data, and divide the training speech data into a plurality of batches of data, each batch of data containing a plurality of speech samples;

[0240] Selecting a corresponding language feature expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language feature expert module to obtain an output result;

[0241] Assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0242] Determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language;

[0243] Selecting a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and training the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter;

[0244] Until the plurality of batches of data are trained, a trained target language dependent adapter is generated to obtain a target speech recognition model;

[0245] Speech recognition is performed based on the target speech recognition model.

[0246] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0247] Acquire a pre-trained basic speech recognition model, where the basic speech recognition model can recognize multiple languages, and freeze parameters of the basic speech recognition model;

[0248] Configuring a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter;

[0249] A plurality of language feature expert modules are configured in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature;

[0250] Acquire training speech data, and divide the training speech data into a plurality of batches of data, each batch of data containing a plurality of speech samples;

[0251] Selecting a corresponding language feature expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language feature expert module to obtain an output result;

[0252] Assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs;

[0253] Determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language;

[0254] Selecting a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and training the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter;

[0255] Until the plurality of batches of data are trained, a trained target language dependent adapter is generated to obtain a target speech recognition model;

[0256] Speech recognition is performed based on the target speech recognition model.

[0257] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0258] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0259] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0260] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A multilingual speech recognition method, characterized in that: The following steps are involved: Acquire a pre-trained basic speech recognition model, where the basic speech recognition model can recognize multiple languages, and freeze parameters of the basic speech recognition model; Configuring a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter; A plurality of language feature expert modules are configured in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature; Acquire training speech data, and divide the training speech data into a plurality of batches of data, each batch of data containing a plurality of speech samples; Selecting a corresponding language feature expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language feature expert module to obtain an output result; Assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs; Determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language; Selecting a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and training the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter; Until the plurality of batches of data are trained, a trained target language dependent adapter is generated to obtain a target speech recognition model; Speech recognition is performed based on the target speech recognition model.

2. The multilingual speech recognition method according to claim 1, wherein: After obtaining the training speech data, it also includes: The training speech data is compressed by a down-projection matrix to reduce the feature dimension of the training speech data; The compressed training speech data is processed using an activation function; The training speech data processed by the activation function is processed by an up-projection matrix to restore the original feature dimension of the training speech data.

3. The multilingual speech recognition method according to claim 1, wherein: Selecting a corresponding language feature expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language feature expert module to obtain an output result, including: Determining, by a router module, a matching score between each speech sample in the training speech data and each language feature expert module; Select a preset number of language feature expert modules ranked top by matching scores; Input the speech samples into each selected language feature expert module respectively to obtain multiple expert output results; Based on the matching scores, the output results of the multiple experts are weighted to obtain a weighted final output result.

4. The multilingual speech recognition method according to claim 1, wherein: Acquiring a pre-trained basic speech recognition model, where the basic speech recognition model can recognize multiple languages, and freezing parameters of the basic speech recognition model, including: Collect speech datasets in multiple languages; Build deep learning models for multilingual speech recognition; The deep learning model is trained using the speech data set, and the model parameters are adjusted through multiple rounds of iterations to obtain the basic speech recognition model; Freeze the parameters of the basic speech recognition model, wherein the parameters include parameters of an input layer, a hidden layer, and an output layer.

5. The multilingual speech recognition method as claimed in claim 1, characterized in that: Configuring a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter includes: Define a structure of language-dependent adapters for each language; For each language, use the corresponding speech data set to train the corresponding language-dependent adapter, so that the language-dependent adapter learns the acoustic and linguistic features unique to each language; Integrating the language-dependent adapter into the basic speech recognition model; Configure a unique identifier for the language-dependent adapter of each language to correctly select and apply the corresponding language-dependent adapter during model training.

6. The multilingual speech recognition method according to claim 1, wherein: A plurality of language feature expert modules are configured in the basic speech recognition model, each of which is used to process corresponding language features, including: Determine a language list, and define corresponding language features for each language in the language list; Configure the corresponding language feature expert module for each language feature. The language feature expert module is integrated into the middle layer or output layer of the basic speech recognition model.

7. The multilingual speech recognition method according to claim 1, wherein: Performing speech recognition based on the target speech recognition model includes: Receiving a voice signal to be recognized; Performing a preprocessing operation on the speech signal to be recognized, wherein the preprocessing operation includes denoising and normalization processing; Extracting speech features from the speech signal to be recognized after preprocessing, wherein the speech features include Mel frequency cepstral coefficients and pitch features; Determining a corresponding language-dependent adapter in the target speech recognition model based on the language type of the speech signal to be recognized; Processing the speech features through the language-dependent adapter, the processing including feature space mapping, non-linear activation and normalization processing; The processed speech features are recognized by the target speech recognition model to generate a recognition result.

8. A multilingual speech recognition device, characterized in that: The multilingual speech recognition device comprises: A basic model acquisition module, used to acquire a pre-trained basic speech recognition model, the basic speech recognition model being able to recognize multiple languages, and freeze the parameters of the basic speech recognition model; A language adapter configuration module, used to configure a language-dependent adapter in the basic speech recognition model so that each language is configured with a corresponding language-dependent adapter; An expert module configuration module, used to configure a plurality of language feature expert modules in the basic speech recognition model, each language feature expert module being used to process a corresponding language feature; A data batch division module, used to obtain training speech data and divide the training speech data into a plurality of batches of data, each of which contains a plurality of speech samples; A training data processing module, used for selecting a corresponding language characteristic expert module for each speech sample in the training speech data, and inputting the speech sample into the corresponding language characteristic expert module to obtain an output result; A language identifier assignment module is used to assign a corresponding language identifier to each speech sample in each batch of data to indicate the language type to which the speech sample belongs; A low-resource language adapter control module, configured to determine a target low-resource language, freeze parameters of language-dependent adapters that do not correspond to the target low-resource language, and train and optimize only parameters of the language-dependent adapters corresponding to the target low-resource language; A low-resource language adapter training module, used to select a corresponding low-resource language-dependent adapter based on the language identifier of the speech sample in each batch of data, and train the corresponding low-resource language-dependent adapter based on the output result corresponding to the speech sample to optimize and adjust the parameters of the low-resource language-dependent adapter; An adapter training completion module, used to generate a trained target language dependent adapter until the plurality of batches of data are trained to obtain a target speech recognition model; A speech recognition module is used to perform speech recognition based on the target speech recognition model.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a multilingual speech recognition program stored in the memory and executable on the processor. When the multilingual speech recognition program is executed by the processor, the steps of the multilingual speech recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a multi-language speech recognition program, which, when executed by the processor, implements the steps of the multi-language speech recognition method according to any one of claims 1 to 7.

Images