Multi-language recognition model training method, language recognition method and electronic equipment
By injecting low-rank matrix parameters and mixed training data into the language recognition model and combining it with data augmentation technology, the problems of high resource consumption and decreased accuracy in traditional model training are solved, and efficient multi-language recognition model training and recognition are achieved.
Patent Information
- Application Number
- CN202510857357.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-25
- Publication Date
- 2025-09-19
AI Technical Summary
When traditional language recognition models add low-resource language recognition functions, they consume large computing resources, have high training costs and low training efficiency, and may cause the original language recognition accuracy to decline.
By injecting low-rank matrix parameters into the current recognition model, training with mixed training data, and adjusting the model in combination with validation data until the set requirements are met, the classifier and word segmentation modules are dynamically adjusted to adapt to the newly added languages, and data enhancement technology is combined to enrich the training data.
While reducing computing resources and training costs, it improves the recognition accuracy of new languages, reduces the recognition error rate, and maintains the recognition ability of the original languages, achieving efficient multilingual recognition model training.
Smart Images
Figure CN120673750A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech recognition technology, and in particular to a multi-language recognition model training method, a language recognition method, and an electronic device. Background Art
[0002] To add recognition capabilities for new, low-resource languages, traditional language recognition models may require unfreezing all parameters of the original multilingual language recognition model, fine-tuning the model on low-resource languages (such as Russian and Arabic), and fixing the output layer classifier to the Russian task dimensions. This approach consumes significant computing resources, requiring updates to all language recognition model parameters, resulting in high training costs and low training efficiency. Summary of the Invention
[0003] The purpose of this application is to provide a multi-language recognition model training method, a language recognition method and an electronic device, which can improve the efficiency of updating and training the multi-language recognition model.
[0004] In a first aspect, the present invention provides a multilingual recognition model training method, comprising: adjusting model parameters of a current recognition model to obtain an updated recognition model; wherein, when this step is performed for the first time, the current recognition model is the original recognition model, and when this step is performed for the nth time, the current recognition model is the recognition model obtained by n-1 training updates; the original recognition model is a recognition model of a specified language, the specified language does not include a target language, and n is a positive integer greater than or equal to 2; using mixed training data to train low-rank matrix parameters in the updated recognition model to obtain an updated current recognition model; wherein the mixed training data is formed by mixing training data of the target language and training data of the specified language; using verification data of the target language to verify the current recognition model; if the current recognition model does not meet the set requirements, re-performing the above steps to update the current recognition model until the current recognition model meets the set requirements and the current recognition model is used as the target recognition model; wherein the target recognition model is used to recognize the target language and the specified language.
[0005] In the above implementation method, the underlying feature extraction layer learning can be realized, and the amount of training parameters can be far less than the amount of full fine-tuning parameters at the same time. Compared with the learning method of the full fine-tuning parameters of the ring beam, the training efficiency is much higher; in addition, since the underlying feature extraction layer learning is realized, the recognition ability of the obtained target recognition model for the newly added target language can also be improved.
[0006] In an optional embodiment, the original recognition model includes a classifier for outputting classification results; the model parameters of the current recognition model are adjusted to obtain an updated recognition model, including: adjusting the model parameters of the current recognition model to inject low-rank matrix parameters; adjusting the classification dimension of the classifier to obtain an updated recognition model.
[0007] In the above implementation, the length of the classifier can also be adaptively set to better utilize the recognition of the newly added target language, while also maintaining the recognition of the original designated language.
[0008] In an optional embodiment, the original recognition model includes a first classifier for outputting classification results; the first classifier is a pluggable module; the model parameters of the current recognition model are adjusted to obtain an updated recognition model, including: adjusting the model parameters of the current recognition model to inject low-rank matrix parameters; based on current recognition requirements, designing a second classifier, and using the second classifier to replace the first classifier to obtain an updated recognition model; wherein, the second classifier is a pluggable module.
[0009] In the above implementation, the classifier can be a flexible and pluggable module, which can better adapt to the recognition requirements of different tasks and can also facilitate the update of the classifier.
[0010] In an optional embodiment, the model parameters of the current recognition model are adjusted to inject low-rank matrix parameters, including: adjusting the model parameters of the current recognition model according to the recognition accuracy of the target language and the recognition accuracy of the specified language to inject low-rank matrix parameters.
[0011] In the above implementation method, it is also possible to adapt to the requirements of recognition accuracy and dynamically adjust the range of injected low-rank matrix parameters, so as to gradually release more neural network layers of the original recognition model, and not only unfreeze the classifier, but also gradually fine-tune the strategy to reflect the task-driven working mode.
[0012] In an optional embodiment, before using the mixed training data to train the low-rank matrix parameters in the updated recognition model, the method further includes: obtaining initial training data of the target language; and performing enhancement processing on the initial training data to obtain training data of the target language.
[0013] In the above implementation, data enhancement methods can also be used to enrich the training data of the target language and reduce the difficulty of obtaining training data.
[0014] In an optional embodiment, the enhancement processing of the initial training data includes: adjusting one or more enhancement processing of speech rate, pitch, background noise on the initial training data to obtain supplementary training data; wherein, the supplementary training data and the initial training data are used as training data for training the updated recognition model.
[0015] In the above implementation, data enhancement can be achieved from multiple dimensions, which can enrich the training data of the target language and improve the training effect.
[0016] In an optional embodiment, the adjusting one or more enhancement processes among speech rate, pitch, and background noise on the initial training data to obtain supplementary training data includes: inputting the initial training data and obtaining supplementary training data through a pre-trained language synthesis model; wherein, the adjusting one or more enhancement processes among speech rate, pitch, and background noise are implemented by adjusting parameters in the language synthesis model.
[0017] In the above implementation, the language synthesis model can be combined to enhance the training data and improve the efficiency of data enhancement.
[0018] In a second aspect, the present invention provides a language recognition method, comprising: inputting a speech to be recognized into a target recognition model to obtain a recognition result; wherein, the target recognition model is trained using the method described in any one of the aforementioned implementation methods, and the speech to be recognized is the language of the target language or the speech of a specified language.
[0019] In a third aspect, the present invention provides an electronic device comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method described in any one of the aforementioned embodiments.
[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which executes the steps of the method described in any one of the aforementioned embodiments when the computer program is executed by a processor.
[0021] In a fifth aspect, the present invention provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method described in any one of the aforementioned embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application; Figure 2 A flowchart of the multilingual recognition model training method provided in an embodiment of the present application; Figure 3 A partial flow chart of the multilingual recognition model training method provided in an embodiment of the present application; Figure 4 This is a flowchart of the language recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.
[0025] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.
[0026] With the development of neural network models, their application areas are expanding. Models trained based on neural networks can also be used for speech recognition. However, unlike other fields, the language field is diverse. Depending on the needs, a trained speech recognition model may need to recognize more languages. In this case, completely retraining the recognition model can require significant computing resources and time. To address this, there are currently two approaches: full parameter fine-tuning: all parameters of a pre-trained multilingual speech model (such as SenseVoice) are unfrozen and fully fine-tuned using training data for the newly added language to be recognized (such as Russian or Arabic). The output layer classifier is fixed to the newly added language task dimensions. The second approach is to fine-tune only the output layer classifier of the pre-trained multilingual speech model: all parameters of the other components of the pre-trained multilingual speech model are frozen, and only the newly added classifier (such as the fully connected layer) is trained to directly adapt to the newly added language task dimensions.
[0027] However, the above approach has some drawbacks. First, full-parameter fine-tuning consumes a lot of computational resources: all model parameters must be updated, resulting in high training costs. Furthermore, there is the potential for catastrophic forgetting, where fine-tuning the task for a newly added language can significantly reduce the recognition accuracy of the original languages (such as English and Chinese). Furthermore, fine-tuning the classifier can result in phonemes unique to the newly added language (such as soft consonants and stress rules) not being learned by the underlying feature extraction layer, resulting in higher recognition error rates than with full-parameter fine-tuning.
[0028] Based on the above research, the embodiments of the present application can provide a multi-language recognition model training method, a language recognition method, and an electronic device, which can reduce the training cost and also reduce the recognition error rate.
[0029] To facilitate understanding of this embodiment, the electronic device that executes the multilingual recognition model training method and language recognition method disclosed in the embodiment of this application is first introduced in detail.
[0030] like Figure 1 , which is a block diagram of an electronic device. The electronic device 100 may include a memory 111 and a processor 113. A person skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.
[0031] The memory 111 and processor 113 are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable modules stored in the memory.
[0032] The memory 111 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the programs after receiving an execution instruction. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application may be applied to the processor 113 or implemented by the processor 113.
[0033] The processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor.
[0034] Based on different actual needs, the electronic device 100 may further include more components.
[0035] Exemplarily, after executing the steps in the language recognition method, the execution results need to be displayed, and the electronic device 100 may further include a display unit. The display unit provides an interactive interface (such as a user operation interface) between the electronic device 100 and the user or is used to display image data for the user's reference. In this embodiment, the display unit may be a liquid crystal display or a touch display. If it is a touch display, it may be a capacitive touch screen or a resistive touch screen that supports single-point and multi-touch operations. Supporting single-point and multi-touch operations means that the touch display can sense touch operations generated simultaneously from one or more positions on the touch display, and the sensed touch operations are handed over to the processor for calculation and processing.
[0036] Exemplarily, the electronic device 100 may transmit the execution results of each method to other devices. The electronic device 100 may further include a communication unit for communicating with other devices.
[0037] The electronic device 100 in this embodiment can be used to execute each step of each method provided in the embodiments of this application. The following describes the implementation process of the multi-language recognition model training method and the language recognition method through several embodiments.
[0038] See also Figure 2 , is a flow chart of the multi-language recognition model training method provided by the embodiment of the present application. The multi-language recognition model training method provided by the embodiment of the present application can be applied to an electronic device, and the steps in the multi-language recognition model training method are performed by the electronic device. Figure 2 The specific process shown is described in detail.
[0039] Step 210 , adjusting the model parameters of the current recognition model to inject low-rank matrix parameters to obtain an updated recognition model.
[0040] When this step is performed for the first time, the current recognition model is the original recognition model. When this step is performed for the nth time, the current recognition model is the recognition model updated after the n-1 training in step 220. n is a positive integer greater than or equal to 2.
[0041] The original recognition model is a recognition model of a specified language, which does not include the target language. For example, the specified language can be a variety of commonly used languages, such as Chinese, English, Japanese, French, etc.
[0042] Optionally, low-rank matrix parameters may be injected into some levels of the current recognition model.
[0043] Exemplarily, the current recognition model may include an attention mechanism layer, and the parameters of the attention mechanism layer may be adjusted to inject low-rank matrix parameters. Alternatively, the low-rank matrix parameters may be injected into all attention mechanism layers. Alternatively, the low-rank matrix parameters may be injected into only some key attention mechanism layers. Some key attention mechanism layers may be some layers located in the front end or back end of the current recognition model.
[0044] In this embodiment, the above-mentioned injected low-rank matrix parameters can be adjusted using LoRA (Low-Rank Adaptation) technology, and the parameters of the low-rank matrix are injected through the LoRA technology.
[0045] Step 220: Use the mixed training data to train the low-rank matrix parameters in the updated recognition model to obtain an updated current recognition model.
[0046] The mixed training data is formed by mixing the training data of the target language and the training data of the specified language.
[0047] Alternatively, the mixed training data may be obtained by mixing the target language and the designated language in a proportional ratio. For example, the ratio of the target language training data to the designated language training data may be 3:10. Alternatively, the designated language training data may be the data used to train the original recognition model. For example, if the target language is Russian, the Russian data may be 150 hours of speech, and the designated language training data may be 500 hours of speech.
[0048] Of course, based on different actual training needs and focuses, the proportion of training data can also be adaptively adjusted.
[0049] Optionally, the low-rank matrix parameters in the updated recognition model are updated using the above-mentioned mixed training data to obtain an updated current recognition model.
[0050] Step 230: Verify the current recognition model using verification data in the target language.
[0051] In the case that the current recognition model does not meet the set requirements, the above steps 210 and 220 are re-executed to update the current recognition model until the current recognition model meets the set requirements, and the current recognition model is used as the target recognition model.
[0052] The set requirements may include an error rate of the recognition result. The error rate may include an error rate for the recognition of the designated language and an error rate for the target language. For example, the error rate for the recognition of the designated language and the error rate for the target language may be pre-set. If, during the verification in step 230, it is determined that either the error rate for the recognition of the designated language or the error rate for the target language does not meet the requirements, then it may be determined that the set requirements are not met.
[0053] Optionally, a target number of rounds for training the current recognition model may be pre-set. When the number of rounds for training the current recognition model reaches the target number, the cycle of steps 210 to 230 may be terminated.
[0054] In this embodiment, each round of step 210 can adjust the parameters of different structural layers of the current recognition model. For example, if it is determined that the error rate of the current recognition model is too high, the parameters of more structural layers can be adaptively unfrozen to inject low-rank matrix parameters.
[0055] Illustratively, the logic of executing step 210 in different rounds may include: adjusting model parameters of the current recognition model according to the recognition accuracy of the target language and the recognition accuracy of the specified language to inject low-rank matrix parameters.
[0056] For example, if the recognition accuracy of the target language obtained by verification is low, the model parameters of more structural layers of the current recognition model may be adjusted to inject low-rank matrix parameters into more structural layers.
[0057] In an embodiment of the present application, a progressive fine-tuning strategy can be adopted. According to different needs, if the priority for Russian is higher, more neural network layers can be gradually released so that the obtained target recognition model can better meet the needs.
[0058] The target recognition model is used to recognize the target language and the specified language.
[0059] In this embodiment, the original recognition model may include a classifier for outputting a classification result. When adjusting the model, the classifier may also be adaptively adjusted to enable it to adapt to the recognition of the newly added target language.
[0060] In one embodiment, the above-mentioned step 210 may include: adjusting the model parameters of the current recognition model to inject low-rank matrix parameters; and adjusting the classification dimension of the classifier to obtain an updated recognition model.
[0061] The Classification Head can be the last layer or multiple layers of the original recognition model, used to map training features or recognition features to the final category output. When recognition of a new target language is added, its output dimension also needs to be adaptively increased. This can be achieved by extending its length, thereby increasing the output dimension of the classifier.
[0062] In one instance, the target language For example, since the recognition requirement of the target language is newly added, the categories that need to be output also increase, and the output categories of the classifier can be adaptively increased.
[0063] The classification task of the classifier of the original recognition model is M1. After the recognition of the target language is added, it can become M2. The output dimension of the classifier can be adjusted to M2, so that the output can be achieved using the M2 recognition results.
[0064] For example, the classifier may include an output layer, which may be a fully connected layer, with an output dimension equal to the number of categories. The dimension of the output layer may be modified directly: the output dimension of the last layer may be changed from M1 to M2.
[0065] Optionally, the original recognition model includes a first classifier for outputting a classification result; the first classifier is a pluggable module.
[0066] In one embodiment, the above-mentioned step 210 may include: adjusting the model parameters of the current recognition model to inject low-rank matrix parameters; designing a second classifier based on the current recognition requirements, and using the second classifier to replace the first classifier to obtain an updated recognition model; wherein the second classifier is a pluggable module.
[0067] Exemplarily, the target recognition model to be obtained can switch classifiers between different tasks. For example, when a specified language needs to be recognized, the first classifier can be used for output, and when a target language needs to be recognized, the second classifier can be used for output.
[0068] For example, if it is known in advance that the language recognition model needs to dynamically switch the recognition of different languages, or to facilitate the expansion of recognition of different languages, the classifier of the original recognition model can be designed as a pluggable module, allowing different classifiers to be dynamically loaded at runtime.
[0069] With this approach, the underlying feature extraction layer of the recognition model can be shared, and only the classifier at the output layer can be replaced to adapt to the recognition needs of different languages. This implementation method can improve the flexibility of the language recognition model.
[0070] In this embodiment, the updated current recognition model may further add an attention mechanism to interact with the fine-tuned low-rank matrix parameters.
[0071] In order to improve the recognition accuracy and low error rate of the recognition model for the newly added target language, it can also adapt to the target language and provide a corresponding word segmentation module.
[0072] In this embodiment, the original recognition model may include a word segmentation module (bpe.model), which can segment words in the specified language identified by the original recognition model. However, due to the varying word distributions of different languages, the applicability of the original recognition model's word segmentation module to the newly added target language may be relatively low. Therefore, additional target language codebook tokens can be added to train the target language's word segmentation module to adapt to target language recognition.
[0073] After the word segmentation module of the target language is trained, the word segmentation module of the target language can be merged with the word segmentation module of the original recognition model.
[0074] Before training the word segmentation module, you can configure parameters. For different languages, the usage habits may be different, and the corresponding configuration parameters may also be different. In the following example, the target language is Russian. The configuration parameters may include: spm.SentencePieceTrainer.Train( input_format='text', input=' / root / autodl-tmp / combine_for_spp_zh_no_space.txt', model_prefix=' / root / autodl-tmp / expanded_bpesplit_by_whitespace=False_zh_no_space', vocab_size=8000, model_type='bpe', initial_model=' / root / .cache / modelscope / hub / models / iic / SenseVoiceSmall / chn_jpn_yue_eng_ko_spectok.bpe.model',# Load the original model file character_coverage=1,# Cyrillic characters must be covered input_sentence_size=1000000,# Need to increase when the amount of data is large shuffle_input_sentence=True,# Enable sentence shuffle max_sentencepiece_length=512 ,# Common characters in Russian long words byte_fallback=False,# Whether to enable byte fallback split_by_whitespace=False,# Russian has no spaces, so it needs to be disabled split_by_unicode_script=False,# Enable multi-language segmentation num_threads=16,# Improve multi-core utilization minloglevel=2, shrinking_factor = 0.95, # Reduce the probability of floating point overflow By training the target language word segmentation module, the target recognition model obtained can be better adapted to the recognition of the specified language and the target language.
[0075] The scarcity of real speech data for some low-resource languages makes it difficult to obtain training data for these languages, which may lead to insufficient training of the recognition model. Based on this, when the target language is a low-resource language, such as Russian, Arabic, etc., data augmentation can be used to expand the training data. Figure 3 As shown, before step 220, steps 310 and 320 may also be included.
[0076] Step 310: Obtain initial training data of the target language.
[0077] The initial training data can be speech in the target language that has been stored in the database.
[0078] Step 320: Perform enhancement processing on the initial training data to obtain training data in the target language.
[0079] For example, the speech of the target language may be transformed to expand the training data.
[0080] In the case where step 220 needs to be performed multiple times, the above steps 310 and 320 may be performed only when step 220 is performed for the first time to obtain more training data of the target language.
[0081] Of course, when it is determined that the training data of the target language is insufficient, the training data of the target language may be further supplemented by step 320 if the training data of the target language has been obtained.
[0082] Optionally, the above-mentioned step 320 may include: performing one or more enhancement processes of adjusting speech rate, pitch, and background noise on the initial training data to obtain supplementary training data.
[0083] Optionally, the enhancement process may also include other processes, such as the timbre of the speech in the target language, the rhythm of the speech in the target language, etc.
[0084] The supplementary training data and the initial training data are used as training data for training and updating the recognition model.
[0085] For example, the background noise may include street noise, telephone channel distortion, etc.
[0086] Optionally, the initial training data may be input and supplementary training data may be obtained through a pre-trained speech synthesis model; wherein, the parameters in the speech synthesis model may be adjusted to implement one or more enhancement processing such as speech rate, pitch, and background noise.
[0087] Exemplarily, the speech synthesis model may be an IndexTTS model. Optionally, the speech synthesis model may also be a cosyvoice model.
[0088] By inputting the actual target language into the IndexTTS model or cosyvoice model, diverse synthesized speech can be generated.
[0089] Based on actual needs, if more speech in different languages is required, other language synthesis models can also be used.
[0090] Optionally, the robustness of the language synthesis model can be improved by controlling the synthetic language parameters in the synthetic speech model to adjust the speaking speed, pitch, and background noise.
[0091] Through the above implementation method, it is possible to add the recognition function of the target language on the basis of the language recognition model that can already recognize the specified language, and there is no need to follow the training logic of the original full model. It is only necessary to train the low-rank matrix parameters. It can achieve high accuracy and low error rate in recognizing the newly added target language without forgetting the original specified language. The amount of training parameters is only a small part of the full fine-tuning, and the computing resources occupied by the model training process are relatively small.
[0092] Please refer to 4, which is a flow chart of the language recognition method provided in the embodiment of the present application. The language recognition method provided in the embodiment of the present application can be applied to an electronic device, and the steps in the language recognition method are performed by the electronic device. The electronic device that performs the steps of the language recognition method and the electronic device that performs the steps of the multi-language recognition model training method can be the same device or different devices. Figure 4 The specific process shown is described in detail.
[0093] Step 410: Input the speech to be recognized into the target recognition model to obtain a recognition result.
[0094] The target recognition model is a model trained using the multilingual recognition model training method provided in the aforementioned embodiment.
[0095] The speech to be recognized can be the target language or a specified language.
[0096] Regarding the method for determining the target recognition model, please refer to the description of the aforementioned multi-language recognition model training method embodiment, which will not be repeated here.
[0097] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it executes the steps of the multilingual recognition model training method or language recognition method described in the above method embodiment.
[0098] The computer program products of the multilingual recognition model training method and the language recognition method provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the multilingual recognition model training method or the language recognition method described in the above method embodiments. For details, please refer to the above method embodiments and will not be repeated here.
[0099] In the several embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the methods and computer program products according to the multiple embodiments of the application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a portion of code, and the module, program segment, or a portion of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.
[0100] In addition, each method step in each embodiment of the present application can be integrated together to form an independent part for execution, or each method step can be executed by a separate module, or two or more steps can be formed into an independent part for execution.
[0101] If the functions are implemented in the form of software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks. It should be noted that, in this document, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. The foregoing description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.
[0102] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A multilingual recognition model training method, characterized in that: include: Adjust the model parameters of the current recognition model to inject low-rank matrix parameters to obtain an updated recognition model; wherein, when this step is performed for the first time, the current recognition model is the original recognition model, and when this step is performed for the nth time, the current recognition model is the recognition model obtained by n-1 training updates; the original recognition model is the recognition model of the specified language, the specified language does not include the target language, and n is a positive integer greater than or equal to 2; Using mixed training data, training the low-rank matrix parameters in the updated recognition model to obtain an updated current recognition model; wherein the mixed training data is formed by mixing the training data of the target language and the training data of the specified language; Verifying the current recognition model using verification data in the target language; If the current recognition model does not meet the set requirements, the above steps are re-executed to update the current recognition model until the current recognition model meets the set requirements, and the current recognition model is used as the target recognition model; wherein the target recognition model is used to recognize the target language and the specified language.
2. The method according to claim 1, characterized in that The original recognition model includes a classifier for outputting a classification result; The adjusting of the model parameters of the current recognition model to inject low-rank matrix parameters to obtain an updated recognition model includes: Adjust the model parameters of the current recognition model to inject low-rank matrix parameters; The classification dimension of the classifier is adjusted to obtain an updated recognition model.
3. The method according to claim 1, characterized in that The original recognition model includes a first classifier for outputting a classification result; the first classifier is a pluggable module; The adjusting of the model parameters of the current recognition model to inject low-rank matrix parameters to obtain an updated recognition model includes: Adjust the model parameters of the current recognition model to inject low-rank matrix parameters; Based on the current recognition requirements, a second classifier is designed, and the first classifier is replaced by the second classifier to obtain an updated recognition model; wherein, the second classifier is a pluggable module.
4. The method according to claim 2 or 3, characterized in that The adjusting of the model parameters of the current recognition model to inject low-rank matrix parameters includes: According to the recognition accuracy of the target language and the recognition accuracy of the designated language, model parameters of the current recognition model are adjusted to inject low-rank matrix parameters.
5. The method according to claim 1, wherein Before using the mixed training data to train the low-rank matrix parameters in the updated recognition model, the method further includes: obtaining initial training data in the target language; The initial training data is enhanced to obtain training data in the target language.
6. The method according to claim 5, characterized in that The performing enhancement processing on the initial training data includes: The initial training data is subjected to one or more enhancement processes including adjusting speech rate, pitch, and background noise to obtain supplementary training data; wherein the supplementary training data and the initial training data are used as training data for training the updated recognition model.
7. The method according to claim 6, characterized in that The step of adjusting one or more enhancement processes of speech rate, pitch, and background noise on the initial training data to obtain supplementary training data includes: The initial training data is input and supplementary training data is obtained through a pre-trained language synthesis model; wherein, the parameters in the language synthesis model are adjusted to implement one or more enhancement processing such as speech rate, pitch, and background noise.
8. A language recognition method, characterized in that: include: The speech to be recognized is input into a target recognition model to obtain a recognition result; wherein, the target recognition model is trained using the method described in any one of claims 1 to 7, and the speech to be recognized is a speech in a target language or a specified language.
9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method according to any one of claims 1 to 8.