Training method and device of bilingual speech recognition model, medium, equipment and vehicle

By extracting the acoustic features and word segmentation sequences of bilingual voice audio, pre-training and initializing the dual encoder, the problem of poor bilingual voice recognition effect in the prior art is solved, and better bilingual voice recognition effect is achieved, and the mixed voice recognition is reduced. Error rate.

CN119028350APending Publication Date: 2024-11-26BEIJING CO WHEELS TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310611936.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing speech recognition technology has poor effect in bilingual speech recognition, especially the hybrid speech recognition method based on DNN-HMM. The modeling is complex and the optimization goals are inconsistent, resulting in poor bilingual speech recognition effect.

Method used

A training method of bilingual speech recognition model is adopted. By obtaining the speech audio and text information of bilinguals, acoustic features are extracted and word segmentation is performed, single encoders of different languages ​​are pre-trained, and the dual encoder is initialized, and the word segmentation sequence is used to perform double Encoder training to improve the bilingual speech recognition effect.

Benefits of technology

By pre-training the single encoder of two languages ​​and then initializing the dual encoder, the word error rate and word error rate of mixed speech recognition can be effectively reduced, the accuracy of bilingual speech recognition can be improved, and the pronunciation of bilingual language habits can be met by users of bilingual language habits. Identify requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028350A_ABST
    Figure CN119028350A_ABST
Patent Text Reader

Abstract

The invention relates to a bilingual speech recognition model training method and device, a medium, equipment and a vehicle. The method comprises the following steps: acquiring bilingual voice audios and text information corresponding to the voice audios; extracting acoustic features of the voice audio, and performing word segmentation on the text information to obtain a word segmentation sequence; pre-training two single encoders matched with different languages according to the acoustic features and the word segmentation sequence; the pre-trained single encoder is adopted to initialize the double encoders; and training the initialized double encoders according to the voice audio and the corresponding word segmentation sequence to obtain a bilingual voice recognition model. According to the technical scheme, the bilingual speech recognition effect can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a training method, device, medium, equipment and vehicle for a bilingual speech recognition model. Background Art

[0002] With the globalization of information in modern society, bilingual mixed language phenomena are becoming more and more common. For example, in the field of in-vehicle, bilingual mixed speech human-computer interaction is often used. Some existing speech recognition technologies on the market have poor recognition effects. For example, the hybrid speech recognition method based on DNN-HMM requires the construction of different submodules. Not only is the modeling complex, but the optimization goals of different modules are inconsistent, making it difficult to obtain an optimal solution, resulting in poor bilingual speech recognition effects. Therefore, it is very necessary to provide a bilingual speech recognition with better recognition effects. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a method, apparatus, medium, equipment and vehicle for training a bilingual speech recognition model to improve the bilingual speech recognition effect.

[0004] The present disclosure provides a method for training a bilingual speech recognition model, comprising:

[0005] Obtaining bilingual voice audio and text information corresponding to the voice audio;

[0006] Extracting acoustic features of the speech audio, and segmenting the text information into words to obtain a word segmentation sequence;

[0007] Pre-training two single encoders matched to different languages ​​according to the acoustic features and the word segmentation sequence;

[0008] Initializing the dual encoder using the pre-trained single encoder;

[0009] The initialized dual encoder is trained according to the speech audio and the corresponding word segmentation sequence to obtain a bilingual speech recognition model.

[0010] In some embodiments, the bilingual voice audio includes the following:

[0011] a first voice audio in a first language;

[0012] a second audio voice in a second language;

[0013] Includes the first language and mixed speech audio of the first language.

[0014] In some embodiments, the pre-training of two single encoders matching different languages ​​according to the acoustic features and the word segmentation sequence includes:

[0015] Pre-training a first encoder matching the first language according to the acoustic features and word segmentation sequence corresponding to the first speech audio; and

[0016] A second encoder matched to the second language is pre-trained according to the acoustic features and word segmentation sequence corresponding to the second speech audio.

[0017] In some embodiments, the first encoder includes a convolutional layer, a plurality of speech recognition modules and a first decoder, and the pre-training of the first encoder matching the first language according to the acoustic features and word segmentation sequence corresponding to the first speech audio includes performing the following process multiple times:

[0018] Inputting the acoustic features of the first speech audio to the convolution layer of the first encoder, sampling the acoustic features of the first speech audio through the convolution layer to obtain acoustic sampling features;

[0019] Performing speech recognition on the acoustic sampling features through the multiple speech recognition modules to obtain a speech recognition vector;

[0020] The speech recognition vector and the word segmentation sequence of the first speech audio are input into a first decoder; the first decoder is used to convert the speech recognition vector into an original data space according to the word segmentation sequence of the first speech audio.

[0021] In some embodiments, the initializing the dual encoder using the pre-trained single encoder includes:

[0022] The dual encoder is initialized based on a first encoder matching the first language, a second encoder matching the second language, a preset feature fusion layer and a second decoder.

[0023] In some embodiments, the training of the initialized dual encoder according to the speech audio and the corresponding word segmentation sequence to obtain a bilingual speech recognition model includes performing the following process multiple times:

[0024] Inputting the speech audio into the first encoder and the second encoder simultaneously;

[0025] Encoding the speech audio by the first encoder and the second encoder respectively to obtain a first feature vector and a second feature vector;

[0026] fusing the first feature vector and the second feature vector into a target feature vector through the feature fusion layer;

[0027] The target feature vector and the word segmentation sequence corresponding to the speech audio are input into the second decoder; the second decoder is used to convert the target feature vector into the original data space according to the word segmentation sequence corresponding to the speech audio.

[0028] In some embodiments, the single encoder is a squeezeformer-based speech recognition model.

[0029] The present disclosure provides a training device for a bilingual speech recognition model, comprising:

[0030] A data acquisition module, used to acquire bilingual voice audio and text information corresponding to the voice audio;

[0031] A data processing module, used to extract the acoustic features of the speech audio, and segment the text information into words to obtain a word segmentation sequence;

[0032] A first training module is used to pre-train two single encoders matched to different languages ​​according to the acoustic features and the word segmentation sequence;

[0033] An initialization module, used to initialize the dual encoder using the pre-trained single encoder;

[0034] The second training module is used to train the initialized dual encoder according to the speech audio and its corresponding word segmentation sequence to obtain a bilingual speech recognition model.

[0035] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of any of the above methods.

[0036] The present disclosure also provides an electronic device, comprising:

[0037] one or more processors;

[0038] A memory for storing one or more programs or instructions;

[0039] The processor is used to execute the steps of any of the above methods by calling the program or instruction stored in the memory.

[0040] The present disclosure also provides a vehicle, comprising the training device for the bilingual speech recognition model as described above.

[0041] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0042] The technical solution provided by the embodiment of the present disclosure first obtains bilingual speech audio and text information corresponding to the speech audio; then extracts the acoustic features of the speech audio, and divides the text information into words to obtain a word segmentation sequence; finally, pre-trains two single encoders matching different languages ​​according to the acoustic features and the word segmentation sequence; uses the pre-trained single encoder to initialize the dual encoder; and trains the initialized dual encoder according to the speech audio and its corresponding word segmentation sequence to obtain a bilingual speech recognition model. This technical solution first pre-trains single encoders for two languages, and then uses the two single encoders to initialize the dual encoder. The dual encoder obtained thereby can better achieve a bilingual mixed recognition effect and meet the speech recognition needs of users with bilingual language habits. Using the word segmentation sequence in the dual encoder training process can reduce the character error rate and word error rate of mixed speech recognition and achieve better speech recognition effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of a method for training a bilingual speech recognition model provided in an embodiment of the present disclosure;

[0046] Figure 2 A schematic diagram of the structure of a single encoder provided in an embodiment of the present disclosure;

[0047] Figure 3 A schematic diagram of the structure of a dual encoder provided in an embodiment of the present disclosure;

[0048] Figure 4 A structural block diagram of a training device for a bilingual speech recognition model provided in an embodiment of the present disclosure;

[0049] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0052] As bilingual mixed language phenomenon becomes more and more common, bilingual speech recognition is very necessary. Bilingual speech recognition means that the input speech to be recognized is not a single language speech, but a mixed speech of two languages ​​(such as Chinese and English). In order to achieve a better bilingual mixed recognition effect, the embodiment of the present disclosure provides a training method for a bilingual speech recognition model, which trains a dual encoder to apply it to bilingual mixed recognition and achieve a better recognition effect.

[0053] Figure 1 The present invention provides a flow chart of a method for training a bilingual speech recognition model. The method can be performed by a training device for a bilingual speech recognition model, and the training device for a bilingual speech recognition model can be implemented in software and / or hardware. Figure 1 As shown, the method comprises the following steps:

[0054] Step S102, obtaining bilingual speech audio and text information corresponding to the speech audio.

[0055] The bilingual speech audio includes the following: a first speech audio in a first language; a second speech audio in a second language; and a mixed speech audio including the first language and the first language. Exemplarily, the first language and the second language are Chinese and English respectively, the first speech audio is Chinese speech audio, the second speech audio is English speech audio, and the mixed speech audio is Chinese and English speech audio. The above speech audio can be obtained by acquisition, or by using open source Chinese data sets and English data sets.

[0056] In this embodiment, a speech recognition tool such as Kaldi can be used to process the speech audio to obtain text information corresponding to the speech audio; using the above Chinese and English examples, the text information includes: Chinese text information and English text information. It can be understood that each speech sentence included in the speech audio corresponds to each text content included in the text information.

[0057] Step S104, extracting acoustic features of the speech audio, and segmenting the text information into words to obtain a word segmentation sequence.

[0058] In an exemplary manner, the espnet tool can be used to obtain Fbank features from speech audio by performing a fast Fourier transform, taking the power spectrum, taking the squared magnitude, passing through a Mel filter bank, and taking the logarithm. The obtained Fbank features are used as the acoustic features of the speech audio.

[0059] When this embodiment performs word segmentation on text information, different tokens for each language can be set first. A token can be understood as a text segment used to divide the smallest basic unit of text. Then, according to the tokens corresponding to each language, the text information of each language is segmented separately to obtain a word segmentation sequence.

[0060] Taking Chinese and English as examples for illustration. The token for Chinese text information can be set as a single character; according to the single characters, the Chinese text information is divided into a Chinese word segmentation sequence composed of multiple single characters. For example, the text "你好吗" is divided into a word segmentation sequence composed of the three single characters "你", "好", and "吗". Also, the token for English text information is set as BPE (byte-pair encoding); the BPE algorithm is used to segment the English text information to obtain an English word segmentation sequence.

[0061] Considering that in the text information corresponding to Chinese-English mixed speech audio, the Chinese text information will include some single characters in the Chinese word segmentation sequence, and the English text information will include some English characters in the English word segmentation sequence. Then, to facilitate the use of the above word segmentation sequences and share the words in the word segmentation sequences, this embodiment can record each word in the word segmentation process into a word segmentation list, that is, the word segmentation list includes: multiple single characters after segmenting the Chinese text information, and multiple English characters after segmenting the English text information.

[0062] By segmenting the text information in this embodiment, the smallest basic unit of the text information is a single character or an English character or other words. The single encoder trained using such a word segmentation sequence can reduce the character error rate and word error rate when used for speech recognition.

[0063] Step S106, pre-train two single encoders matching different languages according to the acoustic features and the word segmentation sequence.

[0064] In some ways, the dual-encoder technology can be applied to bilingual mixed recognition; the optimization of the dual encoder is generally in the training process. For example, adding a language classifier to the dual encoder for joint training, or adding some language restrictions to each single encoder during the training process. In this way, it will default to using the already very mature transformer model or conformer model to construct the single encoder in the dual encoder.

[0065] This embodiment takes into account that squeezeformer, as an end-to-end speech recognition model, has shown good advantages on the LibriSpeech dataset, such as better recognition accuracy and faster decoding speed. Based on this, the single encoder in this embodiment can be a squeezeformer-based speech recognition model.

[0066] Methods for pre-training a single encoder can include:

[0067] A first encoder matching a first language is pre-trained based on acoustic features and word segmentation sequences corresponding to the first speech audio; and a second encoder matching a second language is pre-trained based on acoustic features and word segmentation sequences corresponding to the second speech audio.

[0068] Here, the training process of the above single encoder is introduced by taking the training of the first encoder as an example, as shown below.

[0069] like Figure 2 As shown, the first encoder includes a convolutional layer, multiple speech recognition modules and a first decoder; wherein the convolutional layer adopts DW Sep.Conv Subsampling, and the speech recognition module adopts squeezeformer.

[0070] Based on the above structure, this embodiment can perform the following process multiple times to implement pre-training of the first encoder:

[0071] The acoustic features of the first speech audio are input into the convolution layer of the first encoder, and the acoustic features of the first speech audio are sampled through the convolution layer to obtain acoustic sampling features; speech recognition is performed on the acoustic sampling features through multiple speech recognition modules to obtain a speech recognition vector; the speech recognition vector and the word segmentation sequence of the first speech audio are input into the first decoder; the first decoder is used to convert the speech recognition vector into the original data space according to the word segmentation sequence of the first speech audio.

[0072] It can be understood that the first speech audio and its word segmentation sequence are one-to-one corresponding, and accordingly, the speech recognition vector and the word segmentation sequence of the first speech audio are also one-to-one corresponding; the above-mentioned speech recognition vector and the word segmentation sequence of the first speech audio with a corresponding relationship are input into the first decoder, and the first decoder can convert the speech recognition vector into the original data space according to the word segmentation sequence of the first speech audio, that is, obtain the generated speech audio similar to the first speech audio. Correspondingly, the parameters of the first encoder and the first decoder are trained by comparing the difference between the first speech audio and the generated speech audio and minimizing the difference. Therefore, according to the above method, the acoustic features and word segmentation sequence corresponding to the first speech audio are continuously input into the first encoder, and the reconstruction error is minimized multiple times during the training process until the pre-training is completed, and the first encoder can learn the ability to accurately recognize speech.

[0073] The method of pre-training the second encoder can refer to the above-mentioned process of pre-training the first encoder. In this embodiment, the second encoder includes a convolutional layer, multiple speech recognition modules and a third decoder; accordingly, according to the acoustic features and word segmentation sequence corresponding to the second speech audio, the second encoder matching the second language is pre-trained, including performing the following process multiple times:

[0074] The acoustic features of the second speech audio are input into the convolution layer of the second encoder, and the acoustic features of the second speech audio are sampled through the convolution layer of the second encoder to obtain acoustic sampling features; the acoustic sampling features of this embodiment may also be referred to as second acoustic sampling features, so as to distinguish them from the acoustic sampling features in the pre-training process of the first encoder.

[0075] The second acoustic sampling features are subjected to speech recognition by multiple speech recognition modules of the second encoder to obtain a speech recognition vector; the speech recognition vector of this embodiment may also be referred to as a second speech recognition vector to facilitate distinction from the speech recognition vector in the pre-training process of the first encoder.

[0076] The second speech recognition vector and the word segmentation sequence of the second speech audio are input into a third decoder; the third decoder is used to convert the second speech recognition vector into the original data space according to the word segmentation sequence of the second speech audio.

[0077] Step S108: Initialize the dual encoder using the pre-trained single encoder.

[0078] Dual encoders means that there are two encoders in one network model. Figure 3As shown, the dual encoder may include the above-mentioned pre-trained two monolingual single encoders, and also include a feature fusion layer and a second decoder. The single encoders may be a first encoder for Chinese speech recognition and a second encoder for English speech recognition; the feature fusion layer may use concatenation.

[0079] According to the above situation, the dual encoder can be initialized based on the first encoder matching the first language, the second encoder matching the second language, the preset feature fusion layer and the second decoder. This partial initialization can provide the dual encoder with initial speech recognition capabilities, that is, speech recognition capabilities for single languages ​​of Chinese and English.

[0080] Step S110, training the initialized dual encoder according to the speech audio and its corresponding word segmentation sequence to obtain a bilingual speech recognition model.

[0081] After initializing the dual encoder, the initialized dual encoder may be trained with reference to the following embodiments, including:

[0082] The speech audio is input into the first encoder and the second encoder at the same time. The speech audio can be the first speech audio, the second speech audio and the mixed speech audio. The speech audio is encoded by the first encoder and the second encoder respectively to obtain the first feature vector and the second feature vector.

[0083] The first feature vector and the second feature vector are fused into a target feature vector through a feature fusion layer; and the target feature vector and the word segmentation sequence corresponding to the speech audio are input into a second decoder.

[0084] This embodiment continuously inputs speech audio and its corresponding word segmentation sequence into the initialized dual encoder, and iteratively minimizes the reconstruction error multiple times during the training process until the dual encoder converges. The dual encoder can learn the ability to accurately recognize bilingual speech.

[0085] In this embodiment, since the two single encoders in the dual encoder are trained using the word segmentation sequence after word segmentation, and the dual encoder is also trained using the word segmentation sequence, the dual encoder can better reduce the character error rate and word error rate of mixed speech recognition when performing mixed speech recognition of two languages, and achieve better speech recognition effect.

[0086] In summary, the training method of the bilingual speech recognition model provided by the embodiment of the present disclosure first obtains the bilingual speech audio and the text information corresponding to the speech audio; then extracts the acoustic features of the speech audio, and divides the text information into words to obtain a word segmentation sequence; finally, pre-trains two single encoders matching different languages ​​according to the acoustic features and the word segmentation sequence; uses the pre-trained single encoder to initialize the dual encoder; and trains the initialized dual encoder according to the speech audio and its corresponding word segmentation sequence to obtain a bilingual speech recognition model. This technical solution first pre-trains single encoders for two languages, and then uses the two single encoders to initialize the dual encoder. The dual encoder obtained thereby can better achieve a bilingual mixed recognition effect and meet the speech recognition needs of users with bilingual language habits. Using the word segmentation sequence in the dual encoder training process can reduce the character error rate and word error rate of mixed speech recognition and achieve better speech recognition effects.

[0087] In addition, this technical solution uses squeezeformer as a single encoder to improve the bilingual mixed speech recognition effect and decoding speed. In addition, this technical solution can improve the optimization of bilingual speech recognition in vehicle-mounted scenarios.

[0088] Corresponding to the method for training a bilingual speech recognition model provided in the embodiment of the present disclosure, the embodiment of the present disclosure also provides a device for training a bilingual speech recognition model. Figure 4 A structural block diagram of a training device for a bilingual speech recognition model provided in an embodiment of the present disclosure, such as Figure 4 As shown, the training device of the bilingual speech recognition model includes:

[0089] The data acquisition module 202 is used to acquire bilingual speech audio and text information corresponding to the speech audio;

[0090] The data processing module 204 is used to extract the acoustic features of the speech audio and segment the text information to obtain a segmentation sequence;

[0091] A first training module 206, configured to pre-train two single encoders matched to different languages ​​according to the acoustic features and the word segmentation sequence;

[0092] An initialization module 208, configured to initialize the dual encoder using the pre-trained single encoder;

[0093] The second training module 210 is used to train the initialized dual encoder according to the speech audio and the corresponding word segmentation sequence to obtain a bilingual speech recognition model.

[0094] In one embodiment, the bilingual speech audio includes the following:

[0095] a first voice audio in a first language;

[0096] a second audio voice in a second language;

[0097] Includes the first language and mixed speech audio of the first language.

[0098] In one embodiment, the first training module 206 is specifically used to pre-train a first encoder matching the first language based on the acoustic features and word segmentation sequence corresponding to the first speech audio; and to pre-train a second encoder matching the second language based on the acoustic features and word segmentation sequence corresponding to the second speech audio.

[0099] In one embodiment, the first encoder includes a convolutional layer, a plurality of speech recognition modules and a first decoder, and the first training module 206 is further configured to perform the following process multiple times:

[0100] The acoustic features of the first speech audio are input into the convolution layer of the first encoder, and the acoustic features of the first speech audio are sampled through the convolution layer to obtain acoustic sampling features; speech recognition is performed on the acoustic sampling features through the multiple speech recognition modules to obtain a speech recognition vector; the speech recognition vector and the word segmentation sequence of the first speech audio are input into the first decoder; the first decoder is used to convert the speech recognition vector into the original data space according to the word segmentation sequence of the first speech audio.

[0101] In one embodiment, the initialization module 208 is specifically configured to initialize the dual encoder based on a first encoder matching the first language, a second encoder matching the second language, a preset feature fusion layer, and a second decoder.

[0102] In one embodiment, the second training module 210 is specifically configured to perform the following process multiple times:

[0103] The speech audio is input into the first encoder and the second encoder at the same time; the speech audio is encoded by the first encoder and the second encoder respectively to obtain a first feature vector and a second feature vector; the first feature vector and the second feature vector are fused into a target feature vector through the feature fusion layer; the target feature vector and the word segmentation sequence corresponding to the speech audio are input into the second decoder; the second decoder is used to convert the target feature vector to the original data space according to the word segmentation sequence corresponding to the speech audio.

[0104] In one embodiment, the single encoder is a squeezeformer-based speech recognition model.

[0105] The training device for the bilingual speech recognition model disclosed in the above embodiments can execute the training method for the bilingual speech recognition model disclosed in the above embodiments, and has the same or corresponding beneficial effects. To avoid repetition, they will not be described here.

[0106] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of any of the above methods.

[0107] For example, bilingual speech audio and text information corresponding to the speech audio are obtained; the acoustic features of the speech audio are extracted, and the text information is segmented to obtain a segmentation sequence; two single encoders matching different languages ​​are pre-trained according to the acoustic features and the segmentation sequence; the dual encoder is initialized using the pre-trained single encoder; the initialized dual encoder is trained according to the speech audio and its corresponding segmentation sequence to obtain a bilingual speech recognition model.

[0108] Optionally, when executed by a computer processor, the computer executable instructions may also be used to execute the technical solution of the training method of any bilingual speech recognition model provided in the embodiments of the present disclosure, thereby achieving corresponding beneficial effects.

[0109] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the embodiments of the present disclosure can be implemented with the help of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0110] The embodiment of the present disclosure also provides a vehicle, which includes the training device of the bilingual speech recognition model as described above.

[0111] The embodiment of the present disclosure also provides an electronic device, including: one or more processors; a memory for storing one or more programs or instructions; the processor calls the programs or instructions stored in the memory to execute the steps of any of the above methods to achieve corresponding beneficial effects.

[0112] Figure 5 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present disclosure. Figure 5 As shown, the electronic device includes one or more processors 301 and a memory 302 .

[0113] The processor 301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0114] The memory 302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 301 may run the program instructions to implement the training method of the bilingual speech recognition model of the embodiment of the present disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0115] In one example, the electronic device may further include: an input device 303 and an output device 304, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0116] In addition, the input device 303 may also include, for example, a keyboard, a mouse, and the like.

[0117] The output device 304 can output various information to the outside, including the determined distance information, direction information, etc. The output device 304 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0118] Of course, to simplify, Figure 5 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0119] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0120] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a bilingual speech recognition model, characterized in that: include: Obtaining bilingual voice audio and text information corresponding to the voice audio; Extracting acoustic features of the speech audio, and segmenting the text information into words to obtain a word segmentation sequence; Pre-training two single encoders matched to different languages ​​according to the acoustic features and the word segmentation sequence; Initializing the dual encoder using the pre-trained single encoder; The initialized dual encoder is trained according to the speech audio and the corresponding word segmentation sequence to obtain a bilingual speech recognition model.

2. The method according to claim 1, characterized in that: The bilingual audio includes the following: a first voice audio in a first language; a second audio voice in a second language; Includes the first language and mixed speech audio of the first language.

3. The method according to claim 2, characterized in that Pre-training two single encoders matching different languages ​​according to the acoustic features and the word segmentation sequence includes: Pre-training a first encoder matching the first language according to the acoustic features and word segmentation sequence corresponding to the first speech audio; and A second encoder matched to the second language is pre-trained according to the acoustic features and word segmentation sequence corresponding to the second speech audio.

4. The method according to claim 3, characterized in that The first encoder includes a convolutional layer, a plurality of speech recognition modules and a first decoder. The pre-training of the first encoder matched to the first language according to the acoustic features and word segmentation sequence corresponding to the first speech audio includes performing the following process multiple times: Inputting the acoustic features of the first speech audio to the convolution layer of the first encoder, sampling the acoustic features of the first speech audio through the convolution layer to obtain acoustic sampling features; Performing speech recognition on the acoustic sampling features through the multiple speech recognition modules to obtain a speech recognition vector; The speech recognition vector and the word segmentation sequence of the first speech audio are input into a first decoder; the first decoder is used to convert the speech recognition vector into an original data space according to the word segmentation sequence of the first speech audio.

5. The method according to claim 2, characterized in that: The step of initializing the dual encoder using the pre-trained single encoder includes: The dual encoder is initialized based on a first encoder matching the first language, a second encoder matching the second language, a preset feature fusion layer and a second decoder.

6. The method according to claim 5, characterized in that The step of training the initialized dual encoder according to the speech audio and the corresponding word segmentation sequence to obtain a bilingual speech recognition model includes performing the following process multiple times: Inputting the speech audio into the first encoder and the second encoder simultaneously; Encoding the speech audio by the first encoder and the second encoder respectively to obtain a first feature vector and a second feature vector; fusing the first feature vector and the second feature vector into a target feature vector through the feature fusion layer; The target feature vector and the word segmentation sequence corresponding to the speech audio are input into the second decoder; the second decoder is used to convert the target feature vector into the original data space according to the word segmentation sequence corresponding to the speech audio.

7. The method according to any one of claims 1 to 6, characterized in that: The single encoder is a squeezeformer-based speech recognition model.

8. A training device for a bilingual speech recognition model, characterized in that: include: A data acquisition module, used to acquire bilingual voice audio and text information corresponding to the voice audio; A data processing module, used to extract the acoustic features of the speech audio, and segment the text information into words to obtain a word segmentation sequence; A first training module is used to pre-train two single encoders matched to different languages ​​according to the acoustic features and the word segmentation sequence; An initialization module, used to initialize the dual encoder using the pre-trained single encoder; The second training module is used to train the initialized dual encoder according to the speech audio and its corresponding word segmentation sequence to obtain a bilingual speech recognition model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs or instructions; The processor is used to execute the steps of the method according to any one of claims 1 to 7 by calling the program or instruction stored in the memory.

11. A vehicle, characterized in that: The vehicle includes a training device for a bilingual speech recognition model as claimed in claim 8.

Citation Information

Cited By

  • Language classification model training method and device, language recognition method and device and intelligent equipment

    CN119785773A