A speech recognition method, apparatus and electronic device

Abstract

Embodiments of the present application provide a speech recognition method, device and electronic equipment. The method comprises: inputting a target language speech to be recognized into a speech quantizer to obtain a first word vector sequence corresponding to the speech to be recognized; the speech quantizer is obtained by joint training of a speech dequantizer, a text translator; the speech quantizer is used to convert speech into a word vector sequence; the speech dequantizer is used to convert a word vector sequence into speech; the text translator is used to convert a first language text into a second language text; comparing the first word vector sequence with a second word vector sequence to obtain a comparison result; the second word vector sequence is obtained by inputting a preset speech into the speech quantizer, and the preset speech contains a target instruction word; and determining whether the target instruction word is contained in the speech to be recognized according to the comparison result. Embodiments of the present application can realize the speech recognition function of a small language without collecting a large amount of small language speech data containing the target instruction word.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more particularly to a speech recognition method, apparatus, and electronic device. Background Technology

[0002] Speech recognition aims to accurately detect and identify specific command words from input speech signals, thereby enabling efficient human-computer interaction. Its applications are wide-ranging, covering smart home control (such as controlling home appliances via voice commands), smart vehicle systems (such as drivers operating navigation and playing music via voice commands), and intelligent customer service (such as automatically recognizing and answering key questions in a user's voice).

[0003] Data collection and model training are crucial steps in achieving speech recognition functionality. A common approach is to collect a large amount of speech data containing target command words, process and analyze this large volume of speech samples, and then use machine learning algorithms to train a speech recognition model capable of accurately recognizing the command words.

[0004] However, for applications involving less commonly spoken languages, the relatively small number of speakers and the scarcity of related speech data resources make it difficult to collect a sufficient number of speech sample data for model training, thus hindering the implementation of speech recognition functions for these languages. Summary of the Invention

[0005] This invention provides a speech recognition method, apparatus, and electronic device that can achieve speech recognition of minority languages ​​without collecting a large amount of speech data containing target instruction words.

[0006] In a first aspect, embodiments of the present invention disclose a speech recognition method, the method comprising:

[0007] The speech to be recognized in the target language is input into a speech quantizer to obtain a first word vector sequence corresponding to the speech; the speech quantizer is a neural network model jointly trained with a speech dequantizer and a text translator; the speech quantizer is used to convert speech into a word vector sequence; the speech dequantizer is used to convert the word vector sequence into speech; the text translator is used to convert text in the first language into text in the second language;

[0008] The first word vector sequence is compared with the second word vector sequence to obtain the comparison result; the second word vector sequence is obtained by inputting the preset speech of the target language into the speech quantizer for conversion, and the preset speech contains the target instruction word;

[0009] Based on the comparison results, it is determined whether the speech to be recognized contains the target instruction word.

[0010] Secondly, embodiments of the present invention disclose a voice recognition device, the device comprising:

[0011] The speech conversion module is used to input the speech to be recognized in the target language into a speech quantizer to obtain a first word vector sequence corresponding to the speech to be recognized; the speech quantizer is a neural network model jointly trained with a speech dequantizer and a text translator; the speech quantizer is used to convert speech into a word vector sequence; the speech dequantizer is used to convert the word vector sequence into speech; the text translator is used to convert text in the first language into text in the second language.

[0012] The feature comparison module is used to compare the first word vector sequence with the second word vector sequence to obtain the comparison result; the second word vector sequence is obtained by inputting a preset speech of the target language into the speech quantizer and converting it, and the preset speech contains target instruction words;

[0013] The target recognition module is used to determine whether the speech to be recognized contains the target instruction word based on the comparison result.

[0014] Thirdly, embodiments of the present invention disclose an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform the steps of the speech recognition method as described above.

[0015] The embodiments of the present invention have the following advantages:

[0016] Addressing the challenge of collecting speech data, especially for speech data in less commonly spoken languages, which is difficult not only to collect but also to annotate, this invention proposes a method to jointly train a speech quantizer with a text translator. This quantizer aligns the speech encoding features obtained by the quantizer with the text encoding features of the corresponding languages, resulting in the final speech quantizer. Since this speech quantizer is jointly trained with the text translator, which has text translation capabilities (e.g., translating text in a first language to a second language, which can be the target language (e.g., a less commonly spoken language), the joint training of the speech quantizer and the text translator allows the use of the text translator's cross-language text conversion function to provide cross-language alignment supervision signals for speech recognition. Therefore, this speech quantizer can be used to extract a second word vector sequence from a pre-set speech in the target language. When performing speech recognition on speech in a target language, the speech in the target language can be input into a speech quantizer to obtain a first word vector sequence corresponding to the speech. By comparing the first word vector sequence with a second word vector sequence, it can be determined whether the speech contains the target instruction word, thereby realizing the keyword speech recognition function for minority languages.

[0017] Through the embodiments of this invention, it is only necessary to collect the pronunciation of the target language (a minor language) covering all texts of that language, without collecting a large amount of minor language speech data containing the target instruction words, to achieve speech recognition capability for that minor language. The translated text data, such as the first language text and its corresponding translated second language text, is relatively easy to collect. Therefore, the translated text data can be collected first to train a text translator capable of text-to-text translation. Based on the text translator, a speech quantizer is jointly trained. Thus, for a minor language, only a small amount (e.g., at least one) of pre-set speech data containing the target instruction words is needed to achieve speech recognition functionality for that minor language, without needing to collect a large amount of speech data to train the speech recognition model. This reduces the difficulty of data collection and improves model training efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of an embodiment of the speech recognition method of the present invention;

[0020] Figure 2This is a schematic diagram of the process for calculating the first loss of the text transcriber in this invention;

[0021] Figure 3 This is a schematic diagram of the process of jointly training the speech quantizer and the speech dequantizer of the present invention;

[0022] Figure 4 This is a flowchart illustrating the process of jointly training the text transcriber and the speech quantizer of the present invention;

[0023] Figure 5 This is a schematic diagram of the process of extracting the second word vector sequence through a speech quantizer according to the present invention;

[0024] Figure 6 This is a structural block diagram of an embodiment of the speech recognition device of the present invention;

[0025] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0028] To facilitate understanding of the technical solutions of this invention, some technical terms involved in this invention will be introduced below.

[0029] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a speech recognition method according to the present invention. The method may include the following steps:

[0030] Step 101: Input the speech to be recognized in the target language into the speech quantizer to obtain the first word vector sequence corresponding to the speech to be recognized; the speech quantizer is a neural network model jointly trained with the speech dequantizer and the text translator respectively; the speech quantizer is used to convert speech into word vector sequence; the speech dequantizer is used to convert word vector sequence into speech; the text translator is used to convert first language text into second language text;

[0031] Step 102: Compare the first word vector sequence with the second word vector sequence to obtain the comparison result; the second word vector sequence is obtained by inputting the preset speech of the target language into the speech quantizer for conversion, and the preset speech contains the target instruction word;

[0032] Step 103: Based on the comparison results, determine whether the speech to be recognized contains the target instruction word.

[0033] The speech recognition method provided by this invention can be applied to electronic devices. These electronic devices may include smartphones, desktop computers, tablets, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, in-vehicle terminal devices, and other terminal devices. The electronic device can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0034] In this embodiment of the invention, a speech quantizer, a speech dequantizer, and a text translator are pre-trained. The speech quantizer is used to serialize the received speech to obtain a quantized sequence. Serialization is the process of converting the state information of an object into a form that can be stored or transmitted. For example, in a speech processing scenario, serialization converts speech data (which may be an analog signal) into digital form and may compress it to reduce the resources required for storage and transmission. For speech data, this typically involves sampling, quantizing, encoding, and possibly compressing analog signals for storage or transmission over a network. The speech quantizer can be a pre-trained neural network model.

[0035] In this embodiment of the invention, the speech quantizer can convert speech into a sequence of word vectors. For example, a speech data is divided into several equal-length speech segments and sequentially input into the speech quantizer. The speech quantizer can predict the classification result corresponding to the speech coding features of each speech segment. This classification result can be used to characterize the category of the speech coding features of the speech segment. Further, the classification result corresponding to each speech segment can correspond to a word vector, which is the speech coding feature representation of the speech segment. Thus, for a speech data, the speech quantizer can convert it into a sequence of word vectors, which contains the word vectors corresponding to the classification results of each speech segment in the speech data. A word vector can be the speech coding feature representation of a fixed-length speech segment, corresponding to a classification result.

[0036] A speech dequantizer is used to deserialize a quantized sequence to obtain speech data. Deserialization is the process of extracting data from a stored or transmitted form and reconstructing the original object. In speech processing scenarios, deserialization involves decoding the quantized sequence, possibly decompressing it, and converting it back to its original analog or digital form for playback or further processing. The speech dequantizer can be a pre-trained neural network model. In this embodiment of the invention, the speech dequantizer can be used to convert word vector sequences into speech.

[0037] In one example, suppose speech A is segmented into 5 fixed-length speech segments and sequentially input into a speech quantizer. The speech quantizer predicts a classification result of {1,3,2,1,4} for each speech segment. Assume the word vector corresponding to classification result 1 is v1, classification result 3 is v3, classification result 2 is v2, classification result 1 is v1, and classification result 4 is v4. Thus, the word vector sequence corresponding to speech A is {v1,v3,v2,v1,v4}. Inputting the word vector sequence {v1,v3,v2,v1,v4} into a speech dequantizer will output speech A. It is understood that the use of numbers to represent classification results in this embodiment is merely illustrative, and this embodiment does not limit the representation of the classification results.

[0038] A text translator is a neural network model capable of translating text. A text translator can translate text in one natural language (first language text) into text in one or more other natural languages ​​(second language text). For example, it can translate English text into Chinese text, or Chinese text into Japanese text, French text, etc. This invention does not limit the types of the first and second languages.

[0039] The embodiments of the present invention do not limit the model structures of the speech quantizer, the speech dequantizer, and the text translator. In one example, the speech quantizer may be implemented using a ResNet50 (Residual Network) structure, the speech dequantizer may be implemented using a HiFi-GAN structure, and the text translator may be implemented using a Transformer structure.

[0040] In specific implementations, the speech quantizer, the speech dequantizer, and the text translator may be neural network models. The embodiments of the present invention do not limit the network structures of the speech quantizer, the speech dequantizer, and the text translator, which may adopt the same network structure or different network structures. The neural network includes, but is not limited to, at least one or a combination, superposition, and nesting of at least two of the following: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory) network, RNN (Simple Recurrent Neural Network), Transformer variant network, attention neural network, and the like.

[0041] The embodiments of the present invention pre-train the speech quantizer, the speech dequantizer, and the text translator, wherein the speech quantizer and the speech dequantizer are obtained through joint training. Since information loss is inevitable during the speech quantization process, the joint training enables the speech quantizer and the speech dequantizer to work better in coordination, optimizes the accuracy of speech data during the quantization and dequantization processes, allows the speech quantizer to minimize information loss as much as possible, and improves the accuracy of output results.

[0042] Further, after the speech quantizer, the speech dequantizer, and the text translator are obtained through training, the embodiments of the present invention further perform joint training on the speech quantizer and the text translator, so that under the supervision of the text translator, the speech encoding feature obtained by the speech quantizer by encoding a segment of speech is aligned with the text encoding feature corresponding to the segment of speech. For example, if the content of a segment of speech is "ni hao" (meaning "hello"), after the segment of speech is input into the speech quantizer, the speech encoding feature obtained by the speech quantizer by encoding the speech segment pronounced as "ni" corresponds to the text encoding feature of the character "ni", and the speech encoding feature obtained by the speech quantizer by encoding the speech segment pronounced as "hao" corresponds to the text encoding feature of the character "hao". By performing joint training on the speech quantizer and the text translator, the embodiments of the present invention can utilize the cross-language text conversion function of the text translator (such as Chinese→Arabic, or English→Arabic, etc.) to provide a cross-language aligned supervision signal for speech recognition of low-resource languages, and provide an implementation basis for speech recognition of low-resource languages.

[0043] In practice, voice data collection is quite difficult, especially for minority languages, where not only is collection challenging but annotation is also difficult. However, text translation data, such as first-language text and its corresponding second-language translation, is easier to collect. Therefore, this embodiment of the invention first collects the translated text data to train a text translator with text-to-text conversion capabilities. Based on the text translator, a speech quantizer is jointly trained, aligning the speech encoding features obtained by the speech quantizer with the text encoding features of the corresponding languages, providing cross-language alignment supervision signals, and yielding the final speech quantizer. Thus, for a particular minority language, only a small amount (e.g., at least one) of pre-set speech data containing the target instruction word is needed, eliminating the need to collect a large amount of speech data containing the target instruction word to train the speech recognition model. The speech recognition function for that minority language can be achieved using only the text translator and the speech quantizer, reducing the difficulty of data collection and improving model training efficiency.

[0044] In this embodiment of the invention, the target language refers to a less commonly spoken language that requires the use of speech recognition. This embodiment of the invention does not limit the target language. For example, the target language is Arabic. In one example, it is assumed that a speech quantizer and a text translator have been trained. The Arabic speech to be recognized is input into the speech quantizer to obtain the first word vector sequence corresponding to the speech. Assuming that a preset Arabic speech containing the target instruction word "turn on the TV" has been collected beforehand, and assuming that the preset speech is divided into speech segments of fixed length as follows {s1, s2, s3, s4, s5}, these segments are sequentially input into the speech quantizer to obtain the word vector corresponding to each speech segment, forming the following second word vector sequence {v1, v2, v3, v4, v5}. Wherein, v1 is the word vector corresponding to speech segment s1, v2 is the word vector corresponding to speech segment s2, and so on.

[0045] After inputting the Arabic speech to be recognized into the speech quantizer to obtain the first word vector sequence corresponding to the speech, the first word vector sequence is compared with the second word vector sequence to obtain the comparison result. This comparison result can indicate the degree of matching between the first word vector sequence and the second word vector sequence. If the degree of matching meets the preset requirements, it can be considered that the speech to be recognized contains the target instruction word; otherwise, it is considered that the speech to be recognized does not contain the target instruction word.

[0046] Optionally, comparing the first word vector sequence with the second word vector sequence to obtain the comparison result may include:

[0047] Each word vector in the first word vector sequence is compared with the corresponding word vector in the second word vector sequence. If the similarity between each word vector in the first word vector sequence and the corresponding word vector in the second word vector sequence reaches a preset threshold, it is determined that the first word vector sequence and the second word vector sequence match, and the speech to be recognized contains the target instruction word. Otherwise, if the similarity between a word vector in the first word vector sequence and the corresponding word vector in the second word vector sequence does not reach the preset threshold, it is determined that the first word vector sequence and the second word vector sequence do not match, and the speech to be recognized does not contain the target instruction word.

[0048] For example, the Arabic speech to be recognized is segmented into speech segments of fixed length as follows: {t1, t2, t3, t4, t5}. These segments are then sequentially input into a speech quantizer to obtain word vectors corresponding to each speech segment, forming the following first word vector sequence: {e1, e2, e3, e4, e5}. Here, e1 is the word vector corresponding to speech segment t1, e2 is the word vector corresponding to speech segment t2, and so on.

[0049] The first word vector sequence {e1,e2,e3,e4,e5} is compared with the second word vector sequence {v1,v2,v3,v4,v5}. For example, e1 is compared with v1. If their similarity meets a preset threshold, the next word vector is compared, e2 is compared with v2, and so on. If the similarity between a word vector in the first word vector sequence and a corresponding word vector in the second word vector sequence does not reach the preset threshold, the comparison is terminated, resulting in a mismatch between the first and second word vector sequences, indicating that the target instruction word is not contained in the speech to be recognized. If the similarity between each word vector in the first word vector sequence and a corresponding word vector in the second word vector sequence reaches the preset threshold, a matching comparison is obtained, indicating that the target instruction word is contained in the speech to be recognized.

[0050] In an optional embodiment of the present invention, the method may further include:

[0051] Step S11: Collect a first preset number of preset voice recordings, where the first preset number is greater than 1;

[0052] Step S12: Input each preset speech into the trained speech quantizer to obtain the second word vector sequence corresponding to each preset speech.

[0053] In practice, the number of preset speech samples can be one. However, in real-world applications, the same sentence spoken by the same person in different environments, or spoken by different people, may result in differences in speech characteristics. Therefore, to ensure the accuracy of speech recognition, the number of preset speech samples collected can be greater than one. Taking Arabic as the target language as an example, assuming the first preset number is 5, then 5 preset Arabic speech samples containing the target instruction words are collected. These 5 preset speech samples can come from the same person's voice, or they can come from different people's voices; this invention does not impose any restrictions on this. Each preset speech sample is divided into several speech segments of a fixed length and then sequentially input into the trained speech quantizer to obtain the second word vector sequence corresponding to each preset speech sample.

[0054] Further, the step of comparing the first word vector sequence with the second word vector sequence to obtain the comparison result may include:

[0055] The first word vector sequence is compared with the second word vector sequence corresponding to each preset speech, and the comparison result of whether the first word vector sequence matches the second word vector sequence corresponding to each preset speech is obtained.

[0056] The step of determining whether the target instruction word is contained in the speech to be recognized based on the comparison result may include:

[0057] Determine whether the first word vector sequence matches a second word vector sequence exceeding a second preset number. If yes, determine that the speech to be recognized contains the target instruction word; otherwise, determine that the speech to be recognized does not contain the target instruction word.

[0058] In specific implementations, after obtaining the final speech quantizer through joint training with a speech dequantizer and a text translator, a speech recognition module can be constructed based on this speech quantizer. The speech recognition module internally contains the speech quantizer and can be used to implement speech recognition functions for less commonly spoken languages. The speech recognition module can be a standalone application, a functional module integrated into other applications, or a independently deployed device, etc. This embodiment of the invention does not impose any limitations on these aspects.

[0059] Furthermore, the final speech quantizer can also serve as a standalone tool for generating a second word vector sequence in the target language. This second word vector sequence corresponds to the target instruction word in the target language and is used to determine whether the target language speech to be recognized contains that target instruction word. For example, in the above example, five preset Arabic speech samples containing the target instruction word are collected beforehand; assuming the target instruction word is "turn on the TV". Each preset speech sample is divided into several speech segments of a fixed length and input sequentially into the speech quantizer to obtain the second word vector sequence corresponding to each preset speech sample, i.e., five second word vector sequences are obtained.

[0060] The speech recognition module can pre-set a second word vector sequence of the target language, such as pre-setting the second word vector sequences of the above five Arabic languages.

[0061] Next, the Arabic speech to be recognized is input into the speech recognition module. The speech recognition module divides the speech into several speech segments of fixed length and inputs them sequentially into its internal speech quantizer to obtain a first word vector sequence. The speech recognition module compares this first word vector sequence with each of the five preset second word vector sequences, obtaining five comparison results. If more than a second preset number of these five comparison results match, it is determined that the speech contains the target instruction word, and a successful recognition result can be output; otherwise, it is determined that the speech does not contain the target instruction word, and a failed recognition result can be output.

[0062] Wherein, the second preset number can be greater than or equal to half of the first preset number. For example, in this example, the second preset number can be 3. If the first word vector sequence matches at least 3 of the above 5 second word vector sequences, it is determined that the speech to be recognized contains the target instruction word; otherwise, it is determined that the speech to be recognized does not contain the target instruction word.

[0063] It is understood that the values ​​of the first preset quantity and the second preset quantity mentioned above are only illustrative examples, and the embodiments of the present invention do not limit the values ​​of the first preset quantity and the second preset quantity.

[0064] In an optional embodiment of the present invention, the method may further include:

[0065] Construct a first training set, and train a text translator based on the first training set; the first training set includes several first language texts and second language texts corresponding to each first language text.

[0066] Construct a second training set, and jointly train a speech quantizer and a speech dequantizer based on the second training set; the second training set includes several audio data and text data of different languages ​​corresponding to each audio data.

[0067] Based on the second training set, the parameters of the text translator are fixed, and the text translator and the speech quantizer are jointly trained to obtain the final speech quantizer.

[0068] To achieve speech recognition functionality for less commonly spoken languages, a speech quantizer and a text translator need to be trained first. Specifically, a first training set is constructed, and a text translator is trained based on this set to achieve text translation functionality. A second training set is constructed, and the speech quantizer and speech dequantizer are jointly trained based on this set. The speech quantizer is supervised by the speech dequantizer during training to obtain a more accurate speech quantizer. After training the speech quantizer and text translator, based on the second training set, the parameters of the text translator are fixed, and the text translator and speech quantizer are jointly trained to further optimize the parameters of the speech quantizer, resulting in the final speech quantizer. The text translation function of the text translator is used to supervise the training of the speech quantizer, providing cross-language alignment supervision signals. This ensures that the features encoded by the speech quantizer for speech segments are aligned with the text features of the corresponding languages, providing a foundation for speech recognition functionality for less commonly spoken languages.

[0069] In an optional embodiment of the present invention, constructing the first training set may include:

[0070] Step S21: Collect several first-language texts and the corresponding second-language texts for each first-language text;

[0071] Step S22: Convert the collected first language text into binary encoding and segment it according to a first preset length to obtain the source character sequence; and convert the collected second language text corresponding to the first language text into binary encoding and segment it according to a first preset length to obtain the target character sequence.

[0072] Step S23: Combine each source character sequence with its corresponding target character sequence to form the first training set.

[0073] The first training set constructed in this embodiment of the invention is used to train a text translator. To train a text translator capable of cross-language text conversion (e.g., Chinese → Arabic), it is necessary to collect mutual translation text data in multiple languages. The mutual translation text data includes several first-language texts and corresponding second-language texts for each first-language text.

[0074] Exemplarily, this embodiment of the invention does not limit the source of the translated text data. For example, the source of the translated text data can be different language versions of the same book, such as classic literary works, academic works, popular science books, etc. For the selected book, its different language versions are obtained. Each book is divided into text units, and for each text unit, its corresponding translation in different language versions is extracted. The text unit can be a paragraph, a sentence, or a phrase, etc. For example, for a certain text unit in a Chinese version of a book, this text unit is the first language text (e.g., denoted as first language text 1), where the first language is Chinese. For this text unit, the corresponding translation content is found in different language translation versions of the Chinese version of the book, which is the second language text corresponding to first language text 1. For example, the translation content corresponding to this text unit is found in the Arabic version of the book, which is the second language text (e.g., denoted as second language text 1), where the second language is Arabic. As another example, the translation content corresponding to this text unit is found in the English version of the book, which is the second language text (e.g., denoted as second language text 2), where the second language is English. For the same text unit, there are translation correspondences between the first-language text and each second-language text. Through multilingual text alignment, the text translator can learn the semantic correspondences between different languages ​​during training.

[0075] It should be noted that the first language and the second language are different languages. This embodiment of the invention does not limit the type of the first language or the type of the second language. The second language includes the target language.

[0076] Furthermore, in this embodiment of the invention, the collected first language text is converted into binary encoding and segmented according to a first preset length to obtain source character sequences; and the collected second language text is converted into binary encoding and segmented according to a first preset length to obtain target character sequences; and each source character sequence and its corresponding target character sequence constitute a first training set.

[0077] This invention converts the first language text and the second language text into binary encoding, such as encoding the first language text and the second language text into a binary sequence by character (or word). For example, UTF-8 encoding can be used to convert the text into binary encoding and then segment it according to a first preset length. This invention does not limit the first preset length, but assumes it to be 3 bits. For example, for Arabic text... This can be converted to the following binary encoding: \xd9\x87\xd8\xb0\xd8\xa7\xd9\x87\xd9\x88\xd9\x85\xd8\xab\xd8\xa7\xd9\x84. Each byte (e.g., \xd9) represents an encoding unit, and the length of an encoding unit is a first preset length, which can represent one character (or word) in the text.

[0078] This invention converts the first language text and the second language text into binary encoding, transforming the training data in the first training set into a unified numerical form, which facilitates model processing. Furthermore, by using a unified numerical form, the model does not need to understand the linguistic meaning; it only needs to process the binary encoding, reducing the complexity of cross-language alignment and improving model training efficiency and accuracy. For example, for the Arabic text mentioned above... It is difficult to distinguish a single character. After converting it to binary encoding, each byte (such as \xd9) can represent a character, which makes it easier for the model to process.

[0079] In an optional embodiment of the present invention, the text transcriber may include a first encoder and a first decoder; the text transcriber trained based on the first training set may include:

[0080] Step S31: Initialize the first encoder and the first decoder;

[0081] Step S32: Select the current source character sequence of the first language and the corresponding current target character sequence of the second language from the first training set;

[0082] Step S33: Input the current source character sequence and the first language identifier into the first encoder to obtain the text encoding features corresponding to the current source character sequence;

[0083] Step S34: Input the text encoding features corresponding to the current source character sequence and the second language identifier into the first decoder to obtain the decoded character sequence corresponding to the text encoding features;

[0084] Step S35: Calculate the first loss based on the decoded character sequence and the current target character sequence;

[0085] Step S36: Update the parameters of the first encoder and the first decoder according to the first loss;

[0086] Step S37: Proceed to the next round of iterative training. When the iteration stopping condition is met, the trained text translator is obtained.

[0087] The function of the text translator is to realize cross-language text conversion, converting a first-language text into a second-language text, for example, converting a Chinese text into an Arabic text. The embodiments of the present invention use the text translator to provide a cross-language aligned supervision signal for speech recognition of low-resource languages. The cross-language alignment refers to enabling the model to learn the semantic correspondence between different languages through multilingual text alignment.

[0088] The embodiments of the present invention do not limit the structure of the text translator. For example, the text translator may include a first encoder and a first decoder. The input of the text translator includes a first-language text, a first-language identifier and a second-language identifier. The output of the text translator is a second-language text obtained by translating the input first-language text. Specifically, the input of the text translator is the binary encoding of the first-language text, and the output of the text translator is the binary encoding of the second-language text.

[0089] The embodiments of the present invention perform steps S31 to S37 to train the text translator using the constructed first training set.

[0090] First, the first encoder and the first decoder are initialized. The current source character sequence of the first language and the corresponding current target character sequence of the second language are selected from the first training set. For example, the first language is Chinese, and the second language is Arabic, and the selected current source character sequence is the binary encoding corresponding to the Chinese text "This is an example", such as:

[0091] \xe8\lxb\fx9\9xe\6lx\98l\xaf\xe4\xb8\x80\lxe\4xb\lxa\axe\4xb\elx\8bx\e5x\a dl\x90。

[0092] In this example, the current target character sequence corresponding to the current source character sequence is the Arabic text corresponding binary encoding, such as:

[0093] \xd9\x87\xd8\xb0\xd8\xa7\xd9\x87\xd9\x88\xd9\x85\xd8\xab\xd8\xa7\xd9\x84。

[0094] Wherein, translating the Arabic text into Chinese is exactly "This is an example".

[0095] The current source character sequence and the first language identifier are input into the first encoder to obtain the text encoding features corresponding to the current source character sequence. The language identifier is used to identify the language type. For example, one-hot encoding or embedding vectors can be used as the language identifier, such as Chinese language identifier as [1,0,0] and Arabic language identifier as [0,1,0].

[0096] The input to the first encoder includes a source character sequence (such as UTF-8 encoding of Chinese text) and a first language identifier (such as [1,0,0]). The first encoder encodes the received source character sequence according to the first language identifier to obtain text encoding features. For example, the first encoder can be a Transformer encoder, and the encoding process of the first encoder may include concatenating or weighting the received source character sequence with the first language identifier to generate text encoding features. The text encoding features are a fusion of text semantic information and linguistic information.

[0097] The output of the first encoder serves as the input of the first decoder, which in turn includes the output of the first encoder and a second language identifier (e.g., [0,1,0]). The first decoder decodes the received text encoding features based on the second language identifier to obtain a decoded character sequence (or translation result). For example, the first decoder can be a Transformer decoder, and its decoding process may include: using a self-attention mechanism, combining the text encoding features and the second language identifier, to generate a binary encoding (decoded character sequence) of the second language text character by character, such as the UTF-8 encoding of Arabic text.

[0098] Based on the difference between the decoded character sequence (predicted value) and the current target character sequence (true value), the first loss of the text translator can be calculated. (Refer to...) Figure 2 This illustrates a flowchart of the process for calculating the first loss of a text transcriber according to an embodiment of the present invention. Figure 2 As shown, the text transcriber includes a first encoder and a first decoder. The input to the first encoder includes the current source character sequence and a first language identifier. Assuming the first language text is the Chinese text "This is an example", the corresponding second language text is Arabic text. The first encoder encodes the received current source character sequence and the first language identifier, outputting text encoding features. The first decoder takes as input the output of the first encoder and the second language identifier. The first decoder decodes the received text encoding features and the second language identifier, outputting a decoded character sequence. Based on the difference between this decoded character sequence and the current target character sequence, the first loss of the text transcriber can be calculated.

[0099] Update the parameters of the first encoder and the first decoder according to the first loss; enter the next round of iteration training, reselect the current source character sequence of the first language and the corresponding current target character sequence of the second language from the first training set, execute steps S33 to S37, and obtain the trained text translator when the iteration stopping condition is met.

[0100] This invention does not limit the method for calculating the first loss. For example, a general loss function can be used to calculate the first loss. General loss functions include cross-entropy loss function, cosine similarity loss function, etc. The iteration stopping condition may include the first loss being less than a preset value, or the number of iterations reaching a preset number.

[0101] In an optional embodiment of the present invention, constructing the second training set may include:

[0102] Step S41: Collect several target videos, each target video corresponding to at least two languages;

[0103] Step S42: For each target video, switch the language of the target video, and extract audio data in different languages ​​and text data in different languages ​​corresponding to each audio data from the target video;

[0104] Step S43: The extracted audio data and text data form a second training set.

[0105] In this embodiment of the invention, a second training set is constructed for jointly training the speech quantizer and the speech dequantizer. After the speech quantizer and the speech dequantizer are jointly trained, the second training set can be used to jointly train the text translator and the speech quantizer to obtain the final speech quantizer.

[0106] The second training set includes several audio data sets and corresponding text data in different languages ​​for each audio data set. The audio data in the second training set is used to jointly train the speech quantizer and the speech dequantizer. The audio data and text data in the second training set are used together to jointly train the text translator and the speech quantizer.

[0107] It's important to clarify that audio data generally refers to the electrical signal representation of sound, encompassing all types of sound, including music, environmental noise, animal calls, human speech, and various other sound signals. Speech data specifically refers to the sounds produced by humans through their vocal organs for communication and expressing thoughts. Speech data possesses specific language rules and semantic information, and is the acoustic representation of human language. Audio data has a broader scope, including speech and other non-speech sounds. Any sound signal that can be captured and processed by audio devices can be called audio data.

[0108] In the embodiments of the present invention, the audio data in the second training set may be extracted from videos, and may contain non-speech sounds, so it is described as audio data. It is understandable that in specific implementations, the extracted audio data may be further processed to filter out non-speech sounds such as ambient noise therein, so as to obtain speech data as training data. In the embodiments of the present invention, audio data may be equivalent to speech data.

[0109] The embodiments of the present invention do not limit the data source of the second training set. Exemplarily, the data source of the second data set may be target videos that include comparison versions in multiple languages. For example, various types of target videos with labeled and translated subtitles may be searched. The themes of the target videos may cover multiple types such as movies, TV series, documentaries, educational lectures, news reports, etc., so as to ensure the diversity of data. Each target video includes comparison versions in multiple languages. The multiple languages at least include a first language and a second language, the first language may be the native language, and the second language may be a target language, such as a certain minor language (Arabic, etc.).

[0110] For each target video, the language type of the target video may be switched, and audio data in different languages and text data in different languages corresponding to each piece of audio data are extracted from the target video. For example, a certain target video includes comparison versions in three languages: Chinese, English and Arabic. Assuming that the Chinese version is switched to, Chinese audio data and text data in Chinese, English and Arabic respectively corresponding to the Chinese audio data may be extracted.

[0111] Exemplarily, audio data of a certain duration is extracted from a certain target video, and the speech content of the audio data is "Hello, the weather is nice today" (in Chinese). According to the timestamp information of the subtitles, the start time and end time for intercepting the audio data may be accurately set, whereby by switching the language type of the target video, the subtitles in Chinese, English and Arabic respectively corresponding to the audio data may be extracted as text data in different languages (Chinese, English and Arabic) corresponding to the audio data.

[0112] Exemplarily, the extracted text data includes: "Hello, the weather is nice today" (in Chinese), "Hello, nice weather today" (in English), (in Arabic).

[0113] According to the above method, a plurality of pieces of audio data in different languages and text data in different languages corresponding to each piece of audio data may be extracted from this target video and other target videos to form the second training set.

[0114] In an optional embodiment of the present invention, the step of jointly training the speech quantizer and the speech inverse quantizer based on the second training set may include:

[0115] Step S51: Initialize the speech quantizer and speech dequantizer;

[0116] Step S52: Select an audio data from the second training set as the first input audio, divide it into several speech segments according to the second preset length, and input them into the speech quantizer in sequence to obtain the word vector sequence corresponding to the first input audio;

[0117] Step S53: Input the word vector sequence corresponding to the first input audio into the speech dequantizer to obtain the decoded audio;

[0118] Step S54: Calculate the second loss based on the decoded audio and the first input audio;

[0119] Step S55: Update the parameters of the speech quantizer and the speech inverse quantizer according to the second loss;

[0120] Step S56: Proceed to the next iteration. When the iteration stopping condition is met, the trained speech quantizer and speech dequantizer are obtained.

[0121] This invention does not limit the structure of the speech quantizer and speech dequantizer. Exemplarily, the speech quantizer may include a second encoder and a classifier. Exemplarily, the second encoder may use a UNet encoder, and the classifier may be implemented using a fully connected layer. The second encoder is used to receive a speech segment of a second preset length, encode the received speech segment to obtain speech coding features, and input the speech coding features into the classifier; the classifier is used to output the classification result corresponding to the speech coding features.

[0122] The speech dequantizer may include a second decoder and a generator. For example, the second decoder may use a Unet decoder, and the generator may be implemented using a fully connected layer. The second decoder receives a sequence of word vectors, decodes the received word vector sequence into decoded features, and inputs the decoded features into the generator; the generator converts the decoded features into audio data (decoded audio).

[0123] Furthermore, in this embodiment of the invention, a quantization dictionary is pre-defined, which contains a mapping relationship between pre-defined word vectors and classification results corresponding to speech segments. A word vector can be a feature representation of a speech segment of a preset length. Furthermore, the classification results corresponding to the word vectors in the quantization dictionary can correspond to the classification results predictable by the speech quantizer.

[0124] In this embodiment of the invention, the quantization dictionary can serve as a constraint for jointly training the speech quantizer and the speech dequantizer. Since the model's representations can be infinite, leading to long training times, this invention uses a quantization dictionary to constrain the model's representations, limiting the representations of the speech quantizer and the speech dequantizer to word vectors within the quantization dictionary, thereby accelerating the model training process.

[0125] For each speech segment output by the classifier, the corresponding word vector can be queried in the quantization dictionary to obtain a word vector sequence.

[0126] In this embodiment of the invention, the speech quantizer may include a second encoder and a classifier. The classifier has M categories, where M is a positive integer. The value of M can be set according to actual needs, and M can be the number of word vectors preset in the quantization dictionary. The second encoder receives a speech segment of a second preset length, encodes the received speech segment to obtain speech coding features, and inputs the speech coding features into the classifier. The classifier classifies the received speech coding features and outputs the classification result corresponding to the speech coding features.

[0127] Reference Figure 3 The diagram illustrates the flowchart of the joint training of the speech quantizer and speech inverse quantizer of the present invention. Figure 3 As shown, assuming there are 4 pre-set word vectors in the quantization dictionary, denoted as v1, v2, v3, and v4, these 4 word vectors correspond to the classification results of 4 speech segments as 1, 2, 3, and 4, respectively.

[0128] First, the speech quantizer and speech dequantizer are initialized. An audio data point is taken from the second training set as the first input audio. This first input audio is segmented into several speech segments according to a second preset length, and then sequentially input into the initialized speech quantizer. The speech quantizer predicts the classification result corresponding to each speech segment, thus obtaining the quantized sequence (or classification result vector) corresponding to the first input audio. The quantized sequence corresponding to the first input audio includes the classification result corresponding to each speech segment obtained from the segmentation of the first input audio.

[0129] In one example, for the current first input audio, assume it is divided into 5 speech segments according to a second preset length, which can be set according to actual needs. These 5 speech segments are sequentially input into a speech quantizer for serialization processing. Assume the output classification result for each speech segment is: 1, 3, 2, 1, 4. That is, the classification result for the first speech segment is 1, the classification result for the second speech segment is 3, and so on. The classification result vector for this first input audio is {1,3,2,1,4}.

[0130] In this embodiment of the invention, the speech quantizer can predict the probability value of each classification result in the quantization dictionary corresponding to the received speech segment, and take the one with the highest probability value as the classification result of the speech segment.

[0131] For example, in the example above, after the first speech segment is input into the speech quantizer, the speech quantizer predicts the probability value of each classification result in the quantization dictionary corresponding to the first speech segment. Assuming that the probability value of the first speech segment belonging to classification result 1 is the highest, the predicted classification result for the first speech segment is 1. Similarly, the predicted classification result for the second speech segment is 3. And so on, thus obtaining the quantization sequence of the first input audio as {1,3,2,1,4}.

[0132] Then, based on the classification result of each speech segment, the quantization dictionary is queried, and the word vectors corresponding to each classification result in the quantization sequence {1,3,2,1,4} are obtained from the quantization dictionary. These word vectors are then arranged in order to form the word vector sequence corresponding to the first input audio. For example, for the quantization sequence {1,3,2,1,4} of the first input audio, the word vectors corresponding to each classification result can be extracted from the quantization dictionary and arranged in order to form a word vector sequence, such as {v1,v3,v2,v1,v4}.

[0133] Next, the word vector sequence corresponding to the first input audio is input into the initialized speech dequantizer to predict the decoded audio. Based on the difference between the decoded audio (predicted value) and the first input audio (true value), a second loss can be calculated. The model parameters of the speech quantizer and the speech dequantizer can be updated based on the second loss. In the next iteration, a new audio data point is taken from the second training set as the first input audio, and the above steps are repeated. When the iteration stopping condition is met, the trained speech quantizer and speech dequantizer are obtained.

[0134] This invention does not limit the method for calculating the second loss. For example, a general loss function can be used to calculate the model loss. General loss functions include cross-entropy loss function, cosine similarity loss function, etc. The iteration stopping condition may include the second loss being less than a preset value, or the number of iterations reaching a preset number.

[0135] In an optional embodiment of the present invention, the method may further include: updating the parameters of the quantization dictionary according to the second loss.

[0136] In this embodiment of the invention, the quantization dictionary can be a trainable vector group composed of N vectors of dimension V. The quantization dictionary contains N×V word vectors, each corresponding to its own weight parameters. During the joint training of the speech quantizer and the speech dequantizer, the quantization dictionary can also be trained to continuously optimize the weight parameters of each word vector in the quantization dictionary.

[0137] Specifically, after calculating the second loss, the model parameters of the speech quantizer, the model parameters of the speech dequantizer, and the weight parameters of each word vector in the quantization dictionary can be updated according to the second loss, and the next iteration can be entered. When the iteration stopping condition is met, the trained speech quantizer, speech dequantizer, and quantization dictionary are obtained.

[0138] In an optional embodiment of the present invention, the step of jointly training the text translator and the speech quantizer based on the second training set, fixing the parameters of the text translator, to obtain the final speech quantizer may include:

[0139] Step S61: Select an audio data in the first language from the second training set as the second input audio, and select the text data in the second language corresponding to the second input audio as the first label;

[0140] Step S62: Divide the second input audio into several speech segments according to the second preset length and input them sequentially into the speech quantizer to obtain the word vector sequence corresponding to the second input audio;

[0141] Step S63: Input the word vector sequence corresponding to the second input audio, the first language identifier, and the second language identifier into the text transcriber to obtain the first transcribing result;

[0142] Step S64: Calculate the third loss based on the first translation result and the first label;

[0143] Step S65: Fix the parameters of the text translator and update the parameters of the speech quantizer according to the third loss;

[0144] Step S66: Proceed to the next round of iterative training. When the iteration stopping condition is met, the final speech quantizer is obtained.

[0145] After training the text translator and jointly training the speech quantizer and speech dequantizer, the trained text translator and speech quantizer can be jointly trained. In this process, the parameters of the text translator are fixed, and the parameters of the speech quantizer are further optimized to align the output of the speech quantizer with the information of the target language text, thus obtaining the final speech quantizer.

[0146] In this embodiment of the invention, audio data and text data from a second training set are used to jointly train a text translator and a speech quantizer.

[0147] Reference Figure 4 The diagram illustrates the process of jointly training the text transcriber and the speech quantizer of the present invention.

[0148] First, select an audio data point in the first language from the second training set as the second input audio, and select the corresponding text data in the second language from the second input audio as the first label.

[0149] In this example, assuming the first language is Chinese and the second language is Arabic, the selected second input audio is an audio segment with the voice content "This is an example" (in Chinese). The selected text data (binary encoding) in the corresponding second language (e.g., Arabic) is used as the first tag, such as: \xd9\x87\xd8\xb0\xd8\xa7\xd9\x87\xd9\x88\xd9\x85\xd8\xab\xd8\xa7\xd9\x84.

[0150] The second input audio is divided into several speech segments according to a second preset length and then sequentially input into the speech quantizer. The second encoder of the speech quantizer encodes each speech segment to obtain speech coding features. The classifier of the speech quantizer predicts the classification result corresponding to the speech coding features output by the second encoder. By querying the quantization dictionary, the word vector corresponding to each classification result can be obtained, thus obtaining the word vector sequence corresponding to the second input audio.

[0151] The word vector sequence corresponding to the second input audio, the first language identifier, and the second language identifier (e.g., [0,1,0]) are input into the text transcriber. The first encoder of the text transcriber receives the word vector sequence and the first language identifier corresponding to the second input audio, encodes them, and outputs text encoding features. The first decoder of the text transcriber receives the second language identifier and the text encoding features output by the first encoder, transcribs them, and outputs a first transcribing result. The first transcribing result is a decoded character sequence of the second language.

[0152] Based on the first translation result (predicted value) and the first label (true value), a third loss can be calculated. The parameters of the text translator are fixed, and the parameters of the speech quantizer are updated according to the third loss. The next round of iterative training is then initiated. A second input audio and the corresponding first label are reselected from the second training set, and the above steps are repeated. When the iteration stopping condition is met, the final speech quantizer is obtained.

[0153] This invention does not limit the method for calculating the third loss. For example, a general loss function can be used to calculate the third loss. General loss functions include cross-entropy loss function, cosine similarity loss function, etc. The iteration stopping condition may include the third loss being less than a preset value, or the number of iterations reaching a preset number.

[0154] In an optional embodiment of the present invention, after training the text transcriber, the method may further include:

[0155] Step S71: Select a source character sequence of the first language from the first training set as the input character sequence;

[0156] Step S72: Select the target character sequence of the second language corresponding to the input character sequence from the first training set as the second label;

[0157] Step S73: Insert blank characters into the input character sequence according to a preset rule to obtain the processed character sequence;

[0158] Step S74: Input the processed character sequence, the first language identifier, and the second language identifier into the text translator to obtain the second translation result;

[0159] Step S75: Calculate the fourth loss based on the second translation result and the second label;

[0160] Step S76: Update the parameters of the text transcriber according to the fourth loss;

[0161] Step S77: Proceed to the next round of iterative training. When the iteration stopping condition is met, the fine-tuned text translator is obtained.

[0162] In practice, speech often contains invalid information such as silence or noise, so the length of speech is usually longer than the corresponding text. After the speech quantizer encodes the speech, the resulting speech coding features may contain this invalid information, which in turn affects the text translator's ability to provide cross-language alignment supervision for speech recognition.

[0163] To address this issue, in this embodiment of the invention, after training the text translator, speech quantizer, and speech dequantizer, the trained text translator is fine-tuned before jointly training the text translator and speech quantizer to accommodate invalid information in speech coding.

[0164] Specifically, source character sequences of the first language are selected from the first training set as input character sequences. For example, assuming the first language is Chinese, the first language text is "This is an example"; the second language is Arabic, and the second language text corresponding to the first language text "This is an example" is... The selected input character sequence is the binary encoding corresponding to the Chinese text, such as: \xe8\lxb\fx9\9xe\6lx\98l\xaf\xe4\xb8\x80\lxe\4xb\lxa\axe\4xb\elx\8bx\e5x\adl\x90. The target character sequence of the second language corresponding to this input character sequence is selected from the first training set as the second label, i.e., the second language text is... The corresponding binary encoding is as follows: \xd9\x87\xd8\xb0\xd8\xa7\xd9\x87\xd9\x88\xd9\x85\xd8\xab\xd8\xa7\xd9\x84.

[0165] The input character sequence is processed by inserting blank characters according to a preset rule to obtain the processed character sequence. For example, the preset rule may include: randomly inserting blank characters into the input character sequence (in binary encoding form), where the length of the blank character is a specified number of bytes; the position of the blank character is random, but it must be ensured that a blank character is inserted at least once in the input character sequence. The specified number of bytes may include 1 to 3 bytes.

[0166] Inserted whitespace characters are used to simulate invalid information in speech, and the randomness of whitespace insertion is used to enhance the robustness of the text translator in aligning character lengths of different languages.

[0167] For example, in the example above, inserting whitespace characters into the selected input character sequence according to preset rules results in the following processed character sequence:

[0168] xe8\lxb\fx9\9xc\elx\b8l\xce\lxb\8xe\6lx\98x\afx\cel\xb8\le4\xb8\x80\lxc\elx\b8 x\e4l\xb8\xaa\lxc\elx\b8x\cel\xb8\lxc\elx\b8l\xe4\xbe\lx8\blx\cel\xb8\xe5\lxa\d x9\0lx\cex\b8l\xce\lxb\8lx\cel\xb8.

[0169] The text corresponding to the character sequence after the above processing is: "This 00 is 0-0 examples of 000". Here, the inserted 0 represents a randomly inserted whitespace character.

[0170] The processed character sequence, the first language identifier, and the second language identifier are input into the text transcriber. The first encoder of the text transcriber receives the processed character sequence and the first language identifier, encodes them, and outputs text encoding features. The first decoder of the text transcriber receives the second language identifier and the text encoding features output by the first encoder, transcribs them, and outputs a second transcribing result. The second transcribing result is the decoded character sequence of the second language.

[0171] The fourth loss can be calculated based on the second translation result (predicted value) and the second label (true value); the parameters of the text translator are updated according to the fourth loss; the next round of iteration training is entered, the input character sequence and the second label are reselected from the first training set, and the above steps are repeated. When the iteration stopping condition is met, the fine-tuned text translator is obtained.

[0172] This invention does not limit the method for calculating the fourth loss. For example, a general loss function can be used to calculate the fourth loss. General loss functions include the cross-entropy loss function, the cosine similarity loss function, etc. The iteration stopping condition may include the fourth loss being less than a preset value, or the number of iterations reaching a preset number.

[0173] Further, the step of jointly training the text translator and the speech quantizer based on the second training set, fixing the parameters of the text translator, to obtain the final speech quantizer, may include:

[0174] Based on the second training set, the parameters of the text translator are fixed, and the fine-tuned text translator and the speech quantizer are jointly trained to obtain the final speech quantizer.

[0175] The final speech quantizer can be used to achieve speech recognition for less commonly spoken languages. Specifically, a first preset number of pre-set speech samples are collected, which are speech data of the target language containing target instruction words. The speech quantizer can then be used to extract a second word vector sequence of the first preset number of samples.

[0176] Reference Figure 5 This illustrates a flowchart of the present invention for extracting the second word vector sequence using a speech quantizer. Figure 5As shown, a preset speech is divided into several speech segments according to a second preset length and sequentially input into a speech quantizer. The second encoder of the speech quantizer encodes each speech segment to obtain speech coding features, which are then input into the classifier of the speech quantizer. The classifier predicts a classification result based on the speech coding features corresponding to each speech segment. Furthermore, invalid classification results can be filtered out. Invalid classification results correspond to invalid information in the preset speech and can be represented by a specified classification result, for example, using classification result 0 to represent an invalid classification result. Figure 5 As shown, after filtering out invalid classification results, the resulting classification result vector is {14,1,2}. By querying the quantization dictionary, the word vector corresponding to each classification result in this vector can be obtained, thus yielding the second word vector sequence corresponding to the preset speech.

[0177] Since the speech quantizer is jointly trained with a text translator, and the text translator has text translation capabilities, such as translating first-language text into second-language text, the second-language text can be the target language (such as a minor language), the joint training of the speech quantizer and the text translator allows the use of the text translator's cross-language text conversion (text translation) function to provide cross-language alignment supervision signals for speech recognition. Therefore, the speech quantizer is used to extract a second word vector sequence from a preset speech in the target language. When performing speech recognition on the speech to be recognized in the target language, the speech to be recognized in the target language can be input into the speech quantizer to obtain a first word vector sequence corresponding to the speech to be recognized. By comparing the first word vector sequence with the second word vector sequence, it can be determined whether the speech to be recognized contains the target instruction word, thereby realizing the keyword speech recognition function for minor languages. Through this embodiment of the invention, it is only necessary to collect speech data of the target language (a minor language) covering all texts of that language, without collecting a large amount of minor language speech data containing the target instruction word, to achieve the speech recognition capability for minor languages.

[0178] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0179] Reference Figure 6 The diagram illustrates a structural block diagram of an embodiment of a speech recognition device according to the present invention. The device may include:

[0180] The speech conversion module 601 is used to input the speech to be recognized in the target language into a speech quantizer to obtain a first word vector sequence corresponding to the speech to be recognized; the speech quantizer is a neural network model jointly trained with a speech dequantizer and a text translator; the speech quantizer is used to convert the speech into a word vector sequence; the speech dequantizer is used to convert the word vector sequence into speech; and the text translator is used to convert the first language text into the second language text.

[0181] The feature comparison module 602 is used to compare the first word vector sequence with the second word vector sequence to obtain a comparison result; the second word vector sequence is obtained by inputting a preset speech of the target language into the speech quantizer and converting it, and the preset speech contains target instruction words;

[0182] The target recognition module 603 is used to determine whether the target instruction word is contained in the speech to be recognized based on the comparison result.

[0183] Optionally, the device further includes:

[0184] The first training module is used to construct a first training set and train a text transcriber based on the first training set; the first training set includes several first language texts and second language texts corresponding to each first language text.

[0185] The second training module is used to construct a second training set, and to jointly train a speech quantizer and a speech dequantizer based on the second training set; the second training set includes several audio data and text data in different languages ​​corresponding to each audio data.

[0186] The third training module is used to perform joint training on the text translator and the speech quantizer based on the second training set, with the parameters of the text translator fixed, to obtain the final speech quantizer.

[0187] Optionally, the first training module includes a first construction module for constructing a first training set; the first construction module includes:

[0188] The first collection submodule is used to collect several first-language texts and the corresponding second-language texts for each first-language text.

[0189] The encoding conversion submodule is used to convert the collected first language text into binary encoding and segment it according to a first preset length to obtain the source character sequence; and to convert the collected second language text corresponding to the first language text into binary encoding and segment it according to a first preset length to obtain the target character sequence.

[0190] The first construction submodule is used to construct the first training set by combining each source character sequence with its corresponding target character sequence.

[0191] Optionally, the text transcriber includes a first encoder and a first decoder; the first training module includes:

[0192] The first initialization submodule is used to initialize the first encoder and the first decoder;

[0193] The first selection submodule is used to select the current source character sequence of the first language and the current target character sequence of the corresponding second language from the first training set;

[0194] The first input submodule is used to input the current source character sequence and the first language identifier into the first encoder to obtain the text encoding features corresponding to the current source character sequence;

[0195] The second input submodule is used to input the text encoding features corresponding to the current source character sequence and the second language identifier into the first decoder to obtain the decoded character sequence corresponding to the text encoding features;

[0196] The first calculation submodule is used to calculate a first loss based on the decoded character sequence and the current target character sequence;

[0197] The first update submodule is used to update the parameters of the first encoder and the first decoder based on the first loss;

[0198] The first iteration submodule is used to enter the next round of iteration training. When the iteration stopping condition is met, the trained text translator is obtained.

[0199] Optionally, the second training module includes a second construction module for constructing a second training set; the second construction module includes:

[0200] The second collection submodule is used to collect several target videos, each target video corresponding to at least two languages;

[0201] The data extraction submodule is used to switch the language of each target video and extract audio data in different languages ​​and text data in different languages ​​corresponding to each audio data from the target video.

[0202] The second construction submodule is used to construct a second training set from the extracted audio data and the text data.

[0203] Optionally, the second training module includes:

[0204] The second initialization submodule is used to initialize the speech quantizer and speech dequantizer;

[0205] The third input submodule is used to select an audio data from the second training set as the first input audio, divide it into several speech segments according to the second preset length, and then input them into the speech quantizer in sequence to obtain the word vector sequence corresponding to the first input audio.

[0206] The fourth input submodule is used to input the word vector sequence corresponding to the first input audio into the speech dequantizer to obtain the decoded audio;

[0207] The second calculation submodule is used to calculate the second loss based on the decoded audio and the first input audio;

[0208] The second update submodule is used to update the parameters of the speech quantizer and the speech dequantizer according to the second loss;

[0209] The second iteration submodule is used to enter the next iteration. When the iteration stopping condition is met, the trained speech quantizer and speech dequantizer are obtained.

[0210] Optionally, the third training module includes:

[0211] The second selection submodule is used to select an audio data in the first language from the second training set as the second input audio, and to select the text data in the second language corresponding to the second input audio as the first label;

[0212] The fifth input submodule is used to divide the second input audio into several speech segments according to a second preset length and then input them sequentially into the speech quantizer to obtain the word vector sequence corresponding to the second input audio;

[0213] The sixth input submodule is used to input the word vector sequence corresponding to the second input audio, the first language identifier, and the second language identifier into the text translator to obtain the first translation result;

[0214] The third calculation submodule is used to calculate the third loss based on the first translation result and the first label;

[0215] The third update submodule is used to fix the parameters of the text transcriber and update the parameters of the speech quantizer according to the third loss.

[0216] The third iteration submodule is used to enter the next round of iteration training. When the iteration stopping condition is met, the final speech quantizer is obtained.

[0217] Optionally, the device further includes: a model fine-tuning module for fine-tuning the trained text translator; the model fine-tuning module includes:

[0218] The third selection submodule is used to select a source character sequence of the first language from the first training set as the input character sequence;

[0219] The fourth selection submodule is used to select the target character sequence of the second language corresponding to the input character sequence from the first training set as the second label;

[0220] The data processing submodule is used to insert blank characters into the input character sequence according to a preset rule to obtain the processed character sequence;

[0221] The seventh input submodule is used to input the processed character sequence, the first language identifier, and the second language identifier into the text translator to obtain the second translation result;

[0222] The fourth calculation submodule is used to calculate the fourth loss based on the second translation result and the second label;

[0223] The fourth update submodule is used to update the parameters of the text transcriber based on the fourth loss;

[0224] The fourth iteration submodule is used to enter the next round of iteration training. When the iteration stopping condition is met, the fine-tuned text transcriber is obtained.

[0225] The third training module is specifically used for:

[0226] Based on the second training set, the parameters of the text translator are fixed, and the fine-tuned text translator and the speech quantizer are jointly trained to obtain the final speech quantizer.

[0227] Optionally, the device further includes:

[0228] The voice collection module is used to collect a first preset number of preset voices, wherein the first preset number is greater than 1;

[0229] The target extraction module is used to divide each preset speech into several speech segments according to a second preset length and then input them sequentially into the trained speech quantizer to obtain the second word vector sequence corresponding to each preset speech.

[0230] The feature comparison module is specifically used for:

[0231] The first word vector sequence is compared with the second word vector sequence corresponding to each preset speech, and the comparison result of whether the first word vector sequence matches the second word vector sequence corresponding to each preset speech is obtained.

[0232] The target recognition module is specifically used for:

[0233] Determine whether the first word vector sequence matches a second word vector sequence exceeding a second preset number. If yes, determine that the speech to be recognized contains the target instruction word; otherwise, determine that the speech to be recognized does not contain the target instruction word.

[0234] Addressing the challenge of collecting speech data, especially for speech data in less commonly spoken languages, which is difficult not only to collect but also to annotate, this invention proposes a method to jointly train a speech quantizer with a text translator. This aligns the speech encoding features obtained by the quantizer with the corresponding text encoding features, resulting in the final speech quantizer. Since this speech quantizer is jointly trained with the text translator, which has text translation capabilities (e.g., translating text from a first language to a second language, where the second language can be the target language (e.g., a less commonly spoken language), the joint training of the speech quantizer and text translator allows the use of the text translator's cross-language text conversion function to provide cross-language alignment supervision signals for speech recognition. Therefore, this speech quantizer is used to extract a second word vector sequence from a preset speech in the target language. When performing speech recognition on speech in a target language, the speech in the target language can be input into a speech quantizer to obtain a first word vector sequence corresponding to the speech. By comparing the first word vector sequence with a second word vector sequence, it can be determined whether the speech contains the target instruction word, thereby realizing the keyword speech recognition function for minority languages.

[0235] Through the embodiments of this invention, it is only necessary to collect the pronunciation of the target language (a minor language) covering all texts of that language, without collecting a large amount of minor language speech data containing the target instruction words, to achieve speech recognition capability for that minor language. The translated text data, such as the first language text and its corresponding translated second language text, is relatively easy to collect. Therefore, the translated text data can be collected first to train a text translator capable of text-to-text translation. Based on the text translator, a speech quantizer is jointly trained. Thus, for a minor language, only a small amount (e.g., at least one) of pre-set speech data containing the target instruction words is needed to achieve speech recognition functionality for that minor language, without needing to collect a large amount of speech data to train the speech recognition model. This reduces the difficulty of data collection and improves model training efficiency.

[0236] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0237] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0238] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0239] Reference Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Figure 7 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the steps of the speech recognition method of the aforementioned embodiment.

[0240] This invention provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a program or processor of a terminal, enables the terminal to perform the steps of the speech recognition method described in the foregoing embodiments.

[0241] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0242] Furthermore, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program products or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0243] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0244] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0245] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0246] The above provides a detailed description of the speech recognition method, apparatus, and electronic device provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A speech recognition method, characterized in that, The method includes: The speech to be recognized in the target language is input into a speech quantizer to obtain a first word vector sequence corresponding to the speech; the speech quantizer is a neural network model jointly trained with a speech dequantizer and a text translator; the speech quantizer is used to convert speech into a word vector sequence; the speech dequantizer is used to convert the word vector sequence into speech; the text translator is used to convert text in the first language into text in the second language; The first word vector sequence is compared with the second word vector sequence to obtain the comparison result; the second word vector sequence is obtained by inputting the preset speech of the target language into the speech quantizer for conversion, and the preset speech contains the target instruction word; Based on the comparison results, it is determined whether the speech to be recognized contains the target instruction word.

2. The method according to claim 1, characterized in that, The method further includes: Construct a first training set, and train a text translator based on the first training set; the first training set includes several first language texts and the second language texts corresponding to each first language text. Construct a second training set, and jointly train a speech quantizer and a speech dequantizer based on the second training set; the second training set includes several audio data and text data of different languages ​​corresponding to each audio data. Based on the second training set, the parameters of the text translator are fixed, and the text translator and the speech quantizer are jointly trained to obtain the final speech quantizer.

3. The method according to claim 2, characterized in that, The construction of the first training set includes: Collect several first-language texts and the corresponding second-language texts for each first-language text; The collected first-language text is converted into binary encoding and segmented according to a first preset length to obtain the source character sequence; and the second-language text corresponding to the collected first-language text is converted into binary encoding and segmented according to a first preset length to obtain the target character sequence. The first training set is formed by combining each source character sequence with its corresponding target character sequence.

4. The method according to claim 3, characterized in that, The text translator includes a first encoder and a first decoder; the text translator trained based on the first training set includes: Initialize the first encoder and the first decoder; Select the current source character sequence of the first language and the corresponding current target character sequence of the second language from the first training set; The current source character sequence and the first language identifier are input into the first encoder to obtain the text encoding features corresponding to the current source character sequence; Input the text encoding features and the second language identifier corresponding to the current source character sequence into the first decoder to obtain the decoded character sequence corresponding to the text encoding features; Calculate the first loss based on the decoded character sequence and the current target character sequence; Update the parameters of the first encoder and the first decoder based on the first loss; The training continues in the next iteration. When the iteration stops, the trained text translator is obtained.

5. The method according to claim 2, characterized in that, The construction of the second training set includes: Collect several target videos, each corresponding to at least two languages; For each target video, the language of the target video is switched, and audio data in different languages ​​and text data in different languages ​​corresponding to each audio data are extracted from the target video. The extracted audio data and text data constitute the second training set.

6. The method according to claim 2, characterized in that, The process of jointly training the speech quantizer and speech inverse quantizer based on the second training set includes: Initialize the speech quantizer and speech dequantizer; Select an audio data from the second training set as the first input audio, divide it into several speech segments according to the second preset length, and then input them into the speech quantizer in sequence to obtain the word vector sequence corresponding to the first input audio. The word vector sequence corresponding to the first input audio is input into the speech dequantizer to obtain the decoded audio; Calculate the second loss based on the decoded audio and the first input audio; Update the parameters of the speech quantizer and the speech dequantizer according to the second loss; The next iteration begins, and when the iteration stops, the trained speech quantizer and speech dequantizer are obtained.

7. The method according to claim 2, characterized in that, The step of jointly training the text translator and the speech quantizer based on the second training set, fixing the parameters of the text translator, to obtain the final speech quantizer includes: Select an audio data point in the first language from the second training set as the second input audio, and select the corresponding text data in the second language from the second input audio as the first label; The second input audio is divided into several speech segments according to a second preset length and then input into the speech quantizer in sequence to obtain the word vector sequence corresponding to the second input audio. The word vector sequence corresponding to the second input audio, the first language identifier, and the second language identifier are input into the text transcriber to obtain the first transcribing result; Calculate the third loss based on the first translation result and the first label; The parameters of the text translator are fixed, and the parameters of the speech quantizer are updated according to the third loss. The training continues in the next iteration. When the iteration stops, the final speech quantizer is obtained.

8. The method according to claim 3, characterized in that, After training the text transcriber, the method further includes: Select source character sequences of the first language from the first training set as input character sequences; Select the target character sequence of the second language corresponding to the input character sequence from the first training set as the second label; The input character sequence is inserted with blank characters according to a preset rule to obtain the processed character sequence. The processed character sequence, the first language identifier, and the second language identifier are input into the text translator to obtain the second translation result; Calculate the fourth loss based on the second translation result and the second label; Update the parameters of the text transcriber based on the fourth loss; The next iteration of training begins, and when the iteration stopping condition is met, the fine-tuned text translator is obtained. The step of jointly training the text translator and the speech quantizer based on the second training set, fixing the parameters of the text translator, to obtain the final speech quantizer includes: Based on the second training set, the parameters of the text translator are fixed, and the fine-tuned text translator and the speech quantizer are jointly trained to obtain the final speech quantizer.

9. The method according to claim 1, characterized in that, The method further includes: Collect a first preset number of pre-set voice recordings, where the first preset number is greater than 1; Each preset speech is divided into several speech segments according to the second preset length and then input into the trained speech quantizer in sequence to obtain the second word vector sequence corresponding to each preset speech. The step of comparing the first word vector sequence with the second word vector sequence to obtain the comparison result includes: The first word vector sequence is compared with the second word vector sequence corresponding to each preset speech, and the comparison result of whether the first word vector sequence matches the second word vector sequence corresponding to each preset speech is obtained. The step of determining whether the target instruction word is contained in the speech to be recognized based on the comparison result includes: Determine whether the first word vector sequence matches a second word vector sequence exceeding a second preset number. If yes, determine that the speech to be recognized contains the target instruction word; otherwise, determine that the speech to be recognized does not contain the target instruction word.

10. A voice recognition device, characterized in that, The device includes: The speech conversion module is used to input the speech to be recognized in the target language into a speech quantizer to obtain a first word vector sequence corresponding to the speech to be recognized; the speech quantizer is a neural network model jointly trained with a speech dequantizer and a text translator; the speech quantizer is used to convert speech into a word vector sequence; the speech dequantizer is used to convert the word vector sequence into speech; the text translator is used to convert text in the first language into text in the second language. The feature comparison module is used to compare the first word vector sequence with the second word vector sequence to obtain the comparison result; the second word vector sequence is obtained by inputting a preset speech of the target language into the speech quantizer and converting it, and the preset speech contains target instruction words; The target recognition module is used to determine whether the speech to be recognized contains the target instruction word based on the comparison result.

11. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the speech recognition method as described in any one of claims 1 to 9.

Images