A speech recognition method, apparatus, device, and storage medium

Abstract

This application provides a speech recognition method, apparatus, device, and storage medium. The specific implementation scheme is as follows: Acquired speech data to be recognized is input into a target speech recognition model to obtain the recognition result corresponding to the speech data to be recognized. The target speech recognition model is obtained at least through a first training process on an initial speech recognition model. This first training includes training the initial speech recognition model to predict occluded text using text training data containing occluded text. The first training uses the consistency between the initial speech recognition model's first prediction result for occluded text in the text training data and a pre-trained language model's second prediction result for occluded text in the text training data as the target. The language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict occluded text in the text training data. According to the technical solution of this application, the recognition effect of speech data can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus, device, and storage medium. Background Technology

[0002] Traditional speech recognition mainly consists of an acoustic model, a language model, and a decoder. The acoustic model primarily models the speech data, while the language model, trained on text data, represents the probability of word order and is an indispensable and important component of a speech recognition system.

[0003] Current speech recognition systems are limited by the domain or related information of the speech data, resulting in poor recognition performance. Summary of the Invention

[0004] To address the aforementioned issues, this application proposes a speech recognition method, apparatus, device, and storage medium that can effectively improve the recognition performance of speech data.

[0005] According to a first aspect of the embodiments of this application, a speech recognition method is provided, comprising:

[0006] Acquire the speech data to be recognized;

[0007] The speech data to be recognized is input into the target speech recognition model to obtain the recognition result corresponding to the speech data to be recognized;

[0008] The target speech recognition model is obtained at least by performing a first training on an initial speech recognition model. The first training includes training the initial speech recognition model to predict occluded text using text training data containing occluded text. The first training takes the consistency between the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the pre-trained language model for the occluded text in the text training data as the target. The preset language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict the occluded text in the text training data.

[0009] According to a second aspect of the embodiments of this application, a voice recognition device is provided, comprising:

[0010] The first acquisition module is used to acquire the speech data to be recognized;

[0011] A recognition module is used to input the speech data to be recognized into a target speech recognition model to obtain a recognition result corresponding to the speech data to be recognized; wherein, the target speech recognition model is obtained at least through a first training of an initial speech recognition model, the first training including training the initial speech recognition model to predict occluded text using text training data containing occluded text; the first training takes the consistency between the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the pre-trained language model for the occluded text in the text training data as the target; the language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict the occluded text in the text training data.

[0012] A third aspect of this application provides an electronic device, comprising:

[0013] Memory and processor;

[0014] The memory is connected to the processor and is used to store programs;

[0015] The processor implements the aforementioned speech recognition method by running the program in the memory.

[0016] A fourth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the aforementioned speech recognition method.

[0017] One embodiment of the above application has the following advantages or beneficial effects:

[0018] The target speech recognition model is obtained by first training the initial speech recognition model and using the consistency between the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the pre-trained language model for the occluded text in the text training data. In this way, the target speech recognition model obtained by assisting the initial speech recognition model with the language model based on the knowledge base can adapt to speech recognition in more fields, improve the recognition rate of rare entities or text, and thus improve the recognition effect. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A schematic flowchart of a speech recognition method provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram of the process for performing a first training of an initial speech recognition model according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the process of performing first and second training on an initial speech recognition model according to an embodiment of this application.

[0023] Figure 4 This is a schematic diagram illustrating the process of obtaining occluded text in text training data using a language model in a speech recognition method provided in an embodiment of this application.

[0024] Figure 5 A schematic diagram of the KG-BERT model and the ED model provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram illustrating the conversion of a sentence tree into a sequence structure according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a speech recognition device provided in one embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of this application are applicable to various speech recognition scenarios, such as human-vehicle interaction, human-computer interaction, VR scenarios, online meetings, and online education. Using the technical solutions of this application can improve the recognition effect of speech data.

[0029] The technical solutions of this application can be applied, by way of example, to hardware devices such as processors, electronic devices, and servers (including cloud servers), or packaged as software programs and run. When the hardware device executes the processing procedure of the technical solutions of this application, or when the aforementioned software program is run, a target speech recognition model can be obtained by assisting the initial speech recognition model with training based on a knowledge base language model, so as to adapt to the purpose of speech recognition in more fields. This application only provides an exemplary description of the specific processing procedure of the technical solutions of this application, and does not limit the specific implementation form of the technical solutions of this application. Any technical implementation form that can execute the processing procedure of the technical solutions of this application can be adopted by this application.

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Exemplary methods

[0032] Figure 1 This is a flowchart of a speech recognition method according to an embodiment of this application. In an exemplary embodiment, the speech recognition method specifically includes:

[0033] S110. Obtain the speech data to be recognized;

[0034] S120. Input the speech data to be recognized into the target speech recognition model to obtain the recognition result corresponding to the speech data to be recognized;

[0035] The target speech recognition model is obtained at least by performing a first training on an initial speech recognition model. The first training includes training the initial speech recognition model to predict occluded text using text training data containing occluded text. The first training takes the consistency between the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the pre-trained language model for the occluded text in the text training data as the target. The language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict the occluded text in the text training data.

[0036] In step S110, for example, the voice data to be recognized can be voice data collected directly or indirectly through a terminal device. The terminal device can be any device capable of acquiring and transmitting sound, such as a mobile phone, tablet, or recorder. Optionally, the voice data can be data from any field; the voice data can be the voice of a specific person, or it can be the voice of any person, without limitation.

[0037] In step S120, exemplarily, the language model is used to predict occluded words in a sentence based on contextual information and its extended information. Optionally, the preset knowledge base may include knowledge graphs from multiple domains. A knowledge graph describes various entities and concepts existing in the real world, and the relationships between them. The basic building blocks of a knowledge graph are "entity-relationship-entity" triples, and entities and their associated attribute-value pairs. Entities are interconnected through relations, forming a network-like knowledge structure. In this embodiment, the language model may be a KG-BERT model, etc.

[0038] For example, the initial speech recognition model represents a model capable of recognizing speech as text. The text training data can be text data from any open-source dataset, text data collected from web pages, or manually input text data; there is no limitation here. Optionally, masked text is used to represent the masked words. Therefore, the text training data is text data containing masked words. Optionally, the consistency between the initial speech recognition model's first prediction result for the masked text in the text training data and the language model's second prediction result for the masked text in the text training data can mean that the first prediction result and the second prediction result are exactly the same, or it can mean that the error between the first prediction result and the second prediction result is within a preset range. The error can be calculated using cosine similarity or other methods; there is no limitation here. The preset range is set according to actual needs and is not limited here.

[0039] Specifically, during model training, after acquiring the text training data, the language model uses a knowledge graph index to obtain the corresponding extended text data. Then, based on the extended text data, it predicts the occluded text in the text training data, outputting a second prediction result. The initial speech recognition model is then used to predict the occluded text in the text training data, outputting a first prediction result. The initial speech recognition model is trained with the goal of ensuring consistency between the first and second prediction results. This yields a target speech recognition model with accurate text recognition. In this way, the speech data to be recognized can be recognized more accurately based on the target speech recognition model.

[0040] In the technical solution of this application, the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the language model for the occluded text in the text training data are taken as the target. The initial speech recognition model is trained first to obtain the target speech recognition model. In this way, the target speech recognition model obtained by the language model based on the knowledge base to assist in training the initial speech recognition model can adapt to speech recognition in more fields, improve the recognition rate of rare entities or text, and thus improve the recognition effect.

[0041] In one embodiment, the training process of the target speech recognition model further includes a second training, which includes training the initial speech recognition model on speech recognition based on speech training data and its corresponding text data.

[0042] For example, the speech training data can be data from any open-source speech dataset, or speech data collected by any user-permitted terminal device, such as in-vehicle devices or mobile phones. The text data is the data corresponding to the speech training data. Specifically, it can be text data obtained by processing the speech training data using existing speech recognition technology; or it can be a pair of speech training data and text data directly obtained from an open-source speech dataset.

[0043] For example, the initial speech recognition model can be a neural network model. Optionally, the initial speech recognition model is trained a second time based on the speech training data and its corresponding text data to generate a trained initial speech recognition model. Then, the first prediction result of the trained initial speech recognition model for occluded text in the text training data and the second prediction result of the language model for occluded text in the text training data are taken as the target, and the trained initial speech recognition model is trained a first time to obtain a target speech recognition model. In this way, the target speech recognition model can recognize speech data.

[0044] In one embodiment, the method for obtaining the speech training data and its corresponding text data includes:

[0045] By filtering the general raw data using the text data corresponding to the preset target domain and historical speech training dataset, the target speech training data and text data are obtained.

[0046] For example, the preset target domain can be a pre-specified domain or any domain. The target domain can be the medical field, the biological field, the chemical field, or the artificial intelligence field. The target domain can be a single specified domain or multiple specified domains.

[0047] Optionally, the filtering rules may include filtering the general raw data based on perplexity (PPL) to select data that is more relevant to the target domain, thereby ensuring the recognition rate of the target domain.

[0048] Optionally, the selection rules may also include: filtering the general raw data based on the correlation between each word in the text data corresponding to the speech training dataset and each word in the general raw data. Specifically, the original training data is initially filtered based on word correlation, and then words with low frequency in the text data corresponding to the speech training dataset but high frequency in the general raw data are selected. The sentences corresponding to the selected words are then used as the text training data. This can improve the recognition rate of rare words in actual business applications.

[0049] Furthermore, the data selected by these two methods, along with the corresponding text data for the audio, can be mixed in a certain proportion to serve as the target speech training data and text data. This can further improve the accuracy of model training.

[0050] In one implementation, such as Figure 2 As shown, the initial training of the speech recognition model includes:

[0051] S210. Input the text training data into the initial speech recognition model to obtain a first prediction result for the occluded text in the text training data; and input the text training data into the pre-trained language model to obtain a second prediction result for the occluded text in the text training data.

[0052] S220. Determine the prediction loss function based on the first prediction result and the second prediction result;

[0053] S230. With the goal of reducing the prediction loss function, the parameters of the initial speech recognition model are corrected.

[0054] For example, the prediction loss function is used to guide the training of the initial speech recognition model. For instance, the prediction loss function can be obtained by calculating the first and second prediction results using relative entropy (KL divergence). The initial speech recognition model is adjusted (i.e., parameter correction) by minimizing the KL divergence of each word i between the initial speech recognition model and the language model, as shown in the following formula:

[0055]

[0056] in, This represents the probability that the i-th target (i.e., the occluded word) is predicted as word v by the ED model (i.e., the initial speech recognition model). This represents the probability that the i-th target (i.e., the masked word) is predicted as word v by the KG-BERT model (i.e., the language model). V represents the size of the pre-defined dictionary, which is a collection of words from various domains. The pre-defined dictionary can be a specified dictionary or any dictionary, without any restrictions here.

[0057]

[0058] Among them, y \i =(y1,...,y i-1 [MASK],y i+1 ,...,y l ) represents the i-th token replaced by the mask (i.e., the masked word), and T represents a temperature parameter. It should be noted that the larger the T, the smoother the text. j z represents the output before the softmax layer (i.e., the representation function of words in the dictionary replaced by the i-th token of the mask), v This represents the output before the softmax layer (i.e., the representation function that replaces the i-th token of the MASK with the word v).

[0059] Furthermore, such as Figure 3 As shown, the initial speech recognition model undergoes first and second training, including:

[0060] S310. A prediction loss function is obtained by performing a first training on the initial speech recognition model, and a recognition loss function is obtained by performing a second training on the initial speech recognition model.

[0061] S320. Based on the prediction loss function and the identification loss function, generate the target loss function;

[0062] S330. With the goal of reducing the target loss function, the parameters of the initial speech recognition model are corrected.

[0063] For example, the target loss function is used to adjust the initial speech recognition model to obtain the target speech recognition model. The target loss function represents the degree of dependence on the prediction loss function. That is, since the prediction loss function is obtained from a knowledge-based language model, the degree of dependence of the target speech recognition model on the knowledge-based language model is reflected by setting weights for the prediction loss function and the recognition loss function. The recognition loss function is the loss function used for the second training of the initial speech recognition model. It can be understood that the loss function can be set according to the model used. For example, if the initial speech recognition model uses the BERT model, it can use the negative log-likelihood function as the recognition loss function.

[0064] Specifically, the target loss function is obtained by weighting and summing the prediction loss function and the recognition loss function. This allows for further integration of the recognition loss function from the initial speech recognition model and the prediction loss function from the knowledge-based language model. By using the language model to guide the training of the initial speech recognition model, knowledge information is incorporated into the language model, thereby improving the recognition rate of rare entities.

[0065] In this embodiment, the prediction loss function can be obtained through the above method. because The entire distillation process is fixed. It can be simplified to the following formula:

[0066]

[0067]

[0068] Among them, L ED This represents the recognition loss function. This represents the weighting coefficient, which is neither less than 0 nor greater than 1. The specific value of L depends on the degree of dependence on the language model. That is, if the dependence on the language model is high, then L... KD The coefficient is relatively large, therefore increase it. The value.

[0069] In one implementation, such as Figure 4 As shown, the process by which the pre-trained language model obtains a second prediction result for occluded text in the text training data includes:

[0070] S410. Use the text training data to expand the sentences and obtain expanded text data;

[0071] S420. Using the positional encoding of each word in the extended text data and the visibility of each word in the extended text data, determine the representation vector corresponding to the extended text data;

[0072] S430. Using a preset dictionary and the representation vector corresponding to the extended text data, predict the second prediction result corresponding to the occluded text.

[0073] For example, sentence expansion can be performed on the entire sentence of the text training data. Specifically, the entire sentence is used as an index to search for expanded content in web pages or existing technologies. The expanded content is then filtered, and expanded text data is generated based on the filtered expanded content and the text training data. Optionally, sentence expansion can also be performed on each word in the text training data. Each word in the text training data is used as an index to search for expanded content in web pages or existing technologies. The expanded content is then filtered, and expanded text data is generated based on the filtered expanded content and the text training data. The order information of each word in the expanded text data is determined through soft-position encoding. However, soft-position alone is insufficient because it can confuse words on different branches. Therefore, the visibility of each word in the expanded text data is introduced. The visibility is used to restrict the connections between words, i.e., to determine whether words are on the same branch. This restricts the connections between words, and the expanded text data is represented as a vector based on the visibility of each word. Then, the probability of predicting the i-th target (i.e., the occluded word) as word v is calculated based on the representation vector corresponding to the expanded text data according to a pre-defined dictionary. The pre-defined dictionary is a collection of vocabulary from various fields. This allows for the expansion of knowledge from the text training data, integrating external knowledge into language expression.

[0074] In one implementation, the step of expanding sentences using the text training data to obtain expanded text data includes:

[0075] The vocabulary in the text training data is expanded based on a preset knowledge graph to obtain vocabulary-expanded text data;

[0076] The extended text data is generated based on the vocabulary extended text data and the text training data.

[0077] For example, the pre-defined knowledge graph can contain knowledge from various domains. After acquiring text training data, it is segmented to identify individual words in the training data. Each word is then filtered to select those that can be expanded within the knowledge graph. These selected words are then used as indexes to search for corresponding expanded text within the knowledge graph. Alternatively, the words in the training data can be used as indexes to directly search for corresponding expanded text within the knowledge graph. For example, the expanded text for "Messi" in the knowledge graph is "football player". The expanded text and the training data are then concatenated to generate expanded text data. For example, the expanded text data can be represented as a sentence tree. In this way, the knowledge graph can be integrated into language expression, utilizing the external knowledge of the knowledge graph for language generation.

[0078] In one implementation, determining the representation vector corresponding to the extended text data by utilizing the positional encoding of each word in the extended text data and the visibility of each word in the extended text data includes:

[0079] The extended text data is converted into a generated text sequence using the location encoding.

[0080] Based on the visibility of each word in the text sequence, the representation vector corresponding to the text sequence is determined.

[0081] For example, the extended text data can be subjected to both hard-position encoding and soft-position encoding. Tokens in the sentence tree are flattened into a series of token embeddings using their hard-position encoding, and the soft-position encoding is used together with the token embeddings as positional embeddings. In the segment embedding, all tokens in the first sentence are labeled "A". This generates a text sequence. Then, the visibility of each word in the text sequence is introduced, and the text sequence is encoded using Mask-Transformer encoding to generate corresponding representation vectors. Here, Mask-Transformer is a stack of mask-self-attention layers.

[0082] Preferably, determining the representation vector corresponding to the text sequence based on the visibility of each word in the text sequence includes:

[0083] Based on the visibility of each word in the text sequence, determine the visibility matrix of each word;

[0084] Attention is calculated on the text sequence using the visibility matrix to obtain the corresponding representation vector.

[0085] For example, the visibility matrix is ​​used to represent the visibility between words. For instance, 0 indicates that two words are on the same branch, i.e., they are visible to each other; -∞ indicates that two words are on different branches, i.e., they are not visible to each other. Attention computation is an encoding method of the self-attention transformer, i.e., through the output of three matrices: Q (Query), K (Key), and V (Value).

[0086] In this embodiment, the KG-BERT language model is used as an example for illustration. Figure 5As shown, the KG-BERT model includes: a Knowledge layer, an Embedding layer, a Seeing layer, and a Mask-Transformer Encoder. The Knowledge layer expands the input text into a sentence tree using a Knowledge Graph; the Embedding layer converts the sentence tree into embedded representations; the Seeing layer uses a visible matrix to constrain the relationships between words; and the Mask-Transformer Encoder incorporates structural information from the graph or tree structure into the model. The output of the KG-BERT model... and the output of the ED model Generate loss function L KD To pass through the loss function L KD The final loss function is generated by combining the loss function of the ED model itself. The ED model uses the decoder from seq2seq.

[0087] like Figure 6 As shown, since the sentence tree is a tree structure while BERT takes a sequence as input, it is necessary to convert the sentence tree into a sequence structure by flattening the sentence tree into a sequence. Figure 6 In this diagram, the numbers above represent soft position indices, and the numbers below represent hard position indices. The sentence tree is flattened into token embeddings according to the hard position indices. The soft position indices and token embeddings together form the soft-position embedding. Since only one sentence was input, all segment embeddings are A. In addition to introducing soft position indices, a visibility matrix is ​​also introduced, which can be represented as...

[0088]

[0089] Among them, b i Θb j b i and b j Belonging to the same branch, i and j represent hard position indices. When b i and b j When characters belong to the same branch, the value is 0; otherwise, the value is negative infinity. For example, 'football' and 'is' do not belong to the same branch, so the value is negative infinity; 'football' and 'Messi' belong to the same branch, so the value is 0. After obtaining the visibility matrix B, the attention matrix needs to be masked. The specific calculation formula is as follows, where W...q W k and W v It is a trainable weight matrix, h i It is the hidden state of the i-th mask-self-attention module, d k B is a scaling factor, and B is a visible matrix. If b k For b j If it is invisible, then B jk Attention score A mask value of 0 indicates that b k For state b j It makes no contribution:

[0090] Q i+1 ,K i+1 V i+1 =h i W q ,h i W k ,h i W v ,

[0091]

[0092] h i+1 =S i+1 V i+1 ,

[0093] For the output of the KG-BERT model, the probability of predicting the i-th masked word as word v is:

[0094]

[0095] Among them, y \i =(y1,...,y i-1 [MASK],y i+1 ,...,y l ) represents the i-th token replaced by the mask (i.e., the masked word), and T represents a temperature parameter. It should be noted that the larger the T, the smoother the text. j z represents the output before the softmax layer (i.e., the representation function of words in the dictionary replaced by the i-th token of the mask), vThis represents the output before the softmax layer (i.e., the representation function of the word v after the i-th token of the mask is replaced). Then, the output of the KG-BERT model guides the training of the ED model. Therefore, fusing knowledge graphs with BERT to guide the training of the ED model, integrating knowledge information into the language model, and utilizing the bidirectional encoding structure of BERT to see future information greatly helps improve the performance of speech recognition systems.

[0096] Exemplary device

[0097] Correspondingly, correspondingly Figure 7 This is a schematic diagram of a speech recognition device according to an embodiment of this application. In an exemplary embodiment, this application also proposes a speech recognition device, which includes:

[0098] The first acquisition module 710 is used to acquire the speech data to be recognized;

[0099] The recognition module 720 is used to input the speech data to be recognized into a target speech recognition model to obtain a recognition result corresponding to the speech data to be recognized; wherein, the target speech recognition model is obtained at least through a first training of an initial speech recognition model, the first training including training the initial speech recognition model to predict occluded text using text training data containing occluded text; the first training aims to make the first prediction result of the initial speech recognition model for the occluded text in the text training data consistent with the second prediction result of the pre-trained language model for the occluded text in the text training data; the preset language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict the occluded text in the text training data.

[0100] In one embodiment, the recognition module 720 includes a second training module, used to train the initial speech recognition model for speech recognition based on speech training data and its corresponding text data.

[0101] In one implementation, the second training module is further configured to:

[0102] By filtering the general raw data using the text data corresponding to the preset target domain and historical speech training dataset, the target speech training data and text data are obtained.

[0103] In one implementation, the initial speech recognition model undergoes first training and second training, including:

[0104] By performing a first training on the initial speech recognition model, a prediction loss function is obtained; and by performing a second training on the initial speech recognition model, a recognition loss function is obtained.

[0105] Based on the prediction loss function and the identification loss function, a target loss function is generated;

[0106] The parameters of the initial speech recognition model are adjusted with the goal of reducing the target loss function.

[0107] In one implementation, the initial speech recognition model is first trained, including:

[0108] Input the text training data into an initial speech recognition model to obtain a first prediction result for the occluded text in the text training data; and input the text training data into a pre-trained language model to obtain a second prediction result for the occluded text in the text training data.

[0109] Based on the first prediction result and the second prediction result, determine the prediction loss function;

[0110] The parameters of the initial speech recognition model are adjusted with the goal of reducing the prediction loss function.

[0111] In one implementation, the pre-trained language model obtains a second prediction result for occluded text in the text training data through a processing procedure, including:

[0112] The text training data is used to expand sentences, resulting in expanded text data.

[0113] The representation vector corresponding to the extended text data is determined by using the positional encoding of each word in the extended text data and the visibility of each word in the extended text data;

[0114] Using a preset dictionary and the representation vector corresponding to the extended text data, a second prediction result is predicted for the occluded text.

[0115] In one implementation, the step of expanding sentences using the text training data to obtain expanded text data includes:

[0116] The vocabulary in the text training data is expanded based on a preset knowledge graph to obtain vocabulary-expanded text data;

[0117] The extended text data is generated based on the vocabulary extended text data and the text training data.

[0118] In one implementation, determining the representation vector corresponding to the extended text data by utilizing the positional encoding of each word in the extended text data and the visibility of each word in the extended text data includes:

[0119] The extended text data is converted into a generated text sequence using the location encoding.

[0120] Based on the visibility of each word in the text sequence, the representation vector corresponding to the text sequence is determined.

[0121] In one implementation, determining the representation vector corresponding to the text sequence based on the visibility of each word in the text sequence includes:

[0122] Based on the visibility of each word in the text sequence, determine the visibility matrix of each word;

[0123] Attention is calculated on the text sequence using the visibility matrix to obtain the corresponding representation vector.

[0124] The speech recognition device provided in this embodiment belongs to the same concept as the speech recognition method provided in the above embodiments of this application. It can execute the speech recognition method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the speech recognition method. Technical details not described in detail in this embodiment can be found in the specific processing content of the speech recognition method provided in the above embodiments of this application, and will not be repeated here.

[0125] Exemplary electronic devices

[0126] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 8 As shown, the device includes:

[0127] Memory 800 and processor 810;

[0128] The memory 800 is connected to the processor 810 and is used to store programs;

[0129] The processor 810 is used to implement the speech recognition method disclosed in any of the above embodiments by running the program stored in the memory 800.

[0130] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 820, an input device 830, and an output device 840.

[0131] The processor 810, memory 800, communication interface 820, input device 830, and output device 840 are interconnected via a bus. Among them:

[0132] A bus can include a pathway for transmitting information between various components of a computer system.

[0133] The processor 810 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0134] The processor 810 may include a main processor, as well as a baseband chip, modem, etc.

[0135] The memory 800 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 800 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0136] Input device 830 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0137] Output device 840 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0138] The communication interface 820 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0139] The processor 810 executes the program stored in the memory 800 and calls other devices, which can be used to implement the various steps of any of the speech recognition methods provided in the above embodiments of this application.

[0140] Exemplary computer program products and storage media

[0141] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the speech recognition methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0142] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0143] Furthermore, embodiments of this application may also be storage media storing a computer program thereon, the computer program being executed by a processor of the steps in the speech recognition methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0144] The specific working content of the aforementioned electronic device, as well as the specific working content of the aforementioned computer program product and the computer program on the storage medium being run by the processor, can all be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0145] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0146] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0147] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0148] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0149] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0150] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0151] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0152] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0154] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech data to be recognized; The speech data to be recognized is input into the target speech recognition model to obtain the recognition result corresponding to the speech data to be recognized; The target speech recognition model is obtained at least by performing a first training on an initial speech recognition model. The first training includes training the initial speech recognition model to predict occluded text using text training data containing occluded text. The first training takes the consistency between the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the pre-trained language model for the occluded text in the text training data as the target. The language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict the occluded text in the text training data.

2. The method according to claim 1, characterized in that, The training process of the target speech recognition model also includes a second training, which includes training the initial speech recognition model on speech recognition based on speech training data and its corresponding text data.

3. The method according to claim 2, characterized in that, The method for obtaining the speech training data and its corresponding text data includes: By filtering the general raw data using the text data corresponding to the preset target domain and historical speech training dataset, the target speech training data and text data are obtained.

4. The method according to claim 2, characterized in that, The initial speech recognition model undergoes first and second training, including: By performing a first training on the initial speech recognition model, a prediction loss function is obtained; and by performing a second training on the initial speech recognition model, a recognition loss function is obtained. Based on the prediction loss function and the identification loss function, a target loss function is generated; The parameters of the initial speech recognition model are adjusted with the goal of reducing the target loss function.

5. The method according to claim 1 or 4, characterized in that, The initial training of the speech recognition model includes: The text training data is input into an initial speech recognition model to obtain a first prediction result for the occluded text in the text training data; and the text training data is input into a pre-trained language model to obtain a second prediction result for the occluded text in the text training data. Based on the first prediction result and the second prediction result, determine the prediction loss function; The parameters of the initial speech recognition model are adjusted with the goal of reducing the prediction loss function.

6. The method according to claim 5, characterized in that, The process of obtaining a second prediction result for occluded text in the text training data from the pre-trained language model includes: The text training data is used to expand sentences, resulting in expanded text data. The representation vector corresponding to the extended text data is determined by using the positional encoding of each word in the extended text data and the visibility of each word in the extended text data; Using a preset dictionary and the representation vector corresponding to the extended text data, a second prediction result is predicted for the occluded text.

7. The method according to claim 6, characterized in that, The process of expanding sentences using the text training data to obtain expanded text data includes: The vocabulary in the text training data is expanded based on a preset knowledge graph to obtain vocabulary-expanded text data; The extended text data is generated based on the vocabulary extended text data and the text training data.

8. The method according to claim 6, characterized in that, The step of determining the representation vector corresponding to the extended text data by utilizing the positional encoding of each word in the extended text data and the visibility of each word in the extended text data includes: The extended text data is converted into a generated text sequence using the location encoding. Based on the visibility of each word in the text sequence, the representation vector corresponding to the text sequence is determined.

9. The method according to claim 8, characterized in that, Determining the representation vector corresponding to the text sequence based on the visibility of each word in the text sequence includes: Based on the visibility of each word in the text sequence, determine the visibility matrix of each word; Attention is calculated on the text sequence using the visibility matrix to obtain the corresponding representation vector.

10. A voice recognition device, characterized in that, include: The first acquisition module is used to acquire the speech data to be recognized; A recognition module is used to input the speech data to be recognized into a target speech recognition model to obtain a recognition result corresponding to the speech data to be recognized; wherein, the target speech recognition model is obtained at least through a first training of an initial speech recognition model, the first training including training the initial speech recognition model to predict occluded text using text training data containing occluded text; the first training takes the consistency between the first prediction result of the initial speech recognition model for the occluded text in the text training data and the second prediction result of the pre-trained language model for the occluded text in the text training data as the target; the language model expands the text training data based on a preset knowledge base and uses the expanded text data to predict the occluded text in the text training data.

11. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor implements any one of the speech recognition methods as described in claims 1 to 9 by running the program in the memory.

12. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements any one of the speech recognition methods as described in claims 1 to 9.

Images