Speech recognition method, device and storage medium

Abstract

A speech recognition method, device, electronic device, storage medium and program product, the speech recognition method includes: performing coding on speech data to be recognized to obtain a first feature vector sequence (101); performing coding on each hot word in a predefined hot word thesaurus to obtain a second feature vector sequence (102); performing coding on an audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence (103); performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence (104); and performing a decoding operation according to the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result (105). The method improves hot word recognition accuracy.

Description

[Technical field]

[0001] This application claims priority to an earlier application, application number 202011641751.3, filed on December 31, 2020, entitled "Speech recognition method, device and storage medium," the entire contents of which are hereby incorporated by reference into this application.

[0002] The present application relates to the field of voice recognition technology, and in particular to a voice recognition method, device and storage medium. [Background technology]

[0003] In the field of speech recognition, the effectiveness of traditional hot word scoring incentive methods is limited due to the low scores of low frequency words output from end-to-end models. CLAS (Contextual Listen, Attend and Spell), proposed by Google, is a method to incentivize hot words at the model level. Although good results were obtained by providing incentives for hot words, the method was too simple, and sentences that did not contain hot words were often mistaken for hot words, which reduced the overall recognition rate and made it difficult to directly use the method in an actual system. Therefore, there is an urgent need to improve the accuracy of hot word recognition. Summary of the Invention [Means for solving the problem]

[0004] The embodiments of the present application provide a speech recognition method, device, and storage medium that improve the recognition accuracy of hot words.

[0005] In a first aspect, an embodiment of the present application provides a speech recognition method, the method comprising: encoding the speech data to be recognized to obtain a first feature vector sequence; performing encoding for each hot word in a predefined hot word thesaurus to obtain a second sequence of feature vectors; performing encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence; performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; and performing a decoding operation based on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result.

[0006] In a second aspect, an embodiment of the present application provides a speech recognition device, the speech recognition device comprising: an audio encoder module for encoding speech data to be recognized to obtain a first feature vector sequence; a hot word text encoder module for performing encoding for each hot word in the predefined hot word thesaurus to obtain a second feature vector sequence; a hot word audio encoder module for performing encoding on an audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence; a frame hierarchical attention module that performs a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence and a decoder module for performing a decoding operation based on the sequence of feature vectors to obtain a recognition result.

[0007] In a third aspect, an embodiment of the present application provides an electronic device including a processor, a memory, a communication port, and one or more programs stored in the memory and configured to cause the processor to execute instructions of any of the method steps in the first aspect of the embodiment of the present application.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program for electronic data exchange, the computer program causing a computer to perform some or all of the steps described in the first aspect of the embodiment of the present application.

[0009] In a fifth aspect, embodiments of the present application further provide a computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of the present application, The computer program product may be a software installation package. Effect of the Invention

[0010] The implementation of the present application has the following beneficial effects: According to the above, the speech recognition method, device and related products described in the embodiments of the present application perform coding on the speech data to be recognized, obtain a first feature vector sequence, perform coding on each hot word in a pre-defined hot word thesaurus, obtain a second feature vector sequence, perform coding on the audio segment of each hot word in the pre-defined hot word thesaurus, obtain a third feature vector sequence, perform a first attention operation on the first feature vector sequence and the third feature vector sequence, obtain a fourth feature vector sequence, perform a decoding operation according to the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result, and take not only the hot word text information but also the corresponding audio segment as input, and further perform an attention operation on the speech data to be recognized and the audio segment of the hot word text information as input, which obviously improves the hot word incentive effect, and performs the decoding operation of the three, which improves the hot word recognition effect and enhances the hot word recognition accuracy. [Brief description of the drawings]

[0011] In order to more clearly describe the technical solutions of the embodiments of the present application or the prior art, the drawings necessary for describing the embodiments or the prior art will be briefly described below. Obviously, the drawings described below are only a part of the embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative efforts. [Figure 1A] FIG. 1A is a diagram showing the structure of a speech recognition model according to an embodiment of the present application. [Figure 1B] FIG. 1B is a flow chart illustrating a speech recognition method according to an embodiment of the present application. [Figure 1C] FIG. 1C is a diagram illustrating hot word encoding according to an embodiment of the present application. [Figure 1D] FIG. 1D is a diagram illustrating feature combinations according to an embodiment of the present application. [Diagram 2] FIG. 2 is a flow chart illustrating another speech recognition method according to an embodiment of the present application. [Diagram 3] FIG. 3 is a diagram showing the structure of an electronic device according to an embodiment of the present application. [Figure 4A] FIG. 4A is a block diagram showing a functional unit configuration of a voice recognition device according to an embodiment of the present application. [Figure 4B] FIG. 4B is a block diagram showing a functional unit configuration of another voice recognition device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In order to make those skilled in the art better understand the solution of the present application, the technical solutions of the embodiments of the present application are described below in a clear and detailed manner in conjunction with the drawings of the embodiments of the present application. It is clear that the described embodiments are only some of the embodiments of the present application, and are not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts are within the protection scope of the present application.

[0013] The terms "first", "second", and the like in the specification, claims, and drawings of this application are intended to distinguish different objects, and not to describe a particular order. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusively. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, and the examples further include steps or units that are not listed, or other examples further include other steps or units that are specific to these processes, methods, products, or apparatus.

[0014] In this specification, the term "embodiment" means that a particular feature, structure, or characteristic described with reference to the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase anywhere in this specification does not necessarily refer to all the same embodiment, nor does it refer to an independent or alternative embodiment that is mutually exclusive with other embodiments. It should be understood by those skilled in the art, either explicitly or implicitly, that the embodiments described herein may be combined with other embodiments.

[0015] The electronic devices according to the embodiments of the present application include various handheld devices with voice recognition functions, voice recorders, smart robots, smart readers, smart translators, smart headphones, smart dictionaries, smart reading machines, in-vehicle devices, wearable devices, computing devices or other processing devices connected to wireless modems, and various forms of user equipment (UE), mobile stations (MS), terminal devices, etc., and the electronic devices may also be servers or smart The device may be a home device.

[0016] In an embodiment of the present application, the smart home device may be at least one of a refrigerator, a washing machine, a rice cooker, a smart curtain, a smart light, a smart bed, a smart trash can, a microwave, an oven, a steamer, an air conditioner, a range hood, a server, a smart door, a smart window, a wardrobe, a smart speaker, a smart home, a smart chair, a smart clothesline, a smart shower, a water server, a water purifier, an air purifier, a chime, a surveillance system, a smart garage, a television, a projector, a smart dining table, a smart sofa, a massage chair, a running machine, and the like.

[0017] Examples of the present application will be described in detail below.

[0018] As shown in FIG. 1A, FIG. 1A illustrates a speech recognition model according to an embodiment of the present application, which includes an audio encoder module, a hot word text encoder module, a hot word audio encoder module, a frame hierarchical attention module, a word hierarchical attention module, and a decoder module, and the decoder module may include a decoder. The speech recognition model realizes a speech recognition function. It is used to:

[0019] First, an audio feature vector sequence X=[x1, x2, ……, x K ] is encoded to obtain the first feature vector sequence. ance H x =[h1 x ,h2 x ,……,h K x ]. Among them, x k is the speech feature vector for the kth frame, h k x is the output from the last neural network layer of the audio encoder module. represents the input feature vector, and h k x x kis converted by the audio encoder module The electronic device also uses a hot word text encoder module to encode each hot word in the pre-defined hot word thesaurus separately, and realizes encoding hot words of different lengths into vectors of fixed dimensions, and generates a set of feature vector sequences representing hot words, i.e., a second feature vector sequence H z =[h0 z ,h1 z ,……,h N z ]. Among them, h n z is the nth hot word The feature vector is a feature vector after being encoded by the coder module. The predefined hot word thesaurus may be predefined according to the needs of the user. For example, appropriate hot words are extracted from the basic hot word thesaurus according to the user's status or occupation as the predefined hot word thesaurus. The basic hot word thesaurus may be predefined according to the user's status or occupation. The predefined hot word thesaurus may be automatically generated according to the user's past situation. For example, when the user uses the application, hot words that appear during use may be automatically generated into the predefined hot word thesaurus. In addition, for example, in the case of a voice assistant application, names in the user's phone book are read as hot words, and the predefined hot word thesaurus is generated from these hot words. In addition, for example, when using an input tool, text such as place names and names input by the user in pinyin after legal permission is obtained is stored as hot words, and the hot words are generated into the predefined hot word thesaurus. The predefined hot word thesaurus may be stored locally or in the cloud.

[0020] Then, the audio segment of each hot word in the predefined hot word thesaurus can be individually encoded using a hot word audio encoder module, which can be shared with the audio encoder module mentioned above. Sharing means that both are the same encoder. Therefore, hot word audio segments of different lengths can be encoded into vectors of fixed dimension, i.e., the output of the encoder of the last frame of the hot word audio segment or the average output of all frames represents the feature representation vector of the entire hot word audio segment, thereby forming a third feature vector sequence H representing the hot word audio. w =[h0 w ,h1 w ,……,h N w ]. Among them, h n w is the feature vector after the n-th hot word audio segment is encoded by the hot word encoder module. Then, the frame hierarchical attention module performs attention operation on the frame hierarchical level on the audio coding feature representation of each frame (the first feature vector sequence) and the hot word audio coding feature representation (the third feature vector sequence), fuses the hot word information, and generates a new audio coding feature representation, i.e., the fourth feature vector sequence (see the following equation 1).

number

[0021] First, the word hierarchical attention module outputs the word at time t from the decoder module. The state vector d t ,The fourth feature vector output by the frame hierarchical attention module Sequence (number 2 below)

number

[0022] Second, the decoder module calculates the fourth feature vector sequence (see the following formula 3) output by the frame hierarchical attention module.

number

[0023] In a specific implementation, not only the hot word text information but also the corresponding audio segments are input, and the audio segments of the speech data to be recognized and the hot word text information are fused and input after performing an attention operation. This obviously improves the hot word incentive effect, and the three-way decoding operation is performed to improve the hot word recognition effect and increase the recognition accuracy of the hot words.

[0024] Further, as shown in Figure 1B, Figure 1B is a flowchart illustrating a speech recognition method according to an embodiment of the present application. The speech recognition method shown in Figure 1B is applied to the speech recognition model shown in Figure 1A, and the speech recognition model is applied to an electronic device, and the speech recognition method includes:

[0025] 101: Encoding is performed on speech data to be recognized to obtain a first feature vector sequence. Among them, in the embodiment of the present application, the speech data to be recognized may be speech data or speech feature vector sequence stored in advance or collected in real time, and the speech data may be at least one of recorded data, recorded data recorded in real time, recorded data extracted from video data, synthesized recorded data, etc., and the speech feature vector sequence may be, but is not limited to, a filter bank feature, a Mel frequency cepstral feature, a mel ... The feature may be at least one of a Mel Frequency Cepstrum Coefficient (MFCC) feature, a Perceptual Linear Predictive (PLP) coefficient feature, etc. If the speech data to be recognized is speech data, the electronic device extracts features of the speech data to obtain a speech feature vector sequence, and then encodes the speech feature vector sequence to obtain a first feature vector sequence. If the speech data to be recognized is a speech feature vector sequence, the electronic device directly encodes the speech feature vector sequence to obtain a first feature vector sequence.

[0026] In a specific implementation, as shown in FIG. 1B, the electronic device uses an audio encoder module to encode the speech data to be recognized to obtain a first feature vector sequence. The audio encoder module may include one or more coding layers, such as a long short-term memory neural network (LSTM) coding layer. The long short-term memory layer may be a long short-term memory layer or a convolutional layer of a convolutional neural network. The long short-term memory neural network may be a long short-term memory layer in a unidirectional or bidirectional long short-term memory neural network. For example, in the embodiment of the present application, a three-layer unidirectional long short-term memory layer is selected, and the input speech feature vector sequence X=[x1, x2, ……, x K ] to encode the first feature vector sequence H x =[h1 x ,h2 x ,……,h K x ] may be output.

[0027] 102: For each hot word in the predefined hot word thesaurus, encoding is performed to obtain a second feature vector sequence. Wherein, the pre-defined hot word thesaurus may be stored in the electronic device in advance, and the pre-defined hot word thesaurus may include text information of a plurality of hot words. The electronic device uses a hot word text encoder module to perform encoding for each hot word in the pre-defined hot word thesaurus to obtain a second feature vector sequence. Or, in another embodiment, the pre-defined hot word thesaurus may be stored in another server, and the pre-defined hot word thesaurus may be obtained by accessing.

[0028] In a specific implementation, the number of characters included in different hotwords may be the same or different. When the number of characters is different, taking the Japanese hotword "Tokyo" which has 2 characters and "Kanagawa" which has 3 characters as an example, for model processing, the long and variable input may be represented by a vector of a fixed dimension. The role of the hotword text encoder module is to encode hotwords of different lengths into vectors of a fixed dimension, which may include one or more encoding layers. The encoding layer may be a long short-term memory layer in a long short-term memory neural network or a convolutional layer in a convolutional neural network. The long short-term memory neural network may be a long short-term memory layer of a unidirectional or bidirectional long short-term memory neural network.

[0029] In a specific implementation, the encoding effect of the bidirectional long short-term memory layer on hotwords is better than that of the unidirectional long short-term memory layer. Selecting a single layer of bidirectional long short-term memory layer and taking the hotword "Kanagawa" as an example, the hotword is composed of 3 characters "神", "奈", "川". Encoded by the hotword encoder of a single layer of bidirectional long short-term memory layer, as shown in Figure 1C, the left side is the forward part of the bidirectional long short-term memory layer and the right side is the reverse part. The vectors h f z and h b z obtained by combining are used as the encoded vector representation of the hotword. The encoded vector representations of multiple hotwords can form a second feature vector sequence. z If there are a total of N + 1 hotwords Z = [z0, z1, ……, z N , then each hotword is processed individually using the hotword encoder module to obtain a second feature vector sequence H z = [h0 z , h1 z , ……, h N z . Among them, h i ​z is the i-th hot word z i It is important to note that z0 is a special hot word " <no-bias>" and Hotwa This means that the hotword selected during subsequent decryption is " <no-bias>If ", the voice does not contain hot words, or the voice recognition To process if the segment is not a hot word, <no-bias>Incentives for I will not do that.

[0030] 103: Perform encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence. Wherein, the electronic device uses a hot word audio encoder module to perform encoding on the audio segments of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence, the third feature vector sequence representing the features of the audio information contained in the hot word audio segments. It is possible.

[0031] In a specific implementation, the hot word audio encoder module and the audio encoder module may be shared, that is, they may share an algorithm. For example, they are the same encoder. The hot word audio encoder module may include one or more coding layers, and the coding layer may be a long short-term memory layer in a long short-term memory neural network or a convolution layer in a convolution neural network, and the long short-term memory neural network may be a long short-term memory layer in a unidirectional or bidirectional long short-term memory neural network. Or, in other embodiments, the hot word audio encoder module and the audio encoder module may be two independent encoders, and the present application does not particularly limit them.

[0032] Among them, the hot word audio segments can be obtained by extraction from audio, manual collection, synthesis by a speech synthesis system, etc., so there is no particular limitation here. Finally, the hot word audio segments are obtained. For example, the hot word audio segments can be pre-stored, or can be audio segments synthesized from the hot words. To briefly explain by example, in the embodiment of the present application, a three-layer unidirectional long-term short-term memory layer is selected, and the input hot word audio segment feature vector sequence X=[x1, x2, ……, x K ] is encoded, and the output of the last frame is used as the feature representation vector of the audio segment of the hot word. Since the encoding method is LSTM (Recurrent Neural Network), the last frame can represent the information of the entire audio sequence. Of course, in other embodiments, instead of the last frame, for example, the average value of all frames may be taken. Suppose there are a total of N+1 hot words Z=[z0, z1, ……,z N ], then use a hot word audio encoder module to encode each hot word audio separately and generate a set of hot word audio vector sequences, i.e., a third feature vector sequence H w =[h0 w ,h1 w ,……,h N w ]. Among them, h i w is the i-th hot word z i It is worth noting that z0 is a special hot word " <no-bias>", which means that there is no hot word. In a specific implementation, it is replaced by the average value of all hot word vectors, which may be all vectors of at least one of the second feature vector sequence and the third feature vector sequence. If the hot word selected during subsequent decoding is " <no-bias>" If so, then In case there is no hot word in the speech segment, or the speech segment to be recognized is not a hot word, <no-bias>There will be no incentives for

[0033] 104: Perform a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence.

[0034] In a specific implementation, the electronic device uses a frame hierarchical attention module to perform a first attention operation on the first feature vector sequence and the third feature vector sequence, and merges the features of both to obtain a fourth feature vector sequence, thus obviously improving the hot word incentive effect.

[0035] Among them, the role of the frame hierarchical attention module is to fuse the hot word text information of the hot word thesaurus preset with the output of each frame audio encoder module, and then form a fourth feature vector sequence having a feature representation of the hot word information, so that the audio feature representation (first feature vector sequence) of the recognition target speech data of each frame has high hot word robustness. Specifically, the attention mechanism of the first attention operation is to A vector of one frame h i x Let be the query term, and the third feature vector sequence H z =[h0 z ,h1 z ,……,h N z ] and perform attention mechanism operations on it.

[0036] In an assumed example, performing a first attention operation on the first feature vector sequence and the third feature vector sequence in step 104 to obtain a fourth feature vector sequence includes: 41: performing a matching operation on each first feature vector in the first feature vector sequence and each third feature vector in the third feature vector sequence to obtain a matching coefficient corresponding to each third feature vector; 42: performing a calculation on the matching coefficients corresponding to each of the third feature vectors and the corresponding third feature vectors to obtain new feature vectors corresponding to each of the third feature vectors; 43: Combining a new feature vector corresponding to each of the third feature vectors and the corresponding third feature vector to obtain a feature representation vector corresponding to each of the third feature vectors, and the feature representation vector corresponding to each of the third feature vectors is the fourth feature vector sequence.

[0037] Specifically, as shown in FIG. 1D, when one of the first feature vectors in the first feature vector sequence is taken as a query term, the electronic device uses the query term and each feature vector in the third feature vector sequence to calculate a matching coefficient based on an attention mechanism. For example, the matching coefficient is calculated by an inner product method or a feature distance method, and the matching coefficient W is calculated by a normalization method. n Then, the matching coefficient W n and the corresponding feature vector h n z The calculation method is sum of dot products, weighted average, inner After the operation, a new feature vector h i z , i.e., obtain the feature vector that best matches the query term and combine it with the query term. , the final audio coding feature representation after fusion h i x Then, each frame is encoded with audio. The above operations are performed on the vectors output by the da to obtain the final fourth feature vector sequence (see below for the number 4).

number

[0038] 105: Perform a decoding operation based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result.

[0039] In a specific implementation, the electronic device inputs the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence into a decoder to perform a decoding operation and obtain a recognition result; or the electronic device performs a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence, inputs the result into the decoder to perform a decoding operation and obtain a recognition result. The decoder may include multiple neural network layers. The decoding operation may be performed by Beam Search decoding, or may be other decoding methods, which will not be described here. Omitted.

[0040] In an embodiment of the present application, before step 101, audio data with text annotations is amplified. The amount of collected voice data can be used to extract voice features. The voice features may be at least one of PLP, MFCC, FilterBank, etc., and are not particularly limited here. The generated text-annotated audio data may be used to train the hot word audio encoder module. A sequence of audio features and a sequence of text annotations of a given audio data may be represented in the following manner: Speech feature sequence X=[x1, x2, ……, x k ,……,x K ] A text-annotated sequence Y=[y0,y1,……,y t ,……,y T ] Of these, x k represents the speech feature vector of the k-th frame in the speech feature sequence X, where K is the total number of speech frames. t is the t-th character of the text-annotated sequence Y , T+1 is the total number of characters with the text annotation, of which y0 is the start of the sentence tag" <s>" and y T is the end tag of the sentence "< / s> Let us take Japanese speech recognition as an example, and a single character is the modeling unit. The text content of a sentence is "Today, Tokyo is windy." If the text content is 10 characters in total, plus the start and end tags of the sentence, the text-annotated sequence is 12 characters in total. Then, the text-annotated sequence Y=[ <s> ,Today, ,the ,wind ,is ,strong ,in ,Tokyo.< / s> ].

[0041] In a specific implementation, the voice recognition model can have the recognition function of any hot word, which means that the hot word cannot be limited in the model training. Therefore, in the embodiment of the present application, an annotation segment randomly selected from the text annotation of the training data can participate in the entire model training as a hot word. The following will be described in detail by taking the voice data of B sentence as an example, where B is an integer greater than 1, and model training is performed once. For example, two parameters P and N can be set, where P is the probability of selecting a hot word for the training data of a certain sentence, and N is the maximum number of characters of the selected hot word. In the embodiment of the present application, P=0.5 and N=4, that is, The training data of an arbitrary sentence is also selected with a 50% probability, and a maximum of four consecutive characters may be selected from the text annotation as hot words. Taking "It's windy in Tokyo today" as an example, the comparison of annotations before and after selecting hot words from the sentence is shown in the table below.

[0042] [Table 1]

[0043] When "East" and "Tokyo" in the original annotation are selected as hot words, a special tag " <bias>" may be added. <bias>The role of "introduces training error and adjusts the model parameters related to hot words (hot word audio encoder model) during model training. The training step is to forcibly update the model parameters of the hot word audio encoder module or the model parameters of the hot word text encoder module. When "Tokyo" and "King" are selected as hot words, they are added to the hot word list for the current model update and are input to the hot word audio encoder module or the hot word text encoder module. Whenever a model update is performed, hot words are selected individually, and the initial hot word list may be empty. After processing the data, the model parameters may be updated by an optimization method of the neural network. In the training step, sample data and true recognition results corresponding to the sample data are obtained, the sample data are encoded to obtain a first feature vector sequence, each hot word in the pre-set hot word thesaurus is encoded to obtain a second feature vector sequence, and audio segments of each hot word in the pre-set hot word thesaurus are encoded to obtain a third feature vector sequence, and the first feature vector sequence and the third feature vector sequence are encoded to obtain a third feature vector sequence. A first attention operation is performed on the feature vector sequence to obtain a fourth feature vector sequence, and a decoding operation is performed based on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a predicted recognition result, and model parameter updating can be realized based on the deviation between the true recognition result and the preset recognition result.

[0044] In an assumed example, performing a decoding operation to obtain a recognition result based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence in step 105 is as follows: A51: performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; A52: inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence into a decoder, performing a decoding operation, and obtaining a recognition result.

[0045] Wherein, the electronic device performs a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence. The role of the word hierarchical attention module is to extract the audio-related feature vector and the hot word-related feature vector required at the decoding waiting time from the audio feature vector sequence, the hot word text feature vector sequence, and the hot word audio feature vector sequence at each decoding time. Take the tth character as an example, when the model predicts the tth character, the audio-related feature vector is based on the audio content of the character waiting to be decoded at the tth time. The feature vector for the hot word text is the possible hot word text at time t. The feature vector for hot word audio represents the hot word content at time t. It is considered to represent the forward audio content.

[0046] Among them, the word hierarchical attention mechanism takes a vector as a query term and applies attention to a sequence of feature vectors. The feature vector that best matches the query term is extracted as the output by performing a join mechanism operation. Specifically, the matching coefficient between the query term and each feature vector in the feature vector sequence is calculated, and then these matching coefficients are multiplied by the corresponding feature vectors to obtain a sum, and the new feature vector obtained is the feature vector that best matches the query term.

[0047] In an assumed example, performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence in step A51 to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence, includes: A511: Obtaining a first state feature vector of the decoder at a current time; A512: performing an attention operation on the second feature vector sequence based on the first state feature vector to obtain the hot word text context feature vector sequence at the current time; A513: performing an attention operation on the third feature vector sequence according to the first state feature vector to obtain the hot word audio context feature vector sequence at the current time; A514: The method may include performing an attention operation on the fourth feature vector sequence based on the first state feature vector to obtain the audio context feature vector sequence for the current time.

[0048] In a specific implementation, suppose the first state feature vector of the decoder at the current time is d t Then, d t Using the above attention mechanism, the fourth feature vector sequence output by the frame hierarchical attention module is

number

number

number

[0049] Or in another embodiment, based on the first feature vector sequence, a second attention operation is performed on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence, respectively, to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence and an audio context feature vector sequence.

[0050] Further, in the assumed example, the decoder includes a first unidirectional long short-term memory layer, and obtaining a first state feature vector of the decoder at a first past time in step A511 includes: A5111: Obtaining a recognition result at a first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence at the first past time; A5112: The recognition result at the first past time and the hot word text context feature vector sequence at the first past time, the hot word audio context feature vector sequence, and the audio context feature vector sequence are converted into the first unidirectional length feature vector. and inputting the first state feature vector to a short-term memory layer to obtain the first state feature vector.

[0051] Wherein, the first past time is at least one time before the current time, that is, the first past time may be one time before the current time, or multiple times before the current time. The decoder may include two-layer unidirectional long short-term memory layers, and the two-layer unidirectional long short-term memory layers may include a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer. In a specific implementation, the electronic device obtains the recognition result of the decoder at the first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence at the first past time, and inputs them into the first unidirectional long short-term memory layer to obtain a first state feature vector. In addition, memory (feature) fusion is performed using the recognition result of the first past time and the corresponding input content, which is advantageous to improving the model prediction ability.

[0052] In a specific implementation, the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence at the first past time are obtained as follows: obtain a first state feature vector of the decoder at the first past time, perform an attention operation on the second feature vector sequence based on the first state feature vector to obtain a hot word text context feature vector sequence at the first past time, perform an attention operation on the third feature vector sequence based on the first state feature vector to obtain a hot word audio context feature vector sequence at the first past time, and perform an attention operation on the fourth feature vector sequence based on the first state feature vector to obtain an audio context feature vector sequence at the first past time. For example, suppose the first state feature vector of the decoder at the first past time is d t-1 Then, the relevant d t-1 Using the query term, an attention operation may be performed on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input at the first past time. d t-1 may be a first state feature vector obtained by inputting the recognition result of the second past time, and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence at the second past time into a first unidirectional long short-term memory layer. The second past time may be at least one time before the first past time. That is, the second past time may be one time before the first past time, or may be multiple times before the first past time.

[0053] Or, in another embodiment, the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence input to the decoder at the first past time, which are all or a part of the recognition result before the current time, are input to the first unidirectional long short-term memory layer to obtain a first state feature vector, and the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence input to the decoder at the current time are input to the second unidirectional long short-term memory layer to obtain the recognition result at the current time.

[0054] Further, in the assumed example, the decoder includes a second unidirectional long short-term memory layer, and the step A52: inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence into the decoder, performing a decoding operation, and obtaining a recognition result includes: obtaining the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence by performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence at the current time, respectively, using a first state feature vector input into the decoder at the current time; The method may further include inputting the tonal sequence and the audio context feature vector sequence into the second unidirectional long short-term memory layer to obtain a recognition result for the current time.

[0055] In a specific implementation, the current time may be understood to be the current decoding time, for example, the first past time is a time before the current time, when decoding the tth word The time before the current time, that is, the time when the t-1th word is decoded, is the first time. The decoder may include two unidirectional long short-term memory layers. For example, when decoding the t-th character at time t-1, the first long short-term memory layer decodes the recognition result character y t-1 and the output vector c from the word hierarchical attention module t-1 (t-1 The hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence input to the decoder at the time point of the character are used as inputs to calculate the first state feature vector d of the decoder. t d t is input to the word hierarchy attention module, and the word hierarchy attention module at time t is Module output t Calculate c t is the hot word text context feature vector at time t. a hot word audio context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence, and t is input to the second long short-term memory layer, Calculate the decoder output h t d Finally, calculate the posterior probability of the character to be output. Get the recognition result.

[0056] According to the above-mentioned embodiment of the present application, firstly, in addition to the hot word text information as input, the hot word voice segment is added as an additional input incentive source, and the hot word text is matched with the input of the voice segment, which effectively increases the richness of the hot word input information, and naturally enhances the effect of the hot word incentive. Secondly, the introduction of the two-layer incentive, i.e., the two-time attention operation, naturally enhances the effect of the hot word incentive, and the two inputs and the two-layer hot word incentive solution complement each other and jointly enhance the hot word recognition effect, which is beneficial to the recognition accuracy of the hot word.

[0057] In an assumed example, performing a decoding operation to obtain a recognition result based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence in step 105 is as follows: The second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence may be input to a decoder to perform a decoding operation and obtain the recognition result.

[0058] In a specific implementation, the electronic device directly inputs the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence into the decoder to perform the decoding operation and obtain the recognition result. Not only the hot word text information but also the corresponding audio segment is input, and the recognition target voice data and the audio segment of the hot word text information are further input after performing the attention operation, which obviously improves the hot word incentive effect, and performs the three decoding operations to improve the hot word recognition effect and improve the hot word recognition accuracy.

[0059] Further, in the assumed example, the decoder includes a two-layer unidirectional long short-term memory layer, the two-layer unidirectional long short-term memory layer including a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer, and in the above step, inputting the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to the decoder to perform a decoding operation and obtain the recognition result includes B51: a recognition result at a first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time obtaining a vector sequence and the fourth feature vector sequence; B52: inputting the recognition result at the first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time into the first unidirectional long-term short-term memory layer to obtain a second state feature vector; B53: The method may include inputting the fourth feature vector sequence obtained by performing the first attention operation on the first feature vector sequence and the third feature vector sequence at the current time using the second feature vector sequence, the third feature vector sequence, and the second state feature vector input to the decoder at a current time, into the second unidirectional long / short-term memory layer, and obtaining a recognition result for the current time.

[0060] Wherein, the decoder may include multiple neural network layers, for example, the decoder may include two unidirectional long short-term memory layers, including a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer.

[0061] In a specific implementation, the electronic device obtains the recognition result at the first past time and the second, third and fourth feature vector sequences input to the decoder at the first past time, inputs them into the first unidirectional long-short-term memory layer, obtains a second state feature vector, and inputs the second, third and fourth feature vector sequences input to the decoder at the current time into the second unidirectional long-short-term memory layer to obtain the recognition result at the current time. Among them, the fourth feature vector sequence is obtained by using the second state feature vector to perform a first attention operation on at least one of the first and third feature vector sequences at the current time. For example, the second state feature vector may be used to perform a first attention operation on the first and third feature vector sequences at the current time, respectively. In addition to obtaining the output contents from the second unidirectional long-short-term memory layer of the decoder, the device calculates the posterior probability of the output contents to obtain the final decoding result, i.e., the recognition result at the current time.

[0062] According to the above, the speech recognition method described in the embodiment of the present application performs encoding on the speech data to be recognized, obtains a first feature vector sequence, performs encoding on each hot word in a pre-defined hot word thesaurus, obtains a second feature vector sequence, performs encoding on the audio segment of each hot word in the pre-defined hot word thesaurus, obtains a third feature vector sequence, performs a first attention operation on the first feature vector sequence and the third feature vector sequence, obtains a fourth feature vector sequence, performs a decoding operation according to the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result, and takes not only the hot word text information but also the corresponding audio segment as input, and further performs an attention operation on the speech data to be recognized and the audio segment of the hot word text information as input, which obviously improves the hot word incentive effect, and performs the decoding operation of the three to improve the hot word recognition effect and increase the hot word recognition accuracy.

[0063] As shown in FIG. 2, which is a flowchart illustrating a speech recognition method according to an embodiment of the present application, similar to the embodiment shown in FIG. 1B. The speech recognition method shown in FIG. 2 is applied to the speech recognition model shown in FIG. 1A, and the speech recognition model is applied to an electronic device, and the speech recognition method includes: 201: Encoding speech data to be recognized to obtain a first feature vector sequence; 202: For each hot word in the predefined hot word thesaurus, a code obtaining a second sequence of feature vectors; 203: Perform encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence; 204: performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; 205: performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; 206: Inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence into a decoder to perform a decoding operation and obtain a recognition result. Among them, the detailed explanation of steps 201 to 206 can be referred to the steps of the speech recognition method shown in FIG. 1B, so it will be omitted here.

[0064] According to the above, the speech recognition method described in the embodiment of the present application, firstly, in addition to the hot word text information as input, adds the hot word voice segment as an additional input incentive source, and the input of matching the hot word text with the voice segment effectively increases the richness of the hot word input information, and naturally, the effect of the hot word incentive can be greatly improved. Secondly, the introduction of the two-layer incentive, i.e., the two-time attention operation, naturally, can enhance the effect of the hot word incentive, and the two inputs and the two-layer hot word incentive solution complement each other and jointly improve the hot word recognition effect, which is beneficial to the recognition accuracy of the hot word.

[0065] As in the above embodiment, as shown in FIG. 3, FIG. 3 is a diagram showing the structure of an electronic device according to an embodiment of the present application. The electronic device includes a processor, a memory, a communication port, and one or more programs, among which the one or more programs are stored in the memory and configured to be executed by the processor. In the embodiment of the present application, the programs are: encoding the speech data to be recognized to obtain a first feature vector sequence; performing encoding for each hot word in a predefined hot word thesaurus to obtain a second sequence of feature vectors; performing encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence; performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; performing a decoding operation based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result.

[0066] According to the above, the electronic device described in the embodiment of the present application performs encoding on the recognition target voice data to obtain a first feature vector sequence, performs encoding on each hot word in a pre-defined hot word thesaurus to obtain a second feature vector sequence, performs encoding on an audio segment of each hot word in the pre-defined hot word thesaurus to obtain a third feature vector sequence, performs a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence, performs a decoding operation according to the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result, and obtains not only the hot word text information but also the corresponding audio segment information. The input also includes the input of the speech data to be recognized and the hot word text information, and the input is the fusion of the audio segments of the speech data to be recognized and the hot word text information, which obviously improves the hot word incentive effect, and the decoding operation of the three items is performed to improve the hot word recognition effect and increase the recognition accuracy of the hot words.

[0067] In an assumed example, in a configuration in which a first attention operation is performed on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence, the program further comprises: performing a matching operation on each first feature vector in the first feature vector sequence and each third feature vector in the third feature vector sequence to obtain a matching coefficient corresponding to each third feature vector; performing a calculation on the matching coefficients corresponding to each of the third feature vectors and the corresponding third feature vectors to obtain new feature vectors corresponding to each of the third feature vectors; combining a new feature vector corresponding to each of the third feature vectors and the corresponding first feature vector to obtain a feature representation vector corresponding to each of the third feature vectors, and making the feature representation vector corresponding to each of the third feature vectors the fourth feature vector sequence.

[0068] In an assumed example, in a mode in which a decoding operation is performed based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result, the program further comprises: performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; The hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence are input to a decoder, and a decoding operation is performed to obtain a recognition result.

[0069] In a hypothetical example, a second attention operation is performed on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence. In the above-mentioned program, obtaining a first state feature vector of the decoder at a current time; Performing an attention operation on the second feature vector sequence based on the first state feature vector to obtain the hot word text context feature vector sequence at the current time; performing an attention operation on the third feature vector sequence according to the first state feature vector to obtain the hot word audio context feature vector sequence at the current time; performing an attention operation on the fourth feature vector sequence based on the first state feature vector to obtain the audio context feature vector sequence for the current time.

[0070] In a hypothetical example, the decoder includes a first unidirectional long short-term memory layer, and the program is configured to obtain a first state feature vector of the decoder at a first past time, the program further comprising: Context feature vector sequence, the hot word audio context feature vector sequence and obtaining said audio context feature vector sequence; inputting the recognition result at the first past time, the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence at the first past time into the first unidirectional long short-term memory layer to obtain the first state feature vector.

[0071] In an assumed example, the decoder includes a second unidirectional long short-term memory layer, and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence are input to the decoder to perform a decoding operation and obtain a recognition result. In this embodiment, the program includes: and an instruction for executing inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence obtained by performing the second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence at the current time, respectively, using a first state feature vector input to the decoder at the current time, into the second unidirectional long / short-term memory layer to obtain a recognition result at the current time.

[0072] In an assumed example, in a mode in which a decoding operation is performed based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result, the program further comprises: and instructions for executing inputting the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence into a decoder to perform a decoding operation and obtain the recognition result.

[0073] In an assumed example, the decoder includes a two-layer unidirectional long short-term memory layer, the two-layer unidirectional long short-term memory layer including a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer, and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence are input to the decoder to perform a decoding operation and obtain the recognition result, the program further comprising: obtaining a recognition result at a first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time; inputting the recognition result at the first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time into the first unidirectional long / short-term memory layer to obtain a second state feature vector; inputting the fourth feature vector sequence obtained by performing the first attention operation on the first feature vector sequence and the third feature vector sequence at the current time into the second unidirectional long / short-term memory layer using the second feature vector sequence, the third feature vector sequence, and the second state feature vector input to the decoder at a current time, and obtaining a recognition result at the current time.

[0074] In an assumed example, performing encoding on an audio segment of each hot word in the predefined hot word thesaurus to obtain a third sequence of feature vectors includes: The predetermined hot word sorter is generated using an encoding layer including a long short-term memory layer of a long short-term memory neural network or a convolution layer of a convolution neural network, which is a long short-term memory layer of a one-layer or multiple-layer unidirectional or bidirectional long short-term memory neural network. performing encoding on the audio segment of each hot word in the class to obtain the third sequence of feature vectors.

[0075] The above describes the solution of the embodiment of the present application from the implementation of the method. In order for the electronic device to realize the above functions, the necessary hardware structure and / or software to execute each function are provided. It is understood that the present invention includes a module. With reference to the units and algorithm steps of each example described in the embodiments of this specification, a person skilled in the art can easily think of implementing the present application in hardware or a combination of hardware and computer software. Whether a function is implemented by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. A professional can use different methods for each specific application to realize the described functions, but such implementation should not be considered as going beyond the scope of this application.

[0076] In the embodiments of the present application, the functional units of the electronic device may be divided as in the above method example. For example, each functional unit may be divided by function, or two or more functions may be integrated into a processing unit. The integrated unit may be in the form of hardware or a software functional unit. It should be noted that the division of the units in the embodiments of the present application is only a rough outline and is merely a division of logical functions, and there may be other division methods when actually implemented.

[0077] 4A is a block diagram showing a functional unit configuration of a voice recognition device 400 according to an embodiment of the present application. The voice recognition device 400 is applied to an electronic device, and the voice recognition device 400 includes: an audio encoder module 401 for encoding speech data to be recognized to obtain a first feature vector sequence; a hot word text encoder module 402 for encoding each hot word in a predefined hot word thesaurus to obtain a second sequence of feature vectors; a hot word audio encoder module 403 for performing encoding on an audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence; a frame hierarchical attention module 404 for performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; and a decoder module 405 for performing a decoding operation based on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result.

[0078] According to the above, the speech recognition device described in the embodiment of the present application performs coding on the recognition target speech data, obtains a first feature vector sequence, performs coding on each hot word in a pre-defined hot word thesaurus, obtains a second feature vector sequence, performs coding on the audio segment of each hot word in a pre-defined hot word thesaurus, obtains a third feature vector sequence, performs a first attention operation on the first feature vector sequence and the third feature vector sequence, obtains a fourth feature vector sequence, performs a decoding operation based on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a recognition result, and inputs not only hot word text information but also the corresponding audio segment, and further inputs the fusion of the attention operation performed on the recognition target speech data and the audio segment of the hot word text information, thereby obviously improving the hot word incentive effect, and performing the decoding operation of the three. , it can improve the hot word recognition effect and increase the hot word recognition accuracy.

[0079] In a hypothetical example, in a configuration in which a first attention operation is performed on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence, the frame hierarchical attention module 404 specifically performs: performing a matching operation on each first feature vector in the first feature vector sequence and each third feature vector in the third feature vector sequence to obtain a matching coefficient corresponding to each third feature vector; performing a calculation on the matching coefficients corresponding to each of the third feature vectors and the corresponding third feature vectors to obtain new feature vectors corresponding to each of the third feature vectors; The new feature vector corresponding to each of the third feature vectors and the corresponding first feature vector are combined to obtain a feature representation vector corresponding to each of the third feature vectors, and the feature representation vector corresponding to each of the third feature vectors is used as the fourth feature vector sequence.

[0080] In an assumed example, as shown in FIG. 4B, FIG. 4B is another modified structure of the speech recognition device shown in FIG. 4A. Compared with FIG. 4A, its decoder module 405 is: a word hierarchical attention module 4051, which performs a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; and a decoder 4052 for inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence to a decoder 4054 for performing a decoding operation and obtaining a recognition result.

[0081] In an assumed example, in a case where a second attention operation is performed on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence, the word hierarchical attention module 4051 specifically performs: obtaining a first state feature vector of the decoder at a current time; Performing an attention operation on the second feature vector sequence based on the first state feature vector to obtain the hot word text context feature vector sequence at the current time; performing an attention operation on the third feature vector sequence according to the first state feature vector to obtain the hot word audio context feature vector sequence at the current time; An attention operation is performed on the fourth feature vector sequence based on the first state feature vector to obtain the audio context feature vector sequence at the current time.

[0082] In an assumed example, the decoder includes a first unidirectional long short-term memory layer, and in an embodiment in which a first state feature vector of the decoder at a first past time is obtained, the word hierarchy attention module 4051 specifically performs: Obtaining a recognition result at a first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence at the first past time; The recognition result at the first past time and the hot word text at the first past time the hot word audio context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence are input into the first unidirectional long short-term memory layer to obtain the first state feature vector.

[0083] In an assumed example, the decoder includes a second unidirectional long short-term memory layer, and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence are input to the decoder to perform a decoding operation and obtain a recognition result. In this embodiment, the decoder 4052 specifically includes: The hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence obtained by performing the second attention operation on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence at the current time using the first state feature vector input to the decoder at the current time are input to the second unidirectional long / short-term memory layer to obtain a recognition result at the current time.

[0084] In a hypothetical example, in a case where a decoding operation is performed based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result, the decoder module 405 specifically performs the following steps: The second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence are input to a decoder for performing a decoding operation to obtain the recognition result.

[0085] In an assumed example, the decoder includes a two-layer unidirectional long short-term memory layer, the two-layer unidirectional long short-term memory layer including a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer. The second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence are input to the decoder to perform a decoding operation and obtain the recognition result. In this embodiment, the decoder module 405 specifically includes: obtaining a recognition result at a first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time; inputting the recognition result at the first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time into the first unidirectional long / short-term memory layer to obtain a second state feature vector; The fourth feature vector sequence obtained by performing the first attention operation on the first feature vector sequence and the third feature vector sequence at the current time using the second feature vector sequence, the third feature vector sequence, and the second state feature vector input to the decoder at the current time is input to the second unidirectional long / short-term memory layer, and is used to obtain a recognition result at the current time.

[0086] In a hypothetical example, in the aspect of encoding the audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence, the hot word audio encoder module 403 specifically: The audio segment of each hot word in the predefined hot word thesaurus is encoded using one or more coding layers, which may be a long short-term memory layer of a unidirectional or bidirectional long short-term memory neural network, or a convolutional layer of a convolutional neural network, to obtain the third feature vector sequence.

[0087] It is understood that the functions of each program module of the speech recognition device of this embodiment are embodied by the method according to the above-mentioned method embodiment. The specific implementation procedures can be referred to in the above-mentioned method embodiment, so they are omitted here.

[0088] An embodiment of the present application further provides a computer storage medium, in which the computer storage medium stores a computer program for electronic data exchange, the computer program causing a computer to perform some or all of the steps of any of the methods described in the method embodiments above, the computer including an electronic device.

[0089] An embodiment of the present application further provides a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to perform some or all of the steps of any of the methods described in the method embodiments. The computer program product may be a software installation package. The computer includes an electronic device.

[0090] In terms of what should be explained, the embodiments of the above-mentioned methods are described as a combination of a series of operations for ease of explanation, but those skilled in the art should recognize that the present application is not limited by the order of operations described, since some steps may be performed in other orders or simultaneously according to the present application. In addition, those skilled in the art should recognize that the embodiments described in the specification are all preferred embodiments, and the associated operations and modules are not necessarily essential to the present application.

[0091] In the above embodiments, different emphasis is placed on the description of each embodiment, and for parts that are not described in detail in an embodiment, reference can be made to the descriptions of other embodiments.

[0092] It should be understood that in some embodiments of the present application, the disclosed device may be implemented in other forms. For example, the above-described device embodiments are merely exemplary and are merely logical function divisions, such as the division of the above units, and in actual implementation, additional division methods may exist, such as multiple units or components may be combined or integrated into other systems, or some features may be ignored or not implemented. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections via some interfaces, devices or units, or may be electrical or other forms.

[0093] The above units described as separate components may or may not be physically separated. The components shown as units may or may not be physical components, i.e., they may be located in one place or distributed across multiple network components. Some or all of these units may be selected to achieve the objectives of the solution of the present embodiment according to actual needs.

[0094] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated unit may be in the form of hardware or a software functional unit.

[0095] The above-mentioned integrated unit may be realized in the form of a software functional unit and stored in a computer-readable memory when sold or used as an independent product. Based on this understanding, the technical solution of the present application may be essentially or contributes to the prior art, or all or a part of the technical solution may be embodied in the form of a software product. The computer software product is stored in a memory and includes some instructions to cause a computer device (such as a personal computer, a server, or a network device) to execute all or a part of the steps of the above-mentioned methods in each embodiment of the present application. The above-mentioned memory may be a USB memory, a read-only memory (ROM), a random access memory (RAM), a removable hard disk, a magnetic disk, etc. This includes various media for storing program code, such as hard disks or optical disks.

[0096] Those skilled in the art will understand that all or part of the steps of each method in the above embodiments can be achieved by a program instructing relevant hardware. The program can be stored in a computer-readable memory. The memory can include flash drives, read-only memories (hereinafter referred to as "ROM"), Random Access Memory (hereinafter referred to as "RAM"), magnetic data It may include a disk or optical disk.

[0097] The above describes the embodiments of the present application in detail, and the present specification uses specific examples to describe the principles and embodiments of the present application. The above description of the embodiments is only used to help understand the method and core idea of ​​the present application. In addition, those skilled in the art will understand that there are changes in the specific embodiments and application scope according to the idea of ​​the present application, and as stated above, the contents of this specification should not be understood as limitations on the present application.< / bias> < / bias>

Claims

1. 1. A speech recognition method, comprising: encoding the speech data to be recognized to obtain a first feature vector sequence; Encoding text information of each hot word in a predefined hot word thesaurus to obtain a second feature vector sequence; performing encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third sequence of feature vectors; performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; performing a decoding operation based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result; The method according to claim 1, further comprising:

2. performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence, performing a matching operation on each first feature vector in the first sequence of feature vectors and each third feature vector in the third sequence of feature vectors to obtain a matching coefficient corresponding to each third feature vector; performing an operation on the matching coefficients corresponding to each of the third feature vectors and the corresponding third feature vectors to obtain new feature vectors corresponding to each of the third feature vectors; combining a new feature vector corresponding to each of the third feature vectors and the corresponding first feature vector to obtain a feature representation vector corresponding to each of the third feature vectors, and the feature representation vector corresponding to each of the third feature vectors is the fourth feature vector sequence; 2. The method of claim 1, comprising:

3. performing a decoding operation based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result, performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence into a decoder to perform a decoding operation and obtain a recognition result; 3. The method according to claim 1 or 2, comprising:

4. performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; obtaining a first state feature vector of the decoder at a current time; performing an attention operation on the second feature vector sequence based on the first state feature vector to obtain the hot word text context feature vector sequence at the current time; performing an attention operation on the third feature vector sequence based on the first state feature vector to obtain the hot word audio context feature vector sequence at the current time; performing an attention operation on the fourth feature vector sequence based on the first state feature vector to obtain the audio context feature vector sequence for the current time; 4. The method of claim 3, comprising:

5. The decoder includes a first unidirectional long short-term memory layer, and obtaining a first state feature vector at a first past time of the decoder comprises: Obtaining a recognition result of a first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence of the first past time; inputting the recognition result of the first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence of the first past time into the first unidirectional long short-term memory layer to obtain the first state feature vector; 5. The method of claim 4, comprising:

6. The decoder includes a second unidirectional long short-term memory layer, and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence are input to the decoder to perform a decoding operation and obtain a recognition result. inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence obtained by performing the second attention operation on the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence at the current time using a first state feature vector input to the decoder at the current time into the second unidirectional long-term short-term memory layer, thereby obtaining a recognition result at the current time.

4. The method according to claim 3 .

7. the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence performing a decoding operation and obtaining a recognition result based on the feature vector sequences of the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence into a decoder, performing a decoding operation, and obtaining the recognition result.

2. The method of claim 1 .

8. The decoder includes a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer, i.e., two unidirectional long short-term memory layers; inputting the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence into the decoder to perform a decoding operation and obtain the recognition result, obtaining a recognition result at a first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time; inputting the recognition result at the first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time into the first unidirectional long / short-term memory layer to obtain a second state feature vector; and inputting the fourth feature vector sequence obtained by performing the first attention operation on the first feature vector sequence and the third feature vector sequence at the current time using the second feature vector sequence, the third feature vector sequence, and the second state feature vector input to the decoder at a current time into the second unidirectional long / short-term memory layer to obtain a recognition result at the current time.

9. performing encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third sequence of feature vectors; encoding the audio segments of each hot word in the predefined hot word thesaurus using one or more encoding layers, the encoding layers including a long short-term memory layer of a long short-term memory neural network, which is a long short-term memory layer of a unidirectional or bidirectional long short-term memory neural network, or a convolutional layer of a convolutional neural network, to obtain the third feature vector sequence; 9. The method according to any one of claims 1 to 8.

10. A speech recognition device, comprising: an audio encoder module for encoding speech data to be recognized to obtain a first feature vector sequence; a hot word text encoder module for encoding text information of each hot word in a predefined hot word thesaurus to obtain a second feature vector sequence; a hot word audio encoder module for performing encoding on an audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence; a frame hierarchical attention module that performs a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence; a decoder module for performing a decoding operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a recognition result.

1. A speech recognition device comprising:

11. In the aspect of performing a first attention operation on the first feature vector sequence and the third feature vector sequence to obtain a fourth feature vector sequence, the frame hierarchical attention module specifically comprises: performing a matching operation on each first feature vector in the first sequence of feature vectors and each third feature vector in the third sequence of feature vectors to obtain a matching coefficient corresponding to each third feature vector; performing an operation on the matching coefficients corresponding to each of the third feature vectors and the corresponding third feature vectors to obtain new feature vectors corresponding to each of the third feature vectors; a new feature vector corresponding to each of the third feature vectors and the corresponding first feature vector are combined to obtain a feature representation vector corresponding to each of the third feature vectors, and the feature representation vector corresponding to each of the third feature vectors is used as the fourth feature vector sequence; 11. The apparatus of claim 10.

12. In the aspect of performing a decoding operation to obtain a recognition result based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence, the decoder module specifically comprises: performing a second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence; The hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence are input to a decoder to perform a decoding operation and obtain a recognition result; 12. Apparatus according to claim 10 or 11.

13. In an aspect in which a second attention operation is performed on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence to obtain a hot word text context feature vector sequence, a hot word audio context feature vector sequence, and an audio context feature vector sequence, the decoder module specifically comprises: obtaining a first state feature vector of the decoder at a current time; performing an attention operation on the second feature vector sequence based on the first state feature vector to obtain the hot word text context feature vector sequence; performing an attention operation on the third feature vector sequence based on the first state feature vector to obtain the hot word audio context feature vector sequence; performing an attention operation on the fourth feature vector sequence based on the first state feature vector to obtain the audio context feature vector sequence; 13. The apparatus of claim 12.

14. In an embodiment in which the decoder includes a first unidirectional long short-term memory layer and obtains a first state feature vector at a first past time of the decoder, the decoder module specifically comprises: Obtaining a recognition result of a first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence of the first past time; The recognition result of the first past time and the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence of the first past time are input into the first unidirectional long-term short-term memory layer, which is used to obtain the first state feature vector; 14. The apparatus of claim 13.

15. In the embodiment, the decoder includes a second unidirectional long short-term memory layer, and the hot word text context feature vector sequence, the hot word audio context feature vector sequence and the audio context feature vector sequence are input to the decoder to perform a decoding operation and obtain a recognition result. The decoder module specifically comprises: the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence are obtained by performing the second attention operation on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence at the current time using the first state feature vector input to the decoder at the current time, respectively, and inputting the hot word text context feature vector sequence, the hot word audio context feature vector sequence, and the audio context feature vector sequence into the second unidirectional long-term short-term memory layer to obtain a recognition result at the current time; 13. The apparatus of claim 12.

16. In the aspect of performing a decoding operation to obtain a recognition result based on the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence, the decoder module is specifically used for inputting the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence into a decoder, performing a decoding operation, and obtaining the recognition result.

11. The apparatus of claim 10.

17. In the embodiment, the decoder includes a first unidirectional long short-term memory layer and a second unidirectional long short-term memory layer, i.e., two unidirectional long short-term memory layers, and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence are input to the decoder to perform a decoding operation and obtain the recognition result. Specifically, the decoder module includes: obtaining a recognition result at a first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time; inputting the recognition result at the first past time and the second feature vector sequence, the third feature vector sequence, and the fourth feature vector sequence input to the decoder at the first past time into the first unidirectional long / short-term memory layer to obtain a second state feature vector; the fourth feature vector sequence obtained by performing the first attention operation on the first feature vector sequence and the third feature vector sequence at the current time using the second feature vector sequence, the third feature vector sequence, and the second state feature vector input to the decoder at the current time is input to the second unidirectional long / short-term memory layer, and is used to obtain a recognition result at the current time.

17. The apparatus of claim 16.

18. In the aspect of performing encoding on the audio segment of each hot word in the predefined hot word thesaurus to obtain a third feature vector sequence, the hot word audio encoder module specifically includes: A long short-term memory layer of a unidirectional or bidirectional long short-term memory neural network, or a convolution of a convolutional neural network and encoding the audio segments of each hot word in the predefined hot word thesaurus using one or more coding layers, including a first coding layer, to obtain the third feature vector sequence.

18. Apparatus according to any one of claims 10 to 17.

19. An electronic device, A method for controlling a communication between a computer and a network, comprising: a processor; a memory; a communication port; and one or more programs stored in the memory and configured to be executed by the processor with instructions for each step of the method according to any one of claims 1 to 9.

1. An electronic device comprising:

20. 1. A computer-readable storage medium, comprising: A computer program for electronic data exchange that causes a computer to carry out the method according to any one of claims 1 to 9, A computer-readable storage medium comprising:

21. A computer program comprising:

10. A method operative to cause a computer to carry out the method of any one of claims 1 to 9, A computer program comprising:

Images