Decoding method in artificial neural network for speech recognition, training method of artificial neural network and speech recognition device
By using a shared decoding layer and attention network in the speech recognition artificial neural network, the lexical unit is directly recognized from the speech information, and the problems of traditional technology complexity and resource limitation are solved, achieving efficient and accurate speech recognition effect.
Patent Information
- Application Number
- CN202010361194.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-31
- Filing Date
- 2020-04-30
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-04-30
AI Technical Summary
Traditional speech recognition technology is based on the Hidden Markov model, making it difficult to directly recognize text from speech data, and the end-to-end speech recognition technology of deep learning methods is complex and difficult to efficiently implement on resource-constrained devices.
Using a shared decoding layer in an artificial neural network, by adjusting synaptic weights, performing multi-task decoding, combining attention network and coding layer, directly generate features from speech information to identify lexical units, improving the efficiency and accuracy of speech recognition.
Efficient speech recognition on resource-constrained devices is achieved, reducing system size and power consumption, and improving speech recognition performance, especially when processing the same or similar words.
Smart Images

Figure CN112397058B_ABST
Abstract
Description
[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0093066 filed on Jul. 31, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field
[0002] The following description relates to a decoding method and device in an artificial neural network for speech recognition. Background Art
[0003] Speech recognition refers to a technique for recognizing or understanding acoustic speech signals (such as speech uttered by humans) by analyzing them using a computing device. Traditionally, speech recognition has been performed mainly using a Hidden Markov Model (HMM) to recognize speech based on frequency characteristics extracted from speech data. This HMM-based speech recognition method can recognize speech by analyzing pronunciation from speech data and combining words or sentences based on the analyzed pronunciation.
[0004] However, with the development of deep learning-based machine learning methods, research is underway on end-to-end speech recognition technology, which uses an acoustic model constructed as an artificial neural network to directly recognize text (such as words and sentences) from speech data without analyzing pronunciation from the speech data. Summary of the invention
[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0006] In one general aspect, a decoding method in an artificial neural network for speech recognition includes: using a shared decoding layer included in the artificial neural network to perform a first decoding task of decoding features including speech information and at least one lexical unit recognized until a current time point; using the shared decoding layer to perform a second decoding task of decoding the at least one lexical unit; and based on a result of the first decoding task and a result of the second decoding task, determining an output lexical unit to be recognized after the at least one lexical unit.
[0007] The step of performing the first decoding task may include adjusting the weight of a synapse connecting the shared decoding layer and the neuron corresponding to the feature to a first value. The step of performing the second decoding task may include adjusting the weight of the synapse to a second value.
[0008] The step of performing the first decoding task may include: performing the first decoding task by inputting the feature and at least one lexical unit recognized until the current point into a shared decoding layer. The step of performing the second decoding task may include: performing the second decoding task by inputting at least one lexical unit recognized until the current point into a shared decoding layer.
[0009] The step of performing the first decoding task may include: inputting the feature and at least one lexical unit recognized until the current point into a first pre-decoding layer; and inputting the output of the first pre-decoding layer into a shared decoding layer to perform the first decoding task. The step of performing the second decoding task may include: inputting at least one lexical unit recognized until the current point into a second pre-decoding layer; and inputting the output of the second pre-decoding layer into a shared decoding layer to perform the second decoding task.
[0010] The step of performing the first decoding task may include: inputting the features and at least one lexical unit recognized until the current point into a shared decoding layer; and inputting the output of the shared decoding layer into a first post-decoding layer to perform the first decoding task. The step of performing the second decoding task may include: inputting at least one lexical unit recognized until the current point into a shared decoding layer; and inputting the output of the shared decoding layer into a second post-decoding layer to perform the second decoding task.
[0011] The result of the first decoding task may include a first probability of outputting a candidate for a lexical unit. The result of the second decoding task may include a second probability of outputting a candidate for a lexical unit.
[0012] The step of determining the output lexical unit may include: calculating a weighted sum of the first probability and the second probability; and determining a candidate corresponding to a maximum weighted sum among the candidates as the output lexical unit.
[0013] The decoding method may further include determining the output lexical unit as a subsequent input lexical unit.
[0014] The features may be determined using an attention network based on a sequence vector associated with a progression of speech information.
[0015] The decoding method may further include generating the feature by encoding speech information using an encoding layer included in the artificial neural network.
[0016] In another general aspect, a method for training an artificial neural network for speech recognition includes: selecting at least one sample from a batch including a first training set and a second training set, the first training set including pairs of speech and text corresponding to the speech, and the second training set including the text; in response to selecting the sample from the first training set, training an encoding layer and a shared decoding layer included in the artificial neural network based on the sample; and in response to selecting the sample from the second training set, training the shared decoding layer based on the sample.
[0017] The steps of training the encoding layer and the shared decoding layer may include: using the encoding layer, extracting features from the speech included in the sample; using the shared decoding layer, estimating the output lexical unit to be recognized after the at least one lexical unit based on the extracted features and the at least one lexical unit; and training the encoding layer and the shared decoding layer based on the estimated output lexical unit and at least a portion of the text corresponding to the speech included in the sample.
[0018] The step of training a shared decoding layer may include: using the shared decoding layer, estimating an output lexical unit to be recognized after the at least one lexical unit based on the at least one lexical unit; and training the shared decoding layer based on the estimated output lexical unit and at least a portion of the text included in the sample.
[0019] The step of selecting a sample from a first training set may include: extracting features from speech included in the sample using an encoding layer; estimating a first probability of a candidate for an output lexical unit to be recognized after the at least one lexical unit based on the extracted features and at least one lexical unit using a shared decoding layer; estimating a second probability of a candidate for an output lexical unit to be recognized after the at least one lexical unit based on the at least one lexical unit using a shared decoding layer; estimating an output lexical unit based on a weight between the first probability and the second probability; and learning the weight based on at least a portion of a text corresponding to the speech included in the sample.
[0020] In another general aspect, a speech recognition device includes: an encoder configured to generate features from speech information; and a decoder configured to: use a shared decoding layer to perform a first decoding task of decoding the features and at least one lexical unit recognized until a current time point, perform a second decoding task of decoding at least one lexical unit recognized until the current point, and based on a result of the first decoding task and a result of the second decoding task, determine an output lexical unit to be recognized after the at least one lexical unit.
[0021] The speech recognition apparatus may further include a speech preprocessor configured to extract a speech feature vector from the speech information, and the encoder may generate the feature based on the speech feature vector.
[0022] The decoder may perform a first decoding task by adjusting a weight of a synapse connecting a shared decoding layer and a neuron corresponding to the feature to a first value; and perform a second decoding task by adjusting the weight of the synapse to a second value.
[0023] The decoder may perform a first decoding task by receiving the feature and at least one lexical unit recognized until the current point via a shared decoding layer, and perform a second decoding task by receiving at least one lexical unit recognized until the current point via a shared decoding layer.
[0024] The decoder can perform the first decoding task by receiving the feature and at least one lexical unit recognized until the current point through the first pre-decoding layer and receiving the output of the first pre-decoding layer through the shared decoding layer. In addition, the decoder can perform the second decoding task by receiving at least one lexical unit recognized until the current point through the second pre-decoding layer and receiving the output of the second pre-decoding layer through the shared decoding layer.
[0025] The decoder can perform the first decoding task by receiving the feature and at least one lexical unit recognized until the current point through the shared decoding layer and receiving the output of the shared decoding layer through the first post-decoding layer. In addition, the decoder can perform the second decoding task by receiving at least one lexical unit recognized until the current point through the shared decoding layer and receiving the output of the shared decoding layer through the second post-decoding layer.
[0026] The result of the first decoding task may include a first probability of outputting a candidate for a lexical unit. The result of the second decoding task may include a second probability of outputting a candidate for a lexical unit.
[0027] The decoder may calculate a weighted sum of the first probability and the second probability; and determine a candidate corresponding to a maximum weighted sum among the candidates as the output token.
[0028] The decoder may determine the output token as the subsequent input token.
[0029] The features may be determined using an attention network based on a sequence vector associated with a progression of speech information.
[0030] In another general aspect, a speech recognition device includes: an encoder configured to receive a speech feature vector corresponding to speech information and output a context vector; and a decoder configured to: receive the context vector; decode the context vector and a most recently recognized lexical unit using a shared decoding layer included in an artificial neural network to output a first result; decode only the most recently recognized lexical unit using the shared decoding layer to output a second result; and output a current lexical unit based on the first result and the second result.
[0031] The decoder can be configured to decode the context vector and the most recently recognized token using a speech recognition task that shares a decoding layer.
[0032] The decoder can be configured to decode only the most recently recognized tokens using the language model tasks that share the decoding layer.
[0033] The decoder can be configured entirely through the shared decoding layer.
[0034] Only a portion of the decoder may be configured by sharing the decoding layer.
[0035] Other features and aspects will be apparent from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 An example of an artificial neural network operating for speech recognition is shown.
[0037] Figure 2A and Figure 2B An example of combining a language model (LM) artificial neural network with an artificial neural network is shown.
[0038] Figure 3 An example of a speech recognition device is shown.
[0039] Figure 4 A flow chart illustrating an example of a decoding method.
[0040] Figure 5 A flow chart illustrating an example of a speech recognition method.
[0041] Figure 6 An example of a speech recognition device including a decoder that shares a portion of a decoding layer is shown.
[0042] Figure 7 and Figure 8 An example of training an artificial neural network for speech recognition is shown.
[0043] Fig. 9 An example of a speech recognition device is shown.
[0044] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION
[0045] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order.
[0046] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of this application.
[0047] Although terms such as "first", "second" and "third" may be used herein to describe various members, components, regions, layers or parts, these members, components, regions, layers or parts should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer or part from another member, component, region, layer or part. Therefore, without departing from the teachings of the examples, the first member, first component, first region, first layer or first part represented in the examples described herein may also be referred to as the second member, second component, second region, second layer or second part.
[0048] Throughout this specification, when a component is described as being “connected to” or “coupled to” another component, the component may be directly “connected to” or “coupled to” the other component, or one or more other components may be present in between. Conversely, when an element is described as being “directly connected to” or “directly coupled to” another element, there may be no other elements in between. Likewise, similar expressions (e.g., “between” versus “directly between” and “adjacent” versus “directly adjacent”) will also be interpreted in the same manner.
[0049] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.
[0050] The terms used herein are only for describing various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "include", "comprising" and "having" indicate the presence of the stated features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.
[0051] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the disclosure of the present application, and should not be interpreted as an idealized or overly formal meaning.
[0052] Furthermore, in the description of the examples, when it is considered that a detailed description of a structure or function known therefrom after understanding the disclosure of the present application will lead to an obscure interpretation of the examples, such description will be omitted.
[0053] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals denote like elements throughout.
[0054] Figure 1 An example of an artificial neural network operating for speech recognition is shown.
[0055] Reference Figure 1 , the artificial neural network 100 for speech recognition can be implemented by a sequence-to-sequence model. The sequence-to-sequence model can output a sequence of a different domain from the input sequence. For example, the artificial neural network 100 for speech recognition implemented with a sequence-to-sequence structure can directly generate a text corresponding to the input speech from the input speech, and the text is a recognition result corresponding to the input speech. The artificial neural network 100 can be pre-trained to generate a sequence of recognition results from an input sequence. For example, the artificial neural network 100 can be trained using training data including speech and text corresponding to the speech to increase the correlation between speech and text.
[0056] As shown in the figure, the sequence-to-sequence model includes an encoder 110 and a decoder 120. The encoder 110 may not directly receive and process the speech information 111, but may receive a speech feature vector 112 obtained by converting into a vector form corresponding to the speech information 111. The encoder 110 may sequentially receive the speech feature vectors 112, and then finally compress the collection of all speech information to form a single vector referred to as a context vector 113. The context vector 113 may also be referred to as a feature or encoding feature including speech information. When the speech information 111 based on the passage of time is compressed into a single context vector 113, the encoder 110 may send the context vector 113 to the decoder 120. The decoder 120 may receive the context vector 113 and sequentially output the recognized words one by one.
[0057] The encoder 110 and the decoder 120 may be implemented by a neural network (e.g., a recurrent neural network (RNN)). The encoder 110 may be an RNN configured to receive a sequence (such as speech) as an input. The decoder 120 may be an RNN configured to output a sequence (e.g., text as a result of recognizing speech). In addition to RNN, the encoder 110 and the decoder 120 may also be implemented by, for example, a deep neural network (DNN), a recursive deep neural network (RDNN), etc.
[0058] The decoder 120 may perform autoregressive decoding. The autoregressive decoding may be a decoding method that determines an output token based on tokens recognized until the previous step, and is used to calculate an output of an undefined length through an artificial neural network.
[0059] exist Figure 1 In the example of , the decoder 120 receiving the context vector 113 receives the start lexical unit indicating the beginning of the sentence or the start of the sentence (SOS) 121 as an initial input. The decoder 120 receiving the SOS 121 predicts the lexical unit that is likely to appear subsequently. The lexical unit used herein indicates a unit that forms a sequence, and the unit may include, for example, a word, a subword, or a character. In the following, for ease of description, the lexical unit will be described as a word unit. In the first time step, the decoder 120 predicts the word "Hi" 122 that will appear subsequently. That is, the decoder 120 determines "Hi" 122 as an output lexical unit, and outputs the output lexical unit as a speech recognition result. The decoder 120 that determines "Hi" 122 as the output lexical unit inputs "Hi" 122 as the input of the subsequent time step. In the second time step, the decoder 120 predicts the word "Bixby" 123 that will appear subsequently from the input word "Hi" 122. Then, the decoder 120 inputs "Bixby" as the input of the subsequent time step. The decoder 120 may predict the word that follows as described above, and input the predicted word as input for the subsequent time step. The decoder 120 may repeat such an operation until an end token indicating the end of a sentence or an end of a sentence (EOS) is predicted as a subsequent word.
[0060] The decoder 120 may obtain an output lexical unit based on the information calculated from the encoder 110 in each step, and the output lexical unit may be obtained according to the input lexical unit selected until the previous step. For example, the decoder 120 receiving lexical unit 1 may predict the probability of a candidate of the output lexical unit based on the received lexical unit 1, and determine the output lexical unit based on the probability. In this example, the decoder 120 may select a candidate with the maximum probability from among the candidates as the output lexical unit.
[0061] The token candidate t in step i i The conditional probability of can be expressed by Equation 1.
[0062] [Equation 1]
[0063] p(t i |t 1 ,t 2 ,...,t i-1 ,H e )
[0064] In Equation 1, H e represents the hidden representation of the encoder 110 corresponding to the context vector 113, t 1 to i-1 Indicates the lexical units recognized until the current time point. The decoder 120 may determine the subsequent output lexical units based on the context vector 113 and the lexical units recognized until the current time point. The context vector 113 may be determined using an attention network based on a speech feature vector associated with the process of the speech information 111. Since different information may be required for each word when the decoder 120 determines the output lexical unit, using the same context vector at each time step may degrade the performance of speech recognition.
[0065] For example, using an attention network, when the decoder 120 determines the output token t i When the decoder 120 can obtain the hidden state vector h of the encoder 110 1 、h 2 ,...,h n Combination of for each t i Different context vectors are used. Here, when the decoder 120 determines the output token t i The hidden state vector h of the encoder 110 used when j Can be applied with attention weight a ij , and output the lexical unit t i The context vector can be defined as ∑a ij h j .
[0066] In addition to an encoder 110 configured to receive speech information 111 as input and output a context vector 113, and a decoder 120 configured to receive the context vector 113 as input and output text (e.g., a word sequence) as a recognition result, the artificial neural network 100 for speech recognition can also improve the performance of speech recognition by additionally using a language model (LM) artificial neural network configured to predict the probability of a word sequence.
[0067] The LM artificial neural network can predict the probability of the entire sentence. In addition, when the previous word is given, the LM artificial neural network can predict the probability of the word that appears after the word. The LM artificial neural network can be configured only by the decoder without the encoder, and training or learning is performed using only text.
[0068] When there are words with the same or similar pronunciation, the LM artificial neural network can achieve more accurate recognition of such words. For example, when the words "Minchul" and "feels" are given, the LM artificial neural network can determine that the probability of the word "happy" appearing later is greater than the probability of the word "habit" appearing later. In order to distinguish the operation performed by the decoder 120 of the artificial neural network 100 and the operation performed by the LM artificial neural network, the decoding operation performed by the decoder 120 of the artificial neural network 100 for speech recognition will be referred to as the speech recognition task or the first decoding task hereinafter, and the decoding operation performed by the LM artificial neural network will be referred to as the LM task or the second decoding task hereinafter.
[0069] The speech recognition task may process the relationship between the signal characteristics of speech and language elements, and may be, for example, a task of estimating the pronunciation of a speech signal. The LM task may be a task of estimating the level of grammatical or semantic correctness of a word or sentence.
[0070] Combining the LM artificial neural network with the existing artificial neural network 100 for speech recognition or adding the LM artificial neural network to the existing artificial neural network 100 for speech recognition can improve the performance of speech recognition. In addition, the existing artificial neural network 100 for speech recognition may require pairs of speech and text corresponding to the speech for training or learning. However, the number of pairs of speech and text corresponding to the speech may be much less than that of easily available text. Therefore, by additionally training the LM artificial neural network that can be trained using only text, the performance of speech recognition can be supplemented. In the following, a method for improving the performance of speech recognition while preventing an increase in memory bandwidth and power consumption by combining the LM artificial neural network with the existing artificial neural network 100 will be described.
[0071] Figure 2A and Figure 2B An example of combining an LM artificial neural network with an artificial neural network is shown. Before describing the method of sharing a decoder according to the example, reference will be made to Figure 2A Existing methods for combining external LM artificial neural networks with artificial neural networks are briefly described.
[0072] Reference Figure 2A, the artificial neural network includes an encoder 210, a decoder 220 and an external LM artificial neural network 230. The encoder 210 and the decoder 220 may be similar to the above reference Figure 1 The encoder 110 and the decoder 120 are described.
[0073] The artificial neural network may perform speech recognition based on the combined operation of the decoder 220 and the LM artificial neural network 230. The decoder 220 and the LM artificial neural network 230 may output respective lexical unit-unit recognition results, and a final recognition result may be generated by combining the recognition results based on weights. For example, the decoder 220 may determine a candidate for an output lexical unit based on a context vector and at least one lexical unit recognized until the current time point, and the LM artificial neural network 230 may determine a candidate for an output lexical unit based on at least one lexical unit recognized until the current time point. Subsequently, the final output lexical unit may be determined by combining the candidates for the output lexical unit based on the weights.
[0074] In daily life, speech recognition may need to be suitable for privacy protection, and needs to be performed by or in a device on the palm of the user, regardless of network connection. For example, in an on-device environment where speech recognition is performed on the device side instead of a large server, speech recognition can be performed in the user's device, so privacy can be protected without data leakage to the outside. In addition, in such an on-device environment, speech recognition can be performed in real time without delay despite not being connected to the network. The devices used herein may include digital devices (e.g., smart phones, smart TVs (TVs), artificial intelligence (AI) speakers, navigation systems, digital televisions (DTVs), wearable devices, electronic systems, and robots) that include memory and a microprocessor and have computing power.
[0075] Using an external LM artificial neural network 230 may not be suitable for such an on-device environment. When the LM artificial neural network 230 is used, the performance of speech recognition may be improved. However, due to the addition of the LM artificial neural network 230, the size of the entire system may increase, and due to the operation time of the LM artificial neural network 230, the entire time for speech recognition may increase. In addition, the number of parameters that will be determined by humans to appropriately merge two separately trained models into one model may increase.
[0076] In order to perform speech recognition on the device side, it may be necessary to load the parameters of the artificial neural network onto the memory of the device. However, adding the LM artificial neural network 230 may increase the bandwidth for reading and writing the parameters in the memory and increase the power consumed for reading and writing the memory. In addition, the LM artificial neural network 230 may need to be trained separately from the existing artificial neural network.
[0077] Reference Figure 2B According to the example, the multi-task artificial neural network includes an encoder 240 and a decoder 250, and both the speech recognition task and the LM task can be performed in a single decoder 250. The multi-task artificial neural network can perform tasks through a single decoder 250 shared by the speech recognition task and the LM task without using an external LM artificial neural network.
[0078] Since the multi-task artificial neural network is configured to perform two tasks by sharing a single decoder 250, the parameters (e.g., weights) of the LM artificial neural network may not need to be stored, thereby reducing the size of the entire system. In addition, by using the parameters twice for the speech recognition task and the LM task, the use of a multi-task artificial neural network can increase the reuse rate of the parameters of the decoder that are loaded once for decoding. Because it is not limited to calculations but to memory reading and writing, this is beneficial to general sequence generation models. When speech recognition is performed in a device with a limited amount and bandwidth of memory that is loaded once, this effect is even better. In the following, reference will be made to Figures 3 to 6 The decoding method is described in detail and will be referred to Figure 7 and Figure 8 Describe the training method in detail.
[0079] Figure 3 An example of a speech recognition device is shown.
[0080] Reference Figure 3 The speech recognition device includes an artificial neural network 310, a speech preprocessor 340 and a text processor 350. The artificial neural network 310 includes an encoder 320 and a shared decoder 330. The artificial neural network 310, the encoder 320 and the shared decoder 330 can be respectively Figure 2B The multi-task artificial neural network, encoder 240 and decoder 250 are described.
[0081] The speech preprocessor 340 may preprocess the input of the artificial neural network 310 by, for example, removing noise included in the speech information or processing the speech information into a form suitable for input to the artificial neural network 310. The speech preprocessor 340 may extract a speech feature vector from the speech information. For example, the speech preprocessor 340 may extract features by dividing the speech in a specific unit (e.g., 25 milliseconds (ms)) and configuring a frame while sliding to overlap a portion (e.g., 15ms). Subsequently, the speech preprocessor 340 may generate a speech feature vector to be input to the encoder 320 by connecting a preset number of feature frames (e.g., three feature frames).
[0082] The text processor 350 may input at least one lexical unit recognized until the current time point to the shared decoder 330. For example, as a non-limiting example only, the at least one lexical unit recognized until the current time point may be a most recently recognized lexical unit. However, the present invention is not limited thereto, and the at least one lexical unit may be any one or any combination of the lexical units recognized until the current time point.
[0083] Although the above reference Figure 3 An example is described, but various technical modifications and changes may be applicable. For example, the decoder 330 may be a broader concept including the text processor 350 .
[0084] Figure 4 A flow chart illustrating an example of a decoding method.
[0085] Refer to below Figure 4 Operations 410 to 430 described above may be performed by a shared decoder (e.g., Figure 3 The shared decoder 330 described above may be implemented by one or more hardware modules or software modules or various combinations thereof.
[0086] Reference Figure 4 In operation 410, the shared decoder uses the shared decoding layer to perform a first decoding task of decoding a feature or context vector including speech information and at least one lexical unit recognized until the current time point. The shared decoder can be configured by a neural network including a plurality of layers, among which one or more layers that are generally used for both speech recognition tasks and LM tasks are referred to as shared decoding layers. The shared decoder can be configured entirely by the shared decoding layer, or only a portion of the shared decoder can be configured by the shared decoding layer.
[0087] In operation 420, the shared decoder performs a second decoding task of decoding at least one lexical unit recognized up to the current time point using the shared decoding layer. The shared decoder may perform the LM task with the same parameters using the shared decoding layer that performs the first decoding task.
[0088] The shared decoder can perform the existing function of the decoder of the artificial neural network to predict the output lexical unit using the context vector received from the encoder and at least one lexical unit recognized until the current time point as input, and at the same time perform the function of the LM artificial neural network to predict the output lexical unit using only the at least one lexical unit recognized until the current time point as input without considering the context vector calculated from the speech information.
[0089] The shared decoder may determine a first probability of a candidate for an output lexical unit based on the context vector and at least one lexical unit recognized until the current time point, and determine a second probability of a candidate for an output lexical unit based on at least one lexical unit recognized until the current time point. The first probability may be determined as shown in Equation 1 above. The second probability may be determined as shown in Equation 2.
[0090] [Equation 2]
[0091] p(t i t 1 ,t 2 ,…,t i-1 )
[0092] The shared decoder can use a single decoder to perform multiple tasks (e.g., a first decoding task and a second decoding task) by adjusting the weights of the synapses connecting the neurons corresponding to the context vector and the shared decoding layer. When performing the first decoding task, the shared decoder can adjust the weights of the synapses connecting the neurons corresponding to the context vector and the shared decoding layer to a first value. When performing the second decoding task, the shared decoder can adjust the weights of the synapses connecting the neurons corresponding to the context vector and the shared decoding layer to a second value.
[0093] Since the shared decoder determines the first probability of the candidate of the output lexical unit by considering the context vector when performing the first decoding task, the shared decoder may adjust the weight of the synapse connecting the neuron corresponding to the context vector and the shared decoding layer to a value greater than or equal to the preset threshold. For example, the shared decoder may adjust the weight of the synapse connecting the neuron corresponding to the context vector and the shared decoding layer (for example, to 1) so that the context vector is input to the decoder.
[0094] Since the shared decoder determines the second probability of the candidate of the output lexical unit without considering the context vector when performing the second decoding task, the shared decoder may adjust the weight of the synapse connecting the neuron corresponding to the context vector and the shared decoding layer to a value less than the preset threshold. For example, the shared decoder may adjust the weight of the synapse connecting the neuron corresponding to the context vector and the shared decoding layer (for example, to 0) so that the context vector does not affect the decoder.
[0095] In operation 430, the shared decoder determines the output lexical unit to be recognized after at least one lexical unit based on the result of the first decoding task and the result of the second decoding task. The shared decoder may calculate the weighted sum of the first probability and the second probability, and determine the candidate corresponding to the maximum weighted sum among the candidates for the output lexical unit as the output lexical unit. For example, as a result of the first decoding task, "Bixby" and "Vixby", which are candidates for the output lexical unit, may have first probabilities of 0.79 and 0.81, respectively. In addition, as a result of the second decoding task, "Bixby" and "Vixby" may have second probabilities of 0.85 and 0.12, respectively. In this example, the weight of the first probability to the second probability may be 1:0.2. In this example, the weighted sum of the first probability and the second probability of each of "Bixby" and "Vixby" may be 0.96 and 0.834, respectively. Therefore, in this example, the shared decoder may determine "Bixby", which corresponds to the maximum weighted sum, as the output lexical unit. The weights of the first probability and the second probability may be predetermined values, or may be learned or trained through a gate. Figure 7 Describe in detail how the weights are learned or trained.
[0096] Figure 5 A flow chart illustrating an example of a speech recognition method.
[0097] In the following we will refer to Figure 5 The operations 510 to 590 described above can be referred to in Figure 3 The described speech recognition device performs.
[0098] Reference Figure 5 In operation 510, the speech recognition device receives speech as input. The user may utter a speech, and the speech recognition device may receive the speech uttered by the user as input. The speech received by the speech recognition device may also be referred to as speech information or a speech signal.
[0099] In operation 520 , the speech recognition apparatus encodes the speech information and generates a context vector.
[0100] In operation 530, the speech recognition apparatus initializes a time step i and sets a start word unit SOS indicating the start of a sentence as an initial input in the time step i.
[0101] In operation 540, the speech recognition apparatus performs a first decoding task and decodes the context vector and at least one lexical unit recognized up to the current time point.
[0102] In operation 550, the speech recognition apparatus performs a second decoding task. The speech recognition apparatus decodes at least one lexical unit recognized up to the current time point.
[0103] In operation 560 , the speech recognition apparatus calculates a weighted sum of a first probability and a second probability of outputting a candidate of a lexical unit based on a result of the first decoding task and a result of the second decoding task.
[0104] In operation 570, the speech recognition apparatus determines the output lexical unit in time step i. As described above, the speech recognition apparatus determines a candidate corresponding to the maximum weighted sum among the candidates of the output lexical unit as the output lexical unit (eg, t_i).
[0105] In operations 580 and 590 , the speech recognition apparatus increases the time step by 1, and repeats operations 540 to 570 until the output token or the end token EOS in the time step i indicating the end of the sentence is predicted as a subsequent word.
[0106] Figure 6 An example of a speech recognition device including a decoder that shares a portion of a decoding layer is shown.
[0107] Reference Figure 6 , the speech recognition device includes an artificial neural network 610, a speech preprocessor 640 and a text processor 650. The artificial neural network 610 includes an encoder 620 and a decoder 630. The encoder 620, the speech preprocessor 640 and the text processor 650 can be the above reference Figure 3 The encoder 320, speech preprocessor 340 and text processor 350 are described.
[0108] The decoder 630 may use only a portion of the multiple layers as a shared decoding layer 633. For example, the decoder 630 may have a decoding layer for a speech recognition task and a decoding layer for a LM task, respectively, in a front portion of the decoder 630 configured to receive a context vector from the encoder 620, or in a rear portion of the decoder 630 configured to output a recognition result.
[0109] As shown in the figure, the decoder 630 performs the first decoding task by inputting the context vector and at least one lexical unit recognized until the current time point into the first pre-decoding layer 631 for the speech recognition task and inputting the output of the first pre-decoding layer 631 into the shared decoding layer 633. In addition, the decoder 630 performs the second decoding task by inputting at least one lexical unit recognized until the current time point into the second pre-decoding layer 632 for the LM task and inputting the output of the second pre-decoding layer 632 into the shared decoding layer 633.
[0110] Optionally, the decoder 630 performs a first decoding task by inputting the context vector and at least one lexical unit recognized until the current time point into a shared decoding layer 633 and inputting the output of the shared decoding layer 633 into a first post-decoding layer 634 for a speech recognition task. In addition, the decoder 630 performs a second decoding task by inputting at least one lexical unit recognized until the current time point into a shared decoding layer 633 and inputting the output of the shared decoding layer 633 into a second post-decoding layer 635 for a LM task.
[0111] Optionally, the decoder 630 performs the first decoding task by inputting the context vector and at least one lexical unit recognized until the current time point into the first pre-decoding layer 631 for the speech recognition task, inputting the output of the first pre-decoding layer 631 into the shared decoding layer 633, and inputting the output of the shared decoding layer 633 into the first post-decoding layer 634. In addition, the decoder 630 performs the second decoding task by inputting at least one lexical unit recognized until the current time point into the second pre-decoding layer 632 for the LM task, inputting the output of the second pre-decoding layer 632 into the shared decoding layer 633, and inputting the output of the shared decoding layer 633 into the second post-decoding layer 635.
[0112] When only a portion of the multiple layers are used as a shared decoding layer, the portion or area corresponding to the shared decoding layer can be reduced. However, there may be respective areas or portions for two different tasks, thus improving performance at the expense of model size.
[0113] Figure 7 and Figure 8 An example of training an artificial neural network for speech recognition is shown.
[0114] In the following we will refer to Figure 7 The described operations 710 to 730 may be performed by a speech recognition training apparatus.
[0115] Reference Figure 7 In operation 710, the training device selects at least one sample from a batch including a first training set and a second training set, wherein the first training set includes pairs of speech and text corresponding to the speech, and the second training set includes text. The batch may include the first training set and the second training set at a preset ratio or proportion.
[0116] The sample may be data for training an artificial neural network for speech recognition, and may be referred to herein as a training sample. The speech included in the first training set may be implemented in various forms (e.g., speech information or speech signals, data obtained by preprocessing the speech information, features or feature vectors of the speech information, and phonemes or polyphones (senones) in each frame of the speech information). The type of sample may not be limited to a specific type. Based on design intent, the sample of speech information may be defined or designed as various forms or types (e.g., words, phonemes, morphemes, speech symbol units, phrases, chapters, paragraphs, sentences, etc.).
[0117] In operation 720, in response to selecting a sample from the first training set, the training device trains the encoding layer and the shared decoding layer included in the artificial neural network based on the selected sample. For example, the training device uses the encoding layer to extract features from the speech included in the sample. Based on the extracted features and at least one lexical unit, the training device uses the shared decoding layer to estimate the output lexical unit to be recognized after the at least one lexical unit. Subsequently, based on the estimated output lexical unit and at least a portion of the text corresponding to the speech included in the sample, the training device trains the encoding layer and the shared decoding layer. For example, the training device performs such training to increase the correlation between the estimated output lexical unit and at least a portion of the text corresponding to the speech included in the sample.
[0118] In response to selecting a sample from the second training set, the training device trains the shared decoding layer based on the selected sample in operation 730. For example, the training device uses the decoding layer to estimate an output lexical unit to be recognized after the at least one lexical unit based on the at least one lexical unit, and trains the shared decoding layer based on the estimated output lexical unit and at least a portion of the text included in the sample.
[0119] Reference Figure 8 , when selecting samples from a second training set including only text, the training device trains only the decoding layers included in the decoder 820 by preventing the gradient from flowing to the encoder 810.
[0120] When selecting samples from the first training set, the training device may also train only the decoding layer again by preventing the gradient from flowing to the encoder 810 using only the text in the pair of speech and the text corresponding to the speech.
[0121] The training device may include a gate that determines the flow of the gradient and trains the gate. In addition, the training device may train or learn the attention weights.
[0122] In addition, the training device may train or learn the weight of the first probability and the second probability. For example, the training device selects a sample from a first training set, extracts features from the speech included in the sample using an encoding layer, estimates a first probability of a candidate of an output lexical unit to be recognized after the at least one lexical unit based on the extracted features and at least one lexical unit using a shared decoding layer, estimates a second probability of a candidate of an output lexical unit to be recognized after the at least one lexical unit based on the at least one lexical unit using a shared decoding layer, estimates the output lexical unit based on the weight of the first probability and the second probability, and trains or learns the weight based on at least a portion of the text corresponding to the speech included in the sample.
[0123] Fig. 9 An example of a speech recognition device is shown.
[0124] Reference Fig. 9 , the speech recognition device 900 includes a sensor 910, a processor 930, and a communication interface 970. The speech recognition device 900 also includes a memory 950 and a display 990. The sensor 910, the processor 930, the memory 950, the communication interface 970, and the display 990 can communicate with each other through a communication bus 905.
[0125] The sensor 910 may include, for example, a micro sensor and a voice sensor.
[0126] The processor 930 may execute the above reference Figures 1 to 6 The processor 930 may execute a program and control the speech recognition apparatus 900. The code of the program to be executed by the processor 930 may be stored in the memory 950.
[0127] The processor 930 may be configured as, for example, a central processing unit (CPU) or a graphics processing unit (GPU).
[0128] Processor 930 can generate features from speech information, execute a first decoding task to decode the features and at least one lexical unit recognized until the current time point, execute a second decoding task to decode at least one lexical unit recognized until the current time point, and determine an output lexical unit to be recognized after the at least one lexical unit based on a result of the first decoding task and a result of the second decoding task.
[0129] The memory 950 may store data processed by the processor 930. For example, the memory 950 may store a program. The program to be stored may be a grammar set to be encoded to be executable by the processor 930 to perform speech recognition. The memory 950 may be a volatile memory or a non-volatile memory.
[0130] The communication interface 970 may be connected to the sensor 910, the processor 930, and the memory 950 to transmit and receive data. The communication interface 970 may be connected to another external device to transmit and receive data. Hereinafter, the expression "transmit and receive A" may indicate that information or data indicating A is transmitted and received.
[0131] The communication interface 970 may be implemented by a circuit in the speech recognition device 900. For example, the communication interface 970 may include an internal bus and an external bus. As another example, the communication interface 970 may be an element connecting the speech recognition device 900 and an external device. In this example, the communication interface 970 may receive data from the external device and send the received data to the processor 930 and the memory 950.
[0132] The display 990 may display the results from the decoding, for example, the speech recognition results.
[0133] Although not shown, the speech recognition training device may include a sensor, a processor, and a communication interface. The training device may also include a memory and a display. The sensor, processor, memory, communication interface, and display may communicate with each other via a communication bus.
[0134] Sensors may include, for example, microsensors and voice sensors.
[0135] The processor can execute the above reference Figure 7 and Figure 8 One or more of the operations or methods described and an algorithm corresponding to the operation or method. The processor can execute the program and control the training device in the artificial neural network. The code of the program to be executed by the processor can be stored in the memory.
[0136] The processor may be configured as, for example, a CPU or a GPU.
[0137] The processor may select at least one sample from a batch including a first training set and a second training set, the first training set including pairs of speech and text corresponding to the speech, and the second training set including text. When the sample is selected from the first training set, the processor may train an encoding layer and a shared decoding layer included in the artificial neural network based on the selected sample. When the sample is selected from the second training set, the processor may train the shared decoding layer based on the selected sample.
[0138] The memory may store data processed by the processor. For example, the memory may store a program. The program to be stored may be a grammar set that is encoded to be executable by the processor to perform training. The memory may be a volatile memory or a non-volatile memory.
[0139] The communication interface may be connected to the sensor, processor, and memory to send and receive data. The communication interface may be connected to another external device to send and receive data.
[0140] The communication interface may be implemented by a circuit in the training device. For example, the communication interface may include an internal bus and an external bus. For another example, the communication interface may be a component that connects the training device and an external device. In this example, the communication interface may receive data from the external device and send the received data to the processor and the memory.
[0141] In this regard Figure 1 , FIG. 2A to FIG. 2B , Figure 3 , Figure 6 , Figure 8 and Fig. 9The described speech recognition equipment, speech recognition training equipment and other equipment, units, modules, devices and other components are realized by hardware components. Examples of hardware components that can be used to perform the operations described in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are realized by computing hardware (for example, by one or more processors or computers). Processors or computers can be realized by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.
[0142] Figure 4 , Figure 5 and Figure 7The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as instructions or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.
[0143] The instructions or software for controlling the processor or computer to implement the hardware components and perform the methods described above are written as computer programs, code segments, instructions or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by the processor or computer. In another example, the instructions or software include high-level code executed by the processor or computer using an interpreter. Ordinary programmers in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods described above.
[0144] The instructions or software for controlling the processor or computer-implemented hardware components and performing the methods described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device, wherein any other device is configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor and computer can execute the instructions.
[0145] Although the present disclosure includes specific examples, it will be clear to those skilled in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered merely descriptive and not for purposes of limitation. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents.
[0146] Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as being included in the disclosure.
Claims
1. A decoding method in an artificial neural network for speech recognition, the decoding method include: Using a shared decoding layer included in the artificial neural network, a first decoding task of decoding features including speech information and at least one lexical unit recognized up to a current time point is performed; Using the shared decoding layer, performing a second decoding task of decoding the at least one lexical unit; and determining, based on a result of the first decoding task and a result of the second decoding task, an output lexical unit to be recognized after the at least one lexical unit, The step of performing the first decoding task includes: adjusting the weight of the synapse connecting the shared decoding layer and the neuron corresponding to the feature to a first value, and The step of performing the second decoding task includes: adjusting the weight of the synapse to a second value, The step of performing the first decoding task further includes: Processing the relationship between the signal characteristics of speech information and language elements, and The step of performing the second decoding task further includes: Estimating the level of semantic correctness of speech information.
2. The decoding method according to claim 1, in, The step of performing the first decoding task further includes: performing the first decoding task by inputting the feature and the at least one lexical unit identified up to the current time point into a shared decoding layer, and The step of performing the second decoding task further includes performing the second decoding task by inputting the at least one lexical unit recognized up to the current time point into the shared decoding layer.
3. The decoding method according to claim 1, in, The step of performing the first decoding task also includes: inputting the feature and the at least one lexical unit identified up to the current time point into a first pre-decoding layer; and inputting the output of the first pre-decoding layer into a shared decoding layer to perform the first decoding task, and The step of performing the second decoding task also includes: inputting the at least one lexical unit recognized until the current time point into the second pre-decoding layer; and inputting the output of the second pre-decoding layer into the shared decoding layer to perform the second decoding task.
4. The decoding method according to claim 1, in, The step of performing the first decoding task also includes: inputting the feature and the at least one lexical unit identified up to the current time point into a shared decoding layer; and inputting the output of the shared decoding layer into a first post-decoding layer to perform the first decoding task, and The step of performing the second decoding task also includes: inputting the at least one lexical unit recognized until the current time point into a shared decoding layer; and inputting the output of the shared decoding layer into a second post-decoding layer to perform the second decoding task.
5. The decoding method according to claim 1, in, The result of the first decoding task includes outputting a first probability of a candidate for a token, and The result of the second decoding task includes outputting a second probability of a candidate for a lexical unit.
6. The decoding method according to claim 5, in, The steps of determining the output token include: calculating a weighted sum of the first probability and the second probability; and The candidate corresponding to the maximum weighted sum among the candidates is determined as the output token.
7. The decoding method according to claim 1, further comprising: include: The output token is determined to be the subsequent input token.
8. The decoding method according to claim 1, in, The features are determined using an attention network based on a speech feature vector associated with a progression of speech information.
9. The decoding method according to claim 1, further comprising: include: The features are generated by encoding speech information using an encoding layer included in the artificial neural network.
10. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to perform the decoding method of claim 1.
11. A training method for an artificial neural network for speech recognition, the training method include: selecting at least one sample from a batch including a first training set and a second training set, the first training set including pairs of speech and text corresponding to the speech, the second training set including text; In response to selecting a sample from the first training set, training an encoding layer and a shared decoding layer included in the artificial neural network based on the sample; and In response to selecting a sample from the second training set, training a shared decoding layer based on the sample, wherein, after the shared decoding layer is trained, the shared decoding layer performs a first decoding task by adjusting a weight of a synapse connecting the shared decoding layer and a neuron corresponding to a feature including speech information to a first value, and performs a second decoding task by adjusting the weight of the synapse to a second value, The first decoding task includes: processing the relationship between the signal characteristics of speech information and language elements, The second decoding task includes: a task of estimating the level of semantic correctness of speech information.
12. The training method according to claim 11, in, The steps to train the encoding layer and the shared decoding layer include: Using the encoding layer, features are extracted from the speech included in the sample, Using a shared decoding layer, estimating an output token to be recognized after the at least one token based on the extracted features and the at least one token; and The encoding layer and the shared decoding layer are trained based on the estimated output lexical units and at least a portion of the text corresponding to the speech included in the sample.
13. The training method according to claim 11, in, The steps to train the shared decoding layer include: Using a shared decoding layer, estimating an output token to be recognized after the at least one token based on the at least one token; and A shared decoding layer is trained based on the estimated output tokens and at least a portion of the text included in the sample.
14. The training method according to claim 11, wherein the steps of training the encoding layer and the shared decoding layer included in the artificial neural network based on the samples are: include: extracting features from speech included in the sample using an encoding layer; Using a shared decoding layer, estimating a first probability of a candidate of an output token to be identified after the at least one token based on the extracted features and the at least one token; estimating, using the shared decoding layer, a second probability of a candidate of an output token to be recognized after the at least one token based on the at least one token; Based on the weight of the first probability and the second probability, estimating the output token; and The weight is learned based on at least a portion of the text corresponding to the speech included in the sample.
15. A speech recognition device, include: an encoder configured to generate features from speech information; as well as a decoder configured to: perform a first decoding task of decoding the feature and at least one lexical unit recognized until the current time point using a shared decoding layer, perform a second decoding task of decoding the at least one lexical unit recognized until the current time point, and determine an output lexical unit to be recognized after the at least one lexical unit based on a result of the first decoding task and a result of the second decoding task, The decoder is configured as follows: Performing a first decoding task by adjusting a weight of a synapse connecting a shared decoding layer and a neuron corresponding to the feature to a first value; and performing a second decoding task by adjusting the weight of the synapse to a second value, The first decoding task includes: processing the relationship between the signal characteristics of the speech information and the language elements, and The second decoding task includes: a task of estimating the level of semantic correctness of speech information.
16. The speech recognition device according to claim 15, further comprising: include: A speech preprocessor is configured to extract a speech feature vector from the speech information, The encoder is configured to generate the feature based on the speech feature vector.
17. The speech recognition device according to claim 15, in, The decoder is also configured to: performing a first decoding task by receiving the feature and the at least one lexical unit recognized up to a current time point via a shared decoding layer; and The second decoding task is performed by receiving the at least one lexical unit recognized up to the current time point through the shared decoding layer.
18. The speech recognition device according to claim 15, in, The decoder is also configured to: performing a first decoding task by receiving the feature and the at least one lexical unit recognized up to a current time point via a first pre-decoding layer and receiving an output of the first pre-decoding layer via a shared decoding layer; and The second decoding task is performed by receiving the at least one lexical unit recognized up to the current time point through the second pre-decoding layer and receiving an output of the second pre-decoding layer through the shared decoding layer.
19. The speech recognition device according to claim 15, in, The decoder is also configured to: performing a first decoding task by receiving the feature and the at least one lexical unit recognized up to a current time point via a shared decoding layer and receiving an output of the shared decoding layer via a first post-decoding layer; and The second decoding task is performed by receiving the at least one lexical unit recognized up to the current time point via the shared decoding layer and receiving the output of the shared decoding layer via the second post-decoding layer.
20. The speech recognition device according to claim 15, in, The result of the first decoding task includes outputting a first probability of a candidate for a token, and The result of the second decoding task includes outputting a second probability of a candidate for a lexical unit.
21. The speech recognition device according to claim 20, in, The decoder is also configured to: calculating a weighted sum of the first probability and the second probability; and The candidate corresponding to the maximum weighted sum among the candidates is determined as the output token.
22. The speech recognition device according to claim 15, in, The decoder is also configured to: The output token is determined to be the subsequent input token.
23. The speech recognition device according to claim 15, in, The features are determined using an attention network based on a speech feature vector associated with a progression of speech information.
24. A speech recognition device, include: An encoder configured to receive a speech feature vector corresponding to speech information and output a context vector; and The decoder is configured as: Receive a context vector; Decoding the context vector and the most recently recognized lexical unit using a shared decoding layer included in the artificial neural network to output a first result corresponding to a first decoding task; Decoding only the most recently recognized lexical unit using the shared decoding layer to output a second result corresponding to a second decoding task; and Output the current lexical unit based on the first result and the second result, The decoder is further configured as follows: Performing a first decoding task by adjusting a weight of a synapse connecting a shared decoding layer and a neuron corresponding to the context vector to a first value; and performing a second decoding task by adjusting the weight of the synapse to a second value, The first decoding task includes: processing the relationship between the signal characteristics of the speech information and the language elements, and The second decoding task includes: a task of estimating the level of semantic correctness of speech information.
25. The speech recognition device according to claim 24, in, The decoder is completely configured via the shared decoding layer.
26. The speech recognition device according to claim 24, in, Only a portion of the decoder is configured by sharing the decoding layer.
Citation Information
Patent Citations
Method for Providing Service in Car by using Air Cleaning Device for Vehicles
KR1020190093066A
Speech recognition method and device
CN108615525A
Method and apparatus for training language model and recognizing speech
US20160163310A1
System and Method for Multichannel End-to-End Speech Recognition
US20180261225A1
Multilingual, acoustic deep neural networks
US9460711B1
