Method and apparatus for speech time estimation

Abstract

A method and apparatus for speech duration estimation are provided. The method includes determining a plurality of attention weight matrices using an attention-based sequence-to-sequence model, selecting an attention weight matrix from the plurality of attention weight matrices, and estimating a speech duration corresponding to an output sequence based on the selected attention weight matrix.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0058322, filed May 15, 2020, the disclosure of which is incorporated herein in its entirety by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to a method and apparatus for speech time estimation. BACKGROUND

[0003] Speech recognition can refer to a technology for recognizing or understanding an acoustic speech signal by analyzing the acoustic speech signal, such as a vocal utterance made by a human, using a computing device. A method of recognizing speech based on frequency characteristics or features extracted from audio data can apply a hidden Markov model (HMM). Such an HMM-based speech recognition method can recognize speech by analyzing a pronunciation from audio data and by combining a word or a sentence based on the analyzed pronunciation.

[0004] Such an HMM-based speech recognition method can directly recognize text including a word or a sentence from audio data using an acoustic model including an artificial neural network without analyzing a pronunciation from the audio data. SUMMARY

[0005] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0006] In one general aspect, a processor-implemented speech time estimation method includes determining a plurality of attention weight matrices using an attention-based sequence-to-sequence model, selecting an attention weight matrix from the plurality of attention weight matrices, and estimating a speech time corresponding to an output sequence based on the selected attention weight matrix.

[0007] The step of selecting the selected attention weight matrix can include performing a monotonic regression analysis on the plurality of attention weight matrices, and selecting the selected attention weight matrix based on a result of the monotonic regression analysis.

[0008] The step of estimating the speech time can include performing a monotonic regression analysis on the selected attention weight matrix, and estimating the speech time corresponding to the output sequence based on a result of the monotonic regression analysis.

[0009] The selected attention weight matrix can include, as elements, attention weights of input frames of the input sequence and corresponding to output lexical units of the output sequence, and the estimating of the speaking time can include: selecting a predetermined number of high attention weights from the attention weights of the input frames; and estimating the speaking time corresponding to the output sequence based on the selected attention weights.

[0010] The estimating of the speaking time based on the selected attention weights can include: correcting the selected attention weight matrix based on the selected attention weights; and estimating the speaking time corresponding to the output sequence based on the corrected attention weight matrix.

[0011] The predetermined number of high attention weights can be a predetermined number of highest attention weights among the attention weights corresponding to each output lexical unit.

[0012] The estimating of the speaking time can include: performing diffusion correction on the selected attention weight matrix; and estimating the speaking time corresponding to the output sequence based on the selected attention weight matrix on which the diffusion correction is performed.

[0013] The selected attention weight matrix can include, as elements, attention weights of input frames of the input sequence and corresponding to output lexical units of the output sequence, and the estimating of the speaking time can include: selecting a predetermined number of high attention weights from the attention weights of the input frames; performing diffusion correction on the selected attention weight matrix based on the selected attention weights; and estimating the speaking time corresponding to the output sequence based on the selected attention weight matrix on which the diffusion correction is performed.

[0014] The number of the plurality of attention weight matrices can correspond to a product of a number of attention layers of the attention-based sequence-to-sequence model and a number of decoder layers of the attention-based sequence-to-sequence model.

[0015] The method can further include training the attention-based sequence-to-sequence model such that a predetermined matrix among the plurality of attention weight matrices is generated as the selected attention weight matrix.

[0016] The training of the attention-based sequence-to-sequence model can include: masking an attention layer corresponding to the predetermined matrix, and training the attention-based sequence-to-sequence model based on the masked attention layer.

[0017] A non-transitory computer-readable storage medium can store instructions which, when executed by a processor, configure the processor to perform the method.

[0018] In another general aspect, a processor-implemented speaking time estimation method includes receiving an input sequence; generating encoded features by encoding the input sequence; determining, for each attention layer of at least one attention layer, attention weights of input frames of the input sequence; estimating, for each step of one or more steps, an output lexical unit of an output sequence based on the encoded features and the attention weights; determining, based on the attention weights, input frames corresponding to the output lexical unit of each attention layer of the at least one attention layer; and estimating a speaking time corresponding to the output sequence based on the input frames corresponding to the output lexical unit.

[0019] The step of estimating the speaking time can include determining, based on the input frames corresponding to the output lexical unit, an attention weight matrix of each attention layer of the at least one attention layer; selecting, from the attention weight matrix of each attention layer of the at least one attention layer, an attention weight matrix corresponding to monotonicity; and estimating the speaking time corresponding to the output sequence based on the selected attention weight matrix.

[0020] The step of selecting, from the attention weight matrix of each attention layer of the at least one attention layer, the attention weight matrix corresponding to monotonicity can include performing a monotonic regression analysis on the attention weight matrix; and selecting, as the attention weight matrix corresponding to monotonicity, the attention weight matrix having a minimum error of the monotonic regression analysis.

[0021] In another general aspect, a speaking time estimation device includes a processor configured to determine a plurality of attention weight matrices using an attention-based sequence-to-sequence model; select an attention weight matrix from the plurality of attention weight matrices; and estimate a speaking time corresponding to an output sequence based on the selected attention weight matrix.

[0022] To select the selected attention weight matrix, the processor can be configured to perform a monotonic regression analysis on the plurality of attention weight matrices, and select the selected attention weight matrix based on a result of the monotonic regression analysis.

[0023] To estimate the speaking time, the processor can be configured to perform a monotonic regression analysis on the selected attention weight matrix, and estimate the speaking time corresponding to the output sequence based on a result of the monotonic regression analysis.

[0024] The selected attention weight matrix can include, as elements, attention weights of input frames of the input sequence and corresponding to an output lexical unit of the output sequence, and to estimate the speaking time, the processor can be configured to select a predetermined number of high attention weights from the attention weights of the input frames; and estimate the speaking time corresponding to the output sequence based on the selected attention weights.

[0025] To estimate the speaking time, the processor can be configured to correct the selected attention weight matrix based on the selected attention weights, and estimate the speaking time corresponding to the output sequence based on the corrected attention weight matrix.

[0026] To estimate the speaking time, the processor can be configured to perform diffusion correction on the selected attention weight matrix, and estimate the speaking time corresponding to the output sequence based on the selected attention weight matrix on which the diffusion correction is performed.

[0027] The selected attention weight matrix can include, as elements, attention weights of input frames of the input sequence and corresponding to output lexical units of the output sequence, and to estimate the speaking time, the processor can be configured to select a predetermined number of high attention weights from the attention weights of the input frames, perform diffusion correction on the selected attention weight matrix based on the selected attention weights, and estimate the speaking time corresponding to the output sequence based on the selected attention weight matrix on which the diffusion correction is performed.

[0028] The number of the plurality of attention weight matrices can correspond to a product of a number of attention layers of the attention-based sequence-to-sequence model and a number of decoder layers of the attention-based sequence-to-sequence model.

[0029] The processor can be configured to train the attention-based sequence-to-sequence model such that a predetermined matrix among the plurality of attention weight matrices is generated as the selected attention weight matrix.

[0030] The processor can be configured to mask an attention layer corresponding to the predetermined matrix, and train the attention-based sequence-to-sequence model based on the masked attention layer.

[0031] In another general aspect, a mobile device includes at least one sensor configured to receive an input sequence, and a processor configured to generate encoded features by encoding the input sequence, determine, for each of at least one attention layer, corresponding attention weights of input frames of the input sequence, estimate, for each of one or more steps, output lexical units of an output sequence based on the encoded features and the attention weights, determine, based on the attention weights, input frames corresponding to the output lexical units of each of the at least one attention layer, and estimate a speaking time corresponding to the output sequence based on the input frames corresponding to the output lexical units.

[0032] To estimate the speaking time, the processor can be configured to: determine an attention weight matrix of each of the at least one attention layer based on the input frames corresponding to the output lexicon units; select an attention weight matrix corresponding to the monotonicity from the attention weight matrices of each of the at least one attention layer; and estimate the speaking time corresponding to the output sequence based on the selected attention weight matrix.

[0033] The at least one sensor can include a microphone, the input sequence can include speech, and the mobile device further includes a user interface configured to output the output sequence and the speaking time.

[0034] In another general aspect, a processor-implemented method of speaking time estimation includes: generating encoded features by encoding an input sequence including input frames; estimating output lexicon units of an output sequence corresponding to the input sequence based on the encoded features; determining an attention weight matrix including attention weights, each attention weight corresponding to one of the input frames and one of the output lexicon units; and estimating a speaking time corresponding to the output sequence based on the attention weight matrix.

[0035] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 An example of a speech recognition system is shown.

[0037] Figure 2 An example of operation of a speech recognition device having an end-to-end artificial neural network (ANN) with an encoder-decoder structure is shown.

[0038] Figure 3 An example of a speech recognition device is shown.

[0039] Figure 4A And Figure 4B An example of determining attention weights is shown.

[0040] Figure 5 An example of a method of estimating a speaking time is shown.

[0041] Figure 6 An example of selecting a predetermined attention weight matrix from a plurality of attention weight matrices is shown.

[0042] Figure 7 An example of estimating a speaking time corresponding to an output sequence based on a first attention weight matrix is shown.

[0043] Figure 8A An example of a speaking time estimation result is shown.

[0044] Figure 8B An example of using the speaking time estimation result is shown.

[0045] Figure 9A and Figure 9B An example of training an attention-based sequence-to-sequence model is shown.

[0046] Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing references are understood to refer to the same elements, features, and structures. The drawings can not be to scale, and the relative dimensions, proportions, and depiction of elements in the drawings can be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0047] The following DETAILED DESCRIPTION provides implementation insights to assist in gaining an extensive understanding of the methods, devices, and / or systems described herein. After considering this disclosure, those skilled in the art will be able to employ the methods, devices, and / or systems described herein without undue experimentation. For example, the order of operations described herein is merely an example and is not limited to those set forth herein, but can be altered as will be apparent to one of ordinary skill in the art having the benefit of this disclosure. In addition, descriptions of known features can be omitted for the sake of clarity and brevity.

[0048] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, these examples have been provided so that this disclosure will be thorough and complete, and will fully convey the scope of the methods, devices, and / or systems described herein to one of ordinary skill in the art after having the benefit of this disclosure.

[0049] The following detailed structural or functional description merely describes examples and the scope of the disclosure or such examples is not limited to the description provided in this specification. Various changes and modifications can be made to the examples. Here, the examples are not interpreted as being limited to the disclosure, but should be understood to include all changes, equivalents, and substitutions within the scope of the concept and technology of the disclosure.

[0050] The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the terms "include," "comprise," and "have" are meant to be interpreted inclusively rather than exclusively, unless the context clearly indicates otherwise. The use of "can" with respect to what an example or embodiment can include or do, as used herein, means that there is at least one example or embodiment that includes or does the feature, and no example or embodiment is limited to this.

[0051] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs and the same meaning as understood when read in the context of the present disclosure. Unless otherwise expressly defined herein, terms such as, for example, "first" and "second" are used to distinguish one element from another element, not to describe a particular sequential or chronological order. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0052] When the examples are described with reference to the drawings, the same reference numbers are used throughout the drawings and the same reference numbers are used to describe the same elements, components, regions, layers and / or parts throughout the description of the examples. In the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is deemed that such omissions do not cause an ambiguity of the disclosure.

[0053] Although the terms "first" or "second" are used herein to describe various components, elements, regions, layers or parts, the components, elements, regions, layers or parts should not be limited by the terms. Rather, the terms are used only to distinguish one component, element, region, layer or part from another component, element, region, layer or part. Therefore, the first component, first element, first region, first layer or first part referred to in the examples described herein can also be called the second component, second element, second region, second layer or second part without departing from the teachings of the examples.

[0054] Components having the same function as components included in another example are described using the same name in one example. Unless otherwise described, the description made in one example can be applicable to another example, and the detailed description in the overlapping range is omitted.

[0055] Figure 1 An example of a speech recognition system is shown.

[0056] Referring toFigure 1 The speech recognition system can include a user terminal 110 and a speech recognition device 120. In one example, the speech recognition device 120 can be a device (e.g., a server apparatus) external to the user terminal 110, and in another example, the speech recognition device 120 can be embedded in or included in the user terminal 110 according to a non-limiting example.

[0057] In the speech recognition system, the user terminal 110 can be a terminal configured to receive an input sequence 130 (e.g., audio speech of a user or a speaker) of a speaker, transmit the input sequence 130 to the speech recognition device 120, and provide an output sequence 140 (e.g., where are you) to the user (e.g., the speaker), the output sequence 140 being a recognition result determined by the speech recognition device 120. The user terminal 110 can receive the input sequence 130 through at least one sensor (e.g., a microphone) of the user terminal 110, and can provide the output sequence through a user interface or a display of the user terminal 110. Although the user terminal 110 is shown as a smartphone in FIG. 1, Figure 1 The user terminal 110 can be any one or any combination of electronic apparatuses (e.g., a computer, an ultra-mobile personal computer (UMPC), a workstation, a netbook, a personal digital assistant (PDA), a portable computer, a web tablet, a wireless phone, a mobile phone, an e-book, a portable multimedia player (PMP), a portable game console, a navigation apparatus, a black box, and / or a digital camera).

[0058] In the speech recognition system, the speech recognition device 120 can be a computing apparatus configured to receive the input sequence 130 of the speaker and provide the output sequence 140 as a recognition result. The input sequence 130 can include input speech and input speech feature vectors extracted from the input speech. The input speech can comprehensively include mel-frequency cepstral coefficients (MFCCs) or a spectrogram in which a wave file representing an input speech signal of the speaker in a waveform is represented in the form of frequency. The computing apparatus can be a mobile apparatus (e.g., a notebook computer, a laptop computer, a desktop computer, and / or a smartphone), however, examples are not limited thereto. For example, the computing apparatus can include all types of apparatuses including the operation or computing apparatus.

[0059] The speech recognition device 120 can provide end-to-end speech recognition. In typical speech recognition, a hidden Markov model (HMM) can be used. Such a typical HMM-based speech recognition can analyze a pronunciation from audio data and combine a word or a sentence based on the analyzed pronunciation. However, the pronunciation can vary according to a speaker and a type of a speaker's language, and thus such a typical speech recognition by analyzing the pronunciation can disadvantageously include a correction process including a large amount of trial and error. In addition, because the speech is recognized based on the pronunciation, the typical HMM-based speech recognition method can be vulnerable to environmental noise.

[0060] In comparison with the typical speech recognition, the end-to-end artificial neural network (ANN)-based speech recognition system of one or more embodiments can exhibit superior performance by using the ANN that is learned or trained to model signal processing, pronunciation conversion, a language model, and / or decoding for implementing the speech recognition while minimizing intervention of expertise in the signal processing, the pronunciation conversion, the language model, and / or the decoding, thereby improving a function of the speech recognition system and device and improving a technical field of the speech recognition.

[0061] To provide the end-to-end speech recognition, the speech recognition device 120 can construct a sequence-to-sequence model including an ANN, and provide an output sequence 140 as a recognition result using the constructed sequence-to-sequence model in response to an input sequence 130. The ANN can be, for example, a recurrent neural network (RNN), a convolutional neural network (CNN), a self-attention neural network (SANN), a bidirectional RNN (BRNN), a long short-term memory (LSTM) network, a bidirectional LSTM (BLSTM) network, a gated recurrent unit (GRU), and / or a bidirectional GRU (BGRU). However, examples are not limited to the examples described in the foregoing.

[0062] In addition, the speech recognition device 120 can be configured to estimate utterance time corresponding to the output sequence, and / or can include an utterance time estimation device configured to estimate the utterance time corresponding to the output sequence.

[0063] The speech recognition device 120 of one or more embodiments can determine and provide information on a point in time (e.g., an accurate or substantially accurate point in time) at which each word recognized in the input sequence 130 is spoken, using the utterance time estimation device, and the output sequence 140 as a recognition result of the input sequence 130.

[0064] When each recognized word is predicted by comprehensively considering an input sequence, a typical sequence-to-sequence model-based speech recognition method can not be able to determine (or can not be configured to determine) a time point at which each recognized word is spoken. In particular, when a plurality of attention layers trained to implicitly perform various functions are used to predict each recognized word, a typical multi head attention sequence-to-sequence model can not be able to estimate (or can not be configured to estimate) a time point at which each recognized word is spoken.

[0065] In comparison with such typical methods and models, even when a sequence-to-sequence model-based speech recognition method is performed, a speech recognition system of one or more embodiments can use a speaking time estimation device to determine and provide information about a time point at which each recognized word in an output sequence 140 is spoken, thereby improving the functions of a speech recognition system and device, and improving the technical field of speech recognition. Hereinafter, an example of such a speech recognition system of one or more embodiments will be described with reference to Figures 2 to 9B A non-limiting example of such a speech recognition system of one or more embodiments will be further described.

[0066] Figure 2 An example of the operation of a speech recognition device having an ANN with an end-to-end encoder-decoder structure is illustrated.

[0067] Referring to Figure 2 , the ANN can include an encoder 220 and a decoder 230.

[0068] The ANN can be an end-to-end ANN having a structure including the encoder 220 and the decoder 230 (hereinafter, referred to as an "encoder 220-decoder 230 structure"). The encoder 220 and the decoder 230 can be trained simultaneously when a path through which all parameters of the model are trained simultaneously for one loss function is obtained. An input can be received at one side (or input) of the ANN, and an output can be generated at the other side (or output) of the ANN. Network weights can be optimized by directly considering the input and the output. Nodes of a layer in the ANN can have a non-linear influence on each other, and parameters (e.g., values output from nodes or relationships between nodes) of the ANN can be optimized through training.

[0069] The end-to-end ANN having the encoder 220-decoder 230 structure can be a network structure in which the encoder 220 and the decoder 230 are integrated, and the encoder 220 and the decoder 230 can directly generate a recognition result corresponding to input speech from the input speech.

[0070] The speech recognition device can extract an input speech feature vector 215 from the input speech 210, and an encoder 220 can generate encoded features 225 by encoding the input speech feature vector 215. For example, the encoder 220 can generate the encoded features 225 by transforming the dimension of the input speech feature vector 215. In other words, the encoder 220 can be used to effectively summarize the meaning of the input sequence.

[0071] The encoded features 225 can be applied to a decoder 230 of the ANN. The decoder 230 can determine output tokens based on the encoded features 225 and a predetermined recognition result, in units of tokens.

[0072] For example, when the encoded features 225 are received, the decoder 230 can output a recognition result in units of tokens. The tokens can be, for example, words, sub-words, characters, or units of configured characters (e.g., initial, medial, and final sounds of a Korean syllable structure). In the following description, the entire recognition result output by the decoder 230 can be referred to as an output sequence, and the output sequence can include one or more tokens of the output.

[0073] To calculate an output having an unspecified length using an ANN, the decoder 230 can continue to predict subsequent tokens one by one in response to an input of a predicted output token until a special token indicating an end (e.g., an end-of-sentence token) is predicted. The decoder 230 can output the output sequence including the predicted tokens until the end-of-sentence token is predicted. <e>or the end of speech <eos>(or <eos>) until the appearance or determination of the stop. For example, the decoder 230 can determine the output lexicon unit of each step based on the information computed from the encoder 220 (e.g., each step here can be a process for estimating one output lexicon unit). In this example, the decoder 230 can determine the output lexicon unit from the output lexicon unit of the previous step, and such a method can be referred to as "autoregressive decoding".

[0074] For example, in an end-to-end ANN model that performs autoregressive decoding, the lexicon unit selected as the output lexicon unit can be determined as the subsequent input lexicon unit. For example, when recognizing the sentence "tell me a joke" of the input speech 210, "me" can be recognized based on the input "tell" that is the output lexicon unit recognized in the previous step.

[0075] However, when a typical end-to-end ANN having an encoder-decoder structure uses a single vector to process all parts of the input speech, as the estimated speech becomes longer through sequential estimation of the output lexicon units, information of the front or beginning part of the input speech for estimating the subsequent output lexicon unit can be gradually diluted, which can result in a decrease in speech recognition quality. To address the above problem, for each step of estimating the output lexicon unit by the decoder 230, the speech recognition device of one or more embodiments can estimate the output lexicon unit by referring to or using all parts of the input speech (e.g., all parts of the encoded features 225) in the encoder 220. Further, to even more accurately estimate each output lexicon unit, for each step of estimating the output lexicon unit by the decoder 230, the speech recognition device of one or more embodiments can not estimate the output lexicon unit by referring to or using all parts of the input speech at the same rate or attention. Rather, for each step of estimating the output lexicon unit by the decoder 230, the speech recognition device of one or more embodiments can estimate the output lexicon unit by referring to or using parts of the input speech associated with the output lexicon unit with higher attention than other parts of the input speech. Accordingly, the speech recognition device of one or more embodiments can improve the functionality of the speech recognition system and device, and improve the technical field of speech recognition.

[0076] For example, the attention weight of the input frame constituting the input sequence can be set based on the degree of association with the output lexicon unit to be predicted in the corresponding step, and the decoder 230 can estimate the output lexicon unit of each step based on the set attention weight.

[0077] The encoder 220 and the decoder 230 can be pre-trained to generate a sequence of recognition results from the received input speech feature vectors 215. For example, model parameters (e.g., weights of synapse connections and biases of nodes) of the encoder 220 and the decoder 230 can be pre-trained by training the encoder 220 and the decoder 230 to generate a sequence of recognition results from a sequence of correct answer text pairs corresponding to input speech. Also, attention weights of the encoder 220 and the decoder 230 can be pre-trained by training the encoder 220 and the decoder 230 to generate a sequence of recognition results from a sequence of correct answer text pairs corresponding to input speech.

[0078] The speaking time estimation device of one or more embodiments can estimate a speaking time corresponding to an output sequence based on the attention weights. For example, even in a sequence-to-sequence model based on multi-head attention, the speaking time estimation device can estimate a time point at which each recognized word is spoken.

[0079] Figure 3 An example of a speech recognition device is illustrated.

[0080] Referring to Figure 3 The speech recognition device can include a feature vector extraction module 310, an encoder 320, a decoder 330, and a speaking time estimation device 340. The speech recognition device can further include other components. Figure 3 Components of the speech recognition device can be implemented as separate components, or can be integrated. Hereinafter, each of the illustrated components of the speech recognition device will be described in detail.

[0081] The speech recognition device can use the speaking time estimation device 340 to provide an output sequence that is a recognition result of an input sequence and to provide information about a time point at which each word recognized in the output sequence is spoken.

[0082] The feature vector extraction module 310 can extract an input speech feature vector from input speech received from a speaker. The input speech can be a speech signal including information of each of a plurality of frames, the input speech feature vector can be a sequence of information extracted in at least one frame unit, and can be represented as a multi-dimensional vector. In the following description, an input sequence can include input speech and an input speech feature vector extracted from the input speech, and can include information in an input frame unit.

[0083] As a non-limiting example, Figure 3 The encoder 320 and the decoder 330 of the speech recognition device can correspond to the encoder 220 and the decoder 230, respectively. Figure 2 The encoder 320 and the decoder 330 can be configured to perform the same or similar operations as the encoder 220 and the decoder 230 of FIG. 2, and thus further description of the encoder 320 and the decoder 330 is not repeated here. The decoder 330 can include an attention module 331 configured to determine, whenever an output lexicon is estimated in each step, an input frame that is to be further attended to. For example, the attention module 331 can determine attention weights of input frames that constitute an input sequence in a corresponding step and are used when the decoder 330 estimates an output lexicon for each step. The decoder 330 can estimate one or more output lexicons for each step based on the set attention weights. The attention module 331 can be a multi-headed attention module. The multi-headed attention module can split an entire dimension into multiple dimensions and apply attention multiple times, rather than applying attention to the entire dimension once using multiple attention layers trained to perform various functions.

[0084] The speaking time estimation device 340 can include a monotonic regression analysis module 341 and a diffusion module 343. The monotonic regression analysis module 341 can further include a first attention weight matrix selection module. The speaking time estimation device 340 can estimate a speaking time corresponding to an output sequence based on attention weights received from the attention module 331. Before describing operations of the speaking time estimation device 340 using the monotonic regression analysis module 341 and the diffusion module 343, a non-limiting example of determining attention weights will be further described with reference to FIGS. 3A and 3B. Figure 4A and Figure 4B A non-limiting example of determining attention weights will be further described with reference to FIGS. 3A and 3B.

[0085] Figure 4A and Figure 4B An example of determining attention weights will be shown.

[0086] Referring to Figure 4A , the encoder 320 can receive an input sequence including input frames (e.g., frame i1to frame i N , where N denotes a number of the input frames), and can generate hidden state vectors (e.g., vector h1to vector h N ) of the encoder 320.

[0087] The attention module 331 can determine attention weights of the input frames (e.g., frame i1to frame i N ), and can generate context vectors (e.g., vector c1to vector c M , where M denotes a number of the output lexicons) based on the determined attention weights, the input frames (e.g., frame i1to frame i N ) constitutes the input sequence in the corresponding step and is used when the decoder 330 estimates the output token for each step. For example, the attention module 331 can be a multi-head attention module and can include multiple attention layers (e.g., the first attention layer to the Kth attention layer). Each attention layer can determine the attention weight in parallel and can generate a context vector based on the determined attention weight.

[0088] The decoder 330 may receive the hidden state vector s of the decoder 330 of the previous step t-1 , the output lexical unit y of the previous step t-1 and the context vector c of the current step t As input, the hidden state vector s of the decoder 330 of the current step can be extracted based on the received input, for example t and the output lexical unit y of the current step t For example, the decoder 330 extracts the output lexical unit y at the current step t The context vector c of the current step used t It can be determined as shown in Equation 1 below.

[0089] Equation 1:

[0090]

[0091] In Equation 1, represents the attention weight of the i-th input frame at step t, h i represents the hidden state vector of the encoder 320 for the i-th input frame.

[0092] Reference Figure 4B , represents the attention weight of the i-th input frame of step t as the current step, and “1” can be obtained by taking all input frames of step t (e.g., frames i1 to i) as the attention weight of the i-th input frame of the current step. N )’s attention weight to For example, when the attention weight With attention weight to When the highest value among , the decoder 330 may extract the output lexical unit by focusing most on the second input frame at step t.

[0093] Therefore, based on the distribution of the attention weights of the input frames at step t, it is possible to determine which input frame is most focused on when extracting the output token at the corresponding step, and thus the input frame corresponding to the output token at the corresponding step can be determined. For example, the input frame with the highest attention weight in the corresponding step can be mapped to the output token.

[0094] However, two problems can occur when a typical speech recognition device maps an input frame having the highest attention weight in a corresponding step to an output lexicon unit. First, when a typical speech recognition device maps an input frame having the highest attention weight in a corresponding step to an output lexicon unit, a result determined by mapping an input sequence and an output sequence over time in all steps can not always have monotonicity. For example, an input frame mapped in a previous step can be later in time than an input frame mapped in a current step. For example, based on the above-described method, the highest attention weight in a step t can be The highest attention weight in a step t+1 can be where k is a natural number.

[0095] Second, when a time unit of an output lexicon unit is larger than a time unit of an input frame (e.g., at least 5 times (e.g., in a case where a unit of an output lexicon unit is a character) to 30 times (e.g., in a case where a unit of an output lexicon unit is a word) larger than a time unit of an input frame), a typical speech recognition device can inaccurately map an input sequence and an output sequence by determining a single input frame in a single step in which an output lexicon unit is extracted.

[0096] To solve the above problems of a typical speech recognition device, a speaking time estimation device 340 of one or more embodiments can determine an attention weight matrix. The attention weight matrix can have, as elements, attention weights for input frames constituting an input sequence with respect to output lexicon units constituting an output sequence. For example, when a speech recognition device outputs an output sequence (e.g., "tell me a joke"), the attention weight matrix can be represented by Table 1 below.

[0097] Table 1:

[0098] ​ i2 i3 i4 i5 i6 i7 i8 i9 tell 0.9 0.07 0.02 0.004 0.001 0.001 0.001 0.001 0.001 me 0.004 0.02 0.9 0.07 0.001 0.001 0.001 0.001 0.001 a 0.001 0.001 0.001 0.004 0.33 0.3 0.36 0.001 0.001 joke 0.001 0.001 0.001 0.001 0.001 0.62 0.37 0.004 0.001

[0099] In Table 1, the output lexical unit can be a unit of a word, and each input frame can have a unit of 10 milliseconds (ms). In the output lexical unit "tell", the input frame i1 has the highest attention weight "0.9", the input frame i2 has the second highest attention weight "0.07", and the input frame i3 has the third highest attention weight "0.02". "0.99" can be determined by adding all of the attention weights of the input frames i1 to i3, and thus it can be found that the output lexical unit "tell" is extracted by focusing on the input frames i1 to i3. Accordingly, the speaking time of the output lexical unit "tell" can be estimated as an amount of time of "0" to "30" ms. In one example, for a given output lexical unit (e.g., "tell"), the speaking time estimation device 340 can iteratively sum the highest attention weight (e.g., "0.9") with the next highest attention weight until the iterative sum meets a threshold (e.g., "0.99"), and can determine the speaking time of the given output lexical unit as a time from the input frame of the highest attention weight among the attention weights used to obtain the iterative sum to the input frame of the lowest attention weight.

[0100] The speaking time estimation device 340 can correct the attention weight matrix based on a predetermined condition, can perform monotonic regression analysis on the corrected attention weight matrix, and can estimate the speaking time corresponding to the output sentence based on the attention weight matrix.

[0101] For example, when a plurality of weight matrices are generated by performing multi-head attention, the speaking time estimation device 340 can select a first weight matrix that satisfies a predetermined condition, and can estimate the speaking time corresponding to the output sentence based on the selected first weight matrix. Non-limiting examples of estimating the speaking time will be further described below with reference to Figures 5 to 9B Non-limiting examples of the method of estimating the speaking time will be further described.

[0102] Figure 5 Examples of the method of estimating the speaking time are illustrated.

[0103] Referring to Figure 5 , operations 510 to 560 can be performed by, for example, a speech recognition device. As non-limiting examples, the speech recognition device can be implemented by or include at least one hardware module, which can include one or more processors configured to execute instructions to perform operations 510 to 560. As described above, the speech recognition device can include a speaking time estimation device. In one or more non-limiting examples, Figures 1 to 4B The description of Figure 5 may apply to examples of Figure 5 The operations of Figures 1 to 4B may be implemented by a speech recognition device as described above with reference to

[0104] In operation 510, the speech recognition device can receive an input speech.

[0105] In operation 515, the speech recognition device can extract an input speech feature vector corresponding to the input speech.

[0106] In operation 520, the speech recognition device can encode the input speech feature vector, and set a current step (e.g., step t) to "0".

[0107] In operation 525, the speech recognition device can calculate an attention weight at a corresponding step (e.g., the current step).

[0108] In operation 530, the speech recognition device can receive a hidden state vector of a decoder of a previous step, an output token of the previous step, and a context vector of the current step as inputs, and can extract a hidden state vector of a decoder of the current step and an output token of the current step (e.g., output token t ).

[0109] In operation 535, the speech recognition device can estimate an input frame corresponding to the output token of the corresponding step based on relatively high "n" attention weights (e.g., where "n" is a predetermined number of highest attention weights and / or where "n" is a minimum number of highest attention weights satisfying a predetermined threshold) in the corresponding step (e.g., the current step) for each attention layer. In addition, the speech recognition device can store a position of the input frame and the attention weight.

[0110] Through operations 540 and 545, the speech recognition device can repeatedly perform operations 525 to 535 until the output token of the current step becomes a token <eos>The speech time estimation device can determine an attention weight matrix including attention weights of all steps. For example, when multi-head attention is performed or when the decoder includes a plurality of labels, the speech time estimation device can determine a plurality of attention weight matrices. In this example, the number of the plurality of attention weight matrices can correspond to a product of the number of attention layers and the number of decoder layers. In this example, the decoder layers can be associated with output lexemes, for example, the number of decoder layers can be the same as the number of output lexemes. In one example, the speech time estimation device can determine the plurality of attention weight matrices using an attention-based sequence-to-sequence model. In this example, the input of the attention-based sequence-to-sequence model can be the input frames, and the attention-based sequence-to-sequence model can output the output lexemes based on the input frames and the attention weights included in the plurality of attention weight matrices. In one example, the speech time estimation device can determine the attention weight matrix of each of the at least one attention layer based on the input frames corresponding to the output lexemes.

[0111] In operation 550, the speech time estimation device can select a predetermined attention weight matrix by performing monotonic regression analysis on the plurality of attention weight matrices. For example, the speech time estimation device can perform monotonic regression analysis on the plurality of attention weight matrices using a monotonic regression analysis module included in the speech time estimation device. The selection of the predetermined attention weight matrix from the plurality of attention weight matrices will be further described below with reference to FIGS. 6A and 6B. Figure 6 The non-limiting example of selecting the predetermined attention weight matrix from the plurality of attention weight matrices will be further described below.

[0112] In operation 555, the speech time estimation device can perform post-processing on the selected attention weight matrix. For example, the speech time estimation device can correct the selected attention weight matrix based on a preset condition, and can perform monotonic regression analysis on the corrected attention weight matrix. In one example, the speech time estimation device can perform monotonic regression analysis on the selected attention weight matrix, and estimate the speech time corresponding to the output sequence based on the result of the monotonic regression analysis.

[0113] In operation 560, the speech time estimation device can estimate the speech time corresponding to the output sequence by obtaining time information corresponding to the output lexemes, for example, based on the selected attention weight matrix on which the post-processing is performed.

[0114] Figure 6 The example of selecting the predetermined attention weight matrix from the plurality of attention weight matrices will be further described below.

[0115] Referring to FIG. 6A, the speech time estimation device can determine a plurality of attention weight matrices including attention weights of all steps. For example, when multi-head attention is performed or when the decoder includes a plurality of labels, the speech time estimation device can determine a plurality of attention weight matrices. In this example, the number of the plurality of attention weight matrices can correspond to a product of the number of attention layers and the number of decoder layers. In this example, the decoder layers can be associated with output lexemes, for example, the number of decoder layers can be the same as the number of output lexemes. In one example, the speech time estimation device can determine the plurality of attention weight matrices using an attention-based sequence-to-sequence model. In this example, the input of the attention-based sequence-to-sequence model can be the input frames, and the attention-based sequence-to-sequence model can output the output lexemes based on the input frames and the attention weights included in the plurality of attention weight matrices. In one example, the speech time estimation device can determine the attention weight matrix of each of the at least one attention layer based on the input frames corresponding to the output lexemes. Figure 6 When the multi-head attention is performed or when the decoder includes a plurality of labels, the speech time estimation device can determine the same number of attention weight matrices as a product of the number of attention layers and the number of decoder layers.

[0116] The attention weight of the attention weight matrix can be represented in color. For example, when the attention weight increases, the attention weight can be represented as close to white. When the attention weight decreases, the attention weight can be represented as close to black. For example, a higher attention weight can be lighter in shade than a lower attention weight, and a lower attention weight can be darker in shade than a higher attention weight. In one example, an attention weight "1" can be white, and an attention weight "0" can be black.

[0117] When the output sequence is estimated using a plurality of attention weight matrices, the plurality of attention weight matrices can perform different functions. For example, an attention weight matrix can correspond to noise, and another attention weight matrix can correspond to an actual user's utterance.

[0118] The speech time estimation device can select an attention weight matrix (e.g., the attention weight matrix 630) having the most monotonicity from among a plurality of attention weight matrices (e.g., the plurality of attention weight matrices 620), and can estimate the speech time of the actual user. For example, the speech time estimation device can perform a monotonic regression analysis on the plurality of attention weight matrices, and can select an attention weight matrix having the least error and monotonicity based on the analysis results from the plurality of attention weight matrices.

[0119] The speech time estimation device can estimate the speech time corresponding to the output sequence based on the selected attention weight matrix, hereinafter referred to as a "first attention weight matrix". When the first attention weight matrix is selected once, the speech time estimation device can estimate the speech time corresponding to the output sequence based on the selected first attention weight matrix in response to another input sequence. For example, when the first attention weight matrix is selected once, the speech time estimation device can estimate the speech time corresponding to the output sequence based on the selected first attention weight matrix, instead of selecting the first attention weight matrix again. Non-limiting examples of estimating the speech time corresponding to the output sequence based on the first attention weight matrix will be described below with reference to FIGS. 6 to 9. Figure 7 Non-limiting examples of estimating the speech time corresponding to the output sequence based on the first attention weight matrix are further described.

[0120] Figure 7 An example of estimating the speech time corresponding to the output sequence based on the first attention weight matrix is illustrated.

[0121] Referring to FIG. 6, the speech time estimation device 600 can include a plurality of attention weight matrices 610, 620, 630, 640, and 650. Figure 7 The speech time estimation device can select a predetermined number of relatively high attention weights (e.g., "n" highest attention weights among the attention weights, where "n" is the predetermined number) of the input frames corresponding to the output lexical units of each step from the attention weights of the input frames. The speech time estimation device can estimate the speech time corresponding to the output sequence based on the selected attention weights. The speech time estimation device can perform a monotonic regression analysis on the selected attention weights, and can estimate the speech time corresponding to the output sequence.

[0122] In addition, the speech time estimation device can correct the first attention weight matrix based on the selected attention weights, and can estimate the speech time corresponding to the output sequence based on the corrected first attention weight matrix.

[0123] The speech time estimation device of one or more embodiments can select a plurality of attention weights, rather than selecting only the highest attention weight in the corresponding step, and thus the first attention weight matrix can be corrected to have monotonicity in all parts even if the highest attention weight does not have monotonicity in all parts (e.g., frame parts) afterward.

[0124] For example, the image 720 shows that the relatively high attention weights of the three input frames are selected in advance from the attention weights (e.g., an attention weight matrix) of the input frames corresponding to the output lexical units of each step in the image 710, and indicates that the attention weights are higher in the order of the attention weights 721, 722, and 723. For example, the attention weights 721, 722, and 723 of the image 720 can correspond to the attention weights in the image 710. When the input frame having the highest attention weight cannot satisfy the monotonicity in all parts, the speech time estimation device can correct the part that does not satisfy the monotonicity so that the monotonicity can be satisfied in all parts using the second highest attention weight.

[0125] The speech time estimation device can perform diffusion correction on the first attention weight matrix using a diffusion module. In one example, the speech time estimation device can perform diffusion correction on the first attention weight matrix based on the selected attention weight. In one example, the speech time estimation device can estimate the speech time corresponding to the output sequence based on the first attention weight matrix on which the diffusion correction is performed. To determine a continuous input frame portion corresponding to an output lexicon unit rather than a unit input frame, the speech time estimation device can perform diffusion correction on the first attention weight matrix. For example, when the diffusion correction on the first attention weight matrix is performed, the speech time estimation device can determine a continuous portion exceeding a predetermined threshold as an input frame portion corresponding to an output lexicon unit. The diffusion correction can be an operation of multiplying a transposed matrix of the first attention weight matrix with the first attention weight matrix. Through the diffusion correction, a boundary between attention weights close to white (e.g., a higher value) and attention weights close to black (e.g., a lower value) can be further clarified.

[0126] In addition, the speech time estimation device can select a plurality of attention weights, rather than selecting only the highest attention weight in the corresponding step, and thus the speech time estimation device of one or more embodiments can improve the accuracy of estimation by mapping a plurality of input frames to a single output lexicon unit.

[0127] Figure 8A An example of showing a result obtained by estimating a speech time is shown.

[0128] As Figure 8A As shown in the image 810, the speech time estimation device can map an input sequence 820 corresponding to an output sequence 830 (e.g., Hi Bixby remind me to check the air cleaner filters when I get home).

[0129] As shown in the image 810, the speech time estimation device can estimate a start time of an utterance of an output lexicon unit constituting the output sequence 830. In addition, the speech time estimation device can estimate a speech time including a start time and an end time of an utterance of an output lexicon unit constituting the output sequence 830.

[0130] Figure 8B An example of showing a result obtained by estimating a speech time is shown.

[0131] Referring to Figure 8B The speech time estimation device can be used in a voice recorder or can include a voice recorder. For example, a speech recognition device including the speech time estimation device can recognize text of recorded speech, can match the recognized text with a waveform of the recorded speech, and can provide the user with the recognized text matched with the waveform and a corresponding time. Also, the speech recognition device can highlight and show words spoken according to the speech. In response to a user input for selecting a predetermined character, the speech recognition device can jump to a portion in which speech corresponding to the selected character is recorded, and can reproduce the speech. In addition to the voice recorder, the speech time estimation device can be applicable to all fields (e.g., smart speaker, moving picture, or movie) in which speech recognition is performed. The speech recognition device can receive and record speech using at least one sensor (e.g., microphone) of the speech recognition device, and can output the recognized text, waveform, and / or corresponding time using any one or both of a display and a user interface of the speech recognition device.

[0132] Figure 9A and Figure 9B An example of a method of training an attention-based sequence-to-sequence model is shown.

[0133] Referring to Figure 9A The speech recognition device can train the attention-based sequence-to-sequence model such that an attention weight matrix 900 among a plurality of attention weight matrices can become (or can be generated as) a first attention weight matrix. The speech recognition device can train the attention-based sequence-to-sequence model by guiding or training a predetermined attention layer from a training point in time such that an attention weight matrix (e.g., the attention weight matrix 900) corresponding to the predetermined attention layer has monotonicity, and can estimate a speech time corresponding to an output sentence by monotonic regression analysis based on the trained attention weight matrix without selecting the first attention weight matrix. For example, the speech recognition device can mask an attention layer corresponding to the attention weight matrix 900, and can train the attention-based sequence-to-sequence model. In one example, the speech recognition device can train the attention-based sequence-to-sequence model based on the masked attention layer.

[0134] The descriptions of the feature vector extraction module 310, the encoder 320, the decoder 330, the attention module 331, the speech time estimation device 340, the monotonic regression analysis module 341, and the diffusion module 343 are also applicable to the feature vector extraction module 910, the encoder 920, the decoder 930, the attention module 931, the speech time estimation device 940, the monotonic regression analysis module 941, and the diffusion module 943 shown in Figure 9B The descriptions of the feature vector extraction module 310, the encoder 320, the decoder 330, the attention module 331, the speech time estimation device 340, the monotonic regression analysis module 341, and the diffusion module 343 are also applicable to the feature vector extraction module 910, the encoder 920, the decoder 930, the attention module 931, the speech time estimation device 940, the monotonic regression analysis module 941, and the diffusion module 943 shown in

[0135] Referring to Figure 9B , the speech recognition device can include a monotonic attention guiding module 950 instead of the first attention weight matrix selection module. The monotonic attention guiding module 950 can train the attention-based sequence-to-sequence model by guiding the attention weight matrix 900 to have monotonicity from a training time point. In addition, the monotonic attention guiding module 950 can inform the monotonic regression analysis module 941 of the attention weight matrix 900 of the predetermined attention layer of the training, so that the speaking time corresponding to the output sequence can be estimated using the attention weight matrix 900.

[0136] The monotonic attention guiding module 950 can train the attention weight matrix 900 to have a hard monotonic property, or can train the attention weight matrix 900 to have a soft monotonic property by applying a constant weight to a diagonal weight.

[0137] Here, the monotonic attention guiding module 950 can train the attention weight matrix 900 to have a monotonic property with respect to a time axis of the input sequence. Figures 1 to 9B The described user terminal, speech recognition device, encoder, decoder, module, attention module, speaking time estimation device, monotonic regression analysis module, diffusion module, feature vector extraction module, monotonic attention guiding module, user terminal 110, speech recognition device 120, encoder 220, decoder 230, feature vector extraction module 310, encoder 320, decoder 330, attention module 331, speaking time estimation device 340, monotonic regression analysis module 341, diffusion module 343, feature vector extraction module 910, encoder 920, decoder 930, attention module 931, speaking time estimation device 940, monotonic regression analysis module 941, diffusion module 943, monotonic attention guiding module 950, devices, units, apparatuses, and other components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as logic gates arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and perform instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software for execution by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. The one or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components.The hardware components can have any one or more of a variety of processing configurations, examples of which include a single processor, multiple processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

[0138] Figures 1 to 9B The methods of performing the operations described in this application are performed by computing hardware (e.g., by one or more processors or computers) implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. The one or more processors, or a processor and a controller, can perform a single operation or two or more operations.

[0139] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods as described above can be written in a computer program, code segment, instructions, or any combination thereof, to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as produced by a compiler) directly executable by the one or more processors or computers. In another example, the instructions or software include high-level code to be executed by the one or more processors or computers using an interpreter. The instructions or software can be written in any programming language, based on the block diagrams and flowcharts shown in the drawings and corresponding descriptions used herein, which disclose algorithms for performing the operations performed by the hardware components and methods as described above, using any programming language.

[0140] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory such as a multimedia card or a micro card (e.g., a secure digital (SD) or extreme digital (XD)), magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors and computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed manner.

[0141] While the disclosure includes certain examples, it will be apparent that, within the scope of the claims and their equivalents, various changes can be made in form and details of the examples herein. The examples described herein are to be considered as merely illustrative, and not restrictive, in nature. The description of features or aspects in each example is to be considered as applying mutatis mutandis to similar features or aspects in other examples. Wherever appropriate, the described techniques can be implemented in different orders, and / or the components of the described systems, architectures, devices, or circuits can be combined or divided into other components in order to implement suitable results. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.< / eos> < / eos> < / eos> < / e>

Claims

1. A method for speech time estimation, the method comprising: Use an attention-based sequence-to-sequence model to determine multiple attention weight matrices; selecting an attention weight matrix from the plurality of attention weight matrices; and Estimate the speaking time corresponding to the output sequence as the recognition result of the input sequence based on the selected attention weight matrix, Among them, the step of selecting the attention weight matrix includes: performing monotonic regression analysis on the multiple attention weight matrices, and selecting an attention weight matrix corresponding to monotonicity from the multiple attention weight matrices based on the result of the monotonic regression analysis.

2. The method according to claim 1, wherein The step of selecting the attention weight matrix includes: selecting an attention weight matrix having the greatest monotonicity from the plurality of attention weight matrices as the attention weight matrix corresponding to monotonicity.

3. The method according to claim 1, wherein The step of estimating the speaking time includes performing a monotone regression analysis on the selected attention weight matrix, and estimating the speaking time corresponding to the output sequence based on a result of the monotone regression analysis.

4. The method according to claim 1 or 2, wherein: The selected attention weight matrix includes as elements the attention weights of the input frames of the input sequence corresponding to the output tokens of the output sequence, and The step of estimating the speaking time includes: selecting a predetermined number of high attention weights from the attention weights of the input frame; and estimating the speaking time corresponding to the output sequence based on the selected attention weights.

5. The method according to claim 4, wherein The steps to estimate speaking time based on the selected attention weights include: Correcting the selected attention weight matrix based on the selected attention weights; and The speaking time corresponding to the output sequence is estimated based on the corrected attention weight matrix.

6. The method according to claim 4, wherein: The predetermined number of high attention weights are a predetermined number of highest attention weights among the attention weights corresponding to each output lexical unit.

7. The method according to claim 1 or 2, wherein: The steps for estimating speaking time include: Perform diffusion correction on the selected attention weight matrix; and The utterance time corresponding to the output sequence is estimated based on the selected attention weight matrix to which diffusion correction is performed.

8. The method according to claim 1 or 2, wherein: The selected attention weight matrix includes as elements the attention weights of the input frames of the input sequence corresponding to the output tokens of the output sequence, and The step of estimating the speaking time includes: selecting a predetermined number of high attention weights from the attention weights of the input frame; performing diffusion correction on a selected attention weight matrix based on the selected attention weights; and estimating the speaking time corresponding to the output sequence based on the selected attention weight matrix on which the diffusion correction is performed.

9. The method according to any one of claims 1 to 3, wherein: The number of the multiple attention weight matrices corresponds to the product of the number of attention layers of the attention-based sequence-to-sequence model and the number of decoder layers of the attention-based sequence-to-sequence model.

10. The method according to claim 1, further comprising: An attention-based sequence-to-sequence model is trained so that a predetermined attention weight matrix among the plurality of attention weight matrices is generated as a selected attention weight matrix.

11. The method according to claim 10, wherein: The step of training the attention-based sequence-to-sequence model includes masking an attention layer corresponding to a predetermined attention weight matrix, and training the attention-based sequence-to-sequence model based on the masked attention layer.

12. A method for speech time estimation, the method comprising: Receive an input sequence; Generate encoded features by encoding the input sequence; determining, for each attention layer in the at least one attention layer, an attention weight for an input frame of the input sequence; Estimate the output lexical units of the output sequence based on the encoded features and attention weights; Determine the input frame corresponding to the output token based on the attention weights; and Estimate the utterance time corresponding to the output sequence based on the input frames corresponding to the output tokens, Among them, the step of estimating the speaking time includes: determining the attention weight matrix of each attention layer in the at least one attention layer based on the input frame corresponding to the output lexical unit; performing monotonic regression analysis on the attention weight matrix of each attention layer in the at least one attention layer; selecting an attention weight matrix corresponding to monotonicity from the attention weight matrix of each attention layer in the at least one attention layer based on the result of the monotonic regression analysis; and estimating the speaking time corresponding to the output sequence based on the selected attention weight matrix.

13. The method according to claim 12, wherein: The step of selecting an attention weight matrix corresponding to monotonicity from the attention weight matrices of each attention layer in the at least one attention layer comprises: The attention weight matrix with the minimum error of the monotonic regression analysis is selected as the attention weight matrix corresponding to monotonicity.

14. A device for speech time estimation, the device comprising: The processor is configured to: Determine multiple attention weight matrices using an attention-based sequence-to-sequence model; selecting an attention weight matrix from the plurality of attention weight matrices; and Estimate the speaking time corresponding to the output sequence based on the selected attention weight matrix, In order to select the attention weight matrix, the processor is configured to: perform monotonic regression analysis on the multiple attention weight matrices, and select an attention weight matrix corresponding to monotonicity from the multiple attention weight matrices based on the result of the monotonic regression analysis.

15. The apparatus according to claim 14, wherein In order to select the attention weight matrix, the processor is configured to: select an attention weight matrix having the most monotonicity from the plurality of attention weight matrices as the attention weight matrix corresponding to monotonicity.

16. The apparatus according to claim 14, wherein To estimate the speaking time, the processor is configured to perform a monotonic regression analysis on the selected attention weight matrix, and estimate the speaking time corresponding to the output sequence based on a result of the monotonic regression analysis.

17. The apparatus according to claim 14 or 15, wherein The selected attention weight matrix includes as elements the attention weights of the input frames of the input sequence corresponding to the output tokens of the output sequence, and To estimate the speaking time, the processor is configured to: select a predetermined number of high attention weights from the attention weights of the input frame; and estimate the speaking time corresponding to the output sequence based on the selected attention weights.

18. The apparatus according to claim 17, wherein To estimate the speaking time, the processor is configured to: Correcting the selected attention weight matrix based on the selected attention weights; and The speaking time corresponding to the output sequence is estimated based on the corrected attention weight matrix.

19. The apparatus according to claim 14 or 15, wherein To estimate the speaking time, the processor is configured to: Perform diffusion correction on the selected attention weight matrix; and The utterance time corresponding to the output sequence is estimated based on the selected attention weight matrix to which diffusion correction is performed.

20. The apparatus according to claim 14 or 15, wherein The selected attention weight matrix includes as elements the attention weights of the input frames of the input sequence corresponding to the output tokens of the output sequence, and To estimate the speaking time, the processor is configured to: select a predetermined number of high attention weights from the attention weights of the input frame; performing diffusion correction on a selected attention weight matrix based on the selected attention weights; and estimating a utterance time corresponding to the output sequence based on the selected attention weight matrix on which the diffusion correction is performed.

21. The apparatus according to any one of claims 14 to 16, wherein The number of the multiple attention weight matrices corresponds to the product of the number of attention layers of the attention-based sequence-to-sequence model and the number of decoder layers of the attention-based sequence-to-sequence model.

22. The apparatus of claim 14, wherein: The processor is configured to: train an attention-based sequence-to-sequence model so that a predetermined attention weight matrix among the plurality of attention weight matrices is generated as a selected attention weight matrix.

23. The apparatus of claim 22, wherein: The processor is configured to: mask an attention layer corresponding to a predetermined attention weight matrix, and train an attention-based sequence-to-sequence model based on the masked attention layer.

24. A mobile device comprising: at least one sensor configured to receive an input sequence; and The processor is configured to: Generate encoded features by encoding the input sequence; determining, for each attention layer in the at least one attention layer, a corresponding attention weight for an input frame of the input sequence; Estimate the output lexical units of the output sequence based on the encoded features and attention weights; Determine the input frame corresponding to the output token based on the attention weights; and Estimate the utterance time corresponding to the output sequence based on the input frames corresponding to the output tokens, In which, in order to estimate the speaking time, the processor is configured to: determine the attention weight matrix of each attention layer in the at least one attention layer based on the input frame corresponding to the output lexical unit; perform monotonic regression analysis on the attention weight matrix of each attention layer in the at least one attention layer; select an attention weight matrix corresponding to monotonicity from the attention weight matrix of each attention layer in the at least one attention layer based on the result of the monotonic regression analysis; and estimate the speaking time corresponding to the output sequence based on the selected attention weight matrix.

25. The mobile device according to claim 24, in, The at least one sensor includes a microphone, the input sequence includes speech, and The mobile device further includes a user interface configured to output the output sequence and the speaking time.

26. A method for speech time estimation, the method comprising: generating encoded features by encoding an input sequence comprising input frames; estimating an output lexical unit of an output sequence corresponding to the input sequence based on the encoding features; determining an attention weight matrix comprising attention weights for input frames corresponding to output tokens; and Estimate the speaking time corresponding to the output sequence based on the attention weight matrix, Among them, the step of determining the attention weight matrix includes: determining the attention weight matrix of each attention layer in at least one attention layer based on the input frame corresponding to the output lexical unit; performing monotonic regression analysis on the attention weight matrix of each attention layer in the at least one attention layer; and selecting an attention weight matrix corresponding to monotonicity from the attention weight matrix of each attention layer in the at least one attention layer based on the result of the monotonic regression analysis.

27. A non-transitory computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the processor is configured to perform the method of any one of claims 1 to 13 and claim 26.

Images