Encoder / decoder device, inference device and program

Abstract

To provide an encoder / decoder (AED) with a cross-attention mechanism which can correspond to streaming application.SOLUTION: A duplication part duplicates a string of first input feature amounts for a plurality of mask patterns. A mask processing unit masks the string of the first input feature amounts by using a mask pattern different in each duplicated string. An encoder unit encodes the string of the masked first input feature amounts. A decoder unit performs processing of cross attention based on information transferred from the encoder unit and outputs a decoded result. A linear conversion unit linearly converts the decoded result. An output selection unit generates a binary mask. A multiplication unit multiplies a linear conversion result by the binary mask. An output decision unit generates and outputs a matrix of probability values corresponding to a matrix of the linear conversion result. Internal parameters of each unit are configured to be updated based on three types of losses.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an encoder / decoder device, an inference device, and a program. [Background technology]

[0002] An attention encoder-decoder (AED) type speech recognition model equipped with a cross-attention mechanism has been studied and has achieved high recognition accuracy.

[0003] Non-Patent Document 1 describes a technique for determining whether or not to output using a cross-attention score in a model that uses cross-attention. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Mohan Li, Shucong Zhang, Catalin Zorila, Rama Doddipatla, “Transformer-based Streaming ASR with Cumulative Attention”, arXiv:2203.05736, https: / / arxiv.org / pdf / 2203.05736.pdf, ICASSP 2022, 2022. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology has the following problem: AED-type models are basically used for offline speech recognition. A technology is needed to make AED-type speech recognition models compatible with streaming applications.

[0006] The first problem is that if an AED-type model based on conventional technology is used for streaming input as is, the following problem may arise. In other words, in such a case, the range of reference for input data (in the case of speech recognition, the acoustic features) differs between when training the AED-type model and when making inferences using the trained AED-type model. This difference may have an adverse effect on the inference accuracy of the AED-type model (in the case of speech recognition, the speech recognition accuracy). It is desirable to make the AED-type speech recognition model compatible with streaming input while resolving such problems.

[0007] The second issue is that the AED model may output erroneous inference results when the input data contains noise. In the case of speech recognition processing, noise in the input data refers to noise caused by background sounds or unclear speech. In other words, such noise can cause errors in the speech recognition results.

[0008] The present invention was made based on the above-mentioned problem recognition, and aims to provide an encoder-decoder device, an inference device, and a program that use an encoder-decoder (AED) type model equipped with a cross-attention mechanism that can be used for streaming applications. [Means for solving the problem]

[0009] [1] In order to solve the above problem, an encoder-decoder device according to one aspect of the present invention includes a duplication unit that duplicates an input sequence of first input features for a plurality of mask patterns; a masking unit that masks some of the first input features included in the sequence, and uses a different mask pattern for each of the duplicated sequences to make a set of first input features to be masked differently for each of the duplicated sequences; an encoder unit that inputs the masked sequences and generates a plurality of sequences of encoding results corresponding to each of the sequences using the masked sequences and a first internal parameter; and a cross-attention unit that performs cross-attention processing using a second internal parameter based on the input sequence of second input features and the plurality of sequences of the encoding results passed from the encoder unit, and outputs a matrix of decoding results configured with rows corresponding to the columns of the second input features and columns corresponding to the mask pattern. a decoder unit, a linear transformation unit that performs linear transformation processing using a third internal parameter based on the matrix of the decoded result that is output from the decoder unit, and outputs a matrix of linear transformation result data configured in a row direction corresponding to the columns of the second input feature and a column direction corresponding to the mask pattern; an output selector that generates a binary mask that is a matrix having elements of 1 or approximately 1 or 0 or approximately 0 based on the matrix of the linear transformation result data; a multiplication unit that multiplies each element of the matrix of the linear transformation result data by a corresponding element of the binary mask and outputs the result; an output determiner that generates a matrix of probability values ​​corresponding to the matrix of the linear transformation result data based on each element of the matrix of the linear transformation result data and outputs the result; a first loss calculation unit that calculates a first loss (L) that is an error between an output from the multiplication unit and a correct token sequence corresponding to the first input feature; m ), which is an error between the matrix of probability values ​​output by the output determiner and the matrix of correct probability values ​​corresponding to the matrix of linearly transformed data, da third loss calculation unit that calculates the first loss (L) and the second loss (L), wherein the second input feature is a feature of a token sequence corresponding to a past output from the multiplication unit, m ) and the third loss (L d ) and updates the value of the third internal parameter, the value of the second internal parameter, and the value of the first internal parameter by backpropagation based on a total loss obtained by combining the above-mentioned three internal parameters.

[0010] [2] Also, one aspect of the present invention is that in the encoder-decoder device of [1] above, the output selector generates the binary mask by applying a Gumbel-softmax function to each element of the matrix of the linear transformation result data.

[0011] [3] Furthermore, according to one aspect of the present invention, in the encoder-decoder device of the above [1] or [2], the correct solution of the binary mask is a matrix composed of rows corresponding to the columns of the second input features and columns corresponding to the mask pattern, in which elements near the diagonal of the matrix are non-zero and other elements of the matrix are 0 or nearly 0.

[0012] [4] Furthermore, one aspect of the present invention is an encoder-decoder device according to any one of [1] to [3] above, further comprising: a probability calculation unit that generates a probability value matrix by applying a softmax function to each element of the linear transformation result data; and a cumulative distribution transformation unit that generates a matrix of correct probability values ​​based on the probability value matrix calculated by the probability calculation unit, such that the sum of the values ​​from the 1st column to the mth column of the uth row of the probability value matrix becomes the element of the uth row and mth column of the matrix of correct probability values.

[0013] [5] An inference device according to one aspect of the present invention includes an encoder unit that receives a sequence of first input features and generates a sequence of encoding results corresponding to the sequence using the sequence and a first internal parameter; a decoder unit that performs cross-attention processing using a second internal parameter based on the sequence of second input features and the sequence of encoding results passed from the encoder unit, thereby outputting a sequence of decoding results corresponding to the sequence of second input features; and a decoder unit that performs linear transformation processing using a third internal parameter based on the sequence of decoding results output from the decoder unit, thereby outputting linear transformation result data. an output determiner that generates and outputs a probability value corresponding to the linear transformation result data based on the linear transformation result data; and an output control unit that outputs a token based on the output from the linear transformation unit when the output from the output determiner is equal to or greater than a predetermined threshold, wherein the second input feature is a feature of a token sequence corresponding to a past output from the output control unit, and the value of the first internal parameter, the value of the second internal parameter, and the value of the third internal parameter are values ​​obtained as a result of machine learning by the encoder-decoder device according to any one of claims 1 to 4.

[0014] [6] Also, one aspect of the present invention is a method for generating a matrix of decoded features, the matrix being configured with a row direction corresponding to the second input feature and a column direction corresponding to the mask pattern, by performing cross-attention processing based on a second input feature and the encoded result, the matrix being configured with a row direction corresponding to the second input feature and a column direction corresponding to the mask pattern, and a decoder unit configured to perform cross-attention processing based on the second input feature and the encoded result, the matrix being configured with a row direction corresponding to the second input feature and a column direction corresponding to the mask pattern, and a decoder unit configured to perform cross-attention processing based on the second input feature and the encoded result, the matrix being configured with a row direction corresponding to the second input feature and a column direction corresponding to the mask pattern, the decoder unit configured with cross-attention processing based on the second input feature and the encoded result, the matrix being ... a linear transformation unit that performs linear transformation processing using a third internal parameter based on the matrix of the decoded result that is an output from the decoder unit, and outputs a matrix of linear transformation result data configured in a row direction corresponding to the columns of the second input feature and a column direction corresponding to the mask pattern; an output selector that generates a binary mask that is a matrix having elements of 1 or approximately 1 or 0 or approximately 0 based on the matrix of the linear transformation result data; a multiplication unit that multiplies each element of the matrix of the linear transformation result data by a corresponding element of the binary mask, and outputs the result; an output determiner that generates a matrix of probability values ​​corresponding to the matrix of the linear transformation result data based on each element of the matrix of the linear transformation result data, and outputs the result; a first loss calculation unit that calculates a first loss (L) that is an error between an output from the multiplication unit and a correct token sequence corresponding to the first input feature; m ), which is an error between the matrix of probability values ​​output by the output determiner and the matrix of correct probability values ​​corresponding to the matrix of linearly transformed data, da third loss calculation unit that calculates the first loss (L) and the second loss (L), wherein the second input feature is a feature of a token sequence corresponding to a past output from the multiplication unit, m ) and the third loss (L d ) based on a total loss obtained by combining the third internal parameter, the second internal parameter, and the first internal parameter, by using the backpropagation method.

[0015] [7] Also, one aspect of the present invention includes an encoder unit that receives a sequence of first input features and generates a sequence of encoded results corresponding to the sequence using the sequence and a first internal parameter; a decoder unit that performs cross-attention processing using a second internal parameter based on the sequence of second input features and the sequence of encoded results passed from the encoder unit, thereby outputting a sequence of decoded results corresponding to the sequence of second input features; a linear transformation unit that performs linear transformation processing using a third internal parameter based on the sequence of decoded results output from the decoder unit, and outputs linear transformation result data; A program for causing a computer to function as an inference device, comprising: an output determiner that generates and outputs a probability value corresponding to linear transformation result data based on the linear transformation result data; and an output control unit that outputs a token based on the output from the linear transformation unit when the output from the output determiner is equal to or greater than a predetermined threshold, wherein the second input feature is a feature of a token sequence corresponding to a past output from the output control unit, and the value of the first internal parameter, the value of the second internal parameter, and the value of the third internal parameter are values ​​obtained as a result of machine learning by an encoder-decoder device described in any one of claims 1 to 4. [Effects of the Invention]

[0016] According to the present invention, a first input feature can be masked using a plurality of mask patterns. The encoder-decoder device can perform learning using the first input feature masked using the plurality of mask patterns. Therefore, after learning, the encoder-decoder device (inference device) can improve the accuracy of inference for inputs equivalent to the partially masked first input feature. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an encoder / decoder device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a schematic internal functional configuration of one layer of a decoder according to the embodiment. [Figure 3] FIG. 10 is a functional block diagram showing a configuration for generating a correct answer for calculating a third loss Ld used when the encoder-decoder device according to the embodiment performs machine learning. [Figure 4] This is a block diagram for explaining the functions of the machine-learned encoder-decoder device of the same embodiment when operating in inference mode. [Figure 5] 10 is a schematic diagram for explaining the processing of a feature extraction unit (encoder side), a replication unit, and a mask processing unit, and the data configuration according to the embodiment. FIG. [Figure 6] FIG. 2 is a schematic diagram showing the configuration of data output from an encoder (final layer) according to the same embodiment. [Figure 7] FIG. 2 is a schematic diagram for explaining data processing in a decoder and a linear transform unit according to the embodiment. [Figure 8] FIG. 10 is a schematic diagram showing data output by a linear transformation unit corresponding to an input feature sequence (u=1, 2, 3) in the embodiment. [Figure 9] 10 is a schematic diagram for explaining the processing by the output selection unit and the multiplication unit when the encoder-decoder device according to the embodiment operates in a learning mode. FIG. [Figure 10] 10 is a schematic diagram for explaining a process in which a third loss calculation unit calculates a third loss Ld when the encoder-decoder device according to the embodiment operates in a learning mode. FIG. [Figure 11] FIG. 10 is a block diagram showing an example of the internal configuration of the encoder / decoder device according to the embodiment when it is realized using a general-purpose computer. DETAILED DESCRIPTION OF THE INVENTION

[0018] Next, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, an encoder-decoder device will be described as an example of a device that performs speech recognition processing.

[0019] 1 is a block diagram showing a schematic functional configuration of an encoder / decoder device according to this embodiment. As shown in the figure, the encoder / decoder device 1 includes a feature extraction unit 22, a replication unit 23, a mask processing unit 24, and encoder units 25-1, 25-2, . . . , 25-N. e , a feature extraction unit 32, and decoder units 33-1, 33-2, . . . , 33-N d , a linear conversion unit 36, an output selector 41, an output determiner 42, a multiplication unit 43, a first loss calculation unit 51, a second loss calculation unit 52, and a third loss calculation unit 53. e is the number of layers in the encoder, which is greater than or equal to 1. Also, N dis the number of decoder layers and is 1 or more. Each of these functional units can be realized, for example, by a general-purpose computer and a program. Each functional unit also has a storage means as needed. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Also, non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may be used as needed. Also, at least some or all of the functions of each functional unit may be realized as a dedicated electronic circuit (including a dedicated computer) rather than a program. The functions of each unit are as follows:

[0020] A feature of this embodiment is that when training the encoder-decoder device, a mask process is used to erase part of the input information, allowing the encoder-decoder device to train so that it can process input data appropriately without depending on subsequent information.

[0021] The feature extraction unit 22 extracts features of the data input to the encoder-decoder device 1 and outputs feature vectors. When the encoder-decoder device 1 performs speech recognition processing, the features are acoustic features. The acoustic features are represented as vectors for each predetermined time interval. The feature extraction unit 22 passes the series of feature vectors to the duplication unit 23.

[0022] The duplication unit 23 duplicates the series of feature vectors passed from the feature extraction unit 22. That is, the duplication unit 23 creates multiple series of feature vectors and passes them to the mask processing unit 24. The multiple series of feature vectors output by the duplication unit 23 have the same values.

[0023] The mask processing unit 24 receives multiple series of feature vectors from the replication unit 23 and performs mask processing on each of them using a different mask pattern. The mask processing performed by the mask processing unit 24 is a process of replacing some elements of the data with, for example, a fixed number. Data in the areas masked by the mask processing unit 24 loses information. The mask processing unit 24 passes the masked data to the encoder unit 25-1 (the first layer of the encoder).

[0024] Encoder section 25-1, 25-2, 25-N e is an encoder consisting of multiple layers, which sequentially processes the data output from the mask processing unit 24. e is the number of layers of the encoder. From upstream to downstream, the encoder units 25-1, 25-2, . . . , 25-N e are arranged in order. The final layer encoder part 25-N e sends the output (encoded information) to the decoder units 33-1, 33-2, . . . , 33-N d Pass it to each of them.

[0025] Encoder section 25-1, 25-2, 25-N e are configured using, for example, a neural network. That is, the encoder units 25-1, 25-2, . . . , 25-N e The internal parameter values ​​of the encoder units 25-1, 25-2, . . . , 25-N can be updated through learning. e The internal parameters of the encoder units 25-1, 25-2, . . . , 25-N are called "first internal parameters" for convenience. e may be, for example, a transformer encoder. Note that the transformer itself is a well-known technology.

[0026] The feature extraction unit 32 extracts features to be input to the decoder side and passes them to the decoder unit 33-1, which is the first layer of the decoder. When performing speech recognition processing, the feature extraction unit 32 receives as input a token sequence of the previous output from the decoder.

[0027] Decoder section 33-1, 33-2, . . . , 33-N d is a decoder consisting of multiple layers, which sequentially processes the data supplied from the feature extraction unit 32. d is the number of layers of the decoder. From upstream to downstream, the decoder units 33-1, 33-2, . . . , 33-N d are arranged in sequence. The decoder units 33-1, 33-2, . . . , 33-N d The final layer on the encoder side is the encoder section 25-N. e The encoded information is passed to the decoder units 33-1, 33-2, . . . , 33-N. d Each of the decoder units 33-1, 33-2, . . . , 33-N performs cross-attention processing based on this encoded information. d The internal configuration of the decoder unit 33-N in the final layer will be explained in more detail later with reference to another diagram. d passes its output to the linear transformation unit 36.

[0028] Decoder section 33-1, 33-2, . . . , 33-N d are configured using, for example, a neural network. That is, the decoder units 33-1, 33-2, . . . , 33-N d The internal parameter values ​​of the decoder units 33-1, 33-2, . . . , 33-N can be updated through learning. d The internal parameters of the decoder units 33-1, 33-2, . . . , 33-N are called "second internal parameters" for convenience. d may be, for example, a decoder of a transformer. Note that the transformer itself is a well-known technique.

[0029] The linear transform unit 36 ​​is the decoder unit 33-N, which is the final layer of the decoder. dThe linear conversion unit 36 ​​receives the output from the linear conversion unit 36 ​​and performs linear conversion processing. The linear conversion unit 36 ​​is configured using, for example, a neural network. The internal parameters used by the linear conversion unit 36 ​​when performing linear conversion processing can be updated through learning. For convenience, the internal parameters held by the linear conversion unit 36 ​​are referred to as "third internal parameters." In this embodiment, the output from the linear conversion unit 36 ​​is a multidimensional vector. Each element of the vector is a numerical value representing the likelihood (probability) of the output token.

[0030] The output selector 41 performs a predetermined process based on the data output from the linear transformation unit 36, thereby outputting a binary mask. The output selector 41 passes this binary mask to the multiplication unit 43. When the encoder-decoder device 1 operates in the learning mode, the output selector 41 also passes the binary mask to the second loss calculation unit. The binary mask generated by the output selector 41 is data for determining which of the mask patterns used in the masking process by the mask processing unit 24 corresponds to the output from the linear transformation unit 36, and which should be used as the output value. The process of the output selector 41 will be further described later with reference to FIG. 9.

[0031] The output determiner 42 performs a predetermined process based on the data output from the linear transformation unit 36 ​​to generate a probability value for determining the timing at which to output an output token. When the encoder-decoder device 1 operates in the learning mode, the output determiner 42 passes the probability value to the third loss calculation unit 53. The operation of the output determiner 42 when the encoder-decoder device 1 operates in the inference mode will be described later with reference to FIG. 4.

[0032] The multiplication unit 43 multiplies corresponding elements of the data (matrix) passed from the linear transformation unit 36 ​​and the data (binary mask, matrix) passed from the output selector 41, and outputs the result. When the encoder-decoder device 1 operates in the learning mode, the multiplication unit 43 passes the multiplication result to the first loss calculation unit 51. The operation of the multiplication unit 43 when the encoder-decoder device 1 operates in the inference mode will be described later with reference to FIG. 4.

[0033] The first loss calculation unit 51, the second loss calculation unit 52, and the third loss calculation unit 53 each calculate a loss for adjusting the internal parameters during operation in the learning mode.

[0034] The first loss calculation unit 51 calculates the first loss L between the correct token (sequence) and the token (sequence) identified by the data output from the multiplication unit 43. The first loss L may be, for example, a cross-entropy loss.

[0035] The second loss calculation unit 52 calculates the second loss L between the binary mask output from the output selector 41 and the correct binary mask. m Calculate the second loss L m The second loss L m may be, for example, the squared error between elements of the binary mask output from the output selector 41 and the correct binary mask.

[0036] The third loss calculation unit 53 calculates the third loss L between the probability value (matrix) output from the output determiner 42 and the correct solution (matrix) of the probability value. d The method for generating the correct solution (matrix) of the probability values ​​will be described later. d may be, for example, the cross entropy between the probability value (matrix) output from the output determiner 42 and the correct solution (matrix) of the probability value.

[0037] The encoder-decoder device 1 operates in either a learning mode or an inference mode. A control unit (not shown) controls the operation of each unit of the encoder-decoder device 1 in each of the learning mode and the inference mode.

[0038] In the learning mode, the encoder-decoder device 1 performs machine learning processing using training data. In the case of speech recognition processing, the training data is a pair of an input speech and a correct answer (correct token sequence) of the recognition result. In the learning mode, the encoder-decoder device 1 performs machine learning processing using the first loss L and the second loss L described above. m And the third loss L d The internal parameters are updated by backpropagation based on the overall loss obtained by combining these three types of losses. e The first internal parameter of the decoder units 33-1, 33-2, . . . , 33-N d The values ​​of the second internal parameters of the encoder units 25-1, 25-2, . . . , 25-N and the third parameter of the linear transformation unit 36 ​​are optimized according to the training data. e and decoder units 33-1, 33-2, . . . , 33-N d The internal parameter values ​​obtained by the learning may be stored in a storage device as needed. Furthermore, the internal parameter values ​​may be transferred to other encoder-decoder devices 1.

[0039] In the inference mode, the encoder-decoder device 1 performs inference processing based on an unknown input (unknown speech in the case of speech recognition processing) using the values ​​of the internal parameters (first, second, and third internal parameters) obtained by learning. In the case of speech recognition processing, the encoder-decoder device 1 performs inference processing to infer and output a token sequence (text, sentence) corresponding to the unknown input.

[0040] The encoder-decoder device 1 operating in the inference mode may also be called an "inference device."

[0041] FIG. 2 shows the decoder unit 33-i (1≦i≦N) described in FIG. d 1, the decoder unit 33-1 converts the information on language features passed from the feature extraction unit 32 into the information on language features passed from the encoder unit 25-N. e The decoder unit 33-i (2≦i≦N) calculates output information based on the encoded information (encoded audio information) passed from the decoder unit 33-i. d ) is the output from the previous decoder unit 33-(i-1) and the encoder unit 25-N e The decoder unit 33-i (1≦i≦N) calculates output information based on the encoding information (encoded audio information) passed from the decoder unit 33-i. d ) passes the calculated output information to the next stage of processing. d The next stage is the linear transformation unit 36. As shown in Fig. 2, the decoder unit 33-i is configured to include a self-attention unit 34-i and a cross-attention unit 35-i.

[0042] The self-attention unit 34-i performs self-attention processing based on the input values. The output from the self-attention unit 34-i is passed to the cross-attention unit 35-i.

[0043] The cross-attention unit 35-i performs cross-attention processing using the output from the self-attention unit 34-i and the encoded information from the encoder side. The cross-attention unit 35-i passes the result of the cross-attention processing to the next processing stage. 1≦i <N d In this case, the next stage of processing is the cross attention unit 35-(i+1). d In this case, the next stage of processing is the linear transformation unit 36.

[0044] FIG. 3 shows the third loss L used by the encoder-decoder device 1 when performing machine learning. d 3 is a functional block diagram showing a configuration for generating a correct answer to calculate a probability. As shown in the figure, the encoder-decoder device 1 includes a probability calculation unit 60 and a cumulative distribution conversion unit 61. When operating in learning mode, the encoder-decoder device 1 calculates a correct answer using the configuration shown in FIG.

[0045] The probability calculation unit 60 calculates a probability value based on the data output by the linear transformation unit 36. The probability value calculated by the probability calculation unit 60 corresponds to the matrix data output by the linear transformation unit 36. In other words, the probability calculation unit 60 calculates a matrix having the same number of rows and columns as the matrix output by the linear transformation unit 36, and each of the elements is a probability value based on the data output by the linear transformation unit 36.

[0046] The cumulative distribution conversion unit 61 converts the matrix of probability values ​​passed from the probability calculation unit 60 into cumulative probabilities.

[0047] The processing by the probability calculation unit 60 and the cumulative distribution conversion unit 61 will be described in more detail later with reference to FIG.

[0048] FIG. 4 is a block diagram for explaining the function of the machine-learned encoder-decoder device 1 when it operates in inference mode. When operating in inference mode, the encoder-decoder device 1 includes a feature extraction unit 22 and encoder units 25-1, 25-2, . . . , 25-N. e , a feature extraction unit 32, and decoder units 33-1, 33-2, . . . , 33-N d , a linear transformation unit 36, and an output determiner 42. The encoder-decoder device 1 further includes an output control unit 44. When operating in the inference mode, the output control unit 44 controls the output. Here, the encoder units 25-1, 25-2, . . . , 25-N e and decoder units 33-1, 33-2, . . . , 33-N dand the linear transformation unit 36 ​​have already been trained. e and decoder units 33-1, 33-2, . . . , 33-N d and the linear transformation unit 36 ​​perform calculations using internal parameters optimized by learning.

[0049] In the configuration shown in FIG. 4, the encoder-decoder device 1 operates as follows. The output from the linear transformation unit 36 ​​is information that can represent one token (maximum likelihood token). Meanwhile, the output determiner 42 outputs a matrix of probability values. The output determiner 42 controls whether to output a token based on the output from the linear transformation unit 36, depending on the element values ​​(probability values) of the matrix passed from the output determiner 42. Specifically, if the value passed from the output determiner 42 is equal to or greater than a predetermined threshold (a value between 0 and 1, such as 0.8 or 0.9), the output control unit 44 outputs a token corresponding to the output from the linear transformation unit 36 ​​and feeds the output back to the decoder input. Meanwhile, if the value passed from the output determiner 42 is less than the predetermined threshold, the output control unit 44 rejects the output from the linear transformation unit 36.

[0050] By operating the output determiner 42 and the output control unit 44 in this manner, the output control unit 44 can suppress inappropriate output (in the case of voice recognition, recognition results calculated based on silence, background sound, noise, etc.).

[0051] The processing by the encoder / decoder device 1 and the data thereon will be described in more detail below.

[0052] FIG. 5 is a schematic diagram for explaining the processing of the feature extraction unit 22, the replication unit 23, and the mask processing unit 24, and the configuration of their data.

[0053] As shown in FIG. 5, the feature extraction unit 22 extracts features of input data (speech) and outputs an acoustic feature vector 101. The acoustic feature extracted by the feature extraction unit 22 may be, for example, Mel-Frequency Cepstrum Coefficients (MFCC) for a predetermined time interval. In the example shown, the feature extraction unit 22 sequentially outputs acoustic feature vectors as time t progresses. One rectangle shown as the acoustic feature vector 101 in the figure represents a multidimensional vector. In the example shown, the feature extraction unit 22 outputs four vectors (acoustic feature vectors) corresponding to t=1, 2, 3, and 4, respectively, for example.

[0054] That is, the acoustic feature vector 101 output by the feature quantity extraction unit 22 includes information on the acoustic features at all of t=1, 2, 3, and 4.

[0055] The duplication unit 23 reads the acoustic feature vector 101 passed from the feature extraction unit 22 and duplicates it. The duplication unit 23 outputs a duplicated acoustic feature vector created based on the acoustic feature vector 101. That is, the duplication unit 23 outputs, for example, acoustic feature vectors corresponding to m=1, 2, 3, 4 (each of which includes a feature for t=1, 2, 3, 4). That is, the four vectors for m=1, 2, 3, 4 output by the duplication unit 23 have the same value.

[0056] The mask processing unit 24 performs a process of masking a portion of the duplicated acoustic feature vectors passed from the duplication unit 23. The mask processing unit 24 outputs a masked acoustic feature vector 103. In the example shown in the figure, the masked acoustic feature vector 103 has been subjected to masking processing according to the value of m. In the figure, hatched rectangles represent masked portions. That is, at m=1, the mask processing unit 24 masks the acoustic feature vectors for t=2, 3, and 4. That is, the data in the m=1 column does not contain information on the acoustic features at the times t=2, 3, and 4. In other words, the masked data in the m=1 column represents information on the acoustic features before the arrival of times t=2, 3, and 4 (only for t=1). The same applies below. That is, at m=2, the mask processing unit 24 masks the acoustic feature vectors for t=3 and 4. That is, the data in the m=2 column does not contain information on the acoustic features at the times t=3 and 4. Furthermore, at m=3, the masking processing unit 24 masks the acoustic feature vector at t=4. That is, the data in the column at m=3 does not have information on the acoustic feature at time t=4. Furthermore, at m=4, the masking processing unit 24 does not perform masking. That is, the data in the column at m=4 has information on all of the acoustic features at times t=1, 2, 3, and 4.

[0057] That is, the masking unit 24 masks the information of the acoustic feature quantity in accordance with the progress of time. In other words, the masking unit 24 masks the information in accordance with each time point of t=1, 2, 3 so that the data of each column of m=1, 2, 3 does not include information of future acoustic features as viewed from each time point.

[0058] As a specific masking method, the mask processing unit 24 replaces all the numerical values ​​of vector elements of the portion to be masked (the hatched portion in the example shown in FIG. 5) with a constant value, for example.

[0059] Figure 6 shows the encoder section 25-N e5 is a schematic diagram showing the structure of the data output from the final layer of the encoder. That is, the masked acoustic feature vector 103 (FIG. 5) is e The data 104 is output by sequentially processing each layer of the encoder consisting of layers. In other words, the data 104 is information obtained by encoding the input data (acoustic feature data in the case of speech recognition processing) passed from the feature extraction unit 22. Encoder unit 25-N e This data 104 is sent to each layer of the decoder (decoder unit 33-1 to decoder unit 33-N) d The data 104 is passed to each layer of the decoder (decoder unit 33-1 to decoder unit 33-N). d 6 is used for cross-attention processing in (up to t). In the illustrated example, the data 104 includes t=1, 2, 3, 4 and m=1, 2, 3, 4. That is, the size of the data 104 shown in FIG. 6 corresponds to the size of the masked acoustic feature vector 103 shown in FIG. 5.

[0060] FIG. 7 shows the decoder side (decoder units 33-1 to 33-N d 1 and 2, and a linear transformation unit 36). In the figure, u is a variable representing the order of tokens input to the feature extraction unit 32. That is, in (1) of the figure, a token u=1 is input to the feature extraction unit 32, and the feature extraction unit 32 outputs a corresponding language feature vector 121-1. Also, in (2) of the figure, tokens u=1 and 2 are input to the feature extraction unit 32, and the feature extraction unit 32 outputs a corresponding language feature vector 121-2. Also, in (3) of the figure, tokens u=1, 2, and 3 are input to the feature extraction unit 32, and the feature extraction unit 32 outputs a corresponding language feature vector 121-3.

[0061] In each of the above (1), (2), and (3), the decoder units 33-1 to 33-N dEach of the decoder units 33-1 to 33-N performs calculations using the internal parameter values ​​at that time. d Each of the decoder units 33-1 to 33-N performs an operation using a mask pattern for m=1, 2, 3, 4, and calculates the result. d Each of the decoder units 33-1 to 33-N calculates cross-attention for each mask pattern m=1, 2, 3, 4 based on the value (vector) of u=1 in the language feature vector 121-1. The resulting data is represented in the form of data 122-1. That is, the decoder units 33-1 to 33-N d Each of the decoder units 33-1 to 33-N outputs data for m=1, 2, 3, and 4, respectively, for u=1. d Each of the decoder units 33-1 to 33-N passes the output data to the next stage of processing. Based on the decoder output calculated in this way, the linear transformation unit 36 ​​processes the decoder, selecting the most likely token and outputting it as the decoded result. The output most likely token is added to the input for the next decoder process. Also, in FIG. 7(2), the decoder units 33-1 to 33-N d Each of the decoder units 33-1 to 33-N calculates cross-attention for each mask pattern m=1, 2, 3, 4 based on the values ​​(vectors) of u=1 and 2 of the language feature vector 121-2. The resulting data is represented in the form of data 122-2. That is, the decoder units 33-1 to 33-N d Each of the decoder units 33-1 to 33-N outputs data for m=1, 2, 3, and 4 for u=1 and 2, respectively. d Each of the decoder units 33-1 to 33-N passes the output data to the next stage of processing. Based on the decoder output calculated in this way, the linear transformation unit 36 ​​processes the decoder, selecting the most likely token and outputting it as the decoded result. The most likely token that is output is added to the input for the next decoder process. The token based on the output from the decoder is added to the input for the next decoder process. Also, in FIG. 7(3), the decoder units 33-1 to 33-N dEach of the decoder units 33-1 to 33-N calculates cross-attention for each mask pattern m=1, 2, 3, 4 based on the values ​​(vectors) of u=1, 2, and 3 of the language feature vector 121-3. The resulting data is represented in the form of data 122-3. That is, the decoder units 33-1 to 33-N d Each of the decoders outputs data for m=1, 2, 3, 4 for u=1, 2, and 3. Based on the decoder output calculated in this way, the linear transform unit 36 ​​processes the data, and the most likely token is selected and output as the decoding result.

[0062] The linear conversion unit 36 ​​is a decoder unit 33-N d Based on the data passed from, calculations are performed using the internal parameter values ​​at that time.

[0063] FIG. 8 is a schematic diagram showing data output by the linear transformation unit 36 ​​corresponding to u=1, 2, 3. As shown in the figure, data 201 has calculation results corresponding to m=1, 2, 3, 4 for each of u=1, 2, 3. The element in the u-th row and m-th column of data 201 output from the linear transformation unit 36 ​​is expressed as linear(l u,m ) can be expressed as follows. u,m ) is a vector.

[0064] FIG. 9 is a schematic diagram for explaining the details of the processing by the output selector 41 and the multiplication unit 43 when the encoder-decoder device 1 operates in the learning mode. As mentioned above, the data 201 is the output from the linear transformation unit 36. The element in the u-th row and m-th column of this data 201 is linear(l u,m ) The output selector 41 calculates a binary mask 202 based on the data 201. The element in the u-th row and m-th column of the binary mask 202 is calculated by the following equation (1).

[0065]

number

[0066] That is, the output selector 41 selects the linear (l u,m ) in the direction of m, the function Gumbel() is applied. Gumbel() in equation (1) represents the Gumbel softmax function. The Gumbel softmax is a softmax function that changes very sharply. In other words, the Gumbel softmax outputs values ​​that are very close to 0 or very close to 1 in most areas of its domain. In other words, each element of the binary mask 202 is a value that is very close to 0 or very close to 1. In other words, the values ​​of the elements of the binary mask 202 are almost binary (0 or 1).

[0067] The multiplication unit 43 multiplies corresponding elements of the data 201 and the binary mask 202 and outputs the result. In other words, when this binary mask 202 is passed to the multiplication unit 43, an element of the data 201 is selected. The binary mask 202 illustrated here has values ​​of approximately (1,0,0,0) for u=1, approximately (0,0,1,0) for u=2, and approximately (0,0,0,1) for u=2. In other words, when the binary mask 202 according to this example is used, the element u=1, m=1 in the data 201 is selected for u=1. Furthermore, the element u=2, m=3 in the data 201 is selected for u=2. Furthermore, the element u=3, m=4 in the data 201 is selected for u=3.

[0068] As described above, the first loss calculation unit 51 and the second loss calculation unit 52 calculate the first loss L and the second loss L, respectively. mThe first loss calculation unit 51 is given the correct answer of the text of the recognition result as correct answer data. That is, the first loss calculation unit 51 calculates the difference (loss) between the token sequence obtained based on the information passed from the multiplication unit 43 and the token sequence of the correct answer text as the first loss L. The second loss calculation unit 52 is given a predetermined correct answer. The second loss calculation unit 52 calculates the difference between the mask data in matrix format (for example, a matrix such as the binary mask 202 shown in the figure) output from the output selector 41 and the given correct answer as the second loss L. m It is calculated as follows.

[0069] The correct answer given to the second loss calculation unit 52 may be, for example, a matrix as shown in the following equation (2).

[0070]

number

[0071] The rows of the matrix expressed in equation (2) correspond to the values ​​of u, and the columns correspond to the values ​​of m. This matrix does not need to be a square matrix. That is, the number of rows and the number of columns of this matrix may differ. The elements in the diagonal region of this matrix (the region surrounded by the dashed line) are set to non-zero values, and the other elements are set to 0 (or a value close to 0). The sum of the values ​​of the elements in each row may be set to 0. The values ​​of the elements of this matrix may also be limited to 0 or 1. By having the second loss calculation unit 52 calculate the loss based on the correct solution such as equation (2), the encoder-decoder device 1 is trained to process data input at an earlier time (such as speech recognition) using only information available in that earlier time period.

[0072] It should be noted that the matrix shown in equation (2) is merely an example, and any other matrix may be used as the correct solution.

[0073] For example, a correct matrix may be manually created by referring to the value of the binary mask 202 generated by the output selector 41, and the matrix may be provided as the correct matrix to the second loss calculation unit 52. In this case, the correct matrix is ​​a special case of the matrix expressed by the above formula (2).

[0074] Alternatively, for example, for each u (1≦u≦U, U is the number of rows in the binary mask 202), a correct answer may be created so that the value is 1 at the smallest possible stage of m, and the correct answer may be provided to the second loss calculation unit 52. In this case, however, the matrix elements are either 0 or 1, and m that makes the value of the matrix element in the u-th row 1 is set as m u When u1 <u2ならばm u1 ≦m u2 The correct matrix created in this way is also a special case of the matrix expressed by the above formula (2).

[0075] Alternatively, for example, a matrix whose elements are either 0 or 1 and whose m-th column value is 1 for as small an m as possible (and a reasonable m) in the first row (u=1) may be created as a correct solution, and this correct solution may be provided to the second loss calculation unit 52.

[0076] FIG. 10 shows the third loss calculation unit 53 calculating the third loss L when the encoder-decoder device 1 operates in the learning mode. d As described above, the data 201 is data output from the linear transformation unit 36. The element in the u-th row and m-th column of the data 201 is linear(l u,m )

[0077] Based on the data 201, the output determiner 42 generates and outputs a matrix having the same number of rows and columns as the data 201. The value output by the output determiner 42 acts as a score for determining the timing at which the encoder-decoder device 1 outputs the output data. In other words, the value in the u-th row and m-th column of the data output by the output determiner 42 is a score for determining whether or not the encoder-decoder device 1 should output a value based on the decoding result in the m-th column as the u-th row output. Specifically, the output determiner 42 calculates the element (scalar) in the u-th row and m-th column using the following equation (3):

[0078]

number

[0079] Sigmoid() in formula (3) represents the sigmoid function, which outputs a value greater than or equal to 0.0 and less than or equal to 1.0.

[0080] When the encoder-decoder device 1 operates in the learning mode, the output data (the above score) from the output determiner 42 is passed to a third loss calculation unit 53 .

[0081] On the other hand, the third loss L calculated by the third loss calculation unit 53 d is calculated as follows:

[0082] The data 201 is passed to the probability calculation unit 60. The probability calculation unit 60 calculates a probability value based on the data 201. As a specific example, the probability calculation unit 60 calculates the probability value based on the following equation (4).

[0083]

number

[0084] That is, the probability calculation unit 60 calculates the linear(l u,m) to calculate the probability value in the u-th row and the m-th column. The probability calculation unit 60 passes the value calculated in this way to the cumulative distribution conversion unit 61.

[0085] The cumulative distribution conversion unit 61 converts the matrix of probability values ​​passed from the probability calculation unit 60 into cumulative probabilities. In other words, the cumulative distribution conversion unit 61 adds up all the numerical values ​​of the elements in the u-th row, from the first column to the k-th column (where 1≦k≦M; M is the number of columns in the matrix passed from the probability calculation unit 60), and outputs the result as the element in the u-th row and the k-th column. The matrix output by the cumulative distribution conversion unit 61 is correct data that indicates the timing at which the encoder-decoder device 1 should output the decoded result.

[0086] The supervised data exemplified in Fig. 10 is as follows: That is, when u = 1, the values ​​of each column are 0.8, 0.9, 1.0, 1.0. When u = 2, the values ​​of each column are 0.0, 0.0, 0.9, 1.0. When u = 3, the values ​​of each column are 0.0, 0.0, 0.0, 1.0. Since the cumulative distribution conversion unit 61 accumulates probability values ​​in the direction in which m increases, in a row of supervised data, the larger m becomes, the larger the probability value (accumulated probability value) at that m becomes.

[0087] The third loss calculation unit 53 calculates the cross entropy or square error between the data passed from the output determiner 42 and the correct answer passed from the cumulative distribution transformation unit 61, to obtain the third loss L d Ask for.

[0088] Through the above processing, the probability calculation unit 60 and the cumulative distribution conversion unit 61 calculate the third loss L d It provides the correct answer to find the answer.

[0089] As described above, the first loss calculation unit 51 calculates the first loss L, and the second loss calculation unit 52 calculates the second loss L m The third loss calculation unit 53 calculates the third loss L dWhen operating in the learning mode, the encoder-decoder device 1 performs a process of optimizing internal parameters using the backpropagation algorithm based on the loss that comprehensively reflects these three types of loss.

[0090] The total loss used in this case (which may be called the "total loss") is expressed, for example, by the following equation (5).

[0091]

number

[0092] In addition, in equation (5), α and β may be hyperparameters.

[0093] FIG. 11 is a block diagram showing an example of the internal configuration of an encoder / decoder device 1 implemented using a general-purpose computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with these instructions, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element in the RAM 902 has an address and can be accessed using the address. RAM stands for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0094] At least some of the functions of the encoder / decoder device 1 according to the embodiment can be implemented by a computer and a program. In this case, the functions can be implemented by recording a program for implementing the functions on a computer-readable recording medium and loading and executing the program on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an operating system and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client. The program may also be designed to implement some of the functions described above, or may be capable of implementing the functions described above in combination with a program already stored in the computer system.

[0095] The method for realizing a predetermined function using machine learning techniques is as follows. Here, for convenience, a unit of function that can be machine-learned is referred to as a "function unit." The function unit calculates data to be output based on input data. The function unit uses internal parameter values ​​when calculating the data to be output. The internal parameters can be updated and optimized through a learning process. When training the function unit, training data is used. The training data includes input data to the function unit and a correct answer for output data calculated based on the input data. During the learning process, the function unit reads the input data included in the training data and references output data based on the input data and using the internal parameter values ​​at that time. This output data is an estimated value calculated based on the internal parameters at that time and does not necessarily coincide with the correct answer. To update the internal parameters, the difference between the estimated value calculated and output by the function unit based on the input data at that time and the correct answer corresponding to the input data is calculated. This difference is called an error, loss, etc. The difference calculated here may be, for example, the absolute value of the difference between scalars, a squared error, a cross-entropy error, or a difference calculated by other methods. Based on the calculated difference, the internal parameter values ​​can be updated using backpropagation. This operation adjusts the internal parameter values ​​so as to reduce the error. By performing the above operation multiple times (multiple times) using a predetermined amount (large amount) of training data, the internal parameter values ​​are optimized. In other words, the functional unit is adjusted to perform the processing exemplified by the training data provided. After a sufficient amount of training has been performed, the training process may be terminated. The set of trained internal parameter values ​​at this time may be stored in a storage device or the like, enabling estimation based on the training results. A function that enables data input and output, including the internal parameters, may be called a "model." In other words, a model includes internal parameters. Training the functional unit may be called "model training."One way to implement the model is to use a neural network, and depending on the type of processing to be performed, an appropriate form of neural network can be used.

[0096] Although the embodiment has been described above, the present invention can also be carried out in the following modified examples.

[0097] In the embodiment described above, the encoder / decoder device 1 receives input speech and performs speech recognition processing. As a variation, the encoder / decoder device 1 may perform processing other than speech recognition. In this case, the data input to the encoder / decoder device 1 is, for example, data that is generated continuously in real time. The input data is passed to the encoder / decoder device 1 in, for example, a streaming format. The encoder / decoder device 1 performs processing based on the input data and generates output data. With the configuration of this embodiment, the encoder / decoder device 1 can proceed with processing of the input data sequentially for each time interval, without waiting for all of the input real-time data to be completed.

[0098] When the encoder-decoder device 1 according to the above embodiment operates in the learning mode, the main components are configured as follows.

[0099] The duplication unit 23 duplicates the sequence of the input first input feature amounts for a plurality of mask patterns.

[0100] The mask processing unit 24 masks some of the first input features included in the columns, and by using a different mask pattern for each of the copied columns, the set of first input features to be masked varies for each of the columns copied by the copying unit 23. An example of how this is done is shown in the masked acoustic feature vector 103 in FIG. 5. In other words, the range to be masked is gradually narrowed over time. The mask pattern may also include a pattern in which no masking is done at all.

[0101] Encoder section 25-1, 25-2, 25-N e inputs the masked columns, and generates a plurality of columns of the encoding result corresponding to each of the columns using the masked columns and a first internal parameter.

[0102] Decoder section 33-1, 33-2, . . . , 33-N d is the sequence of the second input feature values ​​to be input and the encoder unit 25-N e and the multiple columns of the encoding result passed from the multiplication unit 43, and by performing cross-attention processing using a second internal parameter, a matrix of the decoding result is output, the matrix being configured with rows corresponding to the columns of the second input features and columns corresponding to the mask pattern. Note that the second input features are features of a token sequence corresponding to a past output from the multiplication unit 43.

[0103] The linear conversion unit 36 ​​is a decoder unit 33-N d Based on the matrix of the decoding result, which is the output from the (final layer), a linear transformation process is performed using a third internal parameter, and a matrix of linear transformation result data composed of rows corresponding to the columns of the second input features and columns corresponding to the mask pattern is output.

[0104] The output selector 41 generates a binary mask, which is a matrix whose elements have a value of 1 or approximately 1 or a value of 0 or approximately 0, based on the matrix of the linear transformation result data. An example of the binary mask to be generated is the binary mask 202 shown in Fig. 9. The output selector 41 may be configured to generate the binary mask by applying a Gumbel-softmax function to each element of the matrix of the linear transformation result data.

[0105] A multiplication unit 43 multiplies each element of the matrix of the linear transformation result data by the corresponding element of the binary mask, and outputs the result.

[0106] The output decision unit 42 generates and outputs a matrix of probability values ​​corresponding to the matrix of the linear transformation result data based on each element of the matrix of the linear transformation result data.

[0107] The first loss calculation unit 51 calculates a first loss (L) which is an error between the output from the multiplication unit 43 and the correct token sequence corresponding to the first input feature.

[0108] The second loss calculation unit 52 calculates a second loss (L m ) is calculated. An example of the correct binary mask is a matrix configured in the row direction corresponding to the columns of the second input features and in the column direction corresponding to the mask pattern, and the matrix may be a matrix in which elements near the diagonal are non-zero and other elements are 0 or nearly 0.

[0109] The third loss calculation unit 53 calculates a third loss (L d) is calculated. In order to generate this matrix of probability values ​​of the correct answer, a probability calculation unit 60 and a cumulative distribution transformation unit 61 may be provided. The probability calculation unit 60 generates a probability value matrix by applying a softmax function to each element of the linear transformation result data. The cumulative distribution transformation unit 61 generates the matrix of probability values ​​of the correct answer based on the probability value matrix calculated by the probability calculation unit 60, such that the sum of the values ​​from the first column to the m-th column of the u-th row of the probability value matrix becomes the element of the u-th row and m-th column of the matrix of probability values ​​of the correct answer.

[0110] The encoder-decoder device is configured to convert the first loss (L) and the second loss (L m ) and the third loss (L d ) and the value of the third internal parameter, the value of the second internal parameter, and the value of the first internal parameter are updated by back propagation based on the total loss (equation (5) shown in the embodiment) which combines the loss, the loss, and the first internal parameter. With this configuration, the value of the third internal parameter, the value of the second internal parameter, and the value of the first internal parameter are optimized.

[0111] When the encoder-decoder device 1 (inference device) according to the above embodiment operates in inference mode, the main parts are configured as follows.

[0112] Encoder section 25-1, 25-2, 25-N e receives a sequence of first input features and generates a sequence of encoding results corresponding to the sequence using the sequence and a first internal parameter.

[0113] Decoder section 33-1, 33-2, . . . , 33-N d is the sequence of the second input feature values ​​to be input and the encoder unit 25-N eBased on the sequence of the encoding results passed from the final layer, the output control unit 44 performs cross-attention processing using second internal parameters to output a sequence of the decoding results corresponding to the sequence of the second input features. Note that the second input features are features of a token sequence corresponding to a past output from the output control unit 44.

[0114] The linear conversion unit 36 ​​is a decoder unit 33-N d Based on the sequence of the decoding results that are output from the (final layer), a linear transformation process is performed using the third internal parameter, and linear transformation result data is output.

[0115] The output decision unit 42 generates and outputs a probability value corresponding to the linear transformation result data based on the linear transformation result data.

[0116] The output control unit 44 outputs a token based on the output from the linear conversion unit 36 ​​when the output from the output determiner 42 is equal to or greater than a predetermined threshold.

[0117] In the encoder-decoder device 1 (inference device) when operating in inference mode, the value of the first internal parameter, the value of the second internal parameter, and the value of the third internal parameter are values ​​obtained as a result of machine learning by the encoder-decoder device operating in the learning mode.

[0118] The above has described in detail an embodiment of the present invention (including modified examples) with reference to the drawings, but the specific configuration is not limited to this embodiment, and also includes designs within the scope that do not deviate from the gist of the present invention. [Industrial Applicability]

[0119] The present invention can be used for processing data generated in real time, such as speech recognition processing, etc. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0120] 1. Encoder-decoder device (inference device) 22 Feature extraction unit 23 Reproduction Department 24 Mask processing section 25-1, 25-2, , 25-N e Encoder section 32 Feature extraction unit 33-1, 33-2, 33-N d Decoder section 34-1, 34-2, 34-N d Self-Attention Department 35-1, 35-2, 35-N d Cross Attention Section 36 Linear transformation section 41 Output Selector 42 Output judger 43 Multiplication section 44 Output control section 51 1st loss calculation section 52 Second loss calculation section 53 Third loss calculation section 60 Probability Calculation Unit 61 Cumulative distribution transformation part 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a duplication unit that duplicates the sequence of the first input feature amount that is input for a plurality of mask patterns; a mask processing unit that masks some of the first input feature amounts included in the columns, and that uses a different mask pattern for each of the copied columns, thereby making a set of first input feature amounts to be masked different for each of the copied columns; an encoder unit that receives the masked columns and generates a plurality of columns of an encoding result corresponding to each of the columns, using the masked columns and a first internal parameter; a decoder unit that performs cross-attention processing using a second internal parameter based on a sequence of second input features that is input and a plurality of sequences of the encoding result that is passed from the encoder unit, and outputs a matrix of decoded results that is configured in a row direction corresponding to the sequence of the second input features and a column direction corresponding to the mask pattern; a linear transformation unit that performs linear transformation processing using a third internal parameter based on the matrix of the decoding result that is output from the decoder unit, and outputs a matrix of linear transformation result data configured in a row direction corresponding to the columns of the second input feature amounts and in a column direction corresponding to the mask pattern; an output selector that generates a binary mask, which is a matrix having elements of 1 or approximately 1 or 0 or approximately 0, based on the matrix of the linear transformation result data; a multiplication unit that multiplies each element of the matrix of the linear transformation result data by a corresponding element of the binary mask and outputs the result; an output determiner that generates and outputs a matrix of probability values ​​corresponding to the matrix of linear transformation result data based on each element of the matrix of linear transformation result data; a first loss calculation unit that calculates a first loss (L) that is an error between an output from the multiplication unit and a correct token sequence corresponding to the first input feature; The second loss (L ｍ a second loss calculation unit that calculates A third loss (L ｄ a third loss calculation unit that calculates Equipped with the second input feature is a feature of a token sequence corresponding to a past output from the multiplication unit; The first loss (L) and the second loss (L ｍ ) and the third loss (L ｄ ) based on a total loss obtained by combining the third internal parameter, the second internal parameter, and the first internal parameter, by back propagation. Encoder-decoder device.

2. the output selector generates the binary mask by applying a Gumbel softmax function to each element of the matrix of the linear transformation result data.

2. The encoder-decoder device according to claim 1.

3. The correct solution of the binary mask is a matrix configured in a row direction corresponding to the columns of the second input features and in a column direction corresponding to the mask pattern, in which elements near the diagonal of the matrix are non-zero and other elements of the matrix are 0 or approximately 0.

2. The encoder-decoder device according to claim 1.

4. a probability calculation unit that applies a softmax function to each element of the linear transformation result data to generate a probability value matrix; a cumulative distribution conversion unit that generates a matrix of correct probability values ​​based on the probability value matrix calculated by the probability calculation unit, so that the sum of values ​​from the first column to the mth column of the uth row of the probability value matrix becomes the element of the uth row and the mth column of the matrix of correct probability values; The encoder-decoder device of claim 1 , further comprising:

5. an encoder unit that receives a sequence of first input features and generates a sequence of encoding results corresponding to the sequence using the sequence and a first internal parameter; a decoder unit that performs cross-attention processing using a second internal parameter based on the sequence of second input features that is input and the sequence of the encoding results that is passed from the encoder unit, and outputs a sequence of decoded results that correspond to the sequence of second input features; a linear transformation unit that performs a linear transformation process using a third internal parameter based on the sequence of the decoded results output from the decoder unit, and outputs linear transformation result data; an output determiner that generates and outputs a probability value corresponding to the linear transformation result data based on the linear transformation result data; an output control unit that outputs a token based on the output from the linear conversion unit when the output from the output determiner is equal to or greater than a predetermined threshold; Equipped with the second input feature is a feature of a token sequence corresponding to a past output from the output control unit, The value of the first internal parameter, the value of the second internal parameter, and the value of the third internal parameter are values ​​obtained as a result of machine learning by the encoder-decoder device according to any one of claims 1 to 4. Reasoning device.

6. a duplication unit that duplicates the sequence of the first input feature amount that is input for a plurality of mask patterns; a mask processing unit that masks some of the first input feature amounts included in the columns, and that uses a different mask pattern for each of the copied columns, thereby making a set of first input feature amounts to be masked different for each of the copied columns; an encoder unit that receives the masked columns and generates a plurality of columns of an encoding result corresponding to each of the columns, using the masked columns and a first internal parameter; a decoder unit that performs cross-attention processing using a second internal parameter based on a sequence of second input features that is input and a plurality of sequences of the encoding result that is passed from the encoder unit, and outputs a matrix of decoded results that is configured in a row direction corresponding to the sequence of the second input features and a column direction corresponding to the mask pattern; a linear transformation unit that performs linear transformation processing using a third internal parameter based on the matrix of the decoding result that is output from the decoder unit, and outputs a matrix of linear transformation result data configured in a row direction corresponding to the columns of the second input feature amounts and in a column direction corresponding to the mask pattern; an output selector that generates a binary mask, which is a matrix having elements of 1 or approximately 1 or 0 or approximately 0, based on the matrix of the linear transformation result data; a multiplication unit that multiplies each element of the matrix of the linear transformation result data by a corresponding element of the binary mask and outputs the result; an output determiner that generates and outputs a matrix of probability values ​​corresponding to the matrix of linear transformation result data based on each element of the matrix of linear transformation result data; a first loss calculation unit that calculates a first loss (L) that is an error between an output from the multiplication unit and a correct token sequence corresponding to the first input feature; The second loss (L ｍ a second loss calculation unit that calculates A third loss (L ｄ a third loss calculation unit that calculates Equipped with the second input feature is a feature of a token sequence corresponding to a past output from the multiplication unit; The first loss (L) and the second loss (L ｍ ) and the third loss (L ｄ ) based on a total loss obtained by combining the third internal parameter, the second internal parameter, and the first internal parameter, by back propagation. A program that allows a computer to function as an encoder / decoder device.

7. an encoder unit that receives a sequence of first input features and generates a sequence of encoding results corresponding to the sequence using the sequence and a first internal parameter; a decoder unit that performs cross-attention processing using a second internal parameter based on the sequence of second input features that is input and the sequence of the encoding results that is passed from the encoder unit, and outputs a sequence of decoded results that correspond to the sequence of second input features; a linear transformation unit that performs a linear transformation process using a third internal parameter based on the sequence of the decoded results output from the decoder unit, and outputs linear transformation result data; an output determiner that generates and outputs a probability value corresponding to the linear transformation result data based on the linear transformation result data; an output control unit that outputs a token based on the output from the linear conversion unit when the output from the output determiner is equal to or greater than a predetermined threshold; Equipped with the second input feature is a feature of a token sequence corresponding to a past output from the output control unit, The value of the first internal parameter, the value of the second internal parameter, and the value of the third internal parameter are values ​​obtained as a result of machine learning by the encoder-decoder device according to any one of claims 1 to 4. A program that enables a computer to function as an inference device.

Images